Add developer support for using next.js to serve generated static files (#814)

To improve the developer experience for front-end development, we're migrating to Next.js. In order to do this migration page-by-page, we're using static site generation via Next.js. This also helps us avoid making cross site requests from front-end to back-end for the time being, while giving a ramp to separating out server and client if needed for scale down the road.

Dev instructions for using the next.js setup are in the added README.

This adds scaffolding for including the built files in the python package as well as the docker images. Docker setup has been tested locally. In order to verify the build is working as expected, we can navigate to the {khoj_host}:42110/experimental and verify that the experiment page comes up.

This setup works with serving static files included in the src/interface/web folder from the Django app. The key bit for understanding the setup is in the yarn export command in package.json.

.github/workflows/dockerize.yml

@@ -8,6 +8,7 @@ on:
       - master
     paths:
       - src/khoj/**
+      - src/interface/web/**
       - pyproject.toml
       - Dockerfile
       - prod.Dockerfile
@@ -30,8 +31,9 @@ on:
 env:
   # Tag Image with tag name on release
   # else with user specified tag (default 'dev') if triggered via workflow
+  # else with run_id if triggered via a pull request
   # else with 'pre' (if push to master)
-  DOCKER_IMAGE_TAG: ${{ github.ref_type == 'tag' && github.ref_name || github.event_name == 'workflow_dispatch' && github.event.inputs.tag || 'pre' }}
+  DOCKER_IMAGE_TAG: ${{ github.ref_type == 'tag' && github.ref_name || github.event_name == 'workflow_dispatch' && github.event.inputs.tag || github.event_name == 'push' && github.run_id || 'pre' }}
 
 jobs:
   build:

.github/workflows/pypi.yml

@@ -8,6 +8,7 @@ on:
       - 'master'
     paths:
       - src/khoj/**
+      - src/interface/web/**
       - pyproject.toml
       - .github/workflows/pypi.yml
   pull_request:
@@ -15,6 +16,7 @@ on:
       - 'master'
     paths:
       - src/khoj/**
+      - src/interface/web/**
       - pyproject.toml
       - .github/workflows/pypi.yml
 
@@ -37,6 +39,16 @@ jobs:
       - name: ⬇️ Install Application
         run: python -m pip install --upgrade pip && pip install --upgrade .
 
+      - name: Install the Next.js application
+        run: |
+          yarn install
+        working-directory: src/interface/web
+
+      - name: Build & export static Next.js app to Django static assets
+        run: |
+          yarn ciexport
+        working-directory: src/interface/web
+
       - name: ⚙️ Build Python Package
         run: |
           # Setup Environment for Reproducible Builds