216acf545f
- Partition configuration for indexing local data based on user accounts - Store indexed data in an underlying postgres db using the `pgvector` extension - Add migrations for all relevant user data and embeddings generation. Very little performance optimization has been done for the lookup time - Apply filters using SQL queries - Start removing many server-level configuration settings - Configure GitHub test actions to run during any PR. Update the test action to run in a containerized environment with a DB. - Update the Docker image and docker-compose.yml to work with the new application design
Django App
Khoj uses Django as the backend framework primarily for its powerful ORM and the admin interface. The Django app is located in the src/app directory. We have one installed app, under the /database/ directory. This app is responsible for all the database related operations and holds all of our models. You can find the extensive Django documentation here 🌈.
Setup (Docker)
Prerequisites
- Ensure you have Docker installed.
- Ensure you have Docker Compose installed.
Run
Using the docker-compose.yml file in the root directory, you can run the Khoj app using the following command:
docker-compose up
Setup (Local)
Install dependencies
pip install -e '.[dev]'
Setup the database
- Ensure you have Postgres installed. For MacOS, you can use Postgres.app.
- If you're not using Postgres.app, you may have to install the pgvector extension manually. You can find the instructions here. If you're using Postgres.app, you can skip this step. Reproduced instructions below for convenience.
cd /tmp
git clone --branch v0.5.1 https://github.com/pgvector/pgvector.git
cd pgvector
make
make install # may need sudo
- Create a database
Make migrations
This command will create the migrations for the database app. This command should be run whenever a new model is added to the database app or an existing model is modified (updated or deleted).
python3 src/manage.py makemigrations
Run migrations
This command will run any pending migrations in your application.
python3 src/manage.py migrate
Run the server
While we're using Django for the ORM, we're still using the FastAPI server for the API. This command automatically scaffolds the Django application in the backend.
python3 src/khoj/main.py