diff --git a/docs/.nojekyll b/docs/.nojekyll new file mode 100644 index 00000000..e69de29b diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 00000000..59341480 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,59 @@ +

Khoj Logo

+ +[![test](https://github.com/khoj-ai/khoj/actions/workflows/test.yml/badge.svg)](https://github.com/khoj-ai/khoj/actions/workflows/test.yml) +[![dockerize](https://github.com/khoj-ai/khoj/actions/workflows/dockerize.yml/badge.svg)](https://github.com/khoj-ai/khoj/pkgs/container/khoj) +[![pypi](https://github.com/khoj-ai/khoj/actions/workflows/pypi.yml/badge.svg)](https://pypi.org/project/khoj-assistant/) + +# Khoj +*An AI personal assistant for your digital brain* + +Welcome to the Docs! This is the best place to get started with Khoj. Check out our [Github](https://github.com/khoj-ai/khoj) to dive straight into the code. + +Khoj gives you lightning fast, offline search on your personal machine and gives you the power to talk to your notes. + +**Supported Plugins** + +[![Khoj on Obsidian](https://img.shields.io/badge/Obsidian-%23483699.svg?style=for-the-badge&logo=obsidian&logoColor=white)](https://github.com/khoj-ai/khoj/tree/master/src/interface/obsidian#readme) +[![Khoj on Emacs](https://img.shields.io/badge/Emacs-%237F5AB6.svg?&style=for-the-badge&logo=gnu-emacs&logoColor=white)](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#readme) + +## Features +- **Search** + - **Local**: Your personal data stays local. All search and indexing is done on your machine. *Unlike chat which requires access to GPT.* + - **Incremental**: Incremental search for a fast, search-as-you-type experience +- **Chat** + - **Faster answers**: Find answers faster, smoother than search. No need to manually scan through your notes to find answers. + - **Iterative discovery**: Iteratively explore and (re-)discover your notes + - **Assisted creativity**: Smoothly weave across answers retrieval and content generation +- **General** + - **Natural**: Advanced natural language understanding using Transformer based ML Models + - **Pluggable**: Modular architecture makes it easy to plug in new data sources, frontends and ML models + - **Multiple Sources**: Index your Org-mode and Markdown notes, PDF files, Github repositories, and Photos + - **Multiple Interfaces**: Interact from your [Web Browser](./src/khoj/interface/web/index.html), [Emacs](./src/interface/emacs/khoj.el) or [Obsidian](./src/interface/obsidian/) + +## Install +[Click here](./setup.md) for full setup instructions. + +## Supported data sources +- markdown* +- org-mode* +- pdf* +- images* +- github +- notion + +\* These data sources are offline only. + +If you're using Github or Notion, you can get on a waitlist for [Khoj Cloud](https://khoj.dev). + +## Credits + +- [Multi-QA MiniLM Model](https://huggingface.co/sentence-transformers/multi-qa-MiniLM-L6-cos-v1), [All MiniLM Model](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) for Text Search. See [SBert Documentation](https://www.sbert.net/examples/applications/retrieve_rerank/README.html) +- [OpenAI CLIP Model](https://github.com/openai/CLIP) for Image Search. See [SBert Documentation](https://www.sbert.net/examples/applications/image-search/README.html) +- Charles Cave for [OrgNode Parser](http://members.optusnet.com.au/~charles57/GTD/orgnode.html) +- [Org.js](https://mooz.github.io/org-js/) to render Org-mode results on the Web interface +- [Markdown-it](https://github.com/markdown-it/markdown-it) to render Markdown results on the Web interface + + +[^1]: Default Khoj config file @ `~/.khoj/khoj.yml` + +[^2]: Default Khoj url @ http://localhost:42110 diff --git a/docs/_coverpage.md b/docs/_coverpage.md new file mode 100644 index 00000000..9b027fe6 --- /dev/null +++ b/docs/_coverpage.md @@ -0,0 +1,14 @@ + + +![logo](./assets/khoj-logo-sideways-200.png) + +> An open source, AI personal assistant for your notes + +- Lightning fast search +- Multi-turn chat +- Keeps you in control of your data + +[GitHub](https://github.com/khoj-ai/khoj) +[Get Started](#khoj) + +![color](#f9f5de) diff --git a/docs/_sidebar.md b/docs/_sidebar.md new file mode 100644 index 00000000..ebad3d62 --- /dev/null +++ b/docs/_sidebar.md @@ -0,0 +1,18 @@ +- Getting Started + - [Overview](README.md) + - [Install](setup.md) + - [Windows Installation](windows_install.md) +- Learn More + - [Features](features.md) + - [Chat](chat.md) + - [Search](search.md) + - [Demos](demos.md) + - Interfaces + - [Obsidian](obsidian.md) + - [Emacs](emacs.md) + - Data Sources + - [Github](github_integration.md) + - [Notion](notion_integration.md) + - [Advanced](advanced.md) +- Contributing + - [Development](development.md) diff --git a/docs/advanced.md b/docs/advanced.md new file mode 100644 index 00000000..95e6d351 --- /dev/null +++ b/docs/advanced.md @@ -0,0 +1,54 @@ + +## Advanced Usage +### Access Khoj on Mobile +1. [Setup Khoj](#Setup) on your personal server. This can be any always-on machine, i.e an old computer, RaspberryPi(?) etc +2. [Install](https://tailscale.com/kb/installation/) [Tailscale](tailscale.com/) on your personal server and phone +3. Open the Khoj web interface of the server from your phone browser.
It should be `http://tailscale-ip-of-server:42110` or `http://name-of-server:42110` if you've setup [MagicDNS](https://tailscale.com/kb/1081/magicdns/) +4. Click the [Add to Homescreen](https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps/Add_to_home_screen) button +5. Enjoy exploring your notes, documents and images from your phone! + +![](https://github.com/khoj-ai/khoj/blob/master/docs/khoj_pwa_android.png?) + +### Search across Different Languages + To search for notes in multiple, different languages, you can use a [multi-lingual model](https://www.sbert.net/docs/pretrained_models.html#multi-lingual-models).
+ For example, the [paraphrase-multilingual-MiniLM-L12-v2](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2) supports [50+ languages](https://www.sbert.net/docs/pretrained_models.html#:~:text=we%20used%20the%20following%2050%2B%20languages), has good search quality and speed. To use it: + 1. Manually update `search-type > asymmetric > encoder` to `paraphrase-multilingual-MiniLM-L12-v2` in your `~/.khoj/khoj.yml` file for now. See diff of `khoj.yml` below for illustration: + ```diff + asymmetric: +- encoder: "sentence-transformers/multi-qa-MiniLM-L6-cos-v1" ++ encoder: "paraphrase-multilingual-MiniLM-L12-v2" + cross-encoder: "cross-encoder/ms-marco-MiniLM-L-6-v2" + model_directory: "~/.khoj/search/asymmetric/" + ``` + + 2. Regenerate your content index. For example, by opening [\/api/update?t=force](http://localhost:42110/api/update?t=force) + +### Bootstrap Khoj Search for Offline Usage later + + You can bootstrap Khoj pre-emptively to run on machines that do not have internet access. An example use-case would be to run Khoj on an air-gapped machine. + Note: *Only search can currently run in fully offline mode, not chat.* + + - With Internet + 1. Manually download the [asymmetric text](https://huggingface.co/sentence-transformers/multi-qa-MiniLM-L6-cos-v1), [symmetric text](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)and [image search](https://huggingface.co/sentence-transformers/clip-ViT-B-32) models from HuggingFace + 2. Pip install khoj (and dependencies) in an associated virtualenv. E.g `python -m venv .venv && source .venv/bin/activate && pip install khoj-assistant` + - Without Internet + 1. Copy each of the search models into their respective folders, `asymmetric`, `symmetric` and `image` under the `~/.khoj/search/` directory on the air-gapped machine + 2. Copy the khoj virtual environment directory onto the air-gapped machine, activate the environment and start and khoj as normal. E.g `source .venv/bin/activate && khoj` + + +## Miscellaneous +### Set your OpenAI API key in Khoj +If you want, Khoj can be configured to use OpenAI for search and chat.
+Add your OpenAI API to Khoj by using either of the two options below: + - Open your [Khoj settings](http://localhost:42110/config/processor/conversation), add your OpenAI API key, and click *Save*. Then go to your [Khoj settings](http://localhost:42110/config) and click `Configure`. This will refresh Khoj with your OpenAI API key. + - Set `openai-api-key` field under `processor.conversation` section in your `khoj.yml`[^1] to your [OpenAI API key](https://beta.openai.com/account/api-keys) and restart khoj: + ```diff + processor: + conversation: + - openai-api-key: # "YOUR_OPENAI_API_KEY" + + openai-api-key: sk-aaaaaaaaaaaaaaaaaaaaaaaahhhhhhhhhhhhhhhhhhhhhhhh + model: "text-davinci-003" + conversation-logfile: "~/.khoj/processor/conversation/conversation_logs.json" + ``` + +!> **Warning**: This will enable Khoj to send your query and note(s) to OpenAI for processing diff --git a/docs/assets/favicon-128x128.ico b/docs/assets/favicon-128x128.ico new file mode 100644 index 00000000..ed751b2d Binary files /dev/null and b/docs/assets/favicon-128x128.ico differ diff --git a/docs/interfaces.png b/docs/assets/interfaces.png similarity index 100% rename from docs/interfaces.png rename to docs/assets/interfaces.png diff --git a/docs/assets/khoj-logo-sideways-200.png b/docs/assets/khoj-logo-sideways-200.png new file mode 100644 index 00000000..52ccdfa4 Binary files /dev/null and b/docs/assets/khoj-logo-sideways-200.png differ diff --git a/docs/assets/khoj-logo-sideways.svg b/docs/assets/khoj-logo-sideways.svg new file mode 100644 index 00000000..12e065ba --- /dev/null +++ b/docs/assets/khoj-logo-sideways.svg @@ -0,0 +1,5385 @@ + + + + + + + + + + + Logo 1024px + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Logo 1024px + + + + diff --git a/docs/khoj_architecture.png b/docs/assets/khoj_architecture.png similarity index 100% rename from docs/khoj_architecture.png rename to docs/assets/khoj_architecture.png diff --git a/docs/khoj_chat_on_emacs_0.5.0.png b/docs/assets/khoj_chat_on_emacs_0.5.0.png similarity index 100% rename from docs/khoj_chat_on_emacs_0.5.0.png rename to docs/assets/khoj_chat_on_emacs_0.5.0.png diff --git a/src/interface/obsidian/docs/khoj_chat_on_obsidian_0.6.0.png b/docs/assets/khoj_chat_on_obsidian_0.6.0.png similarity index 100% rename from src/interface/obsidian/docs/khoj_chat_on_obsidian_0.6.0.png rename to docs/assets/khoj_chat_on_obsidian_0.6.0.png diff --git a/docs/khoj_chat_web_interface.png b/docs/assets/khoj_chat_web_interface.png similarity index 100% rename from docs/khoj_chat_web_interface.png rename to docs/assets/khoj_chat_web_interface.png diff --git a/docs/khoj_codebase_visualization_0.2.1.png b/docs/assets/khoj_codebase_visualization_0.2.1.png similarity index 100% rename from docs/khoj_codebase_visualization_0.2.1.png rename to docs/assets/khoj_codebase_visualization_0.2.1.png diff --git a/docs/khoj_emacs_menu.png b/docs/assets/khoj_emacs_menu.png similarity index 100% rename from docs/khoj_emacs_menu.png rename to docs/assets/khoj_emacs_menu.png diff --git a/src/interface/obsidian/docs/khoj_obsidian_codebase_visualization_0.2.1.png b/docs/assets/khoj_obsidian_codebase_visualization_0.2.1.png similarity index 100% rename from src/interface/obsidian/docs/khoj_obsidian_codebase_visualization_0.2.1.png rename to docs/assets/khoj_obsidian_codebase_visualization_0.2.1.png diff --git a/docs/khoj_on_emacs.png b/docs/assets/khoj_on_emacs.png similarity index 100% rename from docs/khoj_on_emacs.png rename to docs/assets/khoj_on_emacs.png diff --git a/src/interface/obsidian/docs/khoj_on_obsidian_0.2.5.png b/docs/assets/khoj_on_obsidian_0.2.5.png similarity index 100% rename from src/interface/obsidian/docs/khoj_on_obsidian_0.2.5.png rename to docs/assets/khoj_on_obsidian_0.2.5.png diff --git a/docs/khoj_pwa_android.png b/docs/assets/khoj_pwa_android.png similarity index 100% rename from docs/khoj_pwa_android.png rename to docs/assets/khoj_pwa_android.png diff --git a/docs/chat.md b/docs/chat.md new file mode 100644 index 00000000..f1799fad --- /dev/null +++ b/docs/chat.md @@ -0,0 +1,21 @@ +### Khoj Chat +#### Overview +- Creates a personal assistant for you to inquire and engage with your notes +- Uses [ChatGPT](https://openai.com/blog/chatgpt) and [Khoj search](#khoj-search). [Offline chat](https://github.com/khoj-ai/khoj/issues/201) is coming soon. +- Supports multi-turn conversations with the relevant notes for context +- Shows reference notes used to generate a response +- **Note**: *Your query and top notes from khoj search will be sent to OpenAI for processing* + +#### Setup +- [Setup your OpenAI API key in Khoj](#set-your-openai-api-key-in-khoj) + +#### Use +1. Open [/chat](http://localhost:42110/chat)[^2] +2. Type your queries and see response by Khoj from your notes + +#### Demo +![](./assets/khoj_chat_web_interface.png) + +### Details +1. Your query is used to retrieve the most relevant notes, if any, using Khoj search +2. These notes, the last few messages and associated metadata is passed to ChatGPT along with your query for a response diff --git a/docs/demos.md b/docs/demos.md new file mode 100644 index 00000000..fb680d1a --- /dev/null +++ b/docs/demos.md @@ -0,0 +1,40 @@ +## Demos + +### Interfaces + +![](./assets/interfaces.png) + +### Khoj in Obsidian +[KhojObsidian](https://github-production-user-asset-6210df.s3.amazonaws.com/6413477/240061700-3e33d8ea-25bb-46c8-a3bf-c92f78d0f56b.mp4 ':include :type=mp4') + +#### Installation + +1. Install Khoj via `pip` and start Khoj backend in a terminal (Run `khoj`) + ```bash + python -m pip install khoj-assistant + khoj + ``` +2. Install Khoj plugin via Community Plugins settings pane on Obsidian app + - Check the new Khoj plugin settings + - Let Khoj backend index the markdown, pdf, Github markdown files in the current Vault + - Open Khoj plugin on Obsidian via Search button on Left Pane + - Search \"*Announce plugin to folks*\" in the [Obsidian Plugin docs](https://marcus.se.net/obsidian-plugin-docs/) + - Jump to the [search result](https://marcus.se.net/obsidian-plugin-docs/publishing/submit-your-plugin) + +### Khoj in Emacs, Browser +[KhojEmacs](https://user-images.githubusercontent.com/6413477/184735169-92c78bf1-d827-4663-9087-a1ea194b8f4b.mp4 ':include :type=mp4') + +#### Installation + +- Install Khoj via pip +- Start Khoj app +- Add this readme and [khoj.el readme](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs) as org-mode for Khoj to index +- Search \"*Setup editor*\" on the Web and Emacs. Re-rank the results for better accuracy +- Top result is what we are looking for, the [section to Install Khoj.el on Emacs](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#2-Install-Khojel) + +#### Analysis + +- The results do not have any words used in the query + - *Based on the top result it seems the re-ranking model understands that Emacs is an editor?* +- The results incrementally update as the query is entered +- The results are re-ranked, for better accuracy, once user hits enter diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 00000000..c4ba859a --- /dev/null +++ b/docs/development.md @@ -0,0 +1,114 @@ +# Development + +## Setup +### Using Pip +#### 1. Install + +```shell +# Get Khoj Code +git clone https://github.com/khoj-ai/khoj && cd khoj + +# Create, Activate Virtual Environment +python3 -m venv .venv && source .venv/bin/activate + +# Install Khoj for Development +pip install -e .[dev] +``` + +#### 2. Run +1. Start Khoj + ```shell + khoj -vv + ``` +2. Configure Khoj + - **Via the Settings UI**: Add files, directories to index the [Khoj settings](http://localhost:42110/config) UI once Khoj has started up. Once you've saved all your settings, click `Configure`. + - **Manually**: + - Copy the `config/khoj_sample.yml` to `~/.khoj/khoj.yml` + - Set `input-files` or `input-filter` in each relevant `content-type` section of `~/.khoj/khoj.yml` + - Set `input-directories` field in `image` `content-type` section + - Delete `content-type` and `processor` sub-section(s) irrelevant for your use-case + - Restart khoj + + Note: Wait after configuration for khoj to Load ML model, generate embeddings and expose API to query notes, images, documents etc specified in config YAML + +### Using Docker +#### 1. Clone + +```shell +git clone https://github.com/khoj-ai/khoj && cd khoj +``` + +#### 2. Configure + +- **Required**: Update [docker-compose.yml](./docker-compose.yml) to mount your images, (org-mode or markdown) notes, PDFs and Github repositories +- **Optional**: Edit application configuration in [khoj_docker.yml](./config/khoj_docker.yml) + +#### 3. Run + +```shell +docker-compose up -d +``` + +*Note: The first run will take time. Let it run, it\'s mostly not hung, just generating embeddings* + +#### 4. Upgrade + +```shell +docker-compose build --pull +``` + +## Validate +### Before Making Changes +1. Install Git Hooks for Validation + ```shell + pre-commit install -t pre-push -t pre-commit + ``` + - This ensures standard code formatting fixes and other checks run automatically on every commit and push + - Note 1: If [pre-commit](https://pre-commit.com/#intro) didn't already get installed, [install it](https://pre-commit.com/#install) via `pip install pre-commit` + - Note 2: To run the pre-commit changes manually, use `pre-commit run --hook-stage manual --all` before creating PR + +### Before Creating PR + +1. Run Tests. If you get an error complaining about a missing `fast_tokenizer_file`, follow the solution [in this Github issue](https://github.com/UKPLab/sentence-transformers/issues/1659). + ```shell + pytest + ``` + +2. Run MyPy to check types + ```shell + mypy --config-file pyproject.toml + ``` + +### After Creating PR +- Automated [validation workflows](.github/workflows) run for every PR. + + Ensure any issues seen by them our fixed + +- Test the python packge created for a PR + 1. Download and extract the zipped `.whl` artifact generated from the pypi workflow run for the PR. + 2. Install (in your virtualenv) with `pip install /path/to/download*.whl>` + 3. Start and use the application to see if it works fine + +## Create Khoj Release +Follow the steps below to [release](https://github.com/debanjum/khoj/releases/) Khoj. This will create a stable release of Khoj on [Pypi](https://pypi.org/project/khoj-assistant/), [Melpa](https://stable.melpa.org/#%252Fkhoj) and [Obsidian](https://obsidian.md/plugins?id%253Dkhoj). It will also create desktop apps of Khoj and attach them to the latest release. + +1. Create and tag release commit by running the bump_version script. The release commit sets version number in required metadata files. + ```shell + ./scripts/bump_version.sh -c "" + ``` +2. Push commit and then the tag to trigger the release workflow to create Release with auto generated release notes. + ```shell + git push origin master # push release commit to khoj repository + git push origin # push release tag to khoj repository + ``` +3. [Optional] Update the Release Notes to highlight new features, fixes and updates + +## Architecture + +![](./assets/khoj_architecture.png) + +## Visualize Codebase + +*[Interactive Visualization](https://mango-dune-07a8b7110.1.azurestaticapps.net/?repo=debanjum%2Fkhoj)* + +![](./assets/khoj_codebase_visualization_0.2.1.png) diff --git a/docs/emacs.md b/docs/emacs.md new file mode 100644 index 00000000..3578a849 --- /dev/null +++ b/docs/emacs.md @@ -0,0 +1,152 @@ +

Khoj LogoEmacs

+ +An AI personal assistance for your digital brain + +Melpa Stable Badge +Melpa Badge + +Build Badge +Test Badge + + +## Features +- **Search** + - **Natural**: Advanced natural language understanding using Transformer based ML Models + - **Local**: Your personal data stays local. All search, indexing is done on your machine* + - **Incremental**: Incremental search for a fast, search-as-you-type experience +- **Chat** + - **Faster answers**: Find answers faster than search + - **Iterative discovery**: Iteratively explore and (re-)discover your notes + - **Assisted creativity**: Smoothly weave across answer retrieval and content generation + +## Interface +### Search UI + +![khoj_emacs](./assets/khoj_on_emacs.png) + +### Chat UI +![khoj_chat_emacs](./assets/khoj_chat_on_emacs_0.5.0.png) + +## Setup +- /Make sure [[https://realpython.com/installing-python/][python]] and [[https://pip.pypa.io/en/stable/installation/][pip]] are installed on your machine/ + +- /khoj.el attempts to automatically install, start and configure the khoj server./ + If this fails, follow [[https://github.com/khoj-ai/khoj/tree/master/#Setup][these instructions]] to manually setup the khoj server. + +### Direct Install +```elisp +M-x package-install khoj +``` + +### Minimal Install +Add below snippet to your Emacs config file. +Indexes your org-agenda files, by default. + +```elisp + ;; Install Khoj Package from MELPA Stable + (use-package khoj + :ensure t + :pin melpa-stable + :bind ("C-c s" . 'khoj) +``` + +- Note: Install `khoj.el` from MELPA (instead of MELPA Stable) if you installed the pre-release version of khoj + - That is, use `:pin melpa` to install khoj.el in above snippet if khoj server was installed with `--pre` flag, i.e `pip install --pre khoj-assistant` + - Else use `:pin melpa-stable` to install khoj.el in above snippet if khoj was installed with `pip install khoj-assistant` + - This ensures both khoj.el and khoj app are from the same version (git tagged or latest) + +### Standard Install + Add below snippet to your Emacs config file. + Indexes the specified org files, directories. Sets up OpenAI API key for Khoj Chat + +```elisp +;; Install Khoj Package from MELPA Stable +(use-package khoj + :ensure t + :pin melpa-stable + :bind ("C-c s" . 'khoj) + :config (setq khoj-org-directories '("~/docs/org-roam" "~/docs/notes") + khoj-org-files '("~/docs/todo.org" "~/docs/work.org") + khoj-openai-api-key "YOUR_OPENAI_API_KEY")) ; required to enable chat +``` + +### With [https://github.com/raxod502/straight.el](Straight.el) +Add below snippet to your Emacs config file. +Indexes the specified org files, directories. Sets up OpenAI API key for Khoj Chat + +```elisp + ;; Install Khoj Package using Straight.el + (use-package khoj + :after org + :straight (khoj :type git :host github :repo "khoj-ai/khoj" :files (:defaults "src/interface/emacs/khoj.el")) + :bind ("C-c s" . 'khoj) + :config (setq khoj-org-directories '("~/docs/org-roam" "~/docs/notes") + khoj-org-files '("~/docs/todo.org" "~/docs/work.org") + khoj-openai-api-key "YOUR_OPENAI_API_KEY" ; required to enable chat) + ``` + +## Use +### Search +1. Hit `C-c s s` (or `M-x khoj RET s`) to open khoj search + +2. Enter your query in natural language + + e.g "What is the meaning of life?", "My life goals for 2023" + +### Chat +1. Hit `C-c s c` (or `M-x khoj RET c`) to open khoj chat + +2. Ask questions in a natural, conversational style + + E.g "When did I file my taxes last year?" + + See [Khoj Chat](./README.md#khoj-chat) for more details + +### Find Similar Entries +This feature finds entries similar to the one you are currently on. +1. Move cursor to the org-mode entry, markdown section or text paragraph you want to find similar entries for +2. Hit `C-c s f` (or `M-x khoj RET f`) to find similar entries + +### Advanced Usage +- Add [[https://github.com/khoj-ai/khoj/#query-filters][query filters]] during search to narrow down results further + + e.g `What is the meaning of life? -"god" +"none" dt>"last week"` + +- Use `C-c C-o 2` to open the current result at cursor in its source org file + - This calls `M-x org-open-at-point` on the current entry and opens the second link in the entry. + - The second link is the entries [org-id](https://orgmode.org/manual/Handling-Links.html#FOOT28), if set, or the heading text. + The first link is the line number of the entry in the source file. This link is less robust to file changes. + - Note: If you have [speed keys](https://orgmode.org/manual/Speed-Keys.html) enabled, `o 2` will also work + +### Khoj Menu +![](./assets/khoj_emacs_menu.png) +Hit `C-c s` (or `M-x khoj`) to open the khoj menu above. Then: +- Hit `t` until you preferred content type is selected in the khoj menu + + `Content Type` specifies the content to perform `Search`, `Update` or `Find Similar` actions on +- Hit `n` twice and then enter number of results you want to see + + `Results Count` is used by the `Search` and `Find Similar` actions +- Hit `-f u` to `force` update the khoj content index + + The `Force Update` switch is only used by the `Update` action + +## Upgrade +### Upgrade Khoj Backend +```bash +pip install --upgrade khoj-assistant +``` +### Upgrade Khoj.el +Use your Emacs package manager to upgrade `khoj.el` + +- For `khoj.el` from MELPA + - Method 1 + - Run `M-x package-list-packages` to list all packages + - Press `U` on `khoj` to mark it for upgrade + - Press `x` to execute the marked actions + - Method 2 + - Run `M-x package-refresh-content` + - Run `M-x package-reinstall khoj` + +- For `khoj.el` from Straight + - Run `M-x straight-pull-package khoj` diff --git a/docs/features.md b/docs/features.md new file mode 100644 index 00000000..a1f4184a --- /dev/null +++ b/docs/features.md @@ -0,0 +1,13 @@ +## Features +- **Search** + - **Local**: Your personal data stays local. All search and indexing is done on your machine. *Unlike chat which requires access to GPT.* + - **Incremental**: Incremental search for a fast, search-as-you-type experience +- **Chat** + - **Faster answers**: Find answers faster, smoother than search. No need to manually scan through your notes to find answers. + - **Iterative discovery**: Iteratively explore and (re-)discover your notes + - **Assisted creativity**: Smoothly weave across answers retrieval and content generation +- **General** + - **Natural**: Advanced natural language understanding using Transformer based ML Models + - **Pluggable**: Modular architecture makes it easy to plug in new data sources, frontends and ML models + - **Multiple Sources**: Index your Org-mode and Markdown notes, PDF files, Github repositories, and Photos + - **Multiple Interfaces**: Interact from your [Web Browser](https://github.com/khoj-ai/khoj/blob/master/src/khoj/interface/web/index.html), [Emacs](https://github.com/khoj-ai/khoj/blob/master//src/interface/emacs/khoj.el) or [Obsidian](https://github.com/khoj-ai/khoj/blob/master/src/interface/obsidian/) diff --git a/docs/github_integration.md b/docs/github_integration.md new file mode 100644 index 00000000..6b8dce48 --- /dev/null +++ b/docs/github_integration.md @@ -0,0 +1,14 @@ +# Setup the Github integration + +The Github integration allows you to index as many repositories as you want. It's currently default configured to index Issues, Commits, and all Markdown/Org files in each repository. For large repositories, this takes a fairly long time, but it works well for smaller projects. + +# Configure your settings + +1. Go to [http://localhost:42110/config](http://localhost:42110/config) and enter in settings for the data sources you want to index. You'll have to specify the file paths. + +## Use the Github plugin + +1. Generate a [classic PAT (personal access token)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) from [Github](https://github.com/settings/tokens) with `repo` and `admin:org` scopes at least. +2. Navigate to [http://localhost:42110/config/content_type/github](http://localhost:42110/config/content_type/github) to configure your Github settings. Enter in your PAT, along with details for each repository you want to index. +3. Click `Save`. Go back to the settings page and click `Configure`. +4. Go to [http://localhost:42110/](http://localhost:42110/) and start searching! diff --git a/docs/index.html b/docs/index.html new file mode 100644 index 00000000..be0fdb3f --- /dev/null +++ b/docs/index.html @@ -0,0 +1,35 @@ + + + + + Document + + + + + + + +
+ + + + + + + + + + diff --git a/docs/notion_integration.md b/docs/notion_integration.md new file mode 100644 index 00000000..a2ca6734 --- /dev/null +++ b/docs/notion_integration.md @@ -0,0 +1,14 @@ +## 📜 Notion Integration + +Khoj now supports search/chat with pages in your Notion workspaces. [Notion](notion.so/) is a platform people use for taking notes, especially for collaboration. + +We haven't setup a fancy integration with OAuth yet, so this integration still requires some effort on your end to generate an API key. + +1. Go to https://www.notion.so/my-integrations and create a new integration called Khoj to get an API key. +![setup_new_integration](https://github.com/khoj-ai/khoj/assets/65192171/b056e057-d4dc-47dc-aad3-57b59a22c68b) +3. Share all the workspaces that you want to integrate with the Khoj integration you just made in the previous step +![enable_workspace](https://github.com/khoj-ai/khoj/assets/65192171/98290303-b5b8-4cb0-b32c-f68c6923a3d0) +4. In the first step, you generated an API key. Use the newly generated API Key in your Khoj settings, by default at http://localhost:8000/config/content_type/notion. Click `Save`. +5. Click `Configure` in http://localhost:8000/config to index your Notion workspace(s). + +That's it! You should be ready to start searching and chatting. Make sure you've configured your OpenAI API Key for chat. diff --git a/src/interface/obsidian/README.md b/docs/obsidian.md similarity index 79% rename from src/interface/obsidian/README.md rename to docs/obsidian.md index c63fec7f..95b3c84f 100644 --- a/src/interface/obsidian/README.md +++ b/docs/obsidian.md @@ -1,29 +1,7 @@ -Khoj LogoObsidian +

Khoj LogoObsidian

> An AI personal assistant for your Digital Brain in Obsidian -## Table of Contents - -- [Features](#Features) -- [Demo](#Demo) - - [Search Demo](#Search-Demo) -- [Interfaces](#Interfaces) - - [Search Modal](#Search-Modal) - - [Chat Modal](#Chat-Modal) -- [Setup](#Setup) - - [Setup Backend](#1-Setup-Backend) - - [Setup Plugin](#2-Setup-Plugin) -- [Use](#Use) - - [Search](#search) - - [Chat](#chat) - - [Find Similar Notes](#find-similar-notes) -- [Upgrade](#Upgrade) - - [Upgrade Backend](#1-Upgrade-Backend) - - [Upgrade Plugin](#2-Upgrade-Plugin) -- [Troubleshoot](#Troubleshoot) -- [Visualize Codebase](#Visualize-Codebase) -- [Implementation](#Implementation) - ## Features - **Search** - **Natural**: Advanced natural language understanding using Transformer based ML Models @@ -34,31 +12,6 @@ - **Iterative discovery**: Iteratively explore and (re-)discover your notes - **Assisted creativity**: Smoothly weave across answers retrieval and content generation -## Demo -### Search Demo -https://github.com/khoj-ai/khoj/assets/6413477/3e33d8ea-25bb-46c8-a3bf-c92f78d0f56b - -
Description - -1. Install Khoj via `pip` and start Khoj backend - ```shell - python -m pip install khoj-assistant && khoj - ``` -2. Install Khoj plugin via Community Plugins settings pane on Obsidian app - - Check the new Khoj plugin settings - - Wait for Khoj backend to index markdown, PDF files in the current Vault - - Open Khoj plugin on Obsidian via Search button on Left Pane - - Search \"*Announce plugin to folks*\" in the [Obsidian Plugin docs](https://marcus.se.net/obsidian-plugin-docs/) - - Jump to the [search result](https://marcus.se.net/obsidian-plugin-docs/publishing/submit-your-plugin) - -
- -## Interfaces -### Search Modal -![](https://github.com/khoj-ai/khoj/blob/master/src/interface/obsidian/docs/khoj_on_obsidian_0.2.5.png?) - -### Chat Modal -![](https://github.com/khoj-ai/khoj/blob/master/src/interface/obsidian/docs/khoj_chat_on_obsidian_0.6.0.png?) ## Setup - *Make sure [python](https://realpython.com/installing-python/) and [pip](https://pip.pypa.io/en/stable/installation/) are installed on your machine* @@ -94,16 +47,14 @@ Notes: See [Khoj Chat](https://github.com/khoj-ai/khoj/tree/master/#Khoj-Chat) for more details -![](https://github.com/khoj-ai/khoj/blob/master/src/interface/obsidian/docs/khoj_chat_on_obsidian_0.6.0.png?) - ### Search Click the *Khoj search* icon 🔎 on the [Ribbon](https://help.obsidian.md/User+interface/Workspace/Ribbon) or run *Khoj: Search* from the [Command Palette](https://help.obsidian.md/Plugins/Command+palette) *Note: Ensure the khoj server is running in the background before searching. Execute `khoj` in your terminal if it is not already running* -https://user-images.githubusercontent.com/6413477/218801155-cd67e8b4-a770-404a-8179-d6b61caa0f93.mp4 +[search_demo](https://user-images.githubusercontent.com/6413477/218801155-cd67e8b4-a770-404a-8179-d6b61caa0f93.mp4 ':include :type=mp4') -
Query Filters +#### Query Filters Use structured query syntax to filter the natural language search results - **Word Filter**: Get entries that include/exclude a specified term @@ -123,8 +74,6 @@ Use structured query syntax to filter the natural language search results - excluding words *"big"* and *"brother"* - that best match the natural language query *"what is the meaning of life?"* -
- ### Find Similar Notes To see other notes similar to the current one, run *Khoj: Find Similar Notes* from the [Command Palette](https://help.obsidian.md/Plugins/Command+palette) @@ -138,6 +87,30 @@ To see other notes similar to the current one, run *Khoj: Find Similar Notes* fr 2. Click the *Check for updates* button 3. Click the *Update* button next to Khoj, if available +## Demo +### Search Demo +[demo](https://github-production-user-asset-6210df.s3.amazonaws.com/6413477/240061700-3e33d8ea-25bb-46c8-a3bf-c92f78d0f56b.mp4 ':include :type=mp4') + +#### Description + +1. Install Khoj via `pip` and start Khoj backend + ```shell + python -m pip install khoj-assistant && khoj + ``` +2. Install Khoj plugin via Community Plugins settings pane on Obsidian app + - Check the new Khoj plugin settings + - Wait for Khoj backend to index markdown, PDF files in the current Vault + - Open Khoj plugin on Obsidian via Search button on Left Pane + - Search \"*Announce plugin to folks*\" in the [Obsidian Plugin docs](https://marcus.se.net/obsidian-plugin-docs/) + - Jump to the [search result](https://marcus.se.net/obsidian-plugin-docs/publishing/submit-your-plugin) + +## Interfaces +### Search Modal +![](./assets/khoj_on_obsidian_0.2.5.png) + +### Chat Modal +![](./assets/khoj_chat_on_obsidian_0.6.0.png) + ## Troubleshooting - Open the Khoj plugin settings pane, to configure Khoj - Toggle Enable/Disable Khoj, if setting changes have not applied @@ -148,7 +121,8 @@ To see other notes similar to the current one, run *Khoj: Find Similar Notes* fr So notes across multiple vaults **cannot** be searched at the same time ## Visualize Codebase - + +![](./assets/khoj_obsidian_codebase_visualization_0.2.1.png) ## Implementation The plugin implements the following functionality to search your notes with Khoj: diff --git a/docs/search.md b/docs/search.md new file mode 100644 index 00000000..dc05e01a --- /dev/null +++ b/docs/search.md @@ -0,0 +1,79 @@ +## Khoj Search +- **Khoj via Obsidian** + - Click the *Khoj search* icon 🔎 on the [Ribbon](https://help.obsidian.md/User+interface/Workspace/Ribbon) or Search for *Khoj: Search* in the [Command Palette](https://help.obsidian.md/Plugins/Command+palette) +- **Khoj via Emacs** + - Run `M-x khoj ` +- **Khoj via Web** + - Open directly +- **Khoj via API** + - See the Khoj FastAPI [Swagger Docs](http://localhost:42110/docs), [ReDocs](http://localhost:42110/redocs) + +### Query Filters + +Use structured query syntax to filter the natural language search results +- **Word Filter**: Get entries that include/exclude a specified term + - Entries that contain term_to_include: `+"term_to_include"` + - Entries that contain term_to_exclude: `-"term_to_exclude"` +- **Date Filter**: Get entries containing dates in YYYY-MM-DD format from specified date (range) + - Entries from April 1st 1984: `dt:"1984-04-01"` + - Entries after March 31st 1984: `dt>="1984-04-01"` + - Entries before April 2nd 1984 : `dt<="1984-04-01"` +- **File Filter**: Get entries from a specified file + - Entries from incoming.org file: `file:"incoming.org"` +- Combined Example + - `what is the meaning of life? file:"1984.org" dt>="1984-01-01" dt<="1985-01-01" -"big" -"brother"` + - Adds all filters to the natural language query. It should return entries + - from the file *1984.org* + - containing dates from the year *1984* + - excluding words *"big"* and *"brother"* + - that best match the natural language query *"what is the meaning of life?"* + +## Details +1. Your query is used to retrieve the most relevant notes, if any, using Khoj search +2. These notes, the last few messages and associated metadata is passed to ChatGPT along with your query for a response + + +## Performance + +### Query performance + +- Semantic search using the bi-encoder is fairly fast at \<50 ms +- Reranking using the cross-encoder is slower at \<2s on 15 results. Tweak `top_k` to tradeoff speed for accuracy of results +- Filters in query (e.g by file, word or date) usually add \<20ms to query latency + +### Indexing performance + +- Indexing is more strongly impacted by the size of the source data +- Indexing 100K+ line corpus of notes takes about 10 minutes +- Indexing 4000+ images takes about 15 minutes and more than 8Gb of RAM +- Note: *It should only take this long on the first run* as the index is incrementally updated + +### Miscellaneous + +- Testing done on a Mac M1 and a \>100K line corpus of notes +- Search, indexing on a GPU has not been tested yet + +## Advanced Usage + +### Use OpenAI Models for Search +#### Setup +1. Set `encoder-type`, `encoder` and `model-directory` under `asymmetric` and/or `symmetric` `search-type` in your `khoj.yml`[^1]: + ```diff + asymmetric: + - encoder: "sentence-transformers/multi-qa-MiniLM-L6-cos-v1" + + encoder: text-embedding-ada-002 + + encoder-type: khoj.utils.models.OpenAI + cross-encoder: "cross-encoder/ms-marco-MiniLM-L-6-v2" + - encoder-type: sentence_transformers.SentenceTransformer + - model_directory: "~/.khoj/search/asymmetric/" + + model-directory: null + ``` +2. [Setup your OpenAI API key in Khoj](#set-your-openai-api-key-in-khoj) +3. Restart Khoj server to generate embeddings. It will take longer than with offline models. + +#### Warnings + This configuration *uses an online model* + - It will **send all notes to OpenAI** to generate embeddings + - **All queries will be sent to OpenAI** when you search with Khoj + - You will be **charged by OpenAI** based on the total tokens processed + - It *requires an active internet connection* to search and index diff --git a/docs/setup.md b/docs/setup.md new file mode 100644 index 00000000..539b2300 --- /dev/null +++ b/docs/setup.md @@ -0,0 +1,98 @@ +## Setup +These are the general setup instructions for Khoj. + +- Make sure [python](https://realpython.com/installing-python/) and [pip](https://pip.pypa.io/en/stable/installation/) are installed on your machine +- Check the [Khoj.el Readme](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Setup) to setup Khoj with Emacs
+ Its simpler as it can skip the server *install*, *run* and *configure* step below. +- Check the [Khoj Obsidian Readme](https://github.com/khoj-ai/khoj/tree/master/src/interface/obsidian#Setup) to setup Khoj with Obsidian
+ Its simpler as it can skip the *configure* step below. + +### 1. Install +Run the following command in your terminal to install the Khoj backend. + +- On Linux/MacOS + ```shell + python -m pip install khoj-assistant + ``` + +- On Windows + ```shell + py -m pip install khoj-assistant + ``` +For more detailed Windows installation and troubleshooting, see [Windows Install](./windows_install.md). + + +### 2. Run + +Run the following commmand from your terminal to start the Khoj backend and open Khoj in your browser. + +```shell +khoj --gui +``` + +Note: To start Khoj automatically in the background use [Task scheduler](https://www.windowscentral.com/how-create-automated-task-using-task-scheduler-windows-10) on Windows or [Cron](https://en.wikipedia.org/wiki/Cron) on Mac, Linux (e.g with `@reboot khoj`) + +### 3. Configure +1. Set `File`, `Folder` and hit `Save` in each Plugins you want to enable for Search on the Khoj config page +2. Add your OpenAI API key to Chat Feature settings if you want to use Chat +3. Click `Configure` and wait. The app will download ML models and index the content for search and (optionally) chat + +### 4. Install Interface Plugins (Optional) +Khoj exposes a web interface to search, chat and configure by default.
+The optional steps below allow using Khoj from within an existing application like Obsidian or Emacs. + +- **Khoj Obsidian**:
+[Install](https://github.com/khoj-ai/khoj/tree/master/src/interface/obsidian#2-Setup-Plugin) the Khoj Obsidian plugin + +- **Khoj Emacs**:
+[Install](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#2-Install-Khojel) khoj.el + + +## Upgrade +### Upgrade Khoj Server +```shell +pip install --upgrade khoj-assistant +``` + +*Note: To upgrade to the latest pre-release version of the khoj server run below command* +```shell +# Maps to the latest commit on the master branch +pip install --upgrade --pre khoj-assistant +``` + +### Upgrade Khoj on Emacs +- Use your Emacs Package Manager to Upgrade +- See [khoj.el readme](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Upgrade) for details + +### Upgrade Khoj on Obsidian +- Upgrade via the Community plugins tab on the settings pane in the Obsidian app +- See the [khoj plugin readme](https://github.com/khoj-ai/khoj/tree/master/src/interface/obsidian#2-Setup-Plugin) for details + +## Uninstall +1. (Optional) Hit `Ctrl-C` in the terminal running the khoj server to stop it +2. Delete the khoj directory in your home folder (i.e `~/.khoj` on Linux, Mac or `C:\Users\\.khoj` on Windows) +3. Uninstall the khoj server with `pip uninstall khoj-assistant` +4. (Optional) Uninstall khoj.el or the khoj obsidian plugin in the standard way on Emacs, Obsidian + +## Troubleshoot + +#### Install fails while building Tokenizer dependency +- **Details**: `pip install khoj-assistant` fails while building the `tokenizers` dependency. Complains about Rust. +- **Fix**: Install Rust to build the tokenizers package. For example on Mac run: + ```shell + brew install rustup + rustup-init + source ~/.cargo/env + ``` +- **Refer**: [Issue with Fix](https://github.com/khoj-ai/khoj/issues/82#issuecomment-1241890946) for more details + +#### Search starts giving wonky results +- **Fix**: Open [/api/update?force=true](http://localhost:42110/api/update?force=true)[^2] in browser to regenerate index from scratch +- **Note**: *This is a fix for when you percieve the search results have degraded. Not if you think they've always given wonky results* + +#### Khoj in Docker errors out with \"Killed\" in error message +- **Fix**: Increase RAM available to Docker Containers in Docker Settings +- **Refer**: [StackOverflow Solution](https://stackoverflow.com/a/50770267), [Configure Resources on Docker for Mac](https://docs.docker.com/desktop/mac/#resources) + +#### Khoj errors out complaining about Tensors mismatch or null +- **Mitigation**: Disable `image` search using the desktop GUI diff --git a/docs/windows_install.md b/docs/windows_install.md new file mode 100644 index 00000000..d9da5bbe --- /dev/null +++ b/docs/windows_install.md @@ -0,0 +1,23 @@ +# Windows Installation + +These steps can be used to setup Khoj on a clean, new Windows 11 machine. It has been tested on a Windows VM + +## Prerequisites +1. Ensure you have Visual Studio C++ Build tools installed. You can download it [from Microsoft here](https://visualstudio.microsoft.com/visual-cpp-build-tools/). At the minimum, you should have the following configuration: + Screenshot 2023-07-12 at 3 56 25 PM +3. Ensure you have Python installed. You can check by running `python --version`. If you don't, install the latest version [from here](https://www.python.org/downloads/). + - Ensure you have pip installed: `py -m ensurepip --upgrade`. + +## Quick start +1. Open a PowerShell terminal. +7. Run `pip install khoj-assistant` +8. Start Khoj with `khoj` + +## Installation in a Virtual Environment +Use this if you want to install with a virtual environment. This will make it much easier to manage your dependencies. You can read more about [virtual environments](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/) here. + +1. Open a PowerShell terminal with the `Run as Administrator` privileges. +5. Create a virtual environment: `mkdir khoj && cd khoj && py -m venv .venv` +6. Activate the virtual environment: `.\.venv\Scripts\activate`. If you get a permissions error, then run `Set-ExecutionPolicy -ExecutionPolicy RemoteSigned`. +7. Run `pip install khoj-assistant` +8. Start Khoj with `khoj` diff --git a/src/interface/emacs/README.org b/src/interface/emacs/README.org deleted file mode 100644 index 7dfb3050..00000000 --- a/src/interface/emacs/README.org +++ /dev/null @@ -1,167 +0,0 @@ -[[https://github.com/khoj-ai/khoj/edit/master/src/interface/emacs/README.org][file:/src/khoj/interface/web/assets/icons/khoj-logo-sideways-200.png]] Emacs - - - [[https://stable.melpa.org/#/khoj][file:https://stable.melpa.org/packages/khoj-badge.svg]] [[https://melpa.org/#/khoj][file:https://melpa.org/packages/khoj-badge.svg]] [[https://github.com/khoj-ai/khoj/actions/workflows/build_khoj_el.yml][https://github.com/khoj-ai/khoj/actions/workflows/build_khoj_el.yml/badge.svg?]] [[https://github.com/khoj-ai/khoj/actions/workflows/test_khoj_el.yml][https://github.com/khoj-ai/khoj/actions/workflows/test_khoj_el.yml/badge.svg?]] - - /An AI personal assistant for your digital brain/ - -** Table of Contents - - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#features][Features]] - - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Interface][Interface]] - - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Setup][Setup]] - - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Direct-Install][Direct Install]] - - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Minimal-Install][Minimal Install]] - - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Standard-Install][Standard Install]] - - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#With-Straight.el][With Straight.el]] - - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Use][Use]] - - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Search][Search]] - - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Chat][Chat]] - - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Find-similar-entries][Find Similar Entries]] - - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Advanced-usage][Advanced Usage]] - - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Khoj-menu][Khoj Menu]] - - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Upgrade][Upgrade]] - - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Upgrade-Khoj-Backend][Upgrade Backend]] - - [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Upgrade-Khojel][Upgrade Khoj.el]] - -** Features - - *Search* - - *Natural*: Advanced natural language understanding using Transformer based ML Models - - *Local*: Your personal data stays local. All search, indexing is done on your machine* - - *Incremental*: Incremental search for a fast, search-as-you-type experience - - *Chat* - - *Faster answers*: Find answers faster than search - - *Iterative discovery*: Iteratively explore and (re-)discover your notes - - *Assisted creativity*: Smoothly weave across answer retrieval and content generation - -** Interface -*** Search UI - [[/docs/khoj_on_emacs.png]] - -*** Chat UI - [[/docs/khoj_chat_on_emacs_0.5.0.png]] - -** Setup -- /Make sure [[https://realpython.com/installing-python/][python]] and [[https://pip.pypa.io/en/stable/installation/][pip]] are installed on your machine/ - -- /khoj.el attempts to automatically install, start and configure the khoj server./ - If this fails, follow [[https://github.com/khoj-ai/khoj/tree/master/#Setup][these instructions]] to manually setup the khoj server. - -*** Direct Install - #+begin_src elisp - M-x package-install khoj - #+end_src - -*** Minimal Install - Add below snippet to your Emacs config file. - Indexes your org-agenda files, by default. - - #+begin_src elisp - ;; Install Khoj Package from MELPA Stable - (use-package khoj - :ensure t - :pin melpa-stable - :bind ("C-c s" . 'khoj) - #+end_src - - - Note: Install ~khoj.el~ from MELPA (instead of MELPA Stable) if you installed the pre-release version of khoj - - That is, use ~:pin melpa~ to install khoj.el in above snippet if khoj server was installed with ~--pre~ flag, i.e ~pip install --pre khoj-assistant~ - - Else use ~:pin melpa-stable~ to install khoj.el in above snippet if khoj was installed with ~pip install khoj-assistant~ - - This ensures both khoj.el and khoj app are from the same version (git tagged or latest) - -*** Standard Install - Add below snippet to your Emacs config file. - Indexes the specified org files, directories. Sets up OpenAI API key for Khoj Chat - - #+begin_src elisp - ;; Install Khoj Package from MELPA Stable - (use-package khoj - :ensure t - :pin melpa-stable - :bind ("C-c s" . 'khoj) - :config (setq khoj-org-directories '("~/docs/org-roam" "~/docs/notes") - khoj-org-files '("~/docs/todo.org" "~/docs/work.org") - khoj-openai-api-key "YOUR_OPENAI_API_KEY")) ; required to enable chat - #+end_src - -*** With [[https://github.com/raxod502/straight.el][Straight.el]] - Add below snippet to your Emacs config file. - Indexes the specified org files, directories. Sets up OpenAI API key for Khoj Chat - - #+begin_src elisp - ;; Install Khoj Package using Straight.el - (use-package khoj - :after org - :straight (khoj :type git :host github :repo "khoj-ai/khoj" :files (:defaults "src/interface/emacs/khoj.el")) - :bind ("C-c s" . 'khoj) - :config (setq khoj-org-directories '("~/docs/org-roam" "~/docs/notes") - khoj-org-files '("~/docs/todo.org" "~/docs/work.org") - khoj-openai-api-key "YOUR_OPENAI_API_KEY" ; required to enable chat) - #+end_src - -** Use -*** Search - 1. Hit ~C-c s s~ (or ~M-x khoj RET s~) to open khoj search - - 2. Enter your query in natural language - - e.g "What is the meaning of life?", "My life goals for 2023" - -*** Chat - 1. Hit ~C-c s c~ (or ~M-x khoj RET c~) to open khoj chat - - 2. Ask questions in a natural, conversational style - - E.g "When did I file my taxes last year?" - - See [[https://github.com/khoj-ai/khoj/tree/master/#Khoj-Chat][Khoj Chat]] for more details - -*** Find Similar Entries - This feature finds entries similar to the one you are currently on. - 1. Move cursor to the org-mode entry, markdown section or text paragraph you want to find similar entries for - 2. Hit ~C-c s f~ (or ~M-x khoj RET f~) to find similar entries - -*** Advanced Usage - - Add [[https://github.com/khoj-ai/khoj/#query-filters][query filters]] during search to narrow down results further - - e.g `What is the meaning of life? -"god" +"none" dt>"last week"` - - - Use ~C-c C-o 2~ to open the current result at cursor in its source org file - - - This calls ~M-x org-open-at-point~ on the current entry and opens the second link in the entry. - - The second link is the entries [[https://orgmode.org/manual/Handling-Links.html#FOOT28][org-id]], if set, or the heading text. - The first link is the line number of the entry in the source file. This link is less robust to file changes. - - Note: If you have [[https://orgmode.org/manual/Speed-Keys.html][speed keys]] enabled, ~o 2~ will also work - -*** Khoj Menu - [[/docs/khoj_emacs_menu.png]] - Hit ~C-c s~ (or ~M-x khoj~) to open the khoj menu above. Then: - - Hit ~t~ until you preferred content type is selected in the khoj menu - - ~Content Type~ specifies the content to perform ~Search~, ~Update~ or ~Find Similar~ actions on - - Hit ~n~ twice and then enter number of results you want to see - - ~Results Count~ is used by the ~Search~ and ~Find Similar~ actions - - Hit ~-f u~ to ~force~ update the khoj content index - - The ~Force Update~ switch is only used by the ~Update~ action - -** Upgrade -*** Upgrade Khoj Backend - #+begin_src shell - pip install --upgrade khoj-assistant - #+end_src - -*** Upgrade Khoj.el - Use your Emacs package manager to upgrade ~khoj.el~ - - - For ~khoj.el~ from MELPA - - Method 1 - - Run ~M-x package-list-packages~ to list all packages - - Press ~U~ on ~khoj~ to mark it for upgrade - - Press ~x~ to execute the marked actions - - Method 2 - - Run ~M-x package-refresh-content~ - - Run ~M-x package-reinstall khoj~ - - - For ~khoj.el~ from Straight - - Run ~M-x straight-pull-package khoj~