mirror of
https://github.com/khoaliber/khoj.git
synced 2026-03-02 21:19:12 +00:00
Migrate to using docusaurus, rather than docsify for documentation (#603)
* Add docusaurus documentation (to replace the docsify setup * Remove older docs * Specify documentation as the gh pages build action working directory
This commit is contained in:
sabaimran
8
documentation/docs/clients/_category_.json
Normal file
8
documentation/docs/clients/_category_.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"label": "Clients",
|
||||
"position": 4,
|
||||
"link": {
|
||||
"type": "generated-index",
|
||||
"description": "Different ways for indexing data with the Khoj backend"
|
||||
}
|
||||
}
|
||||
31
documentation/docs/clients/desktop.md
Normal file
31
documentation/docs/clients/desktop.md
Normal file
@@ -0,0 +1,31 @@
|
||||
---
|
||||
sidebar_position: 1
|
||||
---
|
||||
|
||||
# Desktop
|
||||
|
||||
Get started with using the Desktop app to index any supported files with Khoj.
|
||||
<h1><img src="/img/khoj-logo-sideways-500.png" width="200" alt="Khoj Logo" /> Desktop</h1>
|
||||
|
||||
> An AI copilot for your Second Brain
|
||||
|
||||
## Features
|
||||
- **Chat**
|
||||
- **Faster answers**: Find answers quickly, from your private notes or the public internet
|
||||
- **Assisted creativity**: Smoothly weave across retrieving answers and generating content
|
||||
- **Iterative discovery**: Iteratively explore and re-discover your notes
|
||||
- **Search**
|
||||
- **Natural**: Advanced natural language understanding using Transformer based ML Models
|
||||
- **Incremental**: Incremental search for a fast, search-as-you-type experience
|
||||
|
||||
## Setup
|
||||
|
||||
1. Install the [Khoj Desktop app](https://khoj.dev/downloads) for your OS
|
||||
2. Generate an API key on the [Khoj Web App](https://app.khoj.dev/config#clients)
|
||||
3. Set your Khoj API Key on the *Settings* page of the Khoj Desktop app
|
||||
4. [Optional] Add any files, folders you'd like Khoj to be aware of on the *Settings* page and Click *Save*
|
||||
|
||||
## Interface
|
||||
| Chat | Search |
|
||||
|:----:|:------:|
|
||||
|  |  |
|
||||
140
documentation/docs/clients/emacs.md
Normal file
140
documentation/docs/clients/emacs.md
Normal file
@@ -0,0 +1,140 @@
|
||||
---
|
||||
sidebar_position: 2
|
||||
---
|
||||
|
||||
# Emacs
|
||||
|
||||
Use Khoj with Emacs to search your Second Brain from within Emacs.
|
||||
|
||||
<h1><img src="/img/khoj-logo-sideways-500.png" width="200" alt="Khoj Logo" /> Emacs</h1>
|
||||
|
||||
> An AI copilot for your Second Brain in Emacs
|
||||
|
||||
<img src="https://stable.melpa.org/packages/khoj-badge.svg" width="150" alt="Melpa Stable Badge" />
|
||||
<img src="https://melpa.org/packages/khoj-badge.svg" width="150" alt="Melpa Badge" />
|
||||
|
||||
<img src="https://github.com/khoj-ai/khoj/actions/workflows/build_khoj_el.yml/badge.svg" width="150" alt="Build Badge" />
|
||||
<img src="https://github.com/khoj-ai/khoj/actions/workflows/test_khoj_el.yml/badge.svg" width="150" alt="Test Badge" />
|
||||
|
||||
|
||||
## Features
|
||||
- **Chat**
|
||||
- **Faster answers**: Find answers quickly, from your private notes or the public internet
|
||||
- **Assisted creativity**: Smoothly weave across retrieving answers and generating content
|
||||
- **Iterative discovery**: Iteratively explore and re-discover your notes
|
||||
- **Search**
|
||||
- **Natural**: Advanced natural language understanding using Transformer based ML Models
|
||||
- **Incremental**: Incremental search for a fast, search-as-you-type experience
|
||||
|
||||
## Interface
|
||||
|
||||
| Search | Chat |
|
||||
|:------:|:----:|
|
||||
|  |  |
|
||||
|
||||
## Setup
|
||||
1. Generate an API key on the [Khoj Web App](https://app.khoj.dev/config#clients)
|
||||
2. Add below snippet to your Emacs config file, usually at `~/.emacs.d/init.el`
|
||||
|
||||
|
||||
#### **Direct Install**
|
||||
*Khoj will index your org-agenda files, by default*
|
||||
|
||||
```elisp
|
||||
;; Install Khoj.el
|
||||
M-x package-install khoj
|
||||
|
||||
; Set your Khoj API key
|
||||
(setq khoj-api-key "YOUR_KHOJ_CLOUD_API_KEY")
|
||||
```
|
||||
|
||||
#### **Minimal Install**
|
||||
*Khoj will index your org-agenda files, by default*
|
||||
|
||||
```elisp
|
||||
;; Install Khoj client from MELPA Stable
|
||||
(use-package khoj
|
||||
:ensure t
|
||||
:pin melpa-stable
|
||||
:bind ("C-c s" . 'khoj)
|
||||
:config (setq khoj-api-key "YOUR_KHOJ_CLOUD_API_KEY"))
|
||||
```
|
||||
|
||||
#### **Standard Install**
|
||||
*Configures the specified org files, directories to be indexed by Khoj*
|
||||
|
||||
```elisp
|
||||
;; Install Khoj client from MELPA Stable
|
||||
(use-package khoj
|
||||
:ensure t
|
||||
:pin melpa-stable
|
||||
:bind ("C-c s" . 'khoj)
|
||||
:config (setq khoj-api-key "YOUR_KHOJ_CLOUD_API_KEY"
|
||||
khoj-org-directories '("~/docs/org-roam" "~/docs/notes")
|
||||
khoj-org-files '("~/docs/todo.org" "~/docs/work.org")))
|
||||
```
|
||||
|
||||
#### **Straight.el**
|
||||
*Configures the specified org files, directories to be indexed by Khoj*
|
||||
|
||||
```elisp
|
||||
;; Install Khoj client using Straight.el
|
||||
(use-package khoj
|
||||
:after org
|
||||
:straight (khoj :type git :host github :repo "khoj-ai/khoj" :files (:defaults "src/interface/emacs/khoj.el"))
|
||||
:bind ("C-c s" . 'khoj)
|
||||
:config (setq khoj-api-key "YOUR_KHOJ_CLOUD_API_KEY"
|
||||
khoj-org-directories '("~/docs/org-roam" "~/docs/notes")
|
||||
khoj-org-files '("~/docs/todo.org" "~/docs/work.org")))
|
||||
```
|
||||
|
||||
## Use
|
||||
### Search
|
||||
See [Khoj Search](/features/search.md) for details
|
||||
1. Hit `C-c s s` (or `M-x khoj RET s`) to open khoj search
|
||||
2. Enter your query in natural language<br/>
|
||||
E.g *"What is the meaning of life?"*, *"My life goals for 2023"*
|
||||
|
||||
### Chat
|
||||
See [Khoj Chat](/features/chat.md) for details
|
||||
1. Hit `C-c s c` (or `M-x khoj RET c`) to open khoj chat
|
||||
2. Ask questions in a natural, conversational style<br/>
|
||||
E.g *"When did I file my taxes last year?"*
|
||||
|
||||
### Find Similar Entries
|
||||
This feature finds entries similar to the one you are currently on.
|
||||
1. Move cursor to the org-mode entry, markdown section or text paragraph you want to find similar entries for
|
||||
2. Hit `C-c s f` (or `M-x khoj RET f`) to find similar entries
|
||||
|
||||
### Advanced Usage
|
||||
- Add [query filters](https://github.com/khoj-ai/khoj/#query-filters) during search to narrow down results further
|
||||
e.g `What is the meaning of life? -"god" +"none" dt>"last week"`
|
||||
|
||||
- Use `C-c C-o 2` to open the current result at cursor in its source org file
|
||||
- This calls `M-x org-open-at-point` on the current entry and opens the second link in the entry.
|
||||
- The second link is the entries [org-id](https://orgmode.org/manual/Handling-Links.html#FOOT28), if set, or the heading text.
|
||||
The first link is the line number of the entry in the source file. This link is less robust to file changes.
|
||||
- Note: If you have [speed keys](https://orgmode.org/manual/Speed-Keys.html) enabled, `o 2` will also work
|
||||
|
||||
### Khoj Menu
|
||||
|
||||
Hit `C-c s` (or `M-x khoj`) to open the khoj menu above. Then:
|
||||
- Hit `t` until you preferred content type is selected in the khoj menu
|
||||
`Content Type` specifies the content to perform `Search`, `Update` or `Find Similar` actions on
|
||||
- Hit `n` twice and then enter number of results you want to see
|
||||
`Results Count` is used by the `Search` and `Find Similar` actions
|
||||
- Hit `-f u` to `force` update the khoj content index
|
||||
The `Force Update` switch is only used by the `Update` action
|
||||
|
||||
## Upgrade
|
||||
Use your Emacs package manager to upgrade `khoj.el`
|
||||
<!-- tabs:start -->
|
||||
|
||||
#### **With MELPA**
|
||||
1. Run `M-x package-refresh-content`
|
||||
2. Run `M-x package-reinstall khoj`
|
||||
|
||||
#### **With Straight.el**
|
||||
- Run `M-x straight-pull-package khoj`
|
||||
|
||||
<!-- tabs:end -->
|
||||
63
documentation/docs/clients/obsidian.md
Normal file
63
documentation/docs/clients/obsidian.md
Normal file
@@ -0,0 +1,63 @@
|
||||
---
|
||||
sidebar_position: 3
|
||||
---
|
||||
|
||||
# Obsidian
|
||||
|
||||
Connect Obsidian to Khoj to search your notes and the web, and chat with your Second Brain.
|
||||
|
||||
<h1><img src="/img/khoj-logo-sideways-500.png" width="200" alt="Khoj Logo" /> Obsidian</h1>
|
||||
|
||||
> An AI copilot for your Second Brain in Obsidian
|
||||
|
||||
## Features
|
||||
- **Chat**
|
||||
- **Faster answers**: Find answers quickly, from your private notes or the public internet
|
||||
- **Assisted creativity**: Smoothly weave across retrieving answers and generating content
|
||||
- **Iterative discovery**: Iteratively explore and re-discover your notes
|
||||
- **Search**
|
||||
- **Natural**: Advanced natural language understanding using Transformer based ML Models
|
||||
- **Incremental**: Incremental search for a fast, search-as-you-type experience
|
||||
|
||||
## Interface
|
||||
|
||||
| Search | Chat |
|
||||
|:------:|:----:|
|
||||
|  |  |
|
||||
|
||||
|
||||
## Setup
|
||||
|
||||
1. Open [Khoj](https://obsidian.md/plugins?id=khoj) from the *Community plugins* tab in Obsidian settings panel
|
||||
2. Click *Install*, then *Enable* on the Khoj plugin page in Obsidian
|
||||
3. Generate an API key on the [Khoj Web App](https://app.khoj.dev/config#clients)
|
||||
4. Set your Khoj API Key in the Khoj plugin settings in Obsidian
|
||||
|
||||
See the official [Obsidian Plugin Docs](https://help.obsidian.md/Extending+Obsidian/Community+plugins) for more details on installing Obsidian plugins.
|
||||
|
||||
## Use
|
||||
### Chat
|
||||
Run *Khoj: Chat* from the [Command Palette](https://help.obsidian.md/Plugins/Command+palette) and ask questions in a natural, conversational style.<br />
|
||||
E.g *"When did I file my taxes last year?"*
|
||||
|
||||
See [Khoj Chat](/docs/features/chat) for more details
|
||||
|
||||
### Find Similar Notes
|
||||
To see other notes similar to the current one, run *Khoj: Find Similar Notes* from the [Command Palette](https://help.obsidian.md/Plugins/Command+palette)
|
||||
|
||||
### Search
|
||||
Click the *Khoj search* icon 🔎 on the [Ribbon](https://help.obsidian.md/User+interface/Workspace/Ribbon) or run *Khoj: Search* from the [Command Palette](https://help.obsidian.md/Plugins/Command+palette)
|
||||
|
||||
See [Khoj Search](/docs/features/search) for more details. Use [query filters](/docs/miscellaneous/advanced#query-filters) to limit entries to search
|
||||
|
||||
[search_demo](https://user-images.githubusercontent.com/6413477/218801155-cd67e8b4-a770-404a-8179-d6b61caa0f93.mp4 ':include :type=mp4')
|
||||
|
||||
## Upgrade
|
||||
1. Open *Community plugins* tab in Obsidian settings
|
||||
2. Click the *Check for updates* button
|
||||
3. Click the *Update* button next to Khoj, if available
|
||||
|
||||
## Troubleshooting
|
||||
- Open the Khoj plugin settings pane, to configure Khoj
|
||||
- Toggle Enable/Disable Khoj, if setting changes have not applied
|
||||
- Click *Update* button to force index to refresh, if results are failing or stale
|
||||
29
documentation/docs/clients/web.md
Normal file
29
documentation/docs/clients/web.md
Normal file
@@ -0,0 +1,29 @@
|
||||
---
|
||||
sidebar_position: 4
|
||||
---
|
||||
|
||||
# Web
|
||||
|
||||
Without any desktop clients, you can get started with using Khoj on the web for chat. Bear in mind you do need one of the desktop clients in order to index and sync your data with Khoj.
|
||||
|
||||
<h1><img src="/img/khoj-logo-sideways-500.png" width="200" alt="Khoj Logo" /> Web</h1>
|
||||
|
||||
> An AI copilot for your Second Brain
|
||||
|
||||
## Features
|
||||
- **Chat**
|
||||
- **Faster answers**: Find answers quickly, from your private notes or the public internet
|
||||
- **Assisted creativity**: Smoothly weave across retrieving answers and generating content
|
||||
- **Iterative discovery**: Iteratively explore and re-discover your notes
|
||||
- **Search**
|
||||
- **Natural**: Advanced natural language understanding using Transformer based ML Models
|
||||
- **Incremental**: Incremental search for a fast, search-as-you-type experience
|
||||
|
||||
## Setup
|
||||
No setup required. The Khoj web app is the default interface to Khoj. You can access it from any web browser. Try it on [Khoj Cloud](https://app.khoj.dev)
|
||||
|
||||
## Interface
|
||||
|
||||
| Search | Chat |
|
||||
|:------:|:----:|
|
||||
|  |  |
|
||||
8
documentation/docs/contributing/_category_.json
Normal file
8
documentation/docs/contributing/_category_.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"label": "Contributing",
|
||||
"position": 2,
|
||||
"link": {
|
||||
"type": "generated-index",
|
||||
"description": "Development Setup"
|
||||
}
|
||||
}
|
||||
181
documentation/docs/contributing/development.mdx
Normal file
181
documentation/docs/contributing/development.mdx
Normal file
@@ -0,0 +1,181 @@
|
||||
---
|
||||
sidebar_position: 0
|
||||
---
|
||||
|
||||
# Development
|
||||
|
||||
Welcome to the development docs of Khoj! Thanks for you interesting in being a contributor ❤️. Open source contributors are a corner-store of the Khoj community. We welcome all contributions, big or small.
|
||||
|
||||
To get started with contributing, check out the official GitHub docs on [contributing to an open-source project](https://docs.github.com/en/get-started/exploring-projects-on-github/contributing-to-a-project).
|
||||
|
||||
Join the [Discord](https://discord.gg/WaxF3SkFPU) server and click the ✅ for the question "Are you interested in becoming a contributor?" in the `#welcome-and-rules` channel. This will give you access to the `#contributors` channel where you can ask questions and get help from other contributors.
|
||||
|
||||
If you're looking for a place to get started, check out the list of [Github Issues](https://github.com/khoj-ai/khoj/issues) with the tag `good first issue` to find issues that are good for first-time contributors.
|
||||
|
||||
## Local Server Installation
|
||||
### Using Pip
|
||||
#### 1. Install
|
||||
```mdx-code-block
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
```
|
||||
|
||||
```mdx-code-block
|
||||
<Tabs>
|
||||
<TabItem value="macos" label="MacOS">
|
||||
```shell
|
||||
# Get Khoj Code
|
||||
git clone https://github.com/khoj-ai/khoj && cd khoj
|
||||
|
||||
# Create, Activate Virtual Environment
|
||||
python3 -m venv .venv && source .venv/bin/activate
|
||||
|
||||
# For MacOS or zsh users run this
|
||||
pip install -e '.[dev]'
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="win" label="Windows">
|
||||
```shell
|
||||
# Get Khoj Code
|
||||
git clone https://github.com/khoj-ai/khoj && cd khoj
|
||||
|
||||
# Create, Activate Virtual Environment
|
||||
python3 -m venv .venv && .venv\Scripts\activate
|
||||
|
||||
# Install Khoj for Development
|
||||
pip install -e .[dev]
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="unix" label="Linux">
|
||||
```shell
|
||||
# Get Khoj Code
|
||||
git clone https://github.com/khoj-ai/khoj && cd khoj
|
||||
|
||||
# Create, Activate Virtual Environment
|
||||
python3 -m venv .venv && source .venv/bin/activate
|
||||
|
||||
# Install Khoj for Development
|
||||
pip install -e .[dev]
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
```
|
||||
|
||||
|
||||
#### 2. Run
|
||||
1. Start Khoj
|
||||
```bash
|
||||
khoj -vv
|
||||
```
|
||||
2. Configure Khoj
|
||||
- **Via the Desktop application**: Add files, directories to index using the settings page of your desktop application. Click "Save" to immediately trigger indexing.
|
||||
|
||||
Note: Wait after configuration for khoj to Load ML model, generate embeddings and expose API to query notes, images, documents etc specified in config YAML
|
||||
|
||||
### Using Docker
|
||||
|
||||
Make sure you install the latest version of [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/).
|
||||
|
||||
#### 1. Clone
|
||||
|
||||
```shell
|
||||
git clone https://github.com/khoj-ai/khoj && cd khoj
|
||||
```
|
||||
|
||||
#### 2. Configure
|
||||
|
||||
1. Update [docker-compose.yml](https://github.com/khoj-ai/khoj/blob/master/docker-compose.yml) to use relevant environment variables.
|
||||
2. Comment out the `image` line and uncomment the `build` line in the `server` service
|
||||
|
||||
#### 3. Run
|
||||
|
||||
This will start the Khoj server, and the database.
|
||||
|
||||
```shell
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
#### 4. Upgrade
|
||||
|
||||
If you've made changes to the codebase, you'll need to rebuild the Docker image before running the container again.
|
||||
|
||||
```shell
|
||||
docker-compose build --no-cache
|
||||
```
|
||||
|
||||
## Update clients
|
||||
In whichever clients you're using for testing, you'll need to update the server URL to point to your local server. By default, the local server URL should be `http://127.0.0.1:42110`.
|
||||
|
||||
## Validate
|
||||
### Before Making Changes
|
||||
1. Install Git Hooks for Validation
|
||||
```shell
|
||||
pre-commit install -t pre-push -t pre-commit
|
||||
```
|
||||
- This ensures standard code formatting fixes and other checks run automatically on every commit and push
|
||||
- Note 1: If [pre-commit](https://pre-commit.com/#intro) didn't already get installed, [install it](https://pre-commit.com/#install) via `pip install pre-commit`
|
||||
- Note 2: To run the pre-commit changes manually, use `pre-commit run --hook-stage manual --all` before creating PR
|
||||
|
||||
### Before Creating PR
|
||||
|
||||
:::tip[Note]
|
||||
You should be in an active virtual environment for Khoj in order to run the unit tests and linter.
|
||||
:::
|
||||
|
||||
1. Ensure that you have a [Github Issue](https://github.com/khoj-ai/khoj/issues) that can be linked to the PR. If not, create one. Make sure you've tagged one of the maintainers to the issue. This will ensure that the maintainers are notified of the PR and can review it. It's best discuss the code design on an existing issue or Discord thread before creating a PR. This helps get your PR merged faster.
|
||||
1. Run unit tests.
|
||||
```shell
|
||||
pytest
|
||||
```
|
||||
2. Run the linter.
|
||||
```shell
|
||||
mypy
|
||||
```
|
||||
4. Think about how to add unit tests to verify the functionality you're adding in the PR. If you're not sure how to do this, ask for help in the Github issue or on Discord's `#contributors` channel.
|
||||
|
||||
### After Creating PR
|
||||
1. Automated [validation workflows](https://github.com/khoj-ai/khoj/tree/master/.github/workflows) should run for every PR. Tag one of the maintainers in the PR to trigger it.
|
||||
|
||||
## Obsidian Plugin Development
|
||||
### Plugin development setup
|
||||
The core code for the Obsidian plugin is under `src/interface/obsidian`. The file `main.ts` is a good place to start.
|
||||
|
||||
1. In your CLI, go to the directory `src/interface/obsidian` in the Khoj repository.
|
||||
2. Run `yarn install` to install the dependencies.
|
||||
3. Run `yarn dev` to start the development server. This will continually rebuild the plugin as you make changes to the code.
|
||||
- Your code changes will be outputted to a file called `main.js` in the `obsidian` directory.
|
||||
|
||||
### Loading your development plugin in Obsidian
|
||||
1. Make sure you have the Khoj plugin installed in Obsidian. [See the plugin page](https://publish.obsidian.md/hub/02+-+Community+Expansions/02.05+All+Community+Expansions/Plugins/khoj).
|
||||
1. Open Obsidian and go to your settings (gear icon in the bottom left corner)
|
||||
2. Click on 'Community Plugins' in the left panel
|
||||
3. Next to the 'Installed Plugins' heading, click on the folder icon to open the folder with the plugin's source code.
|
||||
4. Open the `khoj` folder in the file explorer that opens. You'll see a file called `main.js` in this folder. To test your changes, replace this file with the `main.js` file that was generated by the development server in the previous section.
|
||||
|
||||
## Create Khoj Release (Only for Maintainers)
|
||||
Follow the steps below to [release](https://github.com/debanjum/khoj/releases/) Khoj. This will create a stable release of Khoj on [Pypi](https://pypi.org/project/khoj-assistant/), [Melpa](https://stable.melpa.org/#%252Fkhoj) and [Obsidian](https://obsidian.md/plugins?id%253Dkhoj). It will also create desktop apps of Khoj and attach them to the latest release.
|
||||
|
||||
1. Create and tag release commit by running the bump_version script. The release commit sets version number in required metadata files.
|
||||
```shell
|
||||
./scripts/bump_version.sh -c "<release_version>"
|
||||
```
|
||||
2. Push commit and then the tag to trigger the release workflow to create Release with auto generated release notes.
|
||||
```shell
|
||||
git push origin master # push release commit to khoj repository
|
||||
git push origin <release_version> # push release tag to khoj repository
|
||||
```
|
||||
3. [Optional] Update the Release Notes to highlight new features, fixes and updates
|
||||
|
||||
## Architecture
|
||||
|
||||
|
||||
|
||||
## Visualize Codebase
|
||||
|
||||
*[Interactive Visualization](https://mango-dune-07a8b7110.1.azurestaticapps.net/?repo=debanjum%2Fkhoj)*
|
||||
|
||||
|
||||
|
||||
## Visualize Khoj Obsidian Plugin Codebase
|
||||
|
||||
|
||||
8
documentation/docs/features/_category_.json
Normal file
8
documentation/docs/features/_category_.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"label": "Features",
|
||||
"position": 3,
|
||||
"link": {
|
||||
"type": "generated-index",
|
||||
"description": "Features supported by Khoj"
|
||||
}
|
||||
}
|
||||
35
documentation/docs/features/all_features.md
Normal file
35
documentation/docs/features/all_features.md
Normal file
@@ -0,0 +1,35 @@
|
||||
---
|
||||
sidebar_position: 1
|
||||
---
|
||||
|
||||
# Features
|
||||
|
||||
Khoj supports a variety of features, including search and chat with a wide range of data sources and interfaces.
|
||||
|
||||
#### [Search](/docs/features/search.md)
|
||||
- **Local**: Your personal data stays local. All search and indexing is done on your machine.
|
||||
- **Incremental**: Incremental search for a fast, search-as-you-type experience
|
||||
|
||||
#### [Chat](/docs/features/chat.md)
|
||||
- **Faster answers**: Find answers faster, smoother than search. No need to manually scan through your notes to find answers.
|
||||
- **Iterative discovery**: Iteratively explore and (re-)discover your notes
|
||||
- **Assisted creativity**: Smoothly weave across answers retrieval and content generation
|
||||
|
||||
#### General
|
||||
- **Natural**: Advanced natural language understanding using Transformer based ML Models
|
||||
- **Pluggable**: Modular architecture makes it easy to plug in new data sources, frontends and ML models
|
||||
- **Multiple Sources**: Index your Org-mode and Markdown notes, PDF files, Github repositories, and Photos
|
||||
- **Multiple Interfaces**: Interact from your [Web Browser](/docs/clients/web.md), [Emacs](/docs/clients/emacs) or [Obsidian](/docs/clients/obsidian)
|
||||
|
||||
### Supported Interfaces
|
||||
|
||||
[](/docs/clients/emacs)
|
||||
<span> </span>
|
||||
[](/docs/clients/obsidian)
|
||||
|
||||
### Supported Data Sources
|
||||
- markdown
|
||||
- org-mode
|
||||
- pdf
|
||||
- [github](/docs/online-data-sources/github_integration.md)
|
||||
- [notion](/docs/online-data-sources/notion_integration.md)
|
||||
64
documentation/docs/features/chat.md
Normal file
64
documentation/docs/features/chat.md
Normal file
@@ -0,0 +1,64 @@
|
||||
---
|
||||
sidebar_position: 2
|
||||
---
|
||||
|
||||
# Chat
|
||||
|
||||
You can configure Khoj to chat with you about anything. When relevant, it'll use any notes or documents you shared with it to respond.
|
||||
|
||||
### Overview
|
||||
- Creates a personal assistant for you to inquire and engage with your notes
|
||||
- You can choose to use Online or Offline Chat depending on your requirements
|
||||
- Supports multi-turn conversations with the relevant notes for context
|
||||
- Shows reference notes used to generate a response
|
||||
|
||||
### Setup (Self-Hosting)
|
||||
#### Offline Chat
|
||||
Offline chat stays completely private and works without internet using open-source models.
|
||||
|
||||
> **System Requirements**:
|
||||
> - Minimum 8 GB RAM. Recommend **16Gb VRAM**
|
||||
> - Minimum **5 GB of Disk** available
|
||||
> - A CPU supporting [AVX or AVX2 instructions](https://en.wikipedia.org/wiki/Advanced_Vector_Extensions) is required
|
||||
> - A Mac M1+ or [Vulcan supported GPU](https://vulkan.gpuinfo.org/) should significantly speed up chat response times
|
||||
|
||||
1. Open your [Khoj offline settings](http://localhost:42110/server/admin/database/offlinechatprocessorconversationconfig/) and click *Enable* on the Offline Chat configuration.
|
||||
2. Open your [Chat model options](http://localhost:42110/server/admin/database/chatmodeloptions/) and add a new option for the offline chat model you want to use. Make sure to use `Offline` as its type. We currently only support offline models that use the [Llama chat prompt](https://replicate.com/blog/how-to-prompt-llama#wrap-user-input-with-inst-inst-tags) format. We recommend using `mistral-7b-instruct-v0.1.Q4_0.gguf`.
|
||||
|
||||
|
||||
:::tip[Note]
|
||||
Offline chat is not supported for a multi-user scenario. The host machine will encounter segmentation faults if multiple users try to use offline chat at the same time.
|
||||
:::
|
||||
|
||||
#### Online Chat
|
||||
Online chat requires internet to use ChatGPT but is faster, higher quality and less compute intensive.
|
||||
|
||||
:::danger[Warning]
|
||||
This will enable Khoj to send your chat queries and query relevant notes to OpenAI for processing.
|
||||
:::
|
||||
|
||||
1. Get your [OpenAI API Key](https://platform.openai.com/account/api-keys)
|
||||
2. Open your [Khoj Online Chat settings](http://localhost:42110/server/admin/database/openaiprocessorconversationconfig/). Add a new setting with your OpenAI API key, and click *Save*. Only one configuration will be used, so make sure that's the only one you have.
|
||||
3. Open your [Chat model options](http://localhost:42110/server/admin/database/chatmodeloptions/) and add a new option for the OpenAI chat model you want to use. Make sure to use `OpenAI` as its type.
|
||||
|
||||
### Use
|
||||
1. Open Khoj Chat
|
||||
- **On Web**: Open [/chat](https://app.khoj.dev/chat) in your web browser
|
||||
- **On Obsidian**: Search for *Khoj: Chat* in the [Command Palette](https://help.obsidian.md/Plugins/Command+palette)
|
||||
- **On Emacs**: Run `M-x khoj <user-query>`
|
||||
2. Enter your queries to chat with Khoj. Use [slash commands](#commands) and [query filters](/docs/miscellaneous/advanced.md#query-filters) to change what Khoj uses to respond
|
||||
|
||||
|
||||
|
||||
#### Details
|
||||
1. Your query is used to retrieve the most relevant notes, if any, using Khoj search
|
||||
2. These notes, the last few messages and associated metadata is passed to the enabled chat model along with your query to generate a response
|
||||
|
||||
#### Commands
|
||||
Slash commands allows you to change what Khoj uses to respond to your query
|
||||
- **/notes**: Limit chat to only respond using your notes, not just Khoj's general world knowledge as reference
|
||||
- **/general**: Limit chat to only respond using Khoj's general world knowledge, not using your notes as reference
|
||||
- **/default**: Allow chat to respond using your notes or it's general knowledge as reference. It's the default behavior when no slash command is used
|
||||
- **/online**: Use online information and incorporate it in the prompt to the LLM to send you a response.
|
||||
- **/image**: Generate an image in response to your query.
|
||||
- **/help**: Use /help to get all available commands and general information about Khoj
|
||||
17
documentation/docs/features/search.md
Normal file
17
documentation/docs/features/search.md
Normal file
@@ -0,0 +1,17 @@
|
||||
---
|
||||
sidebar_position: 3
|
||||
---
|
||||
|
||||
# Search
|
||||
|
||||
Take advantage of super fast search to find relevant notes and documents from your Second Brain.
|
||||
|
||||
### Use
|
||||
1. Open Khoj Search
|
||||
- **On Web**: Open https://app.khoj.dev/ in your web browser
|
||||
- **On Obsidian**: Click the *Khoj search* icon 🔎 on the [Ribbon](https://help.obsidian.md/User+interface/Workspace/Ribbon) or Search for *Khoj: Search* in the [Command Palette](https://help.obsidian.md/Plugins/Command+palette)
|
||||
- **On Emacs**: Run `M-x khoj <user-query>`
|
||||
2. Query using natural language to find relevant entries from your knowledge base. Use [query filters](/docs/miscellaneous/advanced.md#query-filters) to limit entries to search
|
||||
|
||||
### Demo
|
||||
|
||||
8
documentation/docs/get-started/_category_.json
Normal file
8
documentation/docs/get-started/_category_.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"label": "Get Started",
|
||||
"position": 1,
|
||||
"link": {
|
||||
"type": "generated-index",
|
||||
"description": "Learn how to get started with using Khoj"
|
||||
}
|
||||
}
|
||||
51
documentation/docs/get-started/demos.md
Normal file
51
documentation/docs/get-started/demos.md
Normal file
@@ -0,0 +1,51 @@
|
||||
---
|
||||
sidebar_position: 2
|
||||
---
|
||||
|
||||
# Demos
|
||||
|
||||
Check out a couple of demos and screenshots of Khoj in action.
|
||||
|
||||
### Screenshots
|
||||
|
||||
| Web | Obsidian | Emacs |
|
||||
|:---:|:--------:|:-----:|
|
||||
|  |  |  |
|
||||
|  |  |  |
|
||||
|
||||
|
||||
### Videos
|
||||
#### Khoj in Obsidian
|
||||
[Link to Video](https://github-production-user-asset-6210df.s3.amazonaws.com/6413477/240061700-3e33d8ea-25bb-46c8-a3bf-c92f78d0f56b.mp4)
|
||||
|
||||
##### Installation
|
||||
|
||||
1. Install Khoj via `pip` and start Khoj backend in a terminal (Run `khoj`)
|
||||
```bash
|
||||
python -m pip install khoj-assistant
|
||||
khoj
|
||||
```
|
||||
2. Install Khoj plugin via Community Plugins settings pane on Obsidian app
|
||||
- Check the new Khoj plugin settings
|
||||
- Let Khoj backend index the markdown, pdf, Github markdown files in the current Vault
|
||||
- Open Khoj plugin on Obsidian via Search button on Left Pane
|
||||
- Search \"*Announce plugin to folks*\" in the [Obsidian Plugin docs](https://marcus.se.net/obsidian-plugin-docs/)
|
||||
- Jump to the [search result](https://marcus.se.net/obsidian-plugin-docs/publishing/submit-your-plugin)
|
||||
|
||||
#### Khoj in Emacs, Browser
|
||||
[Link to Video](https://user-images.githubusercontent.com/6413477/184735169-92c78bf1-d827-4663-9087-a1ea194b8f4b.mp4)
|
||||
|
||||
##### Installation
|
||||
|
||||
- Install Khoj via pip
|
||||
- Start Khoj app
|
||||
- Add this readme and [khoj.el readme](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs) as org-mode for Khoj to index
|
||||
- Search \"*Setup editor*\" on the Web and Emacs. Re-rank the results for better accuracy
|
||||
- Top result is what we are looking for, the [section to Install Khoj.el on Emacs](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#2-Install-Khojel)
|
||||
|
||||
##### Analysis
|
||||
|
||||
- The results do not have any words used in the query
|
||||
- *Based on the top result it seems the re-ranking model understands that Emacs is an editor?*
|
||||
- The results incrementally update as the query is entered
|
||||
- The results are re-ranked, for better accuracy, once user hits enter
|
||||
61
documentation/docs/get-started/overview.md
Normal file
61
documentation/docs/get-started/overview.md
Normal file
@@ -0,0 +1,61 @@
|
||||
---
|
||||
sidebar_position: 0
|
||||
---
|
||||
|
||||
# Overview
|
||||
|
||||
Learn about Khoj and its features.
|
||||
|
||||
<p align="center"><img src="/img/khoj-logo-sideways-500.png" width="200" alt="Khoj Logo"></img></p>
|
||||
|
||||
<div align="center">
|
||||
|
||||
[](https://github.com/khoj-ai/khoj/actions/workflows/test.yml)
|
||||
[](https://github.com/khoj-ai/khoj/pkgs/container/khoj)
|
||||
[](https://pypi.org/project/khoj-assistant/)
|
||||
|
||||
</div>
|
||||
|
||||
<div align="center">
|
||||
<b>An AI copilot for your Second Brain</b>
|
||||
|
||||
</div>
|
||||
|
||||
<div align="center">
|
||||
|
||||
[📜 Explore Code](https://github.com/khoj-ai/khoj)
|
||||
<span> • </span>
|
||||
[🌍 Try Khoj Cloud](https://khoj.dev)
|
||||
<span> • </span>
|
||||
[💬 Get Involved](https://discord.gg/BDgyabRM6e)
|
||||
|
||||
</div>
|
||||
|
||||
## Introduction
|
||||
Welcome to the Khoj Docs! This is the best place to get setup and explore Khoj's features.
|
||||
|
||||
- Khoj is an open source, personal AI
|
||||
- You can [chat](/features/chat.md) with it about anything. When relevant, it'll use any notes or documents you shared with it to respond
|
||||
- Quickly [find](/features/search.md) relevant notes and documents using natural language
|
||||
- It understands pdf, plaintext, markdown, org-mode files, [notion pages](/online-data-sources/notion_integration.md) and [github repositories](/online-data-sources/github_integration.md)
|
||||
- Access it from your [Emacs](/clients/emacs.md), [Obsidian](/clients/obsidian.md), [Web browser](/clients/web.md) or the [Khoj Desktop app](/clients/desktop.md)
|
||||
- You can self-host Khoj on your consumer hardware or share it with your family, friends or team from your private cloud
|
||||
|
||||
## Quickstart
|
||||
- [Try Khoj Cloud](https://app.khoj.dev) to get started quickly
|
||||
- [Read these instructions](/get-started/setup.mdx) to self-host a private instance of Khoj
|
||||
|
||||
## At a Glance
|
||||
<img src="https://docs.khoj.dev/assets/khoj_search_on_web.png" width="400px" />
|
||||
<span> </span>
|
||||
<img src="https://docs.khoj.dev/assets/khoj_chat_on_web.png" width="400px" />
|
||||
|
||||
#### [Search](/features/search.md)
|
||||
- **Natural**: Use natural language queries to quickly find relevant notes and documents.
|
||||
- **Incremental**: Incremental search for a fast, search-as-you-type experience
|
||||
|
||||
#### [Chat](/features/chat.md)
|
||||
- **Faster answers**: Find answers faster, smoother than search. No need to manually scan through your notes to find answers.
|
||||
- **Iterative discovery**: Iteratively explore and (re-)discover your notes
|
||||
- **Assisted creativity**: Smoothly weave across answers retrieval and content generation
|
||||
- **Online or Offline**: Choose online or offline chat depending on your requirements
|
||||
281
documentation/docs/get-started/setup.mdx
Normal file
281
documentation/docs/get-started/setup.mdx
Normal file
@@ -0,0 +1,281 @@
|
||||
---
|
||||
sidebar_position: 1
|
||||
---
|
||||
|
||||
# Self-Host
|
||||
Learn about how to self-host Khoj on your own machine.
|
||||
|
||||
```mdx-code-block
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
```
|
||||
|
||||
## Setup
|
||||
These are the general setup instructions for Khoj.
|
||||
|
||||
- Make sure [python](https://realpython.com/installing-python/) and [pip](https://pip.pypa.io/en/stable/installation/) are installed on your machine
|
||||
- Check the [Khoj Emacs docs](/docs/clients/emacs?id=setup) to setup Khoj with Emacs<br />
|
||||
It's simpler as it can skip the server *install*, *run* and *configure* step below.
|
||||
- Check the [Khoj Obsidian docs](/docs/clients/obsidian?id=_2-setup-plugin) to setup Khoj with Obsidian<br />
|
||||
Its simpler as it can skip the *configure* step below.
|
||||
|
||||
For Installation, you can either use Docker or install Khoj locally.
|
||||
|
||||
### Installation Option 1 (Docker)
|
||||
|
||||
#### Prerequisites
|
||||
1. Install Docker Engine. See [official instructions](https://docs.docker.com/engine/install/).
|
||||
2. Ensure you have Docker Compose. See [official instructions](https://docs.docker.com/compose/install/).
|
||||
|
||||
#### Setup
|
||||
|
||||
Use the sample docker-compose [in Github](https://github.com/khoj-ai/khoj/blob/master/docker-compose.yml) to run Khoj in Docker. Start by configuring all the environment variables to your choosing. Your admin account will automatically be created based on the admin credentials in that file, so pay attention to those. To start the container, run the following command in the same directory as the docker-compose.yml file. This will automatically setup the database and run the Khoj server.
|
||||
|
||||
```shell
|
||||
docker-compose up
|
||||
```
|
||||
|
||||
Khoj should now be running at http://localhost:42110. You can see the web UI in your browser.
|
||||
|
||||
### Installation Option 2 (Local)
|
||||
|
||||
#### Prerequisites
|
||||
|
||||
##### Install Postgres (with PgVector)
|
||||
|
||||
Khoj uses the `pgvector` package to store embeddings of your index in a Postgres database. In order to use this, you need to have Postgres installed.
|
||||
|
||||
```mdx-code-block
|
||||
<Tabs groupId="operating-systems">
|
||||
<TabItem value="macos" label="MacOS">
|
||||
Install [Postgres.app](https://postgresapp.com/). This comes pre-installed with `pgvector` and relevant dependencies.
|
||||
</TabItem>
|
||||
<TabItem value="win" label="Windows">
|
||||
1. Use the [recommended installer](https://www.postgresql.org/download/windows/).
|
||||
2. Follow instructions to [Install PgVector](https://github.com/pgvector/pgvector#windows) in case you need to manually install it. Windows support is experimental for pgvector currently, so we recommend using Docker.
|
||||
</TabItem>
|
||||
<TabItem value="unix" label="Linux">
|
||||
From [official instructions](https://wiki.postgresql.org/wiki/Apt)
|
||||
</TabItem>
|
||||
<TabItem value="source" label="From Source">
|
||||
1. Follow instructions to [Install Postgres](https://www.postgresql.org/download/)
|
||||
2. Follow instructions to [Install PgVector](https://github.com/pgvector/pgvector#installation) in case you need to manually install it.
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
```
|
||||
|
||||
##### Create the Khoj database
|
||||
|
||||
Make sure to update your environment variables to match your Postgres configuration if you're using a different name. The default values should work for most people. When prompted for a password, you can use the default password `postgres`, or configure it to your preference. Make sure to set the environment variable `POSTGRES_PASSWORD` to the same value as the password you set here.
|
||||
|
||||
```mdx-code-block
|
||||
<Tabs groupId="operating-systems">
|
||||
<TabItem value="macos" label="MacOS">
|
||||
```shell
|
||||
createdb khoj -U postgres --password
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="win" label="Windows">
|
||||
```shell
|
||||
createdb -U postgres khoj --password
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="unix" label="Linux">
|
||||
```shell
|
||||
sudo -u postgres createdb khoj --password
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
```
|
||||
|
||||
|
||||
#### Install package
|
||||
|
||||
##### Local Server Setup
|
||||
- *Make sure [python](https://realpython.com/installing-python/) and [pip](https://pip.pypa.io/en/stable/installation/) are installed on your machine*
|
||||
|
||||
Run the following command in your terminal to install the Khoj backend.
|
||||
|
||||
```mdx-code-block
|
||||
<Tabs groupId="operating-systems">
|
||||
<TabItem value="macos" label="MacOS">
|
||||
```shell
|
||||
python -m pip install khoj-assistant
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="win" label="Windows">
|
||||
```shell
|
||||
py -m pip install khoj-assistant
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="unix" label="Linux">
|
||||
```shell
|
||||
python -m pip install khoj-assistant
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
```
|
||||
|
||||
##### Local Server Start
|
||||
|
||||
Before getting started, configure the following environment variables in your terminal for the first run
|
||||
|
||||
```mdx-code-block
|
||||
<Tabs groupId="operating-systems">
|
||||
<TabItem value="macos" label="MacOS">
|
||||
```shell
|
||||
export KHOJ_ADMIN_EMAIL=<your-email>
|
||||
export KHOJ_ADMIN_PASSWORD=<your-password>
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="win" label="Windows">
|
||||
If you're using PowerShell:
|
||||
```shell
|
||||
$env:KHOJ_ADMIN_EMAIL="<your-email>"
|
||||
$env:KHOJ_ADMIN_PASSWORD="<your-password>"
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="unix" label="Linux">
|
||||
```shell
|
||||
export KHOJ_ADMIN_EMAIL=<your-email>
|
||||
export KHOJ_ADMIN_PASSWORD=<your-password>
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
```
|
||||
|
||||
|
||||
Run the following command from your terminal to start the Khoj backend and open Khoj in your browser.
|
||||
|
||||
```shell
|
||||
khoj --anonymous-mode
|
||||
```
|
||||
`--anonymous-mode` allows you to run the server without setting up Google credentials for login. This allows you to use any of the clients without a login wall. If you want to use Google login, you can skip this flag, but you will have to add your Google developer credentials.
|
||||
|
||||
On the first run, you will be prompted to input credentials for your admin account and do some basic configuration for your chat model settings. Once created, you can go to http://localhost:42110/server/admin and login with the credentials you just created.
|
||||
|
||||
Khoj should now be running at http://localhost:42110. You can see the web UI in your browser.
|
||||
|
||||
Note: To start Khoj automatically in the background use [Task scheduler](https://www.windowscentral.com/how-create-automated-task-using-task-scheduler-windows-10) on Windows or [Cron](https://en.wikipedia.org/wiki/Cron) on Mac, Linux (e.g with `@reboot khoj`)
|
||||
|
||||
|
||||
### 2. Download the desktop client
|
||||
|
||||
You can use our desktop executables to select file paths and folders to index. You can simply select the folders or files, and they'll be automatically uploaded to the server. Once you specify a file or file path, you don't need to update the configuration again; it will grab any data diffs dynamically over time.
|
||||
|
||||
**To download the latest desktop client, go to https://download.khoj.dev** and the correct executable for your OS will automatically start downloading. Once downloaded, you can configure your folders for indexing using the settings tab. To set your chat configuration, you'll have to use the web interface for the Khoj server you setup in the previous step.
|
||||
|
||||
To use the desktop client, you need to go to your Khoj server's settings page (http://localhost:42110/config) and copy the API key. Then, paste it into the desktop client's settings page. Once you've done that, you can select files and folders to index.
|
||||
|
||||
### 3. Configure
|
||||
1. Go to http://localhost:42110/server/admin and login with your admin credentials.
|
||||
1. Go to [OpenAI settings](http://localhost:42110/server/admin/database/openaiprocessorconversationconfig/) in the server admin settings to add an Open AI processor conversation config. This is where you set your API key. Alternatively, you can go to the [offline chat settings](http://localhost:42110/server/admin/database/offlinechatprocessorconversationconfig/) and simply create a new setting with `Enabled` set to `True`.
|
||||
2. Go to the ChatModelOptions if you want to add additional models for chat. For example, you can specify `gpt-4` if you're using OpenAI or `mistral-7b-instruct-v0.1.Q4_0.gguf` if you're using offline chat. Make sure to configure the `type` field to `OpenAI` or `Offline` respectively.
|
||||
1. Select files and folders to index [using the desktop client](/docs/get-started/setup?id=_2-download-the-desktop-client). When you click 'Save', the files will be sent to your server for indexing.
|
||||
- Select Notion workspaces and Github repositories to index using the web interface.
|
||||
|
||||
|
||||
:::tip[Note]
|
||||
Using Safari on Mac? You might not be able to login to the admin panel. Try using Chrome or Firefox instead.
|
||||
:::
|
||||
|
||||
|
||||
### 4. Install Client Plugins (Optional)
|
||||
Khoj exposes a web interface to search, chat and configure by default.<br />
|
||||
The optional steps below allow using Khoj from within an existing application like Obsidian or Emacs.
|
||||
|
||||
- **Khoj Obsidian**:<br />
|
||||
[Install](/docs/clients/obsidian?id=_2-setup-plugin) the Khoj Obsidian plugin
|
||||
|
||||
- **Khoj Emacs**:<br />
|
||||
[Install](/docs/clients/emacs?id=setup) khoj.el
|
||||
|
||||
#### Setup host URL
|
||||
To configure your host URL on your clients when self-hosting, use `http://127.0.0.1:42110`. This is the default value for the `KHOJ_HOST` environment variable. Note that `localhost` will not work.
|
||||
|
||||
### 5. Use Khoj 🚀
|
||||
|
||||
You can head to http://localhost:42110 to use the web interface. You can also use the desktop client to search and chat.
|
||||
|
||||
## Upgrade
|
||||
### Upgrade Khoj Server
|
||||
|
||||
```mdx-code-block
|
||||
<Tabs groupId="environment">
|
||||
<TabItem value="localsetup" label="Local Setup">
|
||||
```shell
|
||||
pip install --upgrade khoj-assistant
|
||||
```
|
||||
*Note: To upgrade to the latest pre-release version of the khoj server run below command*
|
||||
</TabItem>
|
||||
<TabItem value="docker" label="Docker">
|
||||
From the same directory where you have your `docker-compose` file, this will fetch the latest build and upgrade your server.
|
||||
```shell
|
||||
docker-compose up --build
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="emacs" label="Emacs">
|
||||
- Use your Emacs Package Manager to Upgrade
|
||||
- See [khoj.el package setup](/docs/clients/emacs?id=setup) for details
|
||||
</TabItem>
|
||||
<TabItem value="obsidian" label="Obsidian">
|
||||
- Upgrade via the Community plugins tab on the settings pane in the Obsidian app
|
||||
- See the [khoj plugin setup](/docs/clients/obsidian?id=_2-setup-plugin) for details
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
```
|
||||
|
||||
## Uninstall
|
||||
### Uninstall Khoj Server
|
||||
|
||||
```mdx-code-block
|
||||
<Tabs groupId="environment">
|
||||
<TabItem value="localsetup" label="Local Setup">
|
||||
```shell
|
||||
# uninstall khoj server
|
||||
pip uninstall khoj-assistant
|
||||
|
||||
# delete khoj postgres db
|
||||
dropdb khoj -U postgres
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="docker" label="Docker">
|
||||
From the same directory where you have your `docker-compose` file, run the command below to remove the server to delete its containers, networks, images and volumes.
|
||||
|
||||
```shell
|
||||
docker-compose down --volumes
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="emacs" label="Emacs">
|
||||
Uninstall the khoj Emacs, or desktop client in the standard way from Emacs or your OS respectively
|
||||
You can also `rm -rf ~/.khoj` to remove the Khoj data directory if did a local install.
|
||||
</TabItem>
|
||||
<TabItem value="obsidian" label="Obsidian">
|
||||
Uninstall the khoj Obisidan, or desktop client in the standard way from Obsidian or your OS respectively
|
||||
You can also `rm -rf ~/.khoj` to remove the Khoj data directory if did a local install.
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
```
|
||||
|
||||
## Troubleshoot
|
||||
|
||||
#### Install fails while building Tokenizer dependency
|
||||
- **Details**: `pip install khoj-assistant` fails while building the `tokenizers` dependency. Complains about Rust.
|
||||
- **Fix**: Install Rust to build the tokenizers package. For example on Mac run:
|
||||
```shell
|
||||
brew install rustup
|
||||
rustup-init
|
||||
source ~/.cargo/env
|
||||
```
|
||||
- **Refer**: [Issue with Fix](https://github.com/khoj-ai/khoj/issues/82#issuecomment-1241890946) for more details
|
||||
|
||||
#### Search starts giving wonky results
|
||||
- **Fix**: Open [/api/update?force=true](http://localhost:42110/api/update?force=true) in browser to regenerate index from scratch
|
||||
- **Note**: *This is a fix for when you perceive the search results have degraded. Not if you think they've always given wonky results*
|
||||
|
||||
#### Khoj in Docker errors out with \"Killed\" in error message
|
||||
- **Fix**: Increase RAM available to Docker Containers in Docker Settings
|
||||
- **Refer**: [StackOverflow Solution](https://stackoverflow.com/a/50770267), [Configure Resources on Docker for Mac](https://docs.docker.com/desktop/mac/#resources)
|
||||
|
||||
#### Khoj errors out complaining about Tensors mismatch or null
|
||||
- **Mitigation**: Disable `image` search using the desktop GUI
|
||||
8
documentation/docs/miscellaneous/_category_.json
Normal file
8
documentation/docs/miscellaneous/_category_.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"label": "Miscellaneous",
|
||||
"position": 6,
|
||||
"link": {
|
||||
"type": "generated-index",
|
||||
"description": "Additional resources for learning about Khoj"
|
||||
}
|
||||
}
|
||||
32
documentation/docs/miscellaneous/advanced.md
Normal file
32
documentation/docs/miscellaneous/advanced.md
Normal file
@@ -0,0 +1,32 @@
|
||||
---
|
||||
sidebar_position: 3
|
||||
---
|
||||
|
||||
# Advanced Usage
|
||||
|
||||
### Search across Different Languages (Self-Hosting)
|
||||
To search for notes in multiple, different languages, you can use a [multi-lingual model](https://www.sbert.net/docs/pretrained_models.html#multi-lingual-models).<br />
|
||||
For example, the [paraphrase-multilingual-MiniLM-L12-v2](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2) supports [50+ languages](https://www.sbert.net/docs/pretrained_models.html#:~:text=we%20used%20the%20following%2050%2B%20languages), has good search quality and speed. To use it:
|
||||
1. Manually update the search config in server's admin settings page. Go to [the search config](http://localhost:42110/server/admin/database/searchmodelconfig/). Either create a new one, if none exists, or update the existing one. Set the bi_encoder to `sentence-transformers/multi-qa-MiniLM-L6-cos-v1` and the cross_encoder to `cross-encoder/ms-marco-MiniLM-L-6-v2`.
|
||||
2. Regenerate your content index from all the relevant clients. This step is very important, as you'll need to re-encode all your content with the new model.
|
||||
|
||||
### Query Filters
|
||||
|
||||
Use structured query syntax to filter entries from your knowledge based used by search results or chat responses.
|
||||
|
||||
- **Word Filter**: Get entries that include/exclude a specified term
|
||||
- Entries that contain term_to_include: `+"term_to_include"`
|
||||
- Entries that contain term_to_exclude: `-"term_to_exclude"`
|
||||
- **Date Filter**: Get entries containing dates in YYYY-MM-DD format from specified date (range)
|
||||
- Entries from April 1st 1984: `dt:"1984-04-01"`
|
||||
- Entries after March 31st 1984: `dt>="1984-04-01"`
|
||||
- Entries before April 2nd 1984 : `dt<="1984-04-01"`
|
||||
- **File Filter**: Get entries from a specified file
|
||||
- Entries from incoming.org file: `file:"incoming.org"`
|
||||
- Combined Example
|
||||
- `what is the meaning of life? file:"1984.org" dt>="1984-01-01" dt<="1985-01-01" -"big" -"brother"`
|
||||
- Adds all filters to the natural language query. It should return entries
|
||||
- from the file *1984.org*
|
||||
- containing dates from the year *1984*
|
||||
- excluding words *"big"* and *"brother"*
|
||||
- that best match the natural language query *"what is the meaning of life?"*
|
||||
13
documentation/docs/miscellaneous/credits.md
Normal file
13
documentation/docs/miscellaneous/credits.md
Normal file
@@ -0,0 +1,13 @@
|
||||
---
|
||||
sidebar_position: 4
|
||||
---
|
||||
|
||||
# Credits
|
||||
Many Open Source projects are used to power Khoj. Here's a few of them:
|
||||
|
||||
- [Multi-QA MiniLM Model](https://huggingface.co/sentence-transformers/multi-qa-MiniLM-L6-cos-v1), [All MiniLM Model](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) for Text Search. See [SBert Documentation](https://www.sbert.net/examples/applications/retrieve_rerank/README.html)
|
||||
- [OpenAI CLIP Model](https://github.com/openai/CLIP) for Image Search. See [SBert Documentation](https://www.sbert.net/examples/applications/image-search/README.html)
|
||||
- Charles Cave for [OrgNode Parser](http://members.optusnet.com.au/~charles57/GTD/orgnode.html)
|
||||
- [Org.js](https://mooz.github.io/org-js/) to render Org-mode results on the Web interface
|
||||
- [Markdown-it](https://github.com/markdown-it/markdown-it) to render Markdown results on the Web interface
|
||||
- [GPT4All](https://github.com/nomic-ai/gpt4all) to chat with local LLM
|
||||
25
documentation/docs/miscellaneous/performance.md
Normal file
25
documentation/docs/miscellaneous/performance.md
Normal file
@@ -0,0 +1,25 @@
|
||||
---
|
||||
sidebar_position: 2
|
||||
---
|
||||
|
||||
# Performance
|
||||
|
||||
Here are some top-level performance metrics for Khoj. These are rough estimates and will vary based on your hardware and data.
|
||||
|
||||
### Search performance
|
||||
|
||||
- Semantic search using the bi-encoder is fairly fast at \<100 ms across all content types
|
||||
- Reranking using the cross-encoder is slower at \<2s on 15 results. Tweak `top_k` to tradeoff speed for accuracy of results
|
||||
- Filters in query (e.g by file, word or date) usually add \<20ms to query latency
|
||||
|
||||
### Indexing performance
|
||||
|
||||
- Indexing is more strongly impacted by the size of the source data
|
||||
- Indexing 100K+ line corpus of notes takes about 10 minutes
|
||||
- Indexing 4000+ images takes about 15 minutes and more than 8Gb of RAM
|
||||
- Note: *It should only take this long on the first run* as the index is incrementally updated
|
||||
|
||||
### Miscellaneous
|
||||
|
||||
- Testing done on a Mac M1 and a \>100K line corpus of notes
|
||||
- Search, indexing on a GPU has not been tested yet
|
||||
22
documentation/docs/miscellaneous/telemetry.md
Normal file
22
documentation/docs/miscellaneous/telemetry.md
Normal file
@@ -0,0 +1,22 @@
|
||||
---
|
||||
sidebar_position: 1
|
||||
---
|
||||
|
||||
# Telemetry
|
||||
|
||||
We collect some high level, anonymized metadata about usage of Khoj. This includes:
|
||||
- Client (Web, Emacs, Obsidian)
|
||||
- API usage (Search, Chat)
|
||||
- Configured content types (Github, Org, etc)
|
||||
- Request metadata (e.g., host, referrer)
|
||||
|
||||
We don't send any personal information or any information from/about your content. We only send the above metadata. This helps us prioritize feature development and understand how people are using Khoj. Don't just take our word for it -- you can see [the code here](https://github.com/khoj-ai/khoj/tree/master/src/telemetry).
|
||||
|
||||
## Disable Telemetry
|
||||
|
||||
You can opt out of telemetry at any time. To do so,
|
||||
1. Open `~/.khoj/khoj.yml`
|
||||
2. Set `should-log-telemetry` to `false`
|
||||
3. Save the file and restart Khoj
|
||||
|
||||
If you have any questions or concerns, please reach out to us on [Discord](https://discord.gg/BDgyabRM6e).
|
||||
8
documentation/docs/online-data-sources/_category_.json
Normal file
8
documentation/docs/online-data-sources/_category_.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"label": "Online Data Sources",
|
||||
"position": 5,
|
||||
"link": {
|
||||
"type": "generated-index",
|
||||
"description": "Online data sources for indexing via Khoj"
|
||||
}
|
||||
}
|
||||
14
documentation/docs/online-data-sources/github_integration.md
Normal file
14
documentation/docs/online-data-sources/github_integration.md
Normal file
@@ -0,0 +1,14 @@
|
||||
# Setup the Github integration
|
||||
|
||||
The Github integration allows you to index as many repositories as you want. It's currently default configured to index Issues, Commits, and all Markdown/Org files in each repository. For large repositories, this takes a fairly long time, but it works well for smaller projects.
|
||||
|
||||
# Configure your settings
|
||||
|
||||
1. Go to [https://app.khoj.dev/config](https://app.khoj.dev/config) and enter in settings for the data sources you want to index. You'll have to specify the file paths.
|
||||
|
||||
## Use the Github plugin
|
||||
|
||||
1. Generate a [classic PAT (personal access token)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) from [Github](https://github.com/settings/tokens) with `repo` and `admin:org` scopes at least.
|
||||
2. Navigate to [https://app.khoj.dev/config/content-source/github](https://app.khoj.dev/config/content-source/github) to configure your Github settings. Enter in your PAT, along with details for each repository you want to index.
|
||||
3. Click `Save`. Go back to the settings page and click `Configure`.
|
||||
4. Go to [https://app.khoj.dev/](https://app.khoj.dev/) and start searching!
|
||||
14
documentation/docs/online-data-sources/notion_integration.md
Normal file
14
documentation/docs/online-data-sources/notion_integration.md
Normal file
@@ -0,0 +1,14 @@
|
||||
# Notion Integration
|
||||
|
||||
The Notion integration allows you to search/chat with your Notion workspaces. [Notion](https://notion.so/) is a platform people use for taking notes, especially for collaboration.
|
||||
|
||||
We haven't setup a fancy integration with OAuth yet, so this integration still requires some effort on your end to generate an API key.
|
||||
|
||||
1. Go to https://www.notion.so/my-integrations and create a new integration called Khoj to get an API key.
|
||||
|
||||
3. Share all the workspaces that you want to integrate with the Khoj integration you just made in the previous step
|
||||
|
||||
4. In the first step, you generated an API key. Use the newly generated API Key in your Khoj settings, by default at https://app.khoj.dev/config/content-source/notion. Click `Save`.
|
||||
5. Click `Configure` in https://app.khoj.dev/config to index your Notion workspace(s).
|
||||
|
||||
That's it! You should be ready to start searching and chatting. Make sure you've configured your OpenAI API Key for chat.
|
||||
Reference in New Issue
Block a user