mirror of
https://github.com/khoaliber/khoj.git
synced 2026-03-02 13:18:18 +00:00
Merge branch 'master' into fix-1000-file-index-update-limit
This commit is contained in:
Debanjum Singh Solanky
46
.github/workflows/github_pages_deploy.yml
vendored
Normal file
46
.github/workflows/github_pages_deploy.yml
vendored
Normal file
@@ -0,0 +1,46 @@
|
||||
name: build and deploy github pages for documentation
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- 'master'
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pages: write
|
||||
id-token: write
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
environment:
|
||||
name: github-pages
|
||||
url: https://docs.khoj.dev
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v3
|
||||
# 👇 Build steps
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@v3
|
||||
with:
|
||||
node-version: 18.x
|
||||
cache: yarn
|
||||
cache-dependency-path: documentation/yarn.lock
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
cd documentation
|
||||
yarn install --frozen-lockfile --non-interactive
|
||||
- name: Build
|
||||
run: |
|
||||
cd documentation
|
||||
yarn build
|
||||
# 👆 Build steps
|
||||
- name: Setup Pages
|
||||
uses: actions/configure-pages@v3
|
||||
- name: Upload artifact
|
||||
uses: actions/upload-pages-artifact@v2
|
||||
with:
|
||||
# 👇 Specify build output path
|
||||
path: documentation/build
|
||||
- name: Deploy to GitHub Pages
|
||||
id: deployment
|
||||
uses: actions/deploy-pages@v2
|
||||
@@ -40,7 +40,7 @@ It works with jpeg, markdown, notion, org-mode, pdf files and github repositorie
|
||||
|:---------:|:-------:|
|
||||
| Quickly retrieve relevant documents using natural language | Get answers and create content from your existing knowledge base |
|
||||
| Does not need internet | Can be configured to work without internet |
|
||||
| <img src="https://docs.khoj.dev/assets/khoj_search_on_web.png" width="400px"> | <img src="https://docs.khoj.dev/assets/khoj_chat_on_web.png" width="400px"> |
|
||||
| <img src="https://docs.khoj.dev/img/khoj_search_on_web.png" width="400px"> | <img src="https://docs.khoj.dev/img/khoj_chat_on_web.png" width="400px"> |
|
||||
|
||||
## Contributors
|
||||
Cheers to our awesome contributors! 🎉
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
docs.khoj.dev
|
||||
@@ -1,53 +0,0 @@
|
||||
<p align="center"><img src="./assets/khoj-logo-sideways-500.png" width="200" alt="Khoj Logo"></p>
|
||||
|
||||
<div align="center">
|
||||
|
||||
[](https://github.com/khoj-ai/khoj/actions/workflows/test.yml)
|
||||
[](https://github.com/khoj-ai/khoj/pkgs/container/khoj)
|
||||
[](https://pypi.org/project/khoj-assistant/)
|
||||
|
||||
</div>
|
||||
|
||||
<div align="center">
|
||||
<b>An AI copilot for your Second Brain</b>
|
||||
|
||||
</div>
|
||||
|
||||
<div align="center">
|
||||
|
||||
[📜 Explore Code](https://github.com/khoj-ai/khoj)
|
||||
<span> • </span>
|
||||
[🌍 Try Khoj Cloud](https://khoj.dev)
|
||||
<span> • </span>
|
||||
[💬 Get Involved](https://discord.gg/BDgyabRM6e)
|
||||
|
||||
</div>
|
||||
|
||||
## Introduction
|
||||
Welcome to the Khoj Docs! This is the best place to get setup and explore Khoj's features.
|
||||
|
||||
- Khoj is an open source, personal AI
|
||||
- You can [chat](chat.md) with it about anything. When relevant, it'll use any notes or documents you shared with it to respond
|
||||
- Quickly [find](search.md) relevant notes and documents using natural language
|
||||
- It understands pdf, plaintext, markdown, org-mode files, [notion pages](notion_integration.md) and [github repositories](github_integration.md)
|
||||
- Access it from your [Emacs](emacs.md), [Obsidian](obsidian.md), [Web browser](web.md) or the [Khoj Desktop app](desktop.md)
|
||||
- You can self-host Khoj on your consumer hardware or share it with your family, friends or team from your private cloud
|
||||
|
||||
## Quickstart
|
||||
- [Try Khoj Cloud](https://app.khoj.dev) to get started quickly
|
||||
- [Read these instructions](./setup.md) to self-host a private instance of Khoj
|
||||
|
||||
## Overview
|
||||
<img src="https://docs.khoj.dev/assets/khoj_search_on_web.png" width="400px">
|
||||
<span> </span>
|
||||
<img src="https://docs.khoj.dev/assets/khoj_chat_on_web.png" width="400px">
|
||||
|
||||
#### [Search](search.md)
|
||||
- **Natural**: Use natural language queries to quickly find relevant notes and documents.
|
||||
- **Incremental**: Incremental search for a fast, search-as-you-type experience
|
||||
|
||||
#### [Chat](chat.md)
|
||||
- **Faster answers**: Find answers faster, smoother than search. No need to manually scan through your notes to find answers.
|
||||
- **Iterative discovery**: Iteratively explore and (re-)discover your notes
|
||||
- **Assisted creativity**: Smoothly weave across answers retrieval and content generation
|
||||
- **Online or Offline**: Choose online or offline chat depending on your requirements
|
||||
@@ -1,14 +0,0 @@
|
||||
<!-- _coverpage.md -->
|
||||
|
||||
|
||||
|
||||
> An open source, AI personal assistant for your notes
|
||||
|
||||
- Lightning fast search
|
||||
- Multi-turn chat
|
||||
- Keeps you in control of your data
|
||||
|
||||
[GitHub](https://github.com/khoj-ai/khoj)
|
||||
[Get Started](#khoj)
|
||||
|
||||
|
||||
@@ -1,23 +0,0 @@
|
||||
- Get Started
|
||||
- [Overview](README.md)
|
||||
- [Self-Host](setup.md)
|
||||
- [Demos](demos.md)
|
||||
- Use
|
||||
- [Features](features.md)
|
||||
- [Chat](chat.md)
|
||||
- [Search](search.md)
|
||||
- Clients
|
||||
- [Desktop](desktop.md)
|
||||
- [Obsidian](obsidian.md)
|
||||
- [Emacs](emacs.md)
|
||||
- [Web](web.md)
|
||||
- Online Data Sources
|
||||
- [Github](github_integration.md)
|
||||
- [Notion](notion_integration.md)
|
||||
- Contributing
|
||||
- [Development](development.md)
|
||||
- Miscellaneous
|
||||
- [Telemetry](telemetry.md)
|
||||
- [Advanced](advanced.md)
|
||||
- [Performance](performance.md)
|
||||
- [Credits](credits.md)
|
||||
@@ -1,32 +0,0 @@
|
||||
# Installing the Desktop Application [Deprecated -- for 0.11.4 and below]
|
||||
|
||||
We have beta desktop images available for download with new releases. This is recommended if you don't want to bother with the command line. Download the latest release from [here](https://github.com/khoj-ai/khoj/releases). You can find the latest release under the `Assets` section.
|
||||
|
||||
## MacOS
|
||||
|
||||
1. Download the latest release from [here](https://github.com/khoj-ai/khoj/releases).
|
||||
- If your Mac uses one of the Silicon chips, then download the `Khoj_<version>_arm64.dmg` file. Otherwise, download the `Khoj_<version>_amd64.dmg` file.
|
||||
2. Open the downloaded file and drag the Khoj app to your Applications folder.
|
||||
|
||||
## Windows
|
||||
|
||||
Make sure you meet the prerequisites for Windows installation. You can find them [here](windows_install.md#prerequisites).
|
||||
|
||||
1. Download the latest release from [here](https://github.com/khoj-ai/khoj/releases). You'll want the `khoj_<version>_amd64.exe` file.
|
||||
2. Open the downloaded file and double click to install.
|
||||
|
||||
## Linux
|
||||
|
||||
For the Linux installation, you have to have `glibc` version 2.35 or higher. You can check your version with `ldd --version`.
|
||||
|
||||
1. Download the latest release from [here](https://github.com/khoj-ai/khoj/releases). You'll want the `khoj_<version>_amd64.deb` file.
|
||||
2. In your downloads folder, run `sudo dpkg -i khoj_<version>_amd64.deb` to install Khoj.
|
||||
|
||||
|
||||
# Uninstall
|
||||
|
||||
If you decide you want to uninstall the application, you can uninstall it like any other application on your system. For example, on MacOS, you can drag the application to the trash. On Windows, you can uninstall it from the `Add or Remove Programs` menu. On Linux, you can uninstall it with `sudo apt remove khoj`.
|
||||
|
||||
In addition to that, you might want to `rm -rf` the following directories:
|
||||
- `~/.khoj`
|
||||
- `~/.cache/gpt4all`
|
||||
@@ -1,34 +0,0 @@
|
||||
## Features
|
||||
|
||||
#### [Search](search.md)
|
||||
- **Local**: Your personal data stays local. All search and indexing is done on your machine.
|
||||
- **Incremental**: Incremental search for a fast, search-as-you-type experience
|
||||
|
||||
#### [Chat](chat.md)
|
||||
- **Faster answers**: Find answers faster, smoother than search. No need to manually scan through your notes to find answers.
|
||||
- **Iterative discovery**: Iteratively explore and (re-)discover your notes
|
||||
- **Assisted creativity**: Smoothly weave across answers retrieval and content generation
|
||||
|
||||
#### General
|
||||
- **Natural**: Advanced natural language understanding using Transformer based ML Models
|
||||
- **Pluggable**: Modular architecture makes it easy to plug in new data sources, frontends and ML models
|
||||
- **Multiple Sources**: Index your Org-mode and Markdown notes, PDF files, Github repositories, and Photos
|
||||
- **Multiple Interfaces**: Interact from your [Web Browser](./web.md), [Emacs](./emacs.md) or [Obsidian](./obsidian.md)
|
||||
|
||||
### Supported Interfaces
|
||||
|
||||
[](./emacs.md)
|
||||
<span> </span>
|
||||
[](./obsidian.md)
|
||||
|
||||
### Supported Data Sources
|
||||
- markdown*
|
||||
- org-mode*
|
||||
- pdf*
|
||||
- images*
|
||||
- [github](./github_integration.md)
|
||||
- [notion](./notion_integration.md)
|
||||
|
||||
\* These data sources are offline only.
|
||||
|
||||
If you're using Github or Notion, you can get on a waitlist for [Khoj Cloud](https://khoj.dev).
|
||||
@@ -1,47 +0,0 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<title>Document</title>
|
||||
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
|
||||
<meta name="description" content="Description">
|
||||
|
||||
<!-- Open Graph metadata -->
|
||||
<meta property="og:title" content="Khoj Documentation">
|
||||
<meta property="og:type" content="website">
|
||||
<meta property="og:site_name" content="Khoj Documentation">
|
||||
<meta property="og:description" content="Quickly get started with using or self-hosting Khoj">
|
||||
<meta property="og:image" content="https://khoj-web-bucket.s3.amazonaws.com/link_preview_docs.png">
|
||||
<meta property="og:url" content="https://docs.khoj.dev">
|
||||
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, minimum-scale=1.0">
|
||||
<link rel="stylesheet" href="//cdn.jsdelivr.net/npm/docsify/lib/themes/buble.css" />
|
||||
<link rel="icon" href="./assets/favicon-128x128.ico">
|
||||
</head>
|
||||
<body>
|
||||
<div id="app"></div>
|
||||
<script>
|
||||
window.$docsify = {
|
||||
name: 'Khoj',
|
||||
repo: 'https://github.com/khoj-ai/khoj',
|
||||
loadSidebar: true,
|
||||
themeColor: '#c2a600',
|
||||
auto2top: true,
|
||||
// coverpage: true,
|
||||
}
|
||||
</script>
|
||||
<!-- Docsify v4 -->
|
||||
<script src="//cdn.jsdelivr.net/npm/docsify@4"></script>
|
||||
<script src="//cdn.jsdelivr.net/npm/docsify-tabs@1"></script>
|
||||
<script src="//cdn.jsdelivr.net/npm/docsify/lib/plugins/search.min.js"></script>
|
||||
<script src="//cdn.jsdelivr.net/npm/docsify-copy-code/dist/docsify-copy-code.min.js"></script>
|
||||
<script src="//cdn.jsdelivr.net/npm/prismjs@1/components/prism-bash.min.js"></script>
|
||||
<script src="//cdn.jsdelivr.net/npm/prismjs@1/components/prism-diff.min.js"></script>
|
||||
<script defer data-domain="khoj.dev" src="https://plausible.io/js/script.js"></script>
|
||||
</body>
|
||||
<style>
|
||||
video {
|
||||
max-width: 800px;
|
||||
}
|
||||
</style>
|
||||
</html>
|
||||
@@ -1,23 +0,0 @@
|
||||
# Windows Installation
|
||||
|
||||
These steps can be used to setup Khoj on a clean, new Windows 11 machine. It has been tested on a Windows VM
|
||||
|
||||
## Prerequisites
|
||||
1. Ensure you have Visual Studio C++ Build tools installed. You can download it [from Microsoft here](https://visualstudio.microsoft.com/visual-cpp-build-tools/). At the minimum, you should have the following configuration:
|
||||
<img width="1152" alt="Screenshot 2023-07-12 at 3 56 25 PM" src="https://github.com/khoj-ai/khoj/assets/65192171/b506a858-2f5e-4c85-946b-5422d83f112a">
|
||||
2. Ensure you have Python installed. You can check by running `python --version`. If you don't, install the latest version [from here](https://www.python.org/downloads/).
|
||||
- Ensure you have pip installed: `py -m ensurepip --upgrade`.
|
||||
|
||||
## Quick start
|
||||
1. Open a PowerShell terminal.
|
||||
2. Run `pip install khoj-assistant`
|
||||
3. Start Khoj with `khoj`
|
||||
|
||||
## Installation in a Virtual Environment
|
||||
Use this if you want to install with a virtual environment. This will make it much easier to manage your dependencies. You can read more about [virtual environments](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/) here.
|
||||
|
||||
1. Open a PowerShell terminal with the `Run as Administrator` privileges.
|
||||
2. Create a virtual environment: `mkdir khoj && cd khoj && py -m venv .venv`
|
||||
3. Activate the virtual environment: `.\.venv\Scripts\activate`. If you get a permissions error, then run `Set-ExecutionPolicy -ExecutionPolicy RemoteSigned`.
|
||||
4. Run `pip install khoj-assistant`
|
||||
5. Start Khoj with `khoj`
|
||||
20
documentation/.gitignore
vendored
Normal file
20
documentation/.gitignore
vendored
Normal file
@@ -0,0 +1,20 @@
|
||||
# Dependencies
|
||||
/node_modules
|
||||
|
||||
# Production
|
||||
/build
|
||||
|
||||
# Generated files
|
||||
.docusaurus
|
||||
.cache-loader
|
||||
|
||||
# Misc
|
||||
.DS_Store
|
||||
.env.local
|
||||
.env.development.local
|
||||
.env.test.local
|
||||
.env.production.local
|
||||
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
41
documentation/README.md
Normal file
41
documentation/README.md
Normal file
@@ -0,0 +1,41 @@
|
||||
# Website
|
||||
|
||||
This website is built using [Docusaurus](https://docusaurus.io/), a modern static website generator.
|
||||
|
||||
### Installation
|
||||
|
||||
```
|
||||
$ yarn
|
||||
```
|
||||
|
||||
### Local Development
|
||||
|
||||
```
|
||||
$ yarn start
|
||||
```
|
||||
|
||||
This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server.
|
||||
|
||||
### Build
|
||||
|
||||
```
|
||||
$ yarn build
|
||||
```
|
||||
|
||||
This command generates static content into the `build` directory and can be served using any static contents hosting service.
|
||||
|
||||
### Deployment
|
||||
|
||||
Using SSH:
|
||||
|
||||
```
|
||||
$ USE_SSH=true yarn deploy
|
||||
```
|
||||
|
||||
Not using SSH:
|
||||
|
||||
```
|
||||
$ GIT_USER=<Your GitHub username> yarn deploy
|
||||
```
|
||||
|
||||
If you are using GitHub pages for hosting, this command is a convenient way to build the website and push to the `gh-pages` branch.
|
||||
|
Before Width: | Height: | Size: 200 KiB After Width: | Height: | Size: 200 KiB |
|
Before Width: | Height: | Size: 13 KiB After Width: | Height: | Size: 13 KiB |
|
Before Width: | Height: | Size: 36 KiB After Width: | Height: | Size: 36 KiB |
|
Before Width: | Height: | Size: 1.2 MiB After Width: | Height: | Size: 1.2 MiB |
|
Before Width: | Height: | Size: 350 KiB After Width: | Height: | Size: 350 KiB |
|
Before Width: | Height: | Size: 298 KiB After Width: | Height: | Size: 298 KiB |
|
Before Width: | Height: | Size: 302 KiB After Width: | Height: | Size: 302 KiB |
|
Before Width: | Height: | Size: 394 KiB After Width: | Height: | Size: 394 KiB |
|
Before Width: | Height: | Size: 358 KiB After Width: | Height: | Size: 358 KiB |
74
documentation/assets/img/khoj_clients.svg
Normal file
74
documentation/assets/img/khoj_clients.svg
Normal file
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 27 KiB |
|
Before Width: | Height: | Size: 544 KiB After Width: | Height: | Size: 544 KiB |
62
documentation/assets/img/khoj_datasources.svg
Normal file
62
documentation/assets/img/khoj_datasources.svg
Normal file
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 43 KiB |
|
Before Width: | Height: | Size: 49 KiB After Width: | Height: | Size: 49 KiB |
|
Before Width: | Height: | Size: 333 KiB After Width: | Height: | Size: 333 KiB |
|
Before Width: | Height: | Size: 445 KiB After Width: | Height: | Size: 445 KiB |
|
Before Width: | Height: | Size: 333 KiB After Width: | Height: | Size: 333 KiB |
|
Before Width: | Height: | Size: 420 KiB After Width: | Height: | Size: 420 KiB |
|
Before Width: | Height: | Size: 478 KiB After Width: | Height: | Size: 478 KiB |
|
Before Width: | Height: | Size: 268 KiB After Width: | Height: | Size: 268 KiB |
1
documentation/assets/img/logo.svg
Normal file
1
documentation/assets/img/logo.svg
Normal file
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 6.3 KiB |
3
documentation/babel.config.js
Normal file
3
documentation/babel.config.js
Normal file
@@ -0,0 +1,3 @@
|
||||
module.exports = {
|
||||
presets: [require.resolve('@docusaurus/core/lib/babel/preset')],
|
||||
};
|
||||
8
documentation/docs/clients/_category_.json
Normal file
8
documentation/docs/clients/_category_.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"label": "Clients",
|
||||
"position": 4,
|
||||
"link": {
|
||||
"type": "generated-index",
|
||||
"description": "Different ways for indexing data with the Khoj backend"
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,14 @@
|
||||
<h1><img src="./assets/khoj-logo-sideways-500.png" width="200" alt="Khoj Logo"> Desktop</h1>
|
||||
---
|
||||
sidebar_position: 1
|
||||
---
|
||||
|
||||
> An AI copilot for your Second Brain
|
||||
# Desktop
|
||||
|
||||
> Query your Second Brain from your machine
|
||||
|
||||
Use the Desktop app to chat and search with Khoj.
|
||||
You can also sync any relevant files with Khoj using the app.
|
||||
Khoj will use these files to provide contextual reponses when you search or chat.
|
||||
|
||||
## Features
|
||||
- **Chat**
|
||||
@@ -19,5 +27,6 @@
|
||||
4. [Optional] Add any files, folders you'd like Khoj to be aware of on the *Settings* page and Click *Save*
|
||||
|
||||
## Interface
|
||||
|
||||
|
||||
| Chat | Search |
|
||||
|:----:|:------:|
|
||||
|  |  |
|
||||
@@ -1,13 +1,18 @@
|
||||
<h1><img src="./assets/khoj-logo-sideways-500.png" width="200" alt="Khoj Logo"> Emacs</h1>
|
||||
---
|
||||
sidebar_position: 2
|
||||
---
|
||||
|
||||
> An AI copilot for your Second Brain in Emacs
|
||||
# Emacs
|
||||
|
||||
<img src="https://stable.melpa.org/packages/khoj-badge.svg" width="150" alt="Melpa Stable Badge">
|
||||
<img src="https://melpa.org/packages/khoj-badge.svg" width="150" alt="Melpa Badge">
|
||||
<img src="https://stable.melpa.org/packages/khoj-badge.svg" width="130" alt="Melpa Stable Badge" />
|
||||
<img src="https://melpa.org/packages/khoj-badge.svg" width="150" alt="Melpa Badge" />
|
||||
|
||||
<img src="https://github.com/khoj-ai/khoj/actions/workflows/build_khoj_el.yml/badge.svg" width="150" alt="Build Badge">
|
||||
<img src="https://github.com/khoj-ai/khoj/actions/workflows/test_khoj_el.yml/badge.svg" width="150" alt="Test Badge">
|
||||
<img src="https://github.com/khoj-ai/khoj/actions/workflows/build_khoj_el.yml/badge.svg" width="150" alt="Build Badge" />
|
||||
<img src="https://github.com/khoj-ai/khoj/actions/workflows/test_khoj_el.yml/badge.svg" width="150" alt="Test Badge" />
|
||||
|
||||
<br />
|
||||
<br />
|
||||
> Query your Second Brain from Emacs
|
||||
|
||||
## Features
|
||||
- **Chat**
|
||||
@@ -19,19 +24,16 @@
|
||||
- **Incremental**: Incremental search for a fast, search-as-you-type experience
|
||||
|
||||
## Interface
|
||||
#### Search
|
||||
|
||||
|
||||
#### Chat
|
||||
|
||||
| Search | Chat |
|
||||
|:------:|:----:|
|
||||
|  |  |
|
||||
|
||||
## Setup
|
||||
1. Generate an API key on the [Khoj Web App](https://app.khoj.dev/config#clients)
|
||||
2. Add below snippet to your Emacs config file, usually at `~/.emacs.d/init.el`
|
||||
|
||||
|
||||
<!-- tabs:start -->
|
||||
|
||||
#### **Direct Install**
|
||||
*Khoj will index your org-agenda files, by default*
|
||||
|
||||
@@ -83,16 +85,15 @@ M-x package-install khoj
|
||||
khoj-org-files '("~/docs/todo.org" "~/docs/work.org")))
|
||||
```
|
||||
|
||||
<!-- tabs:end -->
|
||||
## Use
|
||||
### Search
|
||||
See [Khoj Search](search.md) for details
|
||||
See [Khoj Search](/features/search) for details
|
||||
1. Hit `C-c s s` (or `M-x khoj RET s`) to open khoj search
|
||||
2. Enter your query in natural language<br/>
|
||||
E.g *"What is the meaning of life?"*, *"My life goals for 2023"*
|
||||
|
||||
### Chat
|
||||
See [Khoj Chat](chat.md) for details
|
||||
See [Khoj Chat](/features/chat) for details
|
||||
1. Hit `C-c s c` (or `M-x khoj RET c`) to open khoj chat
|
||||
2. Ask questions in a natural, conversational style<br/>
|
||||
E.g *"When did I file my taxes last year?"*
|
||||
@@ -113,7 +114,7 @@ This feature finds entries similar to the one you are currently on.
|
||||
- Note: If you have [speed keys](https://orgmode.org/manual/Speed-Keys.html) enabled, `o 2` will also work
|
||||
|
||||
### Khoj Menu
|
||||
|
||||
|
||||
Hit `C-c s` (or `M-x khoj`) to open the khoj menu above. Then:
|
||||
- Hit `t` until you preferred content type is selected in the khoj menu
|
||||
`Content Type` specifies the content to perform `Search`, `Update` or `Find Similar` actions on
|
||||
@@ -1,6 +1,10 @@
|
||||
<h1><img src="./assets/khoj-logo-sideways-500.png" width="200" alt="Khoj Logo"> Obsidian</h1>
|
||||
---
|
||||
sidebar_position: 3
|
||||
---
|
||||
|
||||
> An AI copilot for your Second Brain in Obsidian
|
||||
# Obsidian
|
||||
|
||||
> Query your Second Brain from Obsidian
|
||||
|
||||
## Features
|
||||
- **Chat**
|
||||
@@ -12,8 +16,10 @@
|
||||
- **Incremental**: Incremental search for a fast, search-as-you-type experience
|
||||
|
||||
## Interface
|
||||
|
||||
|
||||
|
||||
| Search | Chat |
|
||||
|:------:|:----:|
|
||||
|  |  |
|
||||
|
||||
|
||||
## Setup
|
||||
@@ -30,7 +36,7 @@ See the official [Obsidian Plugin Docs](https://help.obsidian.md/Extending+Obsid
|
||||
Run *Khoj: Chat* from the [Command Palette](https://help.obsidian.md/Plugins/Command+palette) and ask questions in a natural, conversational style.<br />
|
||||
E.g *"When did I file my taxes last year?"*
|
||||
|
||||
See [Khoj Chat](/chat) for more details
|
||||
See [Khoj Chat](/features/chat) for more details
|
||||
|
||||
### Find Similar Notes
|
||||
To see other notes similar to the current one, run *Khoj: Find Similar Notes* from the [Command Palette](https://help.obsidian.md/Plugins/Command+palette)
|
||||
@@ -38,7 +44,7 @@ To see other notes similar to the current one, run *Khoj: Find Similar Notes* fr
|
||||
### Search
|
||||
Click the *Khoj search* icon 🔎 on the [Ribbon](https://help.obsidian.md/User+interface/Workspace/Ribbon) or run *Khoj: Search* from the [Command Palette](https://help.obsidian.md/Plugins/Command+palette)
|
||||
|
||||
See [Khoj Search](/search) for more details. Use [query filters](/advanced#query-filters) to limit entries to search
|
||||
See [Khoj Search](/features/search) for more details. Use [query filters](/miscellaneous/advanced#query-filters) to limit entries to search
|
||||
|
||||
[search_demo](https://user-images.githubusercontent.com/6413477/218801155-cd67e8b4-a770-404a-8179-d6b61caa0f93.mp4 ':include :type=mp4')
|
||||
|
||||
@@ -1,6 +1,12 @@
|
||||
<h1><img src="./assets/khoj-logo-sideways-500.png" width="200" alt="Khoj Logo"> Web</h1>
|
||||
---
|
||||
sidebar_position: 4
|
||||
---
|
||||
|
||||
> An AI copilot for your Second Brain
|
||||
# Web
|
||||
|
||||
> Query your Second Brain from your Web Browser
|
||||
|
||||
Without any desktop clients, you can start chatting with Khoj on the web. Bear in mind you do need one of the desktop clients in order to share and sync your data with Khoj.
|
||||
|
||||
## Features
|
||||
- **Chat**
|
||||
@@ -15,5 +21,7 @@
|
||||
No setup required. The Khoj web app is the default interface to Khoj. You can access it from any web browser. Try it on [Khoj Cloud](https://app.khoj.dev)
|
||||
|
||||
## Interface
|
||||
|
||||
|
||||
|
||||
| Search | Chat |
|
||||
|:------:|:----:|
|
||||
|  |  |
|
||||
8
documentation/docs/contributing/_category_.json
Normal file
8
documentation/docs/contributing/_category_.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"label": "Contributing",
|
||||
"position": 2,
|
||||
"link": {
|
||||
"type": "generated-index",
|
||||
"description": "Development Setup"
|
||||
}
|
||||
}
|
||||
@@ -1,23 +1,29 @@
|
||||
---
|
||||
sidebar_position: 0
|
||||
---
|
||||
|
||||
# Development
|
||||
|
||||
Welcome to the development docs of Khoj! Thanks for you interesting in being a contributor ❤️. Open source contributors are a corner-store of the Khoj community. We welcome all contributions, big or small.
|
||||
|
||||
To get started with contributing, check out the official GitHub docs on [contributing to an open-source project](https://docs.github.com/en/get-started/exploring-projects-on-github/contributing-to-a-project).
|
||||
|
||||
Join the [Discord](https://discord.gg/WaxF3SkFPU) server and pick up click the ✅ for the question "Are you interested in becoming a contributor?" in the `#welcome-and-rules` channel. This will give you access to the `#contributors` channel where you can ask questions and get help from other contributors.
|
||||
Join the [Discord](https://discord.gg/WaxF3SkFPU) server and click the ✅ for the question "Are you interested in becoming a contributor?" in the `#welcome-and-rules` channel. This will give you access to the `#contributors` channel where you can ask questions and get help from other contributors.
|
||||
|
||||
If you're looking for a place to get started, check out the list of [Github Issues](https://github.com/khoj-ai/khoj/issues) with the tag `good first issue` to find issues that are good for first-time contributors.
|
||||
|
||||
## Local Server Installation
|
||||
### Using Pip
|
||||
#### 1. Install
|
||||
```mdx-code-block
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
```
|
||||
|
||||
|
||||
<!-- tabs:start -->
|
||||
|
||||
#### **MacOS**
|
||||
|
||||
```shell
|
||||
```mdx-code-block
|
||||
<Tabs>
|
||||
<TabItem value="macos" label="MacOS">
|
||||
```shell
|
||||
# Get Khoj Code
|
||||
git clone https://github.com/khoj-ai/khoj && cd khoj
|
||||
|
||||
@@ -26,11 +32,10 @@ python3 -m venv .venv && source .venv/bin/activate
|
||||
|
||||
# For MacOS or zsh users run this
|
||||
pip install -e '.[dev]'
|
||||
```
|
||||
|
||||
#### **Windows**
|
||||
|
||||
```shell
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="win" label="Windows">
|
||||
```shell
|
||||
# Get Khoj Code
|
||||
git clone https://github.com/khoj-ai/khoj && cd khoj
|
||||
|
||||
@@ -39,13 +44,11 @@ python3 -m venv .venv && .venv\Scripts\activate
|
||||
|
||||
# Install Khoj for Development
|
||||
pip install -e .[dev]
|
||||
|
||||
```
|
||||
|
||||
#### **Linux**
|
||||
|
||||
```shell
|
||||
# Get Khoj Code
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="unix" label="Linux">
|
||||
```shell
|
||||
# Get Khoj Code
|
||||
git clone https://github.com/khoj-ai/khoj && cd khoj
|
||||
|
||||
# Create, Activate Virtual Environment
|
||||
@@ -53,15 +56,15 @@ python3 -m venv .venv && source .venv/bin/activate
|
||||
|
||||
# Install Khoj for Development
|
||||
pip install -e .[dev]
|
||||
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
```
|
||||
|
||||
<!-- tabs:end -->
|
||||
|
||||
|
||||
#### 2. Run
|
||||
1. Start Khoj
|
||||
```shell
|
||||
```bash
|
||||
khoj -vv
|
||||
```
|
||||
2. Configure Khoj
|
||||
@@ -115,7 +118,9 @@ In whichever clients you're using for testing, you'll need to update the server
|
||||
|
||||
### Before Creating PR
|
||||
|
||||
!> **Note**: You should be in an active virtual environment for Khoj in order to run the unit tests and linter.
|
||||
:::tip[Note]
|
||||
You should be in an active virtual environment for Khoj in order to run the unit tests and linter.
|
||||
:::
|
||||
|
||||
1. Ensure that you have a [Github Issue](https://github.com/khoj-ai/khoj/issues) that can be linked to the PR. If not, create one. Make sure you've tagged one of the maintainers to the issue. This will ensure that the maintainers are notified of the PR and can review it. It's best discuss the code design on an existing issue or Discord thread before creating a PR. This helps get your PR merged faster.
|
||||
1. Run unit tests.
|
||||
@@ -129,7 +134,7 @@ In whichever clients you're using for testing, you'll need to update the server
|
||||
4. Think about how to add unit tests to verify the functionality you're adding in the PR. If you're not sure how to do this, ask for help in the Github issue or on Discord's `#contributors` channel.
|
||||
|
||||
### After Creating PR
|
||||
1. Automated [validation workflows](.github/workflows) should run for every PR. Tag one of the maintainers in the PR to trigger it.
|
||||
1. Automated [validation workflows](https://github.com/khoj-ai/khoj/tree/master/.github/workflows) should run for every PR. Tag one of the maintainers in the PR to trigger it.
|
||||
|
||||
## Obsidian Plugin Development
|
||||
### Plugin development setup
|
||||
@@ -163,14 +168,14 @@ Follow the steps below to [release](https://github.com/debanjum/khoj/releases/)
|
||||
|
||||
## Architecture
|
||||
|
||||
|
||||
|
||||
|
||||
## Visualize Codebase
|
||||
|
||||
*[Interactive Visualization](https://mango-dune-07a8b7110.1.azurestaticapps.net/?repo=debanjum%2Fkhoj)*
|
||||
|
||||
|
||||
|
||||
|
||||
## Visualize Khoj Obsidian Plugin Codebase
|
||||
|
||||
|
||||
|
||||
8
documentation/docs/features/_category_.json
Normal file
8
documentation/docs/features/_category_.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"label": "Features",
|
||||
"position": 3,
|
||||
"link": {
|
||||
"type": "generated-index",
|
||||
"description": "Features supported by Khoj"
|
||||
}
|
||||
}
|
||||
34
documentation/docs/features/all_features.md
Normal file
34
documentation/docs/features/all_features.md
Normal file
@@ -0,0 +1,34 @@
|
||||
---
|
||||
sidebar_position: 1
|
||||
---
|
||||
|
||||
# Features
|
||||
|
||||
Khoj supports a variety of features, including search and chat with a wide range of data sources and interfaces.
|
||||
|
||||
#### [Search](/features/search)
|
||||
- **Local**: Your personal data stays local. All search and indexing is done on your machine when you [self-host](/get-started/setup)
|
||||
- **Incremental**: Incremental search for a fast, search-as-you-type experience
|
||||
|
||||
#### [Chat](/features/chat)
|
||||
- **Faster answers**: Find answers faster, smoother than search. No need to manually scan through your notes to find answers.
|
||||
- **Iterative discovery**: Iteratively explore and (re-)discover your notes
|
||||
- **Assisted creativity**: Smoothly weave across answers retrieval and content generation
|
||||
- **Works online or offline**: Chat using online or offline AI chat models
|
||||
|
||||
#### General
|
||||
- **Cloud or Self-Host**: Use [cloud](https://app.khoj.dev/login) to use Khoj anytime from anywhere or [self-host](/get-started/setup) for privacy
|
||||
- **Natural**: Advanced natural language understanding using Transformer based ML Models
|
||||
- **Pluggable**: Modular architecture makes it easy to plug in new data sources, frontends and ML models
|
||||
- **Multiple Sources**: Index your Org-mode, Markdown, PDF, plaintext files, Github repos and Notion pages
|
||||
- **Multiple Interfaces**: Interact from your Web Browser, Emacs, Obsidian, Desktop app or even Whatsapp
|
||||
|
||||
### Supported Interfaces
|
||||
Khoj is available as a [Desktop app](/clients/desktop), [Emacs package](/clients/emacs), [Obsidian plugin](/clients/obsidian), [Web app](/clients/web) and a [Whatsapp AI](https://khoj.dev/whatsapp).
|
||||
|
||||
|
||||
|
||||
### Supported Data Sources
|
||||
Khoj can understand your org-mode, markdown, PDF, plaintext files, [Github projects](/online-data-sources/github_integration) and [Notion pages](/online-data-sources/notion_integration).
|
||||
|
||||
|
||||
@@ -1,4 +1,11 @@
|
||||
## Khoj Chat
|
||||
---
|
||||
sidebar_position: 2
|
||||
---
|
||||
|
||||
# Chat
|
||||
|
||||
You can configure Khoj to chat with you about anything. When relevant, it'll use any notes or documents you shared with it to respond.
|
||||
|
||||
### Overview
|
||||
- Creates a personal assistant for you to inquire and engage with your notes
|
||||
- You can choose to use Online or Offline Chat depending on your requirements
|
||||
@@ -18,12 +25,17 @@ Offline chat stays completely private and works without internet using open-sour
|
||||
1. Open your [Khoj offline settings](http://localhost:42110/server/admin/database/offlinechatprocessorconversationconfig/) and click *Enable* on the Offline Chat configuration.
|
||||
2. Open your [Chat model options](http://localhost:42110/server/admin/database/chatmodeloptions/) and add a new option for the offline chat model you want to use. Make sure to use `Offline` as its type. We currently only support offline models that use the [Llama chat prompt](https://replicate.com/blog/how-to-prompt-llama#wrap-user-input-with-inst-inst-tags) format. We recommend using `mistral-7b-instruct-v0.1.Q4_0.gguf`.
|
||||
|
||||
!> **Note**: Offline chat is not supported for a multi-user scenario. The host machine will encounter segmentation faults if multiple users try to use offline chat at the same time.
|
||||
|
||||
:::tip[Note]
|
||||
Offline chat is not supported for a multi-user scenario. The host machine will encounter segmentation faults if multiple users try to use offline chat at the same time.
|
||||
:::
|
||||
|
||||
#### Online Chat
|
||||
Online chat requires internet to use ChatGPT but is faster, higher quality and less compute intensive.
|
||||
|
||||
!> **Warning**: This will enable Khoj to send your chat queries and query relevant notes to OpenAI for processing
|
||||
:::danger[Warning]
|
||||
This will enable Khoj to send your chat queries and query relevant notes to OpenAI for processing.
|
||||
:::
|
||||
|
||||
1. Get your [OpenAI API Key](https://platform.openai.com/account/api-keys)
|
||||
2. Open your [Khoj Online Chat settings](http://localhost:42110/server/admin/database/openaiprocessorconversationconfig/). Add a new setting with your OpenAI API key, and click *Save*. Only one configuration will be used, so make sure that's the only one you have.
|
||||
@@ -34,9 +46,9 @@ Online chat requires internet to use ChatGPT but is faster, higher quality and l
|
||||
- **On Web**: Open [/chat](https://app.khoj.dev/chat) in your web browser
|
||||
- **On Obsidian**: Search for *Khoj: Chat* in the [Command Palette](https://help.obsidian.md/Plugins/Command+palette)
|
||||
- **On Emacs**: Run `M-x khoj <user-query>`
|
||||
2. Enter your queries to chat with Khoj. Use [slash commands](#commands) and [query filters](./advanced.md#query-filters) to change what Khoj uses to respond
|
||||
2. Enter your queries to chat with Khoj. Use [slash commands](#commands) and [query filters](/miscellaneous/advanced#query-filters) to change what Khoj uses to respond
|
||||
|
||||
|
||||
|
||||
|
||||
#### Details
|
||||
1. Your query is used to retrieve the most relevant notes, if any, using Khoj search
|
||||
@@ -47,4 +59,6 @@ Slash commands allows you to change what Khoj uses to respond to your query
|
||||
- **/notes**: Limit chat to only respond using your notes, not just Khoj's general world knowledge as reference
|
||||
- **/general**: Limit chat to only respond using Khoj's general world knowledge, not using your notes as reference
|
||||
- **/default**: Allow chat to respond using your notes or it's general knowledge as reference. It's the default behavior when no slash command is used
|
||||
- **/online**: Use online information and incorporate it in the prompt to the LLM to send you a response.
|
||||
- **/image**: Generate an image in response to your query.
|
||||
- **/help**: Use /help to get all available commands and general information about Khoj
|
||||
@@ -1,10 +1,17 @@
|
||||
## Khoj Search
|
||||
---
|
||||
sidebar_position: 3
|
||||
---
|
||||
|
||||
# Search
|
||||
|
||||
Take advantage of super fast search to find relevant notes and documents from your Second Brain.
|
||||
|
||||
### Use
|
||||
1. Open Khoj Search
|
||||
- **On Web**: Open <https://app.khoj.dev/> in your web browser
|
||||
- **On Web**: Open https://app.khoj.dev/ in your web browser
|
||||
- **On Obsidian**: Click the *Khoj search* icon 🔎 on the [Ribbon](https://help.obsidian.md/User+interface/Workspace/Ribbon) or Search for *Khoj: Search* in the [Command Palette](https://help.obsidian.md/Plugins/Command+palette)
|
||||
- **On Emacs**: Run `M-x khoj <user-query>`
|
||||
2. Query using natural language to find relevant entries from your knowledge base. Use [query filters](./advanced.md#query-filters) to limit entries to search
|
||||
2. Query using natural language to find relevant entries from your knowledge base. Use [query filters](/miscellaneous/advanced#query-filters) to limit entries to search
|
||||
|
||||
### Demo
|
||||
|
||||
|
||||
8
documentation/docs/get-started/_category_.json
Normal file
8
documentation/docs/get-started/_category_.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"label": "Get Started",
|
||||
"position": 1,
|
||||
"link": {
|
||||
"type": "generated-index",
|
||||
"description": "Learn how to get started with using Khoj"
|
||||
}
|
||||
}
|
||||
@@ -1,22 +1,22 @@
|
||||
## Demos
|
||||
---
|
||||
sidebar_position: 2
|
||||
---
|
||||
|
||||
# Demos
|
||||
|
||||
Check out a couple of demos and screenshots of Khoj in action.
|
||||
|
||||
### Screenshots
|
||||
#### Web
|
||||
|
||||
|
||||
|
||||
#### Obsidian
|
||||
|
||||
|
||||
|
||||
#### Emacs
|
||||
|
||||
|
||||
| Web | Obsidian | Emacs |
|
||||
|:---:|:--------:|:-----:|
|
||||
|  |  |  |
|
||||
|  |  |  |
|
||||
|
||||
|
||||
### Videos
|
||||
#### Khoj in Obsidian
|
||||
[KhojObsidian](https://github-production-user-asset-6210df.s3.amazonaws.com/6413477/240061700-3e33d8ea-25bb-46c8-a3bf-c92f78d0f56b.mp4 ':include :type=mp4')
|
||||
[Link to Video](https://github-production-user-asset-6210df.s3.amazonaws.com/6413477/240061700-3e33d8ea-25bb-46c8-a3bf-c92f78d0f56b.mp4)
|
||||
|
||||
##### Installation
|
||||
|
||||
@@ -33,7 +33,7 @@
|
||||
- Jump to the [search result](https://marcus.se.net/obsidian-plugin-docs/publishing/submit-your-plugin)
|
||||
|
||||
#### Khoj in Emacs, Browser
|
||||
[KhojEmacs](https://user-images.githubusercontent.com/6413477/184735169-92c78bf1-d827-4663-9087-a1ea194b8f4b.mp4 ':include :type=mp4')
|
||||
[Link to Video](https://user-images.githubusercontent.com/6413477/184735169-92c78bf1-d827-4663-9087-a1ea194b8f4b.mp4)
|
||||
|
||||
##### Installation
|
||||
|
||||
52
documentation/docs/get-started/overview.md
Normal file
52
documentation/docs/get-started/overview.md
Normal file
@@ -0,0 +1,52 @@
|
||||
---
|
||||
sidebar_position: 0
|
||||
slug: /
|
||||
---
|
||||
|
||||
# Overview
|
||||
|
||||
<p align="center"><img src="/img/khoj-logo-sideways-500.png" width="200" alt="Khoj Logo"></img></p>
|
||||
|
||||
<div align="center">
|
||||
<b>An AI copilot for your Second Brain</b>
|
||||
</div>
|
||||
|
||||
<br />
|
||||
<div align="center">
|
||||
|
||||
[📜 Explore Code](https://github.com/khoj-ai/khoj)
|
||||
<span> • </span>
|
||||
[🌍 Try Khoj Cloud](https://khoj.dev)
|
||||
<span> • </span>
|
||||
[💬 Get Involved](https://discord.gg/BDgyabRM6e)
|
||||
|
||||
</div>
|
||||
|
||||
## Introduction
|
||||
Welcome to the Khoj Docs! This is the best place to get setup and explore Khoj's features.
|
||||
|
||||
- Khoj is an open source, personal AI
|
||||
- You can [chat](/features/chat) with it about anything. It'll use files you shared with it to respond, when relevant
|
||||
- Quickly [find](/features/search) relevant notes and documents using natural language
|
||||
- It understands pdf, plaintext, markdown, org-mode files, [notion pages](/online-data-sources/notion_integration) and [github repositories](/online-data-sources/github_integration)
|
||||
- Access it from your [Emacs](/clients/emacs), [Obsidian](/clients/obsidian), [Web browser](/clients/web) or the [Khoj Desktop app](/clients/desktop)
|
||||
- Use [cloud](https://app.khoj.dev/login) to access your Khoj anytime from anywhere, [self-host](/get-started/setup) on consumer hardware for privacy
|
||||
|
||||
## Quickstart
|
||||
- [Try Khoj Cloud](https://app.khoj.dev) to get started quickly
|
||||
- [Read these instructions](/get-started/setup) to self-host a private instance of Khoj
|
||||
|
||||
## At a Glance
|
||||
<img src="https://docs.khoj.dev/img/khoj_search_on_web.png" width="400px" />
|
||||
<span> </span>
|
||||
<img src="https://docs.khoj.dev/img/khoj_chat_on_web.png" width="400px" />
|
||||
|
||||
#### [Search](/features/search)
|
||||
- **Natural**: Use natural language queries to quickly find relevant notes and documents.
|
||||
- **Incremental**: Incremental search for a fast, search-as-you-type experience
|
||||
|
||||
#### [Chat](/features/chat)
|
||||
- **Faster answers**: Find answers faster, smoother than search. No need to manually scan through your notes to find answers.
|
||||
- **Iterative discovery**: Iteratively explore and (re-)discover your notes
|
||||
- **Assisted creativity**: Smoothly weave across answers retrieval and content generation
|
||||
- **Online or Offline**: Choose online or offline chat depending on your requirements
|
||||
@@ -1,10 +1,22 @@
|
||||
---
|
||||
sidebar_position: 1
|
||||
---
|
||||
|
||||
# Self-Host
|
||||
Learn about how to self-host Khoj on your own machine.
|
||||
|
||||
```mdx-code-block
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
```
|
||||
|
||||
## Setup
|
||||
These are the general setup instructions for Khoj.
|
||||
|
||||
- Make sure [python](https://realpython.com/installing-python/) and [pip](https://pip.pypa.io/en/stable/installation/) are installed on your machine
|
||||
- Check the [Khoj Emacs docs](/emacs?id=setup) to setup Khoj with Emacs<br />
|
||||
- Check the [Khoj Emacs docs](/clients/emacs#setup) to setup Khoj with Emacs<br />
|
||||
It's simpler as it can skip the server *install*, *run* and *configure* step below.
|
||||
- Check the [Khoj Obsidian docs](/obsidian?id=_2-setup-plugin) to setup Khoj with Obsidian<br />
|
||||
- Check the [Khoj Obsidian docs](/clients/obsidian#setup) to setup Khoj with Obsidian<br />
|
||||
Its simpler as it can skip the *configure* step below.
|
||||
|
||||
For Installation, you can either use Docker or install Khoj locally.
|
||||
@@ -33,63 +45,49 @@ Khoj should now be running at http://localhost:42110. You can see the web UI in
|
||||
|
||||
Khoj uses the `pgvector` package to store embeddings of your index in a Postgres database. In order to use this, you need to have Postgres installed.
|
||||
|
||||
<!-- tabs:start -->
|
||||
|
||||
#### **MacOS**
|
||||
|
||||
```mdx-code-block
|
||||
<Tabs groupId="operating-systems">
|
||||
<TabItem value="macos" label="MacOS">
|
||||
Install [Postgres.app](https://postgresapp.com/). This comes pre-installed with `pgvector` and relevant dependencies.
|
||||
|
||||
#### **Windows**
|
||||
|
||||
1. Use the [recommended installer](https://www.postgresql.org/download/windows/)
|
||||
2. Follow instructions to [Install PgVector](https://github.com/pgvector/pgvector#windows) in case you need to manually install it. Windows support is experimental for `pgvector` currently, so we recommend using Docker.
|
||||
|
||||
|
||||
#### **Linux**
|
||||
From [official instructions](https://wiki.postgresql.org/wiki/Apt)
|
||||
|
||||
```bash
|
||||
sudo apt install -y postgresql-common
|
||||
sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh
|
||||
sudo apt install postgres-16 postgresql-16-pgvector
|
||||
</TabItem>
|
||||
<TabItem value="win" label="Windows">
|
||||
1. Use the [recommended installer](https://www.postgresql.org/download/windows/).
|
||||
2. Follow instructions to [Install PgVector](https://github.com/pgvector/pgvector#windows) in case you need to manually install it. Windows support is experimental for pgvector currently, so we recommend using Docker.
|
||||
</TabItem>
|
||||
<TabItem value="unix" label="Linux">
|
||||
From [official instructions](https://wiki.postgresql.org/wiki/Apt)
|
||||
</TabItem>
|
||||
<TabItem value="source" label="From Source">
|
||||
1. Follow instructions to [Install Postgres](https://www.postgresql.org/download/)
|
||||
2. Follow instructions to [Install PgVector](https://github.com/pgvector/pgvector#installation) in case you need to manually install it.
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
```
|
||||
|
||||
##### **From Source**
|
||||
1. Follow instructions to [Install Postgres](https://www.postgresql.org/download/)
|
||||
2. Follow instructions to [Install PgVector](https://github.com/pgvector/pgvector#installation) in case you need to manually install it. Reproduced instructions below for convenience.
|
||||
|
||||
```bash
|
||||
cd /tmp
|
||||
git clone --branch v0.5.1 https://github.com/pgvector/pgvector.git
|
||||
cd pgvector
|
||||
make
|
||||
make install # may need sudo
|
||||
```
|
||||
<!-- tabs:end -->
|
||||
|
||||
|
||||
##### Create the Khoj database
|
||||
|
||||
Make sure to update your environment variables to match your Postgres configuration if you're using a different name. The default values should work for most people. When prompted for a password, you can use the default password `postgres`, or configure it to your preference. Make sure to set the environment variable `POSTGRES_PASSWORD` to the same value as the password you set here.
|
||||
|
||||
<!-- tabs:start -->
|
||||
|
||||
#### **MacOS**
|
||||
```bash
|
||||
```mdx-code-block
|
||||
<Tabs groupId="operating-systems">
|
||||
<TabItem value="macos" label="MacOS">
|
||||
```shell
|
||||
createdb khoj -U postgres --password
|
||||
```
|
||||
|
||||
#### **Windows**
|
||||
```bash
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="win" label="Windows">
|
||||
```shell
|
||||
createdb -U postgres khoj --password
|
||||
```
|
||||
|
||||
#### **Linux**
|
||||
```bash
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="unix" label="Linux">
|
||||
```shell
|
||||
sudo -u postgres createdb khoj --password
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
```
|
||||
|
||||
<!-- tabs:end -->
|
||||
|
||||
#### Install package
|
||||
|
||||
@@ -98,64 +96,54 @@ sudo -u postgres createdb khoj --password
|
||||
|
||||
Run the following command in your terminal to install the Khoj backend.
|
||||
|
||||
<!-- tabs:start -->
|
||||
|
||||
#### **MacOS**
|
||||
|
||||
```shell
|
||||
```mdx-code-block
|
||||
<Tabs groupId="operating-systems">
|
||||
<TabItem value="macos" label="MacOS">
|
||||
```shell
|
||||
python -m pip install khoj-assistant
|
||||
```
|
||||
|
||||
#### **Windows**
|
||||
|
||||
```shell
|
||||
py -m pip install khoj-assistant
|
||||
```
|
||||
For more detailed Windows installation and troubleshooting, see [Windows Install](./windows_install.md).
|
||||
|
||||
#### **Linux**
|
||||
|
||||
```shell
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="win" label="Windows">
|
||||
```shell
|
||||
py -m pip install khoj-assistant
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="unix" label="Linux">
|
||||
```shell
|
||||
python -m pip install khoj-assistant
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
```
|
||||
|
||||
<!-- tabs:end -->
|
||||
|
||||
##### Local Server Start
|
||||
|
||||
Before getting started, configure the following environment variables in your terminal for the first run
|
||||
|
||||
<!-- tabs:start -->
|
||||
|
||||
#### **MacOS**
|
||||
|
||||
```shell
|
||||
export KHOJ_ADMIN_EMAIL=<your-email>
|
||||
export KHOJ_ADMIN_PASSWORD=<your-password>
|
||||
```mdx-code-block
|
||||
<Tabs groupId="operating-systems">
|
||||
<TabItem value="macos" label="MacOS">
|
||||
```shell
|
||||
export KHOJ_ADMIN_EMAIL=<your-email>
|
||||
export KHOJ_ADMIN_PASSWORD=<your-password>
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="win" label="Windows">
|
||||
If you're using PowerShell:
|
||||
```shell
|
||||
$env:KHOJ_ADMIN_EMAIL="<your-email>"
|
||||
$env:KHOJ_ADMIN_PASSWORD="<your-password>"
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="unix" label="Linux">
|
||||
```shell
|
||||
export KHOJ_ADMIN_EMAIL=<your-email>
|
||||
export KHOJ_ADMIN_PASSWORD=<your-password>
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
```
|
||||
|
||||
#### **Windows**
|
||||
|
||||
If you're using PowerShell:
|
||||
```shell
|
||||
$env:KHOJ_ADMIN_EMAIL="<your-email>"
|
||||
$env:KHOJ_ADMIN_PASSWORD="<your-password>"
|
||||
```
|
||||
|
||||
If you're using a Unix shell:
|
||||
```shell
|
||||
export KHOJ_ADMIN_EMAIL="<your-email>"
|
||||
export KHOJ_ADMIN_PASSWORD="<your-password>"
|
||||
```
|
||||
|
||||
#### **Linux**
|
||||
|
||||
```shell
|
||||
export KHOJ_ADMIN_EMAIL=<your-email>
|
||||
export KHOJ_ADMIN_PASSWORD=<your-password>
|
||||
```
|
||||
|
||||
<!-- tabs:end -->
|
||||
|
||||
Run the following command from your terminal to start the Khoj backend and open Khoj in your browser.
|
||||
|
||||
@@ -183,21 +171,24 @@ To use the desktop client, you need to go to your Khoj server's settings page (h
|
||||
1. Go to http://localhost:42110/server/admin and login with your admin credentials.
|
||||
1. Go to [OpenAI settings](http://localhost:42110/server/admin/database/openaiprocessorconversationconfig/) in the server admin settings to add an Open AI processor conversation config. This is where you set your API key. Alternatively, you can go to the [offline chat settings](http://localhost:42110/server/admin/database/offlinechatprocessorconversationconfig/) and simply create a new setting with `Enabled` set to `True`.
|
||||
2. Go to the ChatModelOptions if you want to add additional models for chat. For example, you can specify `gpt-4` if you're using OpenAI or `mistral-7b-instruct-v0.1.Q4_0.gguf` if you're using offline chat. Make sure to configure the `type` field to `OpenAI` or `Offline` respectively.
|
||||
1. Select files and folders to index [using the desktop client](./setup.md?id=_2-download-the-desktop-client). When you click 'Save', the files will be sent to your server for indexing.
|
||||
1. Select files and folders to index [using the desktop client](/get-started/setup#2-download-the-desktop-client). When you click 'Save', the files will be sent to your server for indexing.
|
||||
- Select Notion workspaces and Github repositories to index using the web interface.
|
||||
|
||||
|
||||
> Note: Using Safari on Mac? You might not be able to login to the admin panel. Try using Chrome or Firefox instead.
|
||||
:::tip[Note]
|
||||
Using Safari on Mac? You might not be able to login to the admin panel. Try using Chrome or Firefox instead.
|
||||
:::
|
||||
|
||||
|
||||
### 4. Install Client Plugins (Optional)
|
||||
Khoj exposes a web interface to search, chat and configure by default.<br />
|
||||
The optional steps below allow using Khoj from within an existing application like Obsidian or Emacs.
|
||||
|
||||
- **Khoj Obsidian**:<br />
|
||||
[Install](/obsidian?id=_2-setup-plugin) the Khoj Obsidian plugin
|
||||
[Install](/clients/obsidian#setup) the Khoj Obsidian plugin
|
||||
|
||||
- **Khoj Emacs**:<br />
|
||||
[Install](/emacs?id=setup) khoj.el
|
||||
[Install](/clients/emacs#setup) khoj.el
|
||||
|
||||
#### Setup host URL
|
||||
To configure your host URL on your clients when self-hosting, use `http://127.0.0.1:42110`. This is the default value for the `KHOJ_HOST` environment variable. Note that `localhost` will not work.
|
||||
@@ -209,46 +200,62 @@ You can head to http://localhost:42110 to use the web interface. You can also us
|
||||
## Upgrade
|
||||
### Upgrade Khoj Server
|
||||
|
||||
<!-- tabs:start -->
|
||||
|
||||
#### **Local Setup**
|
||||
|
||||
```shell
|
||||
pip install --upgrade khoj-assistant
|
||||
```mdx-code-block
|
||||
<Tabs groupId="environment">
|
||||
<TabItem value="localsetup" label="Local Setup">
|
||||
```shell
|
||||
pip install --upgrade khoj-assistant
|
||||
```
|
||||
*Note: To upgrade to the latest pre-release version of the khoj server run below command*
|
||||
</TabItem>
|
||||
<TabItem value="docker" label="Docker">
|
||||
From the same directory where you have your `docker-compose` file, this will fetch the latest build and upgrade your server.
|
||||
```shell
|
||||
docker-compose up --build
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="emacs" label="Emacs">
|
||||
- Use your Emacs Package Manager to Upgrade
|
||||
- See [khoj.el package setup](/clients/emacs#setup) for details
|
||||
</TabItem>
|
||||
<TabItem value="obsidian" label="Obsidian">
|
||||
- Upgrade via the Community plugins tab on the settings pane in the Obsidian app
|
||||
- See the [khoj plugin setup](/clients/obsidian#setup) for details
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
```
|
||||
|
||||
*Note: To upgrade to the latest pre-release version of the khoj server run below command*
|
||||
```shell
|
||||
# Maps to the latest commit on the master branch
|
||||
pip install --upgrade --pre khoj-assistant
|
||||
```
|
||||
|
||||
#### **Docker**
|
||||
From the same directory where you have your `docker-compose` file, this will fetch the latest build and upgrade your server.
|
||||
|
||||
```shell
|
||||
docker-compose up --build
|
||||
```
|
||||
|
||||
<!-- tabs:end -->
|
||||
|
||||
|
||||
### Upgrade Khoj on Emacs
|
||||
- Use your Emacs Package Manager to Upgrade
|
||||
- See [khoj.el package setup](/emacs?id=setup) for details
|
||||
|
||||
### Upgrade Khoj on Obsidian
|
||||
- Upgrade via the Community plugins tab on the settings pane in the Obsidian app
|
||||
- See the [khoj plugin setup](/obsidian.md?id=_2-setup-plugin) for details
|
||||
|
||||
## Uninstall
|
||||
1. (Optional) Hit `Ctrl-C` in the terminal running the khoj server to stop it
|
||||
2. Delete the khoj directory in your home folder (i.e `~/.khoj` on Linux, Mac or `C:\Users\<your-username>\.khoj` on Windows)
|
||||
5. You might want to `rm -rf` the following directories:
|
||||
- `~/.khoj`
|
||||
- `~/.cache/gpt4all`
|
||||
3. Uninstall the khoj server with `pip uninstall khoj-assistant`
|
||||
4. (Optional) Uninstall khoj.el or the khoj obsidian plugin in the standard way on Emacs, Obsidian
|
||||
### Uninstall Khoj Server
|
||||
|
||||
```mdx-code-block
|
||||
<Tabs groupId="environment">
|
||||
<TabItem value="localsetup" label="Local Setup">
|
||||
```shell
|
||||
# uninstall khoj server
|
||||
pip uninstall khoj-assistant
|
||||
|
||||
# delete khoj postgres db
|
||||
dropdb khoj -U postgres
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="docker" label="Docker">
|
||||
From the same directory where you have your `docker-compose` file, run the command below to remove the server to delete its containers, networks, images and volumes.
|
||||
|
||||
```shell
|
||||
docker-compose down --volumes
|
||||
```
|
||||
</TabItem>
|
||||
<TabItem value="emacs" label="Emacs">
|
||||
Uninstall the khoj Emacs, or desktop client in the standard way from Emacs or your OS respectively
|
||||
You can also `rm -rf ~/.khoj` to remove the Khoj data directory if did a local install.
|
||||
</TabItem>
|
||||
<TabItem value="obsidian" label="Obsidian">
|
||||
Uninstall the khoj Obisidan, or desktop client in the standard way from Obsidian or your OS respectively
|
||||
You can also `rm -rf ~/.khoj` to remove the Khoj data directory if did a local install.
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
```
|
||||
|
||||
## Troubleshoot
|
||||
|
||||
8
documentation/docs/miscellaneous/_category_.json
Normal file
8
documentation/docs/miscellaneous/_category_.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"label": "Miscellaneous",
|
||||
"position": 6,
|
||||
"link": {
|
||||
"type": "generated-index",
|
||||
"description": "Additional resources for learning about Khoj"
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,8 @@
|
||||
---
|
||||
sidebar_position: 3
|
||||
---
|
||||
|
||||
## Advanced Usage
|
||||
# Advanced Usage
|
||||
|
||||
### Search across Different Languages (Self-Hosting)
|
||||
To search for notes in multiple, different languages, you can use a [multi-lingual model](https://www.sbert.net/docs/pretrained_models.html#multi-lingual-models).<br />
|
||||
@@ -1,4 +1,9 @@
|
||||
## Credits
|
||||
---
|
||||
sidebar_position: 4
|
||||
---
|
||||
|
||||
# Credits
|
||||
Many Open Source projects are used to power Khoj. Here's a few of them:
|
||||
|
||||
- [Multi-QA MiniLM Model](https://huggingface.co/sentence-transformers/multi-qa-MiniLM-L6-cos-v1), [All MiniLM Model](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) for Text Search. See [SBert Documentation](https://www.sbert.net/examples/applications/retrieve_rerank/README.html)
|
||||
- [OpenAI CLIP Model](https://github.com/openai/CLIP) for Image Search. See [SBert Documentation](https://www.sbert.net/examples/applications/image-search/README.html)
|
||||
@@ -1,4 +1,10 @@
|
||||
## Performance
|
||||
---
|
||||
sidebar_position: 2
|
||||
---
|
||||
|
||||
# Performance
|
||||
|
||||
Here are some top-level performance metrics for Khoj. These are rough estimates and will vary based on your hardware and data.
|
||||
|
||||
### Search performance
|
||||
|
||||
@@ -1,4 +1,8 @@
|
||||
# Telemetry (self-hosting)
|
||||
---
|
||||
sidebar_position: 1
|
||||
---
|
||||
|
||||
# Telemetry
|
||||
|
||||
We collect some high level, anonymized metadata about usage of Khoj. This includes:
|
||||
- Client (Web, Emacs, Obsidian)
|
||||
8
documentation/docs/online-data-sources/_category_.json
Normal file
8
documentation/docs/online-data-sources/_category_.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"label": "Online Data Sources",
|
||||
"position": 5,
|
||||
"link": {
|
||||
"type": "generated-index",
|
||||
"description": "Online data sources for indexing via Khoj"
|
||||
}
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
# 🧑🏾💻 Setup the Github integration
|
||||
# Setup the Github integration
|
||||
|
||||
The Github integration allows you to index as many repositories as you want. It's currently default configured to index Issues, Commits, and all Markdown/Org files in each repository. For large repositories, this takes a fairly long time, but it works well for smaller projects.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
## 📜 Notion Integration
|
||||
# Notion Integration
|
||||
|
||||
Khoj now supports search/chat with pages in your Notion workspaces. [Notion](notion.so/) is a platform people use for taking notes, especially for collaboration.
|
||||
The Notion integration allows you to search/chat with your Notion workspaces. [Notion](https://notion.so/) is a platform people use for taking notes, especially for collaboration.
|
||||
|
||||
We haven't setup a fancy integration with OAuth yet, so this integration still requires some effort on your end to generate an API key.
|
||||
|
||||
187
documentation/docusaurus.config.js
Normal file
187
documentation/docusaurus.config.js
Normal file
@@ -0,0 +1,187 @@
|
||||
// @ts-check
|
||||
// `@type` JSDoc annotations allow editor autocompletion and type checking
|
||||
// (when paired with `@ts-check`).
|
||||
// There are various equivalent ways to declare your Docusaurus config.
|
||||
// See: https://docusaurus.io/docs/api/docusaurus-config
|
||||
|
||||
import {themes as prismThemes} from 'prism-react-renderer';
|
||||
|
||||
/** @type {import('@docusaurus/types').Config} */
|
||||
const config = {
|
||||
title: 'Khoj AI',
|
||||
tagline: 'An AI copilot for your Second Brain',
|
||||
|
||||
staticDirectories: ['assets'],
|
||||
|
||||
favicon: 'img/favicon-128x128.ico',
|
||||
|
||||
// Set the production url of your site here
|
||||
url: 'https://docs.khoj.dev',
|
||||
// Set the /<baseUrl>/ pathname under which your site is served
|
||||
// For GitHub pages deployment, it is often '/<projectName>/'
|
||||
baseUrl: '/',
|
||||
|
||||
// GitHub pages deployment config.
|
||||
// If you aren't using GitHub pages, you don't need these.
|
||||
organizationName: 'khoj-ai', // Usually your GitHub org/user name.
|
||||
projectName: 'khoj', // Usually your repo name.
|
||||
|
||||
onBrokenLinks: 'throw',
|
||||
onBrokenMarkdownLinks: 'warn',
|
||||
|
||||
// Even if you don't use internationalization, you can use this field to set
|
||||
// useful metadata like html lang. For example, if your site is Chinese, you
|
||||
// may want to replace "en" with "zh-Hans".
|
||||
i18n: {
|
||||
defaultLocale: 'en',
|
||||
locales: ['en'],
|
||||
},
|
||||
|
||||
presets: [
|
||||
[
|
||||
'classic',
|
||||
/** @type {import('@docusaurus/preset-classic').Options} */
|
||||
({
|
||||
docs: {
|
||||
sidebarPath: './sidebars.js',
|
||||
routeBasePath: '/',
|
||||
// Please change this to your repo.
|
||||
// Remove this to remove the "edit this page" links.
|
||||
editUrl:
|
||||
'https://github.com/khoj-ai/khoj/tree/master/documentation/',
|
||||
},
|
||||
blog: {
|
||||
showReadingTime: true,
|
||||
// Please change this to your repo.
|
||||
// Remove this to remove the "edit this page" links.
|
||||
editUrl:
|
||||
'https://github.com/khoj-ai/khoj/tree/master/documentation/blog/',
|
||||
},
|
||||
theme: {
|
||||
customCss: './src/css/custom.css',
|
||||
},
|
||||
}),
|
||||
],
|
||||
],
|
||||
|
||||
themeConfig:
|
||||
/** @type {import('@docusaurus/preset-classic').ThemeConfig} */
|
||||
({
|
||||
image: 'img/khoj-logo-sideways-500.png',
|
||||
metadata: [
|
||||
{name: 'keywords', content: 'khoj, khoj ai, chatgpt, open ai, open source, productivity'},
|
||||
{name: 'og:title', content: 'Khoj Documentation'},
|
||||
{name: 'og:type', content: 'website'},
|
||||
{name: 'og:site_name', content: 'Khoj Documentation'},
|
||||
{name: 'og:description', content: 'Quickly get started with using or self-hosting Khoj'},
|
||||
{name: 'og:image', content: 'https://khoj-web-bucket.s3.amazonaws.com/link_preview_docs.png'},
|
||||
{name: 'og:url', content: 'https://docs.khoj.dev'},
|
||||
{name: 'keywords', content: 'khoj, khoj ai, chatgpt, open ai, open source, productivity'}
|
||||
],
|
||||
navbar: {
|
||||
title: 'Khoj',
|
||||
logo: {
|
||||
alt: 'Khoj AI',
|
||||
src: 'img/favicon-128x128.ico',
|
||||
},
|
||||
items: [
|
||||
{
|
||||
href: 'https://github.com/khoj-ai/khoj',
|
||||
label: '📜 Code',
|
||||
position: 'right',
|
||||
},
|
||||
{
|
||||
href: 'https://app.khoj.dev/login',
|
||||
label: '🌍 Cloud',
|
||||
position: 'right',
|
||||
},
|
||||
{
|
||||
href: 'https://discord.gg/BDgyabRM6e',
|
||||
label: '💬 Discord',
|
||||
position: 'right',
|
||||
},
|
||||
],
|
||||
},
|
||||
footer: {
|
||||
style: 'dark',
|
||||
links: [
|
||||
{
|
||||
title: 'Docs',
|
||||
items: [
|
||||
{
|
||||
label: 'Get Started',
|
||||
to: '/',
|
||||
},
|
||||
{
|
||||
label: 'Features',
|
||||
to: '/features/all_features',
|
||||
},
|
||||
{
|
||||
label: 'Client Apps',
|
||||
to: '/category/clients',
|
||||
},
|
||||
{
|
||||
label: 'Self-Hosting',
|
||||
to: '/get-started/setup',
|
||||
},
|
||||
{
|
||||
label: 'Contributing',
|
||||
to: '/contributing/development',
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
title: 'Community',
|
||||
items: [
|
||||
{
|
||||
label: 'Discord',
|
||||
href: 'https://discord.gg/BDgyabRM6e',
|
||||
},
|
||||
{
|
||||
label: 'LinkedIn',
|
||||
href: 'https://www.linkedin.com/company/khoj-ai/'
|
||||
},
|
||||
{
|
||||
label: 'Twitter',
|
||||
href: 'https://twitter.com/khoj_ai',
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
title: 'More',
|
||||
items: [
|
||||
// {
|
||||
// label: 'Blog',
|
||||
// to: '/blog',
|
||||
// },
|
||||
{
|
||||
label: 'Cloud',
|
||||
href: 'https://app.khoj.dev/login',
|
||||
},
|
||||
{
|
||||
label: 'Code',
|
||||
href: 'https://github.com/khoj-ai/khoj',
|
||||
},
|
||||
{
|
||||
label: 'Website',
|
||||
href: 'https://khoj.dev',
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
copyright: `Copyright © ${new Date().getFullYear()} Khoj, Inc.`,
|
||||
},
|
||||
prism: {
|
||||
theme: prismThemes.github,
|
||||
darkTheme: prismThemes.dracula,
|
||||
},
|
||||
algolia: {
|
||||
appId: "NBR0FXJNGW",
|
||||
apiKey: "8841b34192a28b2d06f04dd28d768017",
|
||||
indexName: "khoj",
|
||||
contextualSearch: false,
|
||||
}
|
||||
}),
|
||||
};
|
||||
|
||||
export default config;
|
||||
14629
documentation/package-lock.json
generated
Normal file
14629
documentation/package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
44
documentation/package.json
Normal file
44
documentation/package.json
Normal file
@@ -0,0 +1,44 @@
|
||||
{
|
||||
"name": "documentation",
|
||||
"version": "0.0.0",
|
||||
"private": true,
|
||||
"scripts": {
|
||||
"docusaurus": "docusaurus",
|
||||
"start": "docusaurus start",
|
||||
"build": "docusaurus build",
|
||||
"swizzle": "docusaurus swizzle",
|
||||
"deploy": "docusaurus deploy",
|
||||
"clear": "docusaurus clear",
|
||||
"serve": "docusaurus serve",
|
||||
"write-translations": "docusaurus write-translations",
|
||||
"write-heading-ids": "docusaurus write-heading-ids"
|
||||
},
|
||||
"dependencies": {
|
||||
"@docusaurus/core": "3.1.0",
|
||||
"@docusaurus/preset-classic": "3.1.0",
|
||||
"@mdx-js/react": "^3.0.0",
|
||||
"clsx": "^2.0.0",
|
||||
"prism-react-renderer": "^2.3.0",
|
||||
"react": "^18.0.0",
|
||||
"react-dom": "^18.0.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@docusaurus/module-type-aliases": "3.1.0",
|
||||
"@docusaurus/types": "3.1.0"
|
||||
},
|
||||
"browserslist": {
|
||||
"production": [
|
||||
">0.5%",
|
||||
"not dead",
|
||||
"not op_mini all"
|
||||
],
|
||||
"development": [
|
||||
"last 3 chrome version",
|
||||
"last 3 firefox version",
|
||||
"last 5 safari version"
|
||||
]
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=18.0"
|
||||
}
|
||||
}
|
||||
33
documentation/sidebars.js
Normal file
33
documentation/sidebars.js
Normal file
@@ -0,0 +1,33 @@
|
||||
/**
|
||||
* Creating a sidebar enables you to:
|
||||
- create an ordered group of docs
|
||||
- render a sidebar for each doc of that group
|
||||
- provide next/previous navigation
|
||||
|
||||
The sidebars can be generated from the filesystem, or explicitly defined here.
|
||||
|
||||
Create as many sidebars as you want.
|
||||
*/
|
||||
|
||||
// @ts-check
|
||||
|
||||
/** @type {import('@docusaurus/plugin-content-docs').SidebarsConfig} */
|
||||
const sidebars = {
|
||||
// By default, Docusaurus generates a sidebar from the docs folder structure
|
||||
tutorialSidebar: [{type: 'autogenerated', dirName: '.'}],
|
||||
|
||||
// But you can create a sidebar manually
|
||||
/*
|
||||
tutorialSidebar: [
|
||||
'intro',
|
||||
'hello',
|
||||
{
|
||||
type: 'category',
|
||||
label: 'Tutorial',
|
||||
items: ['tutorial-basics/create-a-document'],
|
||||
},
|
||||
],
|
||||
*/
|
||||
};
|
||||
|
||||
export default sidebars;
|
||||
@@ -0,0 +1,11 @@
|
||||
.features {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
padding: 2rem 0;
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
.featureSvg {
|
||||
height: 200px;
|
||||
width: 200px;
|
||||
}
|
||||
37
documentation/src/css/custom.css
Normal file
37
documentation/src/css/custom.css
Normal file
@@ -0,0 +1,37 @@
|
||||
/**
|
||||
* Any CSS included here will be global. The classic template
|
||||
* bundles Infima by default. Infima is a CSS framework designed to
|
||||
* work well for content-centric websites.
|
||||
*/
|
||||
|
||||
@import url('https://fonts.googleapis.com/css2?family=Source+Sans+3&display=swap');
|
||||
|
||||
/* You can override the default Infima variables here. */
|
||||
:root {
|
||||
--ifm-color-primary: #fcc50b;
|
||||
--ifm-color-primary-dark: #fcc50b;
|
||||
--ifm-color-primary-darker: #fcc50b;
|
||||
--ifm-color-primary-darkest: #fcc50b;
|
||||
--ifm-color-primary-light: #fcc50b;
|
||||
--ifm-color-primary-lighter: #fcc50b;
|
||||
--ifm-color-primary-lightest: #fcc50b;
|
||||
--ifm-code-font-size: 95%;
|
||||
--ifm-heading-font-family: 'Source Sans 3', sans-serif;
|
||||
--docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.1);
|
||||
}
|
||||
|
||||
/* For readability concerns, you should choose a lighter palette in dark mode. */
|
||||
[data-theme='dark'] {
|
||||
--ifm-color-primary: #fcc50b;
|
||||
--ifm-color-primary-dark: #fcc50b;
|
||||
--ifm-color-primary-darker: #fcc50b;
|
||||
--ifm-color-primary-darkest: #fcc50b;
|
||||
--ifm-color-primary-light: #fcc50b;
|
||||
--ifm-color-primary-lighter: #fcc50b;
|
||||
--ifm-color-primary-lightest: #fcc50b;
|
||||
--docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.3);
|
||||
}
|
||||
|
||||
body {
|
||||
font-family: 'Source Sans 3', sans-serif;
|
||||
}
|
||||
8344
documentation/yarn.lock
Normal file
8344
documentation/yarn.lock
Normal file
File diff suppressed because it is too large
Load Diff
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"id": "khoj",
|
||||
"name": "Khoj",
|
||||
"version": "1.2.0",
|
||||
"version": "1.2.1",
|
||||
"minAppVersion": "0.15.0",
|
||||
"description": "An AI copilot for your Second Brain",
|
||||
"author": "Khoj Inc.",
|
||||
|
||||
@@ -62,8 +62,8 @@ dependencies = [
|
||||
"pymupdf >= 1.23.5",
|
||||
"django == 4.2.7",
|
||||
"authlib == 1.2.1",
|
||||
"gpt4all >= 2.0.0; platform_system == 'Linux' and platform_machine == 'x86_64'",
|
||||
"gpt4all >= 2.0.0; platform_system == 'Windows' or platform_system == 'Darwin'",
|
||||
"gpt4all >= 2.1.0; platform_system == 'Linux' and platform_machine == 'x86_64'",
|
||||
"gpt4all >= 2.1.0; platform_system == 'Windows' or platform_system == 'Darwin'",
|
||||
"itsdangerous == 2.1.2",
|
||||
"httpx == 0.25.0",
|
||||
"pgvector == 0.2.4",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "Khoj",
|
||||
"version": "1.2.0",
|
||||
"version": "1.2.1",
|
||||
"description": "An AI copilot for your Second Brain",
|
||||
"author": "Saba Imran, Debanjum Singh Solanky <team@khoj.dev>",
|
||||
"license": "GPL-3.0-or-later",
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
;; Saba Imran <saba@khoj.dev>
|
||||
;; Description: An AI copilot for your Second Brain
|
||||
;; Keywords: search, chat, org-mode, outlines, markdown, pdf, image
|
||||
;; Version: 1.2.0
|
||||
;; Version: 1.2.1
|
||||
;; Package-Requires: ((emacs "27.1") (transient "0.3.0") (dash "2.19.1"))
|
||||
;; URL: https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"id": "khoj",
|
||||
"name": "Khoj",
|
||||
"version": "1.2.0",
|
||||
"version": "1.2.1",
|
||||
"minAppVersion": "0.15.0",
|
||||
"description": "An AI copilot for your Second Brain",
|
||||
"author": "Khoj Inc.",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "Khoj",
|
||||
"version": "1.2.0",
|
||||
"version": "1.2.1",
|
||||
"description": "An AI copilot for your Second Brain",
|
||||
"author": "Debanjum Singh Solanky, Saba Imran <team@khoj.dev>",
|
||||
"license": "GPL-3.0-or-later",
|
||||
|
||||
@@ -30,5 +30,6 @@
|
||||
"1.0.0": "0.15.0",
|
||||
"1.0.1": "0.15.0",
|
||||
"1.1.0": "0.15.0",
|
||||
"1.2.0": "0.15.0"
|
||||
"1.2.0": "0.15.0",
|
||||
"1.2.1": "0.15.0"
|
||||
}
|
||||
|
||||
@@ -144,7 +144,15 @@ def configure_server(
|
||||
state.cross_encoder_model = dict()
|
||||
|
||||
for model in search_models:
|
||||
state.embeddings_model.update({model.name: EmbeddingsModel(model.bi_encoder)})
|
||||
state.embeddings_model.update(
|
||||
{
|
||||
model.name: EmbeddingsModel(
|
||||
model.bi_encoder,
|
||||
model.embeddings_inference_endpoint,
|
||||
model.embeddings_inference_endpoint_api_key,
|
||||
)
|
||||
}
|
||||
)
|
||||
state.cross_encoder_model.update({model.name: CrossEncoderModel(model.cross_encoder)})
|
||||
|
||||
state.SearchType = configure_search_types()
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
# Generated by Django 4.2.7 on 2024-01-15 18:12
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
dependencies = [
|
||||
("database", "0024_alter_entry_embeddings"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="searchmodelconfig",
|
||||
name="embeddings_inference_endpoint",
|
||||
field=models.CharField(blank=True, default=None, max_length=200, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="searchmodelconfig",
|
||||
name="embeddings_inference_endpoint_api_key",
|
||||
field=models.CharField(blank=True, default=None, max_length=200, null=True),
|
||||
),
|
||||
]
|
||||
@@ -110,6 +110,8 @@ class SearchModelConfig(BaseModel):
|
||||
model_type = models.CharField(max_length=200, choices=ModelType.choices, default=ModelType.TEXT)
|
||||
bi_encoder = models.CharField(max_length=200, default="thenlper/gte-small")
|
||||
cross_encoder = models.CharField(max_length=200, default="cross-encoder/ms-marco-MiniLM-L-6-v2")
|
||||
embeddings_inference_endpoint = models.CharField(max_length=200, default=None, null=True, blank=True)
|
||||
embeddings_inference_endpoint_api_key = models.CharField(max_length=200, default=None, null=True, blank=True)
|
||||
|
||||
|
||||
class TextToImageModelConfig(BaseModel):
|
||||
|
||||
@@ -123,9 +123,9 @@ def filter_questions(questions: List[str]):
|
||||
|
||||
|
||||
def converse_offline(
|
||||
references,
|
||||
online_results,
|
||||
user_query,
|
||||
references=[],
|
||||
online_results=[],
|
||||
conversation_log={},
|
||||
model: str = "mistral-7b-instruct-v0.1.Q4_0.gguf",
|
||||
loaded_model: Union[Any, None] = None,
|
||||
|
||||
@@ -21,9 +21,11 @@ def download_model(model_name: str):
|
||||
# Try load chat model to GPU if:
|
||||
# 1. Loading chat model to GPU isn't disabled via CLI and
|
||||
# 2. Machine has GPU
|
||||
# 3. GPU has enough free memory to load the chat model
|
||||
# 3. GPU has enough free memory to load the chat model with max context length of 4096
|
||||
device = (
|
||||
"gpu" if state.chat_on_gpu and gpt4all.pyllmodel.LLModel().list_gpu(chat_model_config["path"]) else "cpu"
|
||||
"gpu"
|
||||
if state.chat_on_gpu and gpt4all.pyllmodel.LLModel().list_gpu(chat_model_config["path"], 4096)
|
||||
else "cpu"
|
||||
)
|
||||
except ValueError:
|
||||
device = "cpu"
|
||||
@@ -35,7 +37,7 @@ def download_model(model_name: str):
|
||||
raise e
|
||||
|
||||
# Now load the downloaded chat model onto appropriate device
|
||||
chat_model = gpt4all.GPT4All(model_name=model_name, device=device, allow_download=False)
|
||||
chat_model = gpt4all.GPT4All(model_name=model_name, n_ctx=4096, device=device, allow_download=False)
|
||||
logger.debug(f"Loaded chat model to {device.upper()}.")
|
||||
|
||||
return chat_model
|
||||
|
||||
@@ -1,23 +1,69 @@
|
||||
import logging
|
||||
from typing import List
|
||||
|
||||
import requests
|
||||
import tqdm
|
||||
from sentence_transformers import CrossEncoder, SentenceTransformer
|
||||
from torch import nn
|
||||
|
||||
from khoj.utils.helpers import get_device
|
||||
from khoj.utils.rawconfig import SearchResponse
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class EmbeddingsModel:
|
||||
def __init__(self, model_name: str = "thenlper/gte-small"):
|
||||
def __init__(
|
||||
self,
|
||||
model_name: str = "thenlper/gte-small",
|
||||
embeddings_inference_endpoint: str = None,
|
||||
embeddings_inference_endpoint_api_key: str = None,
|
||||
):
|
||||
self.encode_kwargs = {"normalize_embeddings": True}
|
||||
self.model_kwargs = {"device": get_device()}
|
||||
self.model_name = model_name
|
||||
self.inference_endpoint = embeddings_inference_endpoint
|
||||
self.api_key = embeddings_inference_endpoint_api_key
|
||||
self.embeddings_model = SentenceTransformer(self.model_name, **self.model_kwargs)
|
||||
|
||||
def embed_query(self, query):
|
||||
if self.api_key is not None and self.inference_endpoint is not None:
|
||||
target_url = f"{self.inference_endpoint}"
|
||||
payload = {"inputs": [query]}
|
||||
headers = {"Authorization": f"Bearer {self.api_key}", "Content-Type": "application/json"}
|
||||
response = requests.post(target_url, json=payload, headers=headers)
|
||||
return response.json()["embeddings"][0]
|
||||
return self.embeddings_model.encode([query], show_progress_bar=False, **self.encode_kwargs)[0]
|
||||
|
||||
def embed_documents(self, docs):
|
||||
if self.api_key is not None and self.inference_endpoint is not None:
|
||||
target_url = f"{self.inference_endpoint}"
|
||||
if "huggingface" not in target_url:
|
||||
logger.warning(
|
||||
f"Using custom inference endpoint {target_url} is not yet supported. Please us a HuggingFace inference endpoint."
|
||||
)
|
||||
return self.embeddings_model.encode(docs, show_progress_bar=True, **self.encode_kwargs).tolist()
|
||||
# break up the docs payload in chunks of 1000 to avoid hitting rate limits
|
||||
headers = {
|
||||
"Authorization": f"Bearer {self.api_key}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
with tqdm.tqdm(total=len(docs)) as pbar:
|
||||
for i in range(0, len(docs), 1000):
|
||||
payload = {"inputs": docs[i : i + 1000]}
|
||||
response = requests.post(target_url, json=payload, headers=headers)
|
||||
try:
|
||||
response.raise_for_status()
|
||||
except requests.exceptions.HTTPError as e:
|
||||
print(f"Error: {e}")
|
||||
print(f"Response: {response.json()}")
|
||||
raise e
|
||||
if i == 0:
|
||||
embeddings = response.json()["embeddings"]
|
||||
else:
|
||||
embeddings += response.json()["embeddings"]
|
||||
pbar.update(1000)
|
||||
return embeddings
|
||||
return self.embeddings_model.encode(docs, show_progress_bar=True, **self.encode_kwargs).tolist()
|
||||
|
||||
|
||||
|
||||
@@ -6,6 +6,7 @@ import os
|
||||
import time
|
||||
import uuid
|
||||
from typing import Any, Dict, List, Optional, Union
|
||||
from urllib.parse import unquote
|
||||
|
||||
from asgiref.sync import sync_to_async
|
||||
from fastapi import APIRouter, Depends, File, HTTPException, Request, UploadFile
|
||||
@@ -704,6 +705,7 @@ async def chat(
|
||||
rate_limiter_per_day=Depends(ApiUserRateLimiter(requests=10, subscribed_requests=600, window=60 * 60 * 24)),
|
||||
) -> Response:
|
||||
user: KhojUser = request.user.object
|
||||
q = unquote(q)
|
||||
|
||||
await is_ready_to_chat(user)
|
||||
conversation_command = get_conversation_command(query=q, any_references=True)
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import os
|
||||
import urllib.parse
|
||||
from urllib.parse import quote
|
||||
|
||||
@@ -53,6 +54,7 @@ def test_chat_with_no_chat_history_or_retrieved_content_gpt4all(client_offline_c
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------------------------------
|
||||
@pytest.mark.skipif(os.getenv("SERPER_DEV_API_KEY") is None, reason="requires SERPER_DEV_API_KEY")
|
||||
@pytest.mark.chatquality
|
||||
@pytest.mark.django_db(transaction=True)
|
||||
def test_chat_with_online_content(chat_client):
|
||||
|
||||
@@ -30,5 +30,6 @@
|
||||
"1.0.0": "0.15.0",
|
||||
"1.0.1": "0.15.0",
|
||||
"1.1.0": "0.15.0",
|
||||
"1.2.0": "0.15.0"
|
||||
"1.2.0": "0.15.0",
|
||||
"1.2.1": "0.15.0"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user