klbr/khoj
1
0
Fork 0
mirror of https://github.com/khoaliber/khoj.git synced 2026-03-08 05:39:13 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into upgrade-khoj-on-obsidian

Browse Source 
- Conflicts:
  - src/khoj/interface/web/chat.html
    Use our changes with feedback button changes from master
This commit is contained in:
Debanjum Singh Solanky
2024-06-01 09:58:09 +05:30
parent 92bab9fa61 2667ef4544
commit 7a93599fe8
62 changed files with 4127 additions and 353 deletions
Download Patch File Download Diff File Expand all files Collapse all files

BIN
documentation/assets/img/agents_demo.gif
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 14 MiB

BIN
documentation/assets/img/notion_integration.gif
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 9.6 MiB

2
documentation/docs/clients/desktop.md
View File

@@ -8,7 +8,7 @@ sidebar_position: 1
Use the Desktop app to chat and search with Khoj. Use the Desktop app to chat and search with Khoj.
You can also sync any relevant files with Khoj using the app. You can also sync any relevant files with Khoj using the app.
Khoj will use these files to provide contextual reponses when you search or chat. Khoj will use these files to provide contextual responses when you search or chat.
## Features ## Features
- **Chat** - **Chat**

2
documentation/docs/clients/web.md
View File

@@ -25,7 +25,7 @@ You can upload documents to Khoj from the web interface, one at a time. This is
1. You can drag and drop the document into the chat window. 1. You can drag and drop the document into the chat window.
2. Or click the paperclip icon in the chat window and select the document from your file system. 2. Or click the paperclip icon in the chat window and select the document from your file system.
![demo of dragging and dropping a file](https://khoj-web-bucket.s3.amazonaws.com/drag_drop_file.gif) ![demo of dragging and dropping a file](https://assets.khoj.dev/drag_drop_file.gif)
### Install on Phone ### Install on Phone
You can optionally install Khoj as a [Progressive Web App (PWA)](https://web.dev/learn/pwa/installation). This makes it quick and easy to access Khoj on your phone. You can optionally install Khoj as a [Progressive Web App (PWA)](https://web.dev/learn/pwa/installation). This makes it quick and easy to access Khoj on your phone.

86
documentation/docs/contributing/development.mdx
View File

@@ -14,7 +14,7 @@ If you're looking for a place to get started, check out the list of [Github Issu
## Local Server Installation ## Local Server Installation
### Using Pip ### Using Pip
#### 1. Install #### 1. Khoj Installation
```mdx-code-block ```mdx-code-block
import Tabs from '@theme/Tabs'; import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem'; import TabItem from '@theme/TabItem';
@@ -43,7 +43,7 @@ git clone https://github.com/khoj-ai/khoj && cd khoj
python3 -m venv .venv && .venv\Scripts\activate python3 -m venv .venv && .venv\Scripts\activate
# Install Khoj for Development # Install Khoj for Development
pip install -e .[dev] pip install -e '.[dev]'
``` ```
</TabItem> </TabItem>
<TabItem value="unix" label="Linux"> <TabItem value="unix" label="Linux">
@@ -55,14 +55,59 @@ git clone https://github.com/khoj-ai/khoj && cd khoj
python3 -m venv .venv && source .venv/bin/activate python3 -m venv .venv && source .venv/bin/activate
# Install Khoj for Development # Install Khoj for Development
pip install -e .[dev] pip install -e '.[dev]'
```
</TabItem>
</Tabs>
```
#### 2. Postgres Installation & Setup
Khoj uses the `pgvector` package to store embeddings of your index in a Postgres database. To use this, you need to have Postgres installed.
```mdx-code-block
<Tabs groupId="operating-systems">
<TabItem value="macos" label="MacOS">
Install [Postgres.app](https://postgresapp.com/). This comes pre-installed with `pgvector` and relevant dependencies.
</TabItem>
<TabItem value="win" label="Windows">
1. Use the [recommended installer](https://www.postgresql.org/download/windows/).
2. Follow instructions to [Install PgVector](https://github.com/pgvector/pgvector#windows) in case you need to manually install it. Windows support is experimental for pgvector currently, so we recommend using Docker. Refer to Windows Installation Notes below if there are errors.
</TabItem>
<TabItem value="unix" label="Linux">
From [official instructions](https://wiki.postgresql.org/wiki/Apt)
</TabItem>
<TabItem value="source" label="From Source">
1. Follow instructions to [Install Postgres](https://www.postgresql.org/download/)
2. Follow instructions to [Install PgVector](https://github.com/pgvector/pgvector#installation) in case you need to manually install it.
</TabItem>
</Tabs>
```
##### Create the Khoj database
Make sure to update your environment variables to match your Postgres configuration if you're using a different name. The default values should work for most people. When prompted for a password, you can use the default password `postgres`, or configure it to your preference. Make sure to set the environment variable `POSTGRES_PASSWORD` to the same value as the password you set here.
```mdx-code-block
<Tabs groupId="operating-systems">
<TabItem value="macos" label="MacOS">
```shell
createdb khoj -U postgres --password
```
</TabItem>
<TabItem value="win" label="Windows">
```shell
createdb -U postgres khoj --password
```
</TabItem>
<TabItem value="unix" label="Linux">
```shell
sudo -u postgres createdb khoj --password
``` ```
</TabItem> </TabItem>
</Tabs> </Tabs>
``` ```
#### 3. Run
#### 2. Run
1. Start Khoj 1. Start Khoj
```bash ```bash
khoj -vv khoj -vv
@@ -72,6 +117,37 @@ pip install -e .[dev]
Note: Wait after configuration for khoj to Load ML model, generate embeddings and expose API to query notes, images, documents etc specified in config YAML Note: Wait after configuration for khoj to Load ML model, generate embeddings and expose API to query notes, images, documents etc specified in config YAML
#### Windows Installation Notes
1. Command `khoj` Not Recognized
- Try reactivating the virtual environment and rerunning the `khoj` command.
- If it still doesn't work repeat the installation process.
2. Python Package Missing
- Use `pip install xxx` and try running the `khoj` command.
3. Command `createdb` Not Recognized
- make sure path to postgres binaries is included in environment variables. It usually looks something like
```
C:\Program Files\PostgreSQL\16\bin
```
4. Connection Refused on port xxxx
- Locate the `pg_hba.conf` file in the location where postgres was installed.
- Edit the file to have **trust** as the method for user postgres, local, and host connections.
- Below is an example:
```
host all postgres 127.0.0.1/32 trust
# "local" is for Unix domain socket connections only
local all all trust
# IPv4 local connections:
host all all 127.0.0.1/32 trust
# IPv6 local connections:
host all all ::1/128 trust
```
4. Errors with installing pgvector
- Reinstall Visual Studio 2022 Build Tools with:
1. desktop development with c++ selected in workloads
2. MSVC (C++ Build Tools), Windows 10/11 SDK, and C++/CLI support for build tools selected in individual components.
- Open the x64 Native Tools Command Prompt as an Administrator
- Follow the pgvector windows installation [instructions](https://github.com/pgvector/pgvector?tab=readme-ov-file#windows) in this command prompt.
### Using Docker ### Using Docker
Make sure you install the latest version of [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/). Make sure you install the latest version of [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/).

2
documentation/docs/data-sources/notion_integration.md
View File

@@ -4,7 +4,7 @@ The Notion integration allows you to search/chat with your Notion workspaces. [N
Go to https://app.khoj.dev/config to connect your Notion workspace(s) to Khoj. Go to https://app.khoj.dev/config to connect your Notion workspace(s) to Khoj.
![notion_integration](/img/notion_integration.gif) ![notion_integration](https://assets.khoj.dev/notion_integration.gif)
## Self-Hosted Setup ## Self-Hosted Setup

2
documentation/docs/features/agents.md
View File

@@ -6,7 +6,7 @@ sidebar_position: 4
You can use agents to setup custom system prompts with Khoj. The server host can setup their own agents, which are accessible to all users. You can see ours at https://app.khoj.dev/agents. You can use agents to setup custom system prompts with Khoj. The server host can setup their own agents, which are accessible to all users. You can see ours at https://app.khoj.dev/agents.
![Demo](/img/agents_demo.gif) ![Demo](https://assets.khoj.dev/agents_demo.gif)
## Creating an Agent (Self-Hosted) ## Creating an Agent (Self-Hosted)

7
documentation/docs/features/share.md Normal file
View File

@@ -0,0 +1,7 @@
# Shareable Chat
You can share any of your conversations by going to the three dot menu on the conversation and selecting 'Share'. This will create a **public** link that you can share with anyone. The link will open the conversation in the same state it was when you shared it, so your future messages will not be visible to the person you shared it with.
This means you can easily share a conversation with someone to show them how you solved a problem, or to get help with something you're working on.
![demo of sharing a conversation](https://assets.khoj.dev/shareable_conversations.gif)

2
documentation/docs/get-started/overview.md
View File

@@ -38,7 +38,7 @@ Welcome to the Khoj Docs! This is the best place to get setup and explore Khoj's
- [Read these instructions](/get-started/setup) to self-host a private instance of Khoj - [Read these instructions](/get-started/setup) to self-host a private instance of Khoj
## At a Glance ## At a Glance
![demo_chat](/img/using_khoj_for_studying.gif) ![demo_chat](https://assets.khoj.dev/using_khoj_for_studying.gif)
#### [Search](/features/search) #### [Search](/features/search)
- **Natural**: Use natural language queries to quickly find relevant notes and documents. - **Natural**: Use natural language queries to quickly find relevant notes and documents.

11
documentation/docs/get-started/setup.mdx
View File

@@ -59,6 +59,7 @@ Khoj uses the `pgvector` package to store embeddings of your index in a Postgres
Install [Postgres.app](https://postgresapp.com/). This comes pre-installed with `pgvector` and relevant dependencies. Install [Postgres.app](https://postgresapp.com/). This comes pre-installed with `pgvector` and relevant dependencies.
</TabItem> </TabItem>
<TabItem value="win" label="Windows"> <TabItem value="win" label="Windows">
For detailed instructions and troubleshooting, see [this section](/contributing/development#2-postgres-installation--setup).
1. Use the [recommended installer](https://www.postgresql.org/download/windows/). 1. Use the [recommended installer](https://www.postgresql.org/download/windows/).
2. Follow instructions to [Install PgVector](https://github.com/pgvector/pgvector#windows) in case you need to manually install it. Windows support is experimental for pgvector currently, so we recommend using Docker. 2. Follow instructions to [Install PgVector](https://github.com/pgvector/pgvector#windows) in case you need to manually install it. Windows support is experimental for pgvector currently, so we recommend using Docker.
</TabItem> </TabItem>
@@ -117,13 +118,14 @@ python -m pip install khoj-assistant
``` ```
</TabItem> </TabItem>
<TabItem value="win" label="Windows"> <TabItem value="win" label="Windows">
In PowerShell on Windows
```shell ```shell
# 1. (Optional) To use NVIDIA (CUDA) GPU # 1. (Optional) To use NVIDIA (CUDA) GPU
$env:CMAKE_ARGS = "-DLLAMA_OPENBLAS=on" $env:CMAKE_ARGS = "-DLLAMA_OPENBLAS=on"
# 1. (Optional) To use AMD (ROCm) GPU # 1. (Optional) To use AMD (ROCm) GPU
CMAKE_ARGS="-DLLAMA_HIPBLAS=on" $env:CMAKE_ARGS = "-DLLAMA_HIPBLAS=on"
# 1. (Optional) To use VULCAN GPU # 1. (Optional) To use VULCAN GPU
CMAKE_ARGS="-DLLAMA_VULKAN=on" $env:CMAKE_ARGS = "-DLLAMA_VULKAN=on"
# 2. Install Khoj # 2. Install Khoj
py -m pip install khoj-assistant py -m pip install khoj-assistant
@@ -201,6 +203,11 @@ To disable HTTPS, set the `KHOJ_NO_HTTPS` environment variable to `True`. This c
1. Go to http://localhost:42110/server/admin and login with your admin credentials. 1. Go to http://localhost:42110/server/admin and login with your admin credentials.
#### Configure Chat Model #### Configure Chat Model
##### Configure OpenAI or a custom OpenAI-compatible proxy server ##### Configure OpenAI or a custom OpenAI-compatible proxy server
:::info[Ollama Integration]
Using Ollama? See the [Ollama Integration](/miscellaneous/ollama) section for more custom setup instructions.
:::
1. Go to the [OpenAI settings](http://localhost:42110/server/admin/database/openaiprocessorconversationconfig/) in the server admin settings to add an OpenAI processor conversation config. This is where you set your API key and server API base URL. The API base URL is optional - it's only relevant if you're using another OpenAI-compatible proxy server. 1. Go to the [OpenAI settings](http://localhost:42110/server/admin/database/openaiprocessorconversationconfig/) in the server admin settings to add an OpenAI processor conversation config. This is where you set your API key and server API base URL. The API base URL is optional - it's only relevant if you're using another OpenAI-compatible proxy server.
2. Go over to configure your [chat model options](http://localhost:42110/server/admin/database/chatmodeloptions/). Set the `chat-model` field to a supported chat model[^1] of your choice. For example, you can specify `gpt-4-turbo-preview` if you're using OpenAI. 2. Go over to configure your [chat model options](http://localhost:42110/server/admin/database/chatmodeloptions/). Set the `chat-model` field to a supported chat model[^1] of your choice. For example, you can specify `gpt-4-turbo-preview` if you're using OpenAI.
- Make sure to set the `model-type` field to `OpenAI`. - Make sure to set the `model-type` field to `OpenAI`.

33
documentation/docs/miscellaneous/ollama.md Normal file
View File

@@ -0,0 +1,33 @@
# Ollama / Khoj
You can run your own open source models locally with Ollama and use them with Khoj.
:::info[Ollama Integration]
This is only going to be helpful for self-hosted users. If you're using [Khoj Cloud](https://app.khoj.dev), you're limited to our first-party models.
:::
Khoj supports any OpenAI-API compatible server, which includes [Ollama](http://ollama.ai/). Ollama allows you to start a local server with [several popular open-source LLMs](https://ollama.com/library) directly on your own computer. Combined with Khoj, you can chat with these LLMs and use them to search your notes and documents.
While Khoj also supports local-hosted LLMs downloaded from Hugging Face, the Ollama integration is particularly useful for its ease of setup and multi-model support, especially if you're already using Ollama.
## Setup
1. Setup Ollama: https://ollama.com/
2. Start your preferred model with Ollama. For example,
```bash
ollama run llama3
```
3. Go to Khoj settings at [OpenAI Processor Conversation Config](http://localhost:42110/server/admin/database/openaiprocessorconversationconfig/)
4. Create a new config.
- Name: `ollama`
- Api Key: `any string`
- Api Base Url: `http://localhost:11434/v1/` (default for Ollama)
5. Go to [Chat Model Options](http://localhost:42110/server/admin/database/chatmodeloptions/)
6. Create a new config.
- Name: `llama3` (replace with the name of your local model)
- Model Type: `Openai`
- Openai Config: `<the ollama config you created in step 4>`
- Max prompt size: `1000` (replace with the max prompt size of your model)
7. Go to [your config](http://localhost:42110/config) and select the model you just created in the chat model dropdown.
That's it! You should now be able to chat with your Ollama model from Khoj. If you want to add additional models running on Ollama, repeat step 6 for each model.

6
documentation/docs/miscellaneous/telemetry.md
View File

@@ -16,7 +16,11 @@ We don't send any personal information or any information from/about your conten
If you're self-hosting Khoj, you can opt out of telemetry at any time. To do so, If you're self-hosting Khoj, you can opt out of telemetry at any time. To do so,
1. Open `~/.khoj/khoj.yml` 1. Open `~/.khoj/khoj.yml`
2. Set `should-log-telemetry` to `false` 2. Add the following configuration:
```
app:
should-log-telemetry: false
```
3. Save the file and restart Khoj 3. Save the file and restart Khoj
If you have any questions or concerns, please reach out to us on [Discord](https://discord.gg/BDgyabRM6e). If you have any questions or concerns, please reach out to us on [Discord](https://discord.gg/BDgyabRM6e).

2
manifest.json
View File

@@ -1,7 +1,7 @@
{ {
"id": "khoj", "id": "khoj",
"name": "Khoj", "name": "Khoj",
"version": "1.12.0", "version": "1.12.1",
"minAppVersion": "0.15.0", "minAppVersion": "0.15.0",
"description": "An AI copilot for your Second Brain", "description": "An AI copilot for your Second Brain",
"author": "Khoj Inc.", "author": "Khoj Inc.",

4
pyproject.toml
View File

@@ -59,6 +59,7 @@ dependencies = [
"aiohttp ~= 3.9.0", "aiohttp ~= 3.9.0",
"langchain <= 0.2.0", "langchain <= 0.2.0",
"langchain-openai >= 0.0.5", "langchain-openai >= 0.0.5",
"langchain-community == 0.0.27",
"requests >= 2.26.0", "requests >= 2.26.0",
"anyio == 3.7.1", "anyio == 3.7.1",
"pymupdf >= 1.23.5", "pymupdf >= 1.23.5",
@@ -84,6 +85,7 @@ dependencies = [
"pytz ~= 2024.1", "pytz ~= 2024.1",
"cron-descriptor == 1.4.3", "cron-descriptor == 1.4.3",
"django_apscheduler == 0.6.2", "django_apscheduler == 0.6.2",
"anthropic == 0.26.1",
] ]
dynamic = ["version"] dynamic = ["version"]
@@ -102,7 +104,7 @@ prod = [
"stripe == 7.3.0", "stripe == 7.3.0",
"twilio == 8.11", "twilio == 8.11",
"boto3 >= 1.34.57", "boto3 >= 1.34.57",
"resend >= 0.8.0", "resend == 1.0.1",
] ]
dev = [ dev = [
"khoj-assistant[prod]", "khoj-assistant[prod]",

2
src/interface/desktop/assets/khoj.css
View File

@@ -188,12 +188,14 @@ img.khoj-logo {
.khoj-nav-dropdown-content.show { .khoj-nav-dropdown-content.show {
opacity: 1; opacity: 1;
pointer-events: auto; pointer-events: auto;
border-radius: 20px;
} }
.khoj-nav-dropdown-content a { .khoj-nav-dropdown-content a {
color: black; color: black;
padding: 12px 16px; padding: 12px 16px;
text-decoration: none; text-decoration: none;
display: block; display: block;
border-radius: 20px;
} }
.khoj-nav-dropdown-content a:hover { .khoj-nav-dropdown-content a:hover {
background-color: var(--primary-hover); background-color: var(--primary-hover);

19
src/interface/desktop/chat.html
View File

@@ -334,6 +334,16 @@
let anchorElements = element.querySelectorAll('a'); let anchorElements = element.querySelectorAll('a');
anchorElements.forEach((anchorElement) => { anchorElements.forEach((anchorElement) => {
// Tag external links to open in separate window
if (
!anchorElement.href.startsWith("./") &&
!anchorElement.href.startsWith("#") &&
!anchorElement.href.startsWith("/")
) {
anchorElement.setAttribute('target', '_blank');
anchorElement.setAttribute('rel', 'noopener noreferrer');
}
// Add the class "inline-chat-link" to each element // Add the class "inline-chat-link" to each element
anchorElement.classList.add("inline-chat-link"); anchorElement.classList.add("inline-chat-link");
}); });
@@ -1024,11 +1034,12 @@
threeDotMenu.appendChild(conversationMenu); threeDotMenu.appendChild(conversationMenu);
let deleteButton = document.createElement('button'); let deleteButton = document.createElement('button');
deleteButton.type = "button";
deleteButton.innerHTML = "Delete"; deleteButton.innerHTML = "Delete";
deleteButton.classList.add("delete-conversation-button"); deleteButton.classList.add("delete-conversation-button");
deleteButton.classList.add("three-dot-menu-button-item"); deleteButton.classList.add("three-dot-menu-button-item");
deleteButton.addEventListener('click', function() { deleteButton.addEventListener('click', function(event) {
// Ask for confirmation before deleting chat session event.preventDefault();
let confirmation = confirm('Are you sure you want to delete this chat session?'); let confirmation = confirm('Are you sure you want to delete this chat session?');
if (!confirmation) return; if (!confirmation) return;
let deleteURL = `/api/chat/history?client=web&conversation_id=${incomingConversationId}`; let deleteURL = `/api/chat/history?client=web&conversation_id=${incomingConversationId}`;
@@ -1643,9 +1654,10 @@
content: "▶"; content: "▶";
margin-right: 5px; margin-right: 5px;
display: inline-block; display: inline-block;
transition: transform 0.3s ease-in-out; transition: transform 0.1s ease-in-out;
} }
button.reference-button.expanded::before,
button.reference-button:active:before, button.reference-button:active:before,
button.reference-button[aria-expanded="true"]::before { button.reference-button[aria-expanded="true"]::before {
transform: rotate(90deg); transform: rotate(90deg);
@@ -1876,6 +1888,7 @@
text-align: left; text-align: left;
display: flex; display: flex;
position: relative; position: relative;
margin: 0 8px;
} }
.three-dot-menu { .three-dot-menu {

2
src/interface/desktop/package.json
View File

@@ -1,6 +1,6 @@
{ {
"name": "Khoj", "name": "Khoj",
"version": "1.12.0", "version": "1.12.1",
"description": "An AI copilot for your Second Brain", "description": "An AI copilot for your Second Brain",
"author": "Saba Imran, Debanjum Singh Solanky <team@khoj.dev>", "author": "Saba Imran, Debanjum Singh Solanky <team@khoj.dev>",
"license": "GPL-3.0-or-later", "license": "GPL-3.0-or-later",

1
src/interface/desktop/utils.js
View File

@@ -84,6 +84,7 @@ async function populateHeaderPane() {
`} `}
<div id="khoj-nav-menu" class="khoj-nav-dropdown-content"> <div id="khoj-nav-menu" class="khoj-nav-dropdown-content">
<div class="khoj-nav-username"> ${username} </div> <div class="khoj-nav-username"> ${username} </div>
<a id="github-nav" class="khoj-nav" href="https://github.com/khoj-ai/khoj">GitHub</a>
<a id="settings-nav" class="khoj-nav" href="./config.html">⚙️ Settings</a> <a id="settings-nav" class="khoj-nav" href="./config.html">⚙️ Settings</a>
</div> </div>
</div> </div>

2
src/interface/emacs/khoj.el
View File

@@ -6,7 +6,7 @@
;; Saba Imran <saba@khoj.dev> ;; Saba Imran <saba@khoj.dev>
;; Description: An AI copilot for your Second Brain ;; Description: An AI copilot for your Second Brain
;; Keywords: search, chat, org-mode, outlines, markdown, pdf, image ;; Keywords: search, chat, org-mode, outlines, markdown, pdf, image
;; Version: 1.12.0 ;; Version: 1.12.1
;; Package-Requires: ((emacs "27.1") (transient "0.3.0") (dash "2.19.1")) ;; Package-Requires: ((emacs "27.1") (transient "0.3.0") (dash "2.19.1"))
;; URL: https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs ;; URL: https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs

2
src/interface/obsidian/manifest.json
View File

@@ -1,7 +1,7 @@
{ {
"id": "khoj", "id": "khoj",
"name": "Khoj", "name": "Khoj",
"version": "1.12.0", "version": "1.12.1",
"minAppVersion": "0.15.0", "minAppVersion": "0.15.0",
"description": "An AI copilot for your Second Brain", "description": "An AI copilot for your Second Brain",
"author": "Khoj Inc.", "author": "Khoj Inc.",

2
src/interface/obsidian/package.json
View File

@@ -1,6 +1,6 @@
{ {
"name": "Khoj", "name": "Khoj",
"version": "1.12.0", "version": "1.12.1",
"description": "An AI copilot for your Second Brain", "description": "An AI copilot for your Second Brain",
"author": "Debanjum Singh Solanky, Saba Imran <team@khoj.dev>", "author": "Debanjum Singh Solanky, Saba Imran <team@khoj.dev>",
"license": "GPL-3.0-or-later", "license": "GPL-3.0-or-later",

3
src/interface/obsidian/styles.css
View File

@@ -193,8 +193,9 @@ button.reference-button::before {
content: "▶"; content: "▶";
margin-right: 5px; margin-right: 5px;
display: inline-block; display: inline-block;
transition: transform 0.3s ease-in-out; transition: transform 0.1s ease-in-out;
} }
button.reference-button.expanded::before,
button.reference-button:active:before, button.reference-button:active:before,
button.reference-button[aria-expanded="true"]::before { button.reference-button[aria-expanded="true"]::before {
transform: rotate(90deg); transform: rotate(90deg);

3
src/interface/obsidian/versions.json
View File

@@ -48,5 +48,6 @@
"1.11.0": "0.15.0", "1.11.0": "0.15.0",
"1.11.1": "0.15.0", "1.11.1": "0.15.0",
"1.11.2": "0.15.0", "1.11.2": "0.15.0",
"1.12.0": "0.15.0" "1.12.0": "0.15.0",
"1.12.1": "0.15.0"
} }

2
src/khoj/app/settings.py
View File

@@ -110,7 +110,6 @@ TEMPLATES = [
WSGI_APPLICATION = "app.wsgi.application" WSGI_APPLICATION = "app.wsgi.application"
# Database # Database
# https://docs.djangoproject.com/en/4.2/ref/settings/#databases # https://docs.djangoproject.com/en/4.2/ref/settings/#databases
@@ -122,6 +121,7 @@ DATABASES = {
"USER": os.getenv("POSTGRES_USER", "postgres"), "USER": os.getenv("POSTGRES_USER", "postgres"),
"NAME": os.getenv("POSTGRES_DB", "khoj"), "NAME": os.getenv("POSTGRES_DB", "khoj"),
"PASSWORD": os.getenv("POSTGRES_PASSWORD", "postgres"), "PASSWORD": os.getenv("POSTGRES_PASSWORD", "postgres"),
"CONN_MAX_AGE": 200,
} }
} }

89
src/khoj/database/adapters/__init__.py
View File

@@ -36,8 +36,10 @@ from khoj.database.models import (
NotionConfig, NotionConfig,
OpenAIProcessorConversationConfig, OpenAIProcessorConversationConfig,
ProcessLock, ProcessLock,
PublicConversation,
ReflectiveQuestion, ReflectiveQuestion,
SearchModelConfig, SearchModelConfig,
ServerChatSettings,
SpeechToTextModelOptions, SpeechToTextModelOptions,
Subscription, Subscription,
TextToImageModelConfig, TextToImageModelConfig,
@@ -438,20 +440,21 @@ class ProcessLockAdapters:
return True return True
@staticmethod @staticmethod
def remove_process_lock(process_name: str): def remove_process_lock(process_lock: ProcessLock):
return ProcessLock.objects.filter(name=process_name).delete() return process_lock.delete()
@staticmethod @staticmethod
def run_with_lock(func: Callable, operation: ProcessLock.Operation, max_duration_in_seconds: int = 600, **kwargs): def run_with_lock(func: Callable, operation: ProcessLock.Operation, max_duration_in_seconds: int = 600, **kwargs):
# Exit early if process lock is already taken # Exit early if process lock is already taken
if ProcessLockAdapters.is_process_locked(operation): if ProcessLockAdapters.is_process_locked(operation):
logger.info(f"🔒 Skip executing {func} as {operation} lock is already taken") logger.debug(f"🔒 Skip executing {func} as {operation} lock is already taken")
return return
success = False success = False
process_lock = None
try: try:
# Set process lock # Set process lock
ProcessLockAdapters.set_process_lock(operation, max_duration_in_seconds) process_lock = ProcessLockAdapters.set_process_lock(operation, max_duration_in_seconds)
logger.info(f"🔐 Locked {operation} to execute {func}") logger.info(f"🔐 Locked {operation} to execute {func}")
# Execute Function # Execute Function
@@ -459,15 +462,20 @@ class ProcessLockAdapters:
func(**kwargs) func(**kwargs)
success = True success = True
except IntegrityError as e: except IntegrityError as e:
logger.error(f"⚠️ Unable to create the process lock for {func} with {operation}: {e}", exc_info=True) logger.debug(f"⚠️ Unable to create the process lock for {func} with {operation}: {e}")
success = False success = False
except Exception as e: except Exception as e:
logger.error(f"🚨 Error executing {func} with {operation} process lock: {e}", exc_info=True) logger.error(f"🚨 Error executing {func} with {operation} process lock: {e}", exc_info=True)
success = False success = False
finally: finally:
# Remove Process Lock # Remove Process Lock
ProcessLockAdapters.remove_process_lock(operation) if process_lock:
logger.info(f"🔓 Unlocked {operation} process after executing {func} {'Succeeded' if success else 'Failed'}") ProcessLockAdapters.remove_process_lock(process_lock)
logger.info(
f"🔓 Unlocked {operation} process after executing {func} {'Succeeded' if success else 'Failed'}"
)
else:
logger.debug(f"Skip removing {operation} process lock as it was not set")
def run_with_process_lock(*args, **kwargs): def run_with_process_lock(*args, **kwargs):
@@ -560,7 +568,28 @@ class AgentAdapters:
return await Agent.objects.filter(name=AgentAdapters.DEFAULT_AGENT_NAME).afirst() return await Agent.objects.filter(name=AgentAdapters.DEFAULT_AGENT_NAME).afirst()
class PublicConversationAdapters:
@staticmethod
def get_public_conversation_by_slug(slug: str):
return PublicConversation.objects.filter(slug=slug).first()
@staticmethod
def get_public_conversation_url(public_conversation: PublicConversation):
# Public conversations are viewable by anyone, but not editable.
return f"/share/chat/{public_conversation.slug}/"
class ConversationAdapters: class ConversationAdapters:
@staticmethod
def make_public_conversation_copy(conversation: Conversation):
return PublicConversation.objects.create(
source_owner=conversation.user,
agent=conversation.agent,
conversation_log=conversation.conversation_log,
slug=conversation.slug,
title=conversation.title,
)
@staticmethod @staticmethod
def get_conversation_by_user( def get_conversation_by_user(
user: KhojUser, client_application: ClientApplication = None, conversation_id: int = None user: KhojUser, client_application: ClientApplication = None, conversation_id: int = None
@@ -674,11 +703,49 @@ class ConversationAdapters:
@staticmethod @staticmethod
def get_default_conversation_config(): def get_default_conversation_config():
return ChatModelOptions.objects.filter().first() server_chat_settings = ServerChatSettings.objects.first()
if server_chat_settings is None or server_chat_settings.default_model is None:
return ChatModelOptions.objects.filter().first()
return server_chat_settings.default_model
@staticmethod @staticmethod
async def aget_default_conversation_config(): async def aget_default_conversation_config():
return await ChatModelOptions.objects.filter().prefetch_related("openai_config").afirst() server_chat_settings: ServerChatSettings = (
await ServerChatSettings.objects.filter()
.prefetch_related("default_model", "default_model__openai_config")
.afirst()
)
if server_chat_settings is None or server_chat_settings.default_model is None:
return await ChatModelOptions.objects.filter().prefetch_related("openai_config").afirst()
return server_chat_settings.default_model
@staticmethod
async def aget_summarizer_conversation_config():
server_chat_settings: ServerChatSettings = (
await ServerChatSettings.objects.filter()
.prefetch_related(
"summarizer_model", "default_model", "default_model__openai_config", "summarizer_model__openai_config"
)
.afirst()
)
if server_chat_settings is None or (
server_chat_settings.summarizer_model is None and server_chat_settings.default_model is None
):
return await ChatModelOptions.objects.filter().afirst()
return server_chat_settings.summarizer_model or server_chat_settings.default_model
@staticmethod
def create_conversation_from_public_conversation(
user: KhojUser, public_conversation: PublicConversation, client_app: ClientApplication
):
return Conversation.objects.create(
user=user,
conversation_log=public_conversation.conversation_log,
client=client_app,
slug=public_conversation.slug,
title=public_conversation.title,
agent=public_conversation.agent,
)
@staticmethod @staticmethod
def save_conversation( def save_conversation(
@@ -766,7 +833,9 @@ class ConversationAdapters:
return conversation_config return conversation_config
if conversation_config.model_type == "openai" and conversation_config.openai_config: if (
conversation_config.model_type == "openai" or conversation_config.model_type == "anthropic"
) and conversation_config.openai_config:
return conversation_config return conversation_config
else: else:

79
src/khoj/database/admin.py
View File

@@ -1,9 +1,13 @@
import csv import csv
import json import json
from apscheduler.job import Job
from django.contrib import admin from django.contrib import admin
from django.contrib.auth.admin import UserAdmin from django.contrib.auth.admin import UserAdmin
from django.http import HttpResponse from django.http import HttpResponse
from django_apscheduler.admin import DjangoJobAdmin
from django_apscheduler.jobstores import DjangoJobStore
from django_apscheduler.models import DjangoJob
from khoj.database.models import ( from khoj.database.models import (
Agent, Agent,
@@ -18,6 +22,7 @@ from khoj.database.models import (
ProcessLock, ProcessLock,
ReflectiveQuestion, ReflectiveQuestion,
SearchModelConfig, SearchModelConfig,
ServerChatSettings,
SpeechToTextModelOptions, SpeechToTextModelOptions,
Subscription, Subscription,
TextToImageModelConfig, TextToImageModelConfig,
@@ -25,6 +30,35 @@ from khoj.database.models import (
) )
from khoj.utils.helpers import ImageIntentType from khoj.utils.helpers import ImageIntentType
admin.site.unregister(DjangoJob)
class KhojDjangoJobAdmin(DjangoJobAdmin):
list_display = (
"id",
"next_run_time",
"job_info",
)
search_fields = ("id", "next_run_time")
ordering = ("-next_run_time",)
job_store = DjangoJobStore()
def job_info(self, obj):
job: Job = self.job_store.lookup_job(obj.id)
return f"{job.func_ref} {job.args} {job.kwargs}" if job else "None"
job_info.short_description = "Job Info" # type: ignore
def get_search_results(self, request, queryset, search_term):
queryset, use_distinct = super().get_search_results(request, queryset, search_term)
if search_term:
jobs = [job.id for job in self.job_store.get_all_jobs() if search_term in str(job)]
queryset |= self.model.objects.filter(id__in=jobs)
return queryset, use_distinct
admin.site.register(DjangoJob, KhojDjangoJobAdmin)
class KhojUserAdmin(UserAdmin): class KhojUserAdmin(UserAdmin):
list_display = ( list_display = (
@@ -44,12 +78,9 @@ class KhojUserAdmin(UserAdmin):
admin.site.register(KhojUser, KhojUserAdmin) admin.site.register(KhojUser, KhojUserAdmin)
admin.site.register(ChatModelOptions)
admin.site.register(ProcessLock) admin.site.register(ProcessLock)
admin.site.register(SpeechToTextModelOptions) admin.site.register(SpeechToTextModelOptions)
admin.site.register(OpenAIProcessorConversationConfig)
admin.site.register(SearchModelConfig) admin.site.register(SearchModelConfig)
admin.site.register(Subscription)
admin.site.register(ReflectiveQuestion) admin.site.register(ReflectiveQuestion)
admin.site.register(UserSearchModelConfig) admin.site.register(UserSearchModelConfig)
admin.site.register(TextToImageModelConfig) admin.site.register(TextToImageModelConfig)
@@ -85,6 +116,48 @@ class EntryAdmin(admin.ModelAdmin):
ordering = ("-created_at",) ordering = ("-created_at",)
@admin.register(Subscription)
class KhojUserSubscription(admin.ModelAdmin):
list_display = (
"id",
"user",
"type",
)
search_fields = ("id", "user__email", "user__username", "type")
list_filter = ("type",)
@admin.register(ChatModelOptions)
class ChatModelOptionsAdmin(admin.ModelAdmin):
list_display = (
"id",
"chat_model",
"model_type",
"max_prompt_size",
)
search_fields = ("id", "chat_model", "model_type")
@admin.register(OpenAIProcessorConversationConfig)
class OpenAIProcessorConversationConfigAdmin(admin.ModelAdmin):
list_display = (
"id",
"name",
"api_key",
"api_base_url",
)
search_fields = ("id", "name", "api_key", "api_base_url")
@admin.register(ServerChatSettings)
class ServerChatSettingsAdmin(admin.ModelAdmin):
list_display = (
"default_model",
"summarizer_model",
)
@admin.register(Conversation) @admin.register(Conversation)
class ConversationAdmin(admin.ModelAdmin): class ConversationAdmin(admin.ModelAdmin):
list_display = ( list_display = (

42
src/khoj/database/migrations/0036_publicconversation.py Normal file
View File

@@ -0,0 +1,42 @@
# Generated by Django 4.2.10 on 2024-04-17 13:27
import django.db.models.deletion
from django.conf import settings
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("database", "0035_processlock"),
]
operations = [
migrations.CreateModel(
name="PublicConversation",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("created_at", models.DateTimeField(auto_now_add=True)),
("updated_at", models.DateTimeField(auto_now=True)),
("conversation_log", models.JSONField(default=dict)),
("slug", models.CharField(blank=True, default=None, max_length=200, null=True)),
("title", models.CharField(blank=True, default=None, max_length=200, null=True)),
(
"agent",
models.ForeignKey(
blank=True,
default=None,
null=True,
on_delete=django.db.models.deletion.SET_NULL,
to="database.agent",
),
),
(
"source_owner",
models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL),
),
],
options={
"abstract": False,
},
),
]

14
src/khoj/database/migrations/0040_merge_20240504_1010.py Normal file
View File

@@ -0,0 +1,14 @@
# Generated by Django 4.2.10 on 2024-05-04 10:10
from typing import List
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
("database", "0036_publicconversation"),
("database", "0039_merge_20240501_0301"),
]
operations: List[str] = []

14
src/khoj/database/migrations/0041_merge_20240505_1234.py Normal file
View File

@@ -0,0 +1,14 @@
# Generated by Django 4.2.10 on 2024-05-05 12:34
from typing import List
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
("database", "0040_alter_processlock_name"),
("database", "0040_merge_20240504_1010"),
]
operations: List[str] = []

46
src/khoj/database/migrations/0042_serverchatsettings.py Normal file
View File

@@ -0,0 +1,46 @@
# Generated by Django 4.2.10 on 2024-04-29 11:04
import django.db.models.deletion
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("database", "0041_merge_20240505_1234"),
]
operations = [
migrations.CreateModel(
name="ServerChatSettings",
fields=[
("id", models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("created_at", models.DateTimeField(auto_now_add=True)),
("updated_at", models.DateTimeField(auto_now=True)),
(
"default_model",
models.ForeignKey(
blank=True,
default=None,
null=True,
on_delete=django.db.models.deletion.CASCADE,
related_name="default_model",
to="database.chatmodeloptions",
),
),
(
"summarizer_model",
models.ForeignKey(
blank=True,
default=None,
null=True,
on_delete=django.db.models.deletion.CASCADE,
related_name="summarizer_model",
to="database.chatmodeloptions",
),
),
],
options={
"abstract": False,
},
),
]

21
src/khoj/database/migrations/0043_alter_chatmodeloptions_model_type.py Normal file
View File

@@ -0,0 +1,21 @@
# Generated by Django 4.2.10 on 2024-05-26 12:35
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("database", "0042_serverchatsettings"),
]
operations = [
migrations.AlterField(
model_name="chatmodeloptions",
name="model_type",
field=models.CharField(
choices=[("openai", "Openai"), ("offline", "Offline"), ("anthropic", "Anthropic")],
default="offline",
max_length=200,
),
),
]

41
src/khoj/database/models/__init__.py
View File

@@ -1,3 +1,4 @@
import re
import uuid import uuid
from random import choice from random import choice
@@ -83,6 +84,7 @@ class ChatModelOptions(BaseModel):
class ModelType(models.TextChoices): class ModelType(models.TextChoices):
OPENAI = "openai" OPENAI = "openai"
OFFLINE = "offline" OFFLINE = "offline"
ANTHROPIC = "anthropic"
max_prompt_size = models.IntegerField(default=None, null=True, blank=True) max_prompt_size = models.IntegerField(default=None, null=True, blank=True)
tokenizer = models.CharField(max_length=200, default=None, null=True, blank=True) tokenizer = models.CharField(max_length=200, default=None, null=True, blank=True)
@@ -157,6 +159,15 @@ class GithubRepoConfig(BaseModel):
github_config = models.ForeignKey(GithubConfig, on_delete=models.CASCADE, related_name="githubrepoconfig") github_config = models.ForeignKey(GithubConfig, on_delete=models.CASCADE, related_name="githubrepoconfig")
class ServerChatSettings(BaseModel):
default_model = models.ForeignKey(
ChatModelOptions, on_delete=models.CASCADE, default=None, null=True, blank=True, related_name="default_model"
)
summarizer_model = models.ForeignKey(
ChatModelOptions, on_delete=models.CASCADE, default=None, null=True, blank=True, related_name="summarizer_model"
)
class LocalOrgConfig(BaseModel): class LocalOrgConfig(BaseModel):
input_files = models.JSONField(default=list, null=True) input_files = models.JSONField(default=list, null=True)
input_filter = models.JSONField(default=list, null=True) input_filter = models.JSONField(default=list, null=True)
@@ -249,6 +260,36 @@ class Conversation(BaseModel):
agent = models.ForeignKey(Agent, on_delete=models.SET_NULL, default=None, null=True, blank=True) agent = models.ForeignKey(Agent, on_delete=models.SET_NULL, default=None, null=True, blank=True)
class PublicConversation(BaseModel):
source_owner = models.ForeignKey(KhojUser, on_delete=models.CASCADE)
conversation_log = models.JSONField(default=dict)
slug = models.CharField(max_length=200, default=None, null=True, blank=True)
title = models.CharField(max_length=200, default=None, null=True, blank=True)
agent = models.ForeignKey(Agent, on_delete=models.SET_NULL, default=None, null=True, blank=True)
@receiver(pre_save, sender=PublicConversation)
def verify_public_conversation(sender, instance, **kwargs):
def generate_random_alphanumeric(length):
characters = "0123456789abcdefghijklmnopqrstuvwxyz"
return "".join(choice(characters) for _ in range(length))
# check if this is a new instance
if instance._state.adding:
slug = re.sub(r"\W+", "-", instance.slug.lower())[:50]
observed_random_id = set()
while PublicConversation.objects.filter(slug=slug).exists():
try:
random_id = generate_random_alphanumeric(7)
except IndexError:
raise ValidationError(
"Unable to generate a unique slug for the Public Conversation. Please try again later."
)
observed_random_id.add(random_id)
slug = f"{slug}-{random_id}"
instance.slug = slug
class ReflectiveQuestion(BaseModel): class ReflectiveQuestion(BaseModel):
question = models.CharField(max_length=500) question = models.CharField(max_length=500)
user = models.ForeignKey(KhojUser, on_delete=models.CASCADE, default=None, null=True, blank=True) user = models.ForeignKey(KhojUser, on_delete=models.CASCADE, default=None, null=True, blank=True)

34
src/khoj/interface/email/feedback.html Normal file
View File

@@ -0,0 +1,34 @@
<!DOCTYPE html>
<html>
<head>
<title>Khoj Feedback Form</title>
</head>
<body>
<body style="font-family: 'Verdana', sans-serif; font-weight: 400; font-style: normal; padding: 0; text-align: left; width: 600px; margin: 20px auto;">
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
<a class="logo" href="https://khoj.dev" target="_blank" style="text-decoration: none; text-decoration: underline dotted;">
<img src="https://khoj.dev/khoj-logo-sideways-500.png" alt="Khoj Logo" style="width: 100px;">
</a>
<div class="calls-to-action" style="margin-top: 20px;">
<div>
<h1 style="color: #333; font-size: large; font-weight: bold; margin: 10px; line-height: 1.5; background-color: #fee285; padding: 8px; box-shadow: 6px 6px rgba(0, 0, 0, 1.5);">User Feedback:</h1>
<div>
<h3>User Query</h3>
<p>{{uquery}}</p>
</div>
<div>
<h3>Khoj's Response</h3>
{{kquery}}
</div>
<div>
<h3>Sentiment</h3>
<p>{{sentiment}}</p>
</div>
<div>
<h3>User Email</h3>
<p>{{user_email}}</p>
</div>
</div>
</div>
</body>
</html>

46
src/khoj/interface/web/agents.html
View File

@@ -193,9 +193,55 @@
} }
} }
.loader {
width: 48px;
height: 48px;
border-radius: 50%;
display: inline-block;
border-top: 4px solid var(--primary-color);
border-right: 4px solid transparent;
box-sizing: border-box;
animation: rotation 1s linear infinite;
}
.loader::after {
content: '';
box-sizing: border-box;
position: absolute;
left: 0;
top: 0;
width: 48px;
height: 48px;
border-radius: 50%;
border-left: 4px solid var(--summer-sun);
border-bottom: 4px solid transparent;
animation: rotation 0.5s linear infinite reverse;
}
@keyframes rotation {
0% {
transform: rotate(0deg);
}
100% {
transform: rotate(360deg);
}
}
</style> </style>
<script> <script>
async function openChat(agentSlug) { async function openChat(agentSlug) {
// Create a loading animation
let loading = document.createElement("div");
loading.innerHTML = '<div>Booting your agent...</div><span class="loader"></span>';
loading.style.position = "fixed";
loading.style.top = "0";
loading.style.right = "0";
loading.style.bottom = "0";
loading.style.left = "0";
loading.style.display = "flex";
loading.style.justifyContent = "center";
loading.style.alignItems = "center";
loading.style.backgroundColor = "rgba(0, 0, 0, 0.5)"; // Semi-transparent black
document.body.appendChild(loading);
let response = await fetch(`/api/chat/sessions?agent_slug=${agentSlug}`, { method: "POST" }); let response = await fetch(`/api/chat/sessions?agent_slug=${agentSlug}`, { method: "POST" });
let data = await response.json(); let data = await response.json();
if (response.status == 200) { if (response.status == 200) {

3
src/khoj/interface/web/assets/icons/cancel.svg Normal file
View File

@@ -0,0 +1,3 @@
<?xml version="1.0" encoding="utf-8"?>
<!-- Uploaded to: SVG Repo, www.svgrepo.com, Generator: SVG Repo Mixer Tools -->
<svg width="800px" height="800px" viewBox="0 0 1024 1024" class="icon" version="1.1" xmlns="http://www.w3.org/2000/svg"><path d="M704 288h-281.6l177.6-202.88a32 32 0 0 0-48.32-42.24l-224 256a30.08 30.08 0 0 0-2.24 3.84 32 32 0 0 0-2.88 4.16v1.92a32 32 0 0 0 0 5.12A32 32 0 0 0 320 320a32 32 0 0 0 0 4.8 32 32 0 0 0 0 5.12v1.92a32 32 0 0 0 2.88 4.16 30.08 30.08 0 0 0 2.24 3.84l224 256a32 32 0 1 0 48.32-42.24L422.4 352H704a224 224 0 0 1 224 224v128a224 224 0 0 1-224 224H320a232 232 0 0 1-28.16-1.6 32 32 0 0 0-35.84 27.84 32 32 0 0 0 27.84 35.52A295.04 295.04 0 0 0 320 992h384a288 288 0 0 0 288-288v-128a288 288 0 0 0-288-288zM103.04 760a32 32 0 0 0-62.08 16A289.92 289.92 0 0 0 140.16 928a32 32 0 0 0 40-49.92 225.6 225.6 0 0 1-77.12-118.08zM64 672a32 32 0 0 0 22.72-9.28 37.12 37.12 0 0 0 6.72-10.56A32 32 0 0 0 96 640a33.6 33.6 0 0 0-9.28-22.72 32 32 0 0 0-10.56-6.72 32 32 0 0 0-34.88 6.72A32 32 0 0 0 32 640a32 32 0 0 0 2.56 12.16 37.12 37.12 0 0 0 6.72 10.56A32 32 0 0 0 64 672z" fill="#231815" /></svg>
Side by Side

After

Width:  |  Height:  |  Size: 1.1 KiB

5
src/khoj/interface/web/assets/icons/pencil-edit.svg Normal file
View File

@@ -0,0 +1,5 @@
<?xml version="1.0" encoding="utf-8"?><!-- Uploaded to: SVG Repo, www.svgrepo.com, Generator: SVG Repo Mixer Tools -->
<svg width="800px" height="800px" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M21.2799 6.40005L11.7399 15.94C10.7899 16.89 7.96987 17.33 7.33987 16.7C6.70987 16.07 7.13987 13.25 8.08987 12.3L17.6399 2.75002C17.8754 2.49308 18.1605 2.28654 18.4781 2.14284C18.7956 1.99914 19.139 1.92124 19.4875 1.9139C19.8359 1.90657 20.1823 1.96991 20.5056 2.10012C20.8289 2.23033 21.1225 2.42473 21.3686 2.67153C21.6147 2.91833 21.8083 3.21243 21.9376 3.53609C22.0669 3.85976 22.1294 4.20626 22.1211 4.55471C22.1128 4.90316 22.0339 5.24635 21.8894 5.5635C21.7448 5.88065 21.5375 6.16524 21.2799 6.40005V6.40005Z" stroke="#000000" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"/>
<path d="M11 4H6C4.93913 4 3.92178 4.42142 3.17163 5.17157C2.42149 5.92172 2 6.93913 2 8V18C2 19.0609 2.42149 20.0783 3.17163 20.8284C3.92178 21.5786 4.93913 22 6 22H17C19.21 22 20 20.2 20 18V13" stroke="#000000" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"/>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 1.1 KiB

8
src/khoj/interface/web/assets/icons/share.svg Normal file
View File

@@ -0,0 +1,8 @@
<?xml version="1.0" encoding="utf-8"?><!-- Uploaded to: SVG Repo, www.svgrepo.com, Generator: SVG Repo Mixer Tools -->
<svg width="800px" height="800px" viewBox="-0.5 0 25 25" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M6.75 15C8.13071 15 9.25 13.8807 9.25 12.5C9.25 11.1193 8.13071 10 6.75 10C5.36929 10 4.25 11.1193 4.25 12.5C4.25 13.8807 5.36929 15 6.75 15Z" stroke="#0F0F0F" stroke-miterlimit="10" stroke-linecap="round" stroke-linejoin="round"/>
<path d="M17.25 8C18.6307 8 19.75 6.88071 19.75 5.5C19.75 4.11929 18.6307 3 17.25 3C15.8693 3 14.75 4.11929 14.75 5.5C14.75 6.88071 15.8693 8 17.25 8Z" stroke="#0F0F0F" stroke-miterlimit="10" stroke-linecap="round" stroke-linejoin="round"/>
<path d="M17.25 22C18.6307 22 19.75 20.8807 19.75 19.5C19.75 18.1193 18.6307 17 17.25 17C15.8693 17 14.75 18.1193 14.75 19.5C14.75 20.8807 15.8693 22 17.25 22Z" stroke="#0F0F0F" stroke-miterlimit="10" stroke-linecap="round" stroke-linejoin="round"/>
<path d="M13.49 17.05L10.45 15.06" stroke="#0F0F0F" stroke-miterlimit="10" stroke-linecap="round" stroke-linejoin="round"/>
<path d="M13.48 7.96001L10.46 9.94001" stroke="#0F0F0F" stroke-miterlimit="10" stroke-linecap="round" stroke-linejoin="round"/>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 1.2 KiB

6
src/khoj/interface/web/assets/icons/thumbs-down-svgrepo-com.svg Normal file
View File

@@ -0,0 +1,6 @@
<?xml version="1.0" encoding="utf-8"?>
<!-- Uploaded to: SVG Repo, www.svgrepo.com, Generator: SVG Repo Mixer Tools -->
<svg fill="#000000" width="800px" height="800px" viewBox="-7.5 0 32 32" version="1.1" xmlns="http://www.w3.org/2000/svg">
<title>thumbs-down</title>
<path d="M5.92 25.24c-0.64 0-1.36-0.2-2.2-0.64-0.28-0.16-0.44-0.44-0.44-0.76v-14.96c0-0.36 0.24-0.72 0.6-0.8 0.2-0.040 4.8-1.32 8.16-1.32 1.36 0 2.36 0.2 3 0.6 0.88 0.56 1.44 1.52 1.6 2.92 0.36 2.44-0.52 5.92-1.44 6.96-0.8 0.88-2.36 1.040-3.84 1.2-0.72 0.080-1.72 0.2-2 0.36s-0.44 1.4-0.52 2.12c-0.24 1.84-0.6 4.32-2.92 4.32zM4.92 23.32c0.48 0.2 0.8 0.24 1 0.24 0.72 0 1-0.88 1.24-2.84 0.2-1.36 0.36-2.68 1.24-3.28 0.6-0.4 1.6-0.52 2.76-0.64 0.96-0.12 2.44-0.28 2.8-0.68 0.48-0.52 1.36-3.36 1.040-5.6-0.080-0.6-0.32-1.4-0.84-1.72-0.24-0.12-0.8-0.36-2.12-0.36-2.44 0-5.76 0.76-7.12 1.080 0 0 0 13.8 0 13.8zM0.84 18.64c-0.48 0-0.84-0.36-0.84-0.84v-8.92c0-0.48 0.36-0.84 0.84-0.84s0.84 0.36 0.84 0.84v8.96c0 0.44-0.36 0.8-0.84 0.8z"></path>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 1019 B

6
src/khoj/interface/web/assets/icons/thumbs-up-svgrepo-com.svg Normal file
View File

@@ -0,0 +1,6 @@
<?xml version="1.0" encoding="utf-8"?>
<!-- Uploaded to: SVG Repo, www.svgrepo.com, Generator: SVG Repo Mixer Tools -->
<svg fill="#000000" width="800px" height="800px" viewBox="-7.5 0 32 32" version="1.1" xmlns="http://www.w3.org/2000/svg">
<title>thumbs-up</title>
<path d="M12.040 25.24v0c-3.36 0-7.96-1.24-8.16-1.32-0.36-0.080-0.6-0.44-0.6-0.8v-14.96c0-0.32 0.16-0.6 0.44-0.76 0.84-0.44 1.56-0.64 2.2-0.64 2.32 0 2.68 2.48 2.92 4.28 0.080 0.72 0.24 1.92 0.52 2.12s1.28 0.28 2 0.36c1.52 0.16 3.080 0.32 3.84 1.2 0.92 1.040 1.8 4.52 1.44 6.96-0.2 1.4-0.76 2.36-1.6 2.92-0.68 0.44-1.68 0.64-3 0.64zM4.92 22.48c1.36 0.32 4.64 1.080 7.12 1.080 1.32 0 1.92-0.24 2.12-0.36 0.52-0.32 0.76-1.12 0.84-1.72 0.32-2.24-0.56-5.080-1.040-5.6-0.36-0.4-1.84-0.56-2.8-0.68-1.16-0.12-2.12-0.24-2.76-0.64-0.88-0.6-1.080-1.92-1.24-3.28-0.28-1.96-0.52-2.84-1.24-2.84-0.2 0-0.52 0.040-1 0.24 0 0 0 13.8 0 13.8zM0.84 23.96c-0.48 0-0.84-0.36-0.84-0.84v-8.92c0-0.48 0.36-0.84 0.84-0.84s0.84 0.36 0.84 0.84v8.96c0 0.44-0.36 0.8-0.84 0.8z"></path>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 1.0 KiB

2
src/khoj/interface/web/assets/khoj.css
View File

@@ -156,12 +156,14 @@ img.khoj-logo {
.khoj-nav-dropdown-content.show { .khoj-nav-dropdown-content.show {
opacity: 1; opacity: 1;
pointer-events: auto; pointer-events: auto;
border-radius: 20px;
} }
.khoj-nav-dropdown-content a { .khoj-nav-dropdown-content a {
color: black; color: black;
padding: 12px 16px; padding: 12px 16px;
text-decoration: none; text-decoration: none;
display: block; display: block;
border-radius: 20px;
} }
.khoj-nav-dropdown-content a:hover { .khoj-nav-dropdown-content a:hover {
background-color: var(--primary-hover); background-color: var(--primary-hover);

8
src/khoj/interface/web/assets/utils.js
View File

@@ -8,9 +8,11 @@ function toggleMenu() {
document.addEventListener('click', function(event) { document.addEventListener('click', function(event) {
let menu = document.getElementById("khoj-nav-menu"); let menu = document.getElementById("khoj-nav-menu");
let menuContainer = document.getElementById("khoj-nav-menu-container"); let menuContainer = document.getElementById("khoj-nav-menu-container");
let isClickOnMenu = menuContainer.contains(event.target) || menuContainer === event.target; if (menuContainer) {
if (isClickOnMenu === false && menu.classList.contains("show")) { let isClickOnMenu = menuContainer.contains(event.target) || menuContainer === event.target;
menu.classList.remove("show"); if (isClickOnMenu === false && menu.classList.contains("show")) {
menu.classList.remove("show");
}
} }
}); });

27
src/khoj/interface/web/base_processor_integration.html
View File

@@ -1,27 +0,0 @@
<!DOCTYPE html>
<html>
<head>
<title>Khoj: Processor Settings</title>
<link rel=”stylesheet” href=”static/styles.css”>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@picocss/pico@1/css/pico.min.css">
</head>
<body class="data-integration">
<header class=”header”>
<h1>Configure your processor integrations for Khoj</h1>
</header>
<a href="/config">Go back</a>
<div class=”content”>
{% block content %}
{% endblock %}
</div>
<footer class=”footer”>
</footer>
</body>
<style>
body.data-integration {
padding: 0 10%
}
</style>
</html>

264
src/khoj/interface/web/chat.html
View File

@@ -10,14 +10,13 @@
<link rel="icon" type="image/png" sizes="128x128" href="/static/assets/icons/favicon-128x128.png?v={{ khoj_version }}"> <link rel="icon" type="image/png" sizes="128x128" href="/static/assets/icons/favicon-128x128.png?v={{ khoj_version }}">
<link rel="apple-touch-icon" href="/static/assets/icons/favicon-128x128.png?v={{ khoj_version }}"> <link rel="apple-touch-icon" href="/static/assets/icons/favicon-128x128.png?v={{ khoj_version }}">
<link rel="manifest" href="/static/khoj.webmanifest?v={{ khoj_version }}"> <link rel="manifest" href="/static/khoj.webmanifest?v={{ khoj_version }}">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.10/dist/katex.min.css" integrity="sha384-wcIxkf4k558AjM3Yz3BBFQUbk/zgIYC2R0QpeeYb+TwlBVMrlgLqwRjRtGZiK7ww" crossorigin="anonymous"> <link rel="stylesheet" href="https://assets.khoj.dev/katex/katex.min.css">
<!-- The loading of KaTeX is deferred to speed up page rendering --> <!-- The loading of KaTeX is deferred to speed up page rendering -->
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.10/dist/katex.min.js" integrity="sha384-hIoBPJpTUs74ddyc4bFZSM1TVlQDA60VBbJS0oA934VSz82sBx1X7kSx2ATBDIyd" crossorigin="anonymous"></script> <script defer src="https://assets.khoj.dev/katex/katex.min.js"></script>
<!-- To automatically render math in text elements, include the auto-render extension: --> <!-- To automatically render math in text elements, include the auto-render extension: -->
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.10/dist/contrib/auto-render.min.js" integrity="sha384-43gviWU0YVjaDtb/GhzOouOXtZMP/7XUzwPTstBeZFe/+rCMvRwr4yROQP43s0Xk" crossorigin="anonymous" <script defer src="https://assets.khoj.dev/katex/auto-render.min.js" onload="renderMathInElement(document.body);"></script>
onload="renderMathInElement(document.body);"></script>
</head> </head>
<script type="text/javascript" src="/static/assets/utils.js?v={{ khoj_version }}"></script> <script type="text/javascript" src="/static/assets/utils.js?v={{ khoj_version }}"></script>
<script type="text/javascript" src="/static/assets/markdown-it.min.js?v={{ khoj_version }}"></script> <script type="text/javascript" src="/static/assets/markdown-it.min.js?v={{ khoj_version }}"></script>
@@ -179,11 +178,30 @@ To get started, just start typing below. You can also type / to see a list of co
return referenceButton; return referenceButton;
} }
var khojQuery = "";
function renderMessage(message, by, dt=null, annotations=null, raw=false, renderType="append") { function renderMessage(message, by, dt=null, annotations=null, raw=false, renderType="append", userQuery=null) {
let message_time = formatDate(dt ?? new Date()); let message_time = formatDate(dt ?? new Date());
let by_name = by == "khoj" ? "🏮 Khoj" : "🤔 You"; let by_name = by == "khoj" ? "🏮 Khoj" : "🤔 You";
let formattedMessage = formatHTMLMessage(message, raw); let formattedMessage = formatHTMLMessage(message, raw, true, userQuery);
//update userQuery or khojQuery to latest query for feedback purposes
if(by !== "khoj"){
raw = formattedMessage.innerHTML;
}
//find the thumbs up and thumbs down buttons from the message formatter
var thumbsUpButtons = formattedMessage.querySelectorAll('.thumbs-up-button');
var thumbsDownButtons = formattedMessage.querySelectorAll('.thumbs-down-button');
//only render the feedback options if the message is a response from khoj
if(by !== "khoj"){
thumbsUpButtons.forEach(function(element) {
element.parentNode.removeChild(element);
});
thumbsDownButtons.forEach(function(element) {
element.parentNode.removeChild(element);
});
}
// Create a new div for the chat message // Create a new div for the chat message
let chatMessage = document.createElement('div'); let chatMessage = document.createElement('div');
@@ -303,7 +321,22 @@ To get started, just start typing below. You can also type / to see a list of co
return imageMarkdown; return imageMarkdown;
} }
function formatHTMLMessage(message, raw=false, willReplace=true) { //handler function for posting feedback data to endpoint
function sendFeedback(_uquery="", _kquery="", _sentiment="") {
const uquery = _uquery;
const kquery = _kquery;
const sentiment = _sentiment;
fetch('/api/chat/feedback', {
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify({uquery: uquery, kquery: kquery, sentiment: sentiment})
})
.then(response => response.json())
}
function formatHTMLMessage(message, raw=false, willReplace=true, userQuery) {
var md = window.markdownit(); var md = window.markdownit();
let newHTML = message; let newHTML = message;
@@ -347,7 +380,35 @@ To get started, just start typing below. You can also type / to see a list of co
copyIcon.classList.add("copy-icon"); copyIcon.classList.add("copy-icon");
copyButton.appendChild(copyIcon); copyButton.appendChild(copyIcon);
copyButton.addEventListener('click', createCopyParentText(message)); copyButton.addEventListener('click', createCopyParentText(message));
element.append(copyButton);
//create thumbs-up button
let thumbsUpButton = document.createElement('button');
thumbsUpButton.className = 'thumbs-up-button';
let thumbsUpIcon = document.createElement("img");
thumbsUpIcon.src = "/static/assets/icons/thumbs-up-svgrepo-com.svg";
thumbsUpIcon.classList.add("thumbs-up-icon");
thumbsUpButton.appendChild(thumbsUpIcon);
thumbsUpButton.onclick = function() {
khojQuery = newHTML;
thumbsUpIcon.src = "/static/assets/icons/confirm-icon.svg";
sendFeedback(userQuery ,khojQuery, "Good Response");
};
// Create thumbs-down button
let thumbsDownButton = document.createElement('button');
thumbsDownButton.className = 'thumbs-down-button';
let thumbsDownIcon = document.createElement("img");
thumbsDownIcon.src = "/static/assets/icons/thumbs-down-svgrepo-com.svg";
thumbsDownIcon.classList.add("thumbs-down-icon");
thumbsDownButton.appendChild(thumbsDownIcon);
thumbsDownButton.onclick = function() {
khojQuery = newHTML;
thumbsDownIcon.src = "/static/assets/icons/confirm-icon.svg";
sendFeedback(userQuery, khojQuery, "Bad Response");
};
// Append buttons to parent element
element.append(copyButton, thumbsDownButton, thumbsUpButton);
} }
renderMathInElement(element, { renderMathInElement(element, {
@@ -355,7 +416,6 @@ To get started, just start typing below. You can also type / to see a list of co
// • auto-render specific keys, e.g.: // • auto-render specific keys, e.g.:
delimiters: [ delimiters: [
{left: '$$', right: '$$', display: true}, {left: '$$', right: '$$', display: true},
{left: '$', right: '$', display: false},
{left: '\\(', right: '\\)', display: false}, {left: '\\(', right: '\\)', display: false},
{left: '\\[', right: '\\]', display: true} {left: '\\[', right: '\\]', display: true}
], ],
@@ -546,7 +606,7 @@ To get started, just start typing below. You can also type / to see a list of co
} else { } else {
// If the chunk is not a JSON object, just display it as is // If the chunk is not a JSON object, just display it as is
rawResponse += chunk; rawResponse += chunk;
handleStreamResponse(newResponseText, rawResponse, loadingEllipsis); handleStreamResponse(newResponseText, rawResponse, query, loadingEllipsis);
readStream(); readStream();
} }
}); });
@@ -582,14 +642,14 @@ To get started, just start typing below. You can also type / to see a list of co
return loadingEllipsis; return loadingEllipsis;
} }
function handleStreamResponse(newResponseElement, rawResponse, loadingEllipsis, replace=true) { function handleStreamResponse(newResponseElement, rawResponse, rawQuery, loadingEllipsis, replace=true) {
if (newResponseElement.getElementsByClassName("lds-ellipsis").length > 0 && loadingEllipsis) { if (newResponseElement.getElementsByClassName("lds-ellipsis").length > 0 && loadingEllipsis) {
newResponseElement.removeChild(loadingEllipsis); newResponseElement.removeChild(loadingEllipsis);
} }
if (replace) { if (replace) {
newResponseElement.innerHTML = ""; newResponseElement.innerHTML = "";
} }
newResponseElement.appendChild(formatHTMLMessage(rawResponse, false, replace)); newResponseElement.appendChild(formatHTMLMessage(rawResponse, false, replace, rawQuery));
document.getElementById("chat-body").scrollTop = document.getElementById("chat-body").scrollHeight; document.getElementById("chat-body").scrollTop = document.getElementById("chat-body").scrollHeight;
} }
@@ -889,6 +949,7 @@ To get started, just start typing below. You can also type / to see a list of co
loadingEllipsis: null, loadingEllipsis: null,
references: {}, references: {},
rawResponse: "", rawResponse: "",
rawQuery: "",
} }
if (chatBody.dataset.conversationId) { if (chatBody.dataset.conversationId) {
@@ -907,6 +968,7 @@ To get started, just start typing below. You can also type / to see a list of co
// Append any references after all the data has been streamed // Append any references after all the data has been streamed
finalizeChatBodyResponse(websocketState.references, websocketState.newResponseTextEl); finalizeChatBodyResponse(websocketState.references, websocketState.newResponseTextEl);
const liveQuery = websocketState.rawQuery;
// Reset variables // Reset variables
websocketState = { websocketState = {
newResponseTextEl: null, newResponseTextEl: null,
@@ -914,6 +976,7 @@ To get started, just start typing below. You can also type / to see a list of co
loadingEllipsis: null, loadingEllipsis: null,
references: {}, references: {},
rawResponse: "", rawResponse: "",
rawQuery: liveQuery,
} }
} else { } else {
try { try {
@@ -935,9 +998,9 @@ To get started, just start typing below. You can also type / to see a list of co
websocketState.rawResponse = rawResponse; websocketState.rawResponse = rawResponse;
websocketState.references = references; websocketState.references = references;
} else if (chunk.type == "status") { } else if (chunk.type == "status") {
handleStreamResponse(websocketState.newResponseTextEl, chunk.message, null, false); handleStreamResponse(websocketState.newResponseTextEl, chunk.message, websocketState.rawQuery, null, false);
} else if (chunk.type == "rate_limit") { } else if (chunk.type == "rate_limit") {
handleStreamResponse(websocketState.newResponseTextEl, chunk.message, websocketState.loadingEllipsis, true); handleStreamResponse(websocketState.newResponseTextEl, chunk.message, websocketState.rawQuery, websocketState.loadingEllipsis, true);
} else { } else {
rawResponse = chunk.response; rawResponse = chunk.response;
} }
@@ -960,7 +1023,7 @@ To get started, just start typing below. You can also type / to see a list of co
// If the chunk is not a JSON object, just display it as is // If the chunk is not a JSON object, just display it as is
websocketState.rawResponse += chunk; websocketState.rawResponse += chunk;
if (websocketState.newResponseTextEl) { if (websocketState.newResponseTextEl) {
handleStreamResponse(websocketState.newResponseTextEl, websocketState.rawResponse, websocketState.loadingEllipsis); handleStreamResponse(websocketState.newResponseTextEl, websocketState.rawResponse, websocketState.rawQuery, websocketState.loadingEllipsis);
} }
} }
@@ -1037,6 +1100,7 @@ To get started, just start typing below. You can also type / to see a list of co
loadingEllipsis, loadingEllipsis,
references, references,
rawResponse, rawResponse,
rawQuery: query,
} }
} }
@@ -1154,7 +1218,8 @@ To get started, just start typing below. You can also type / to see a list of co
new Date(chat_log.created + "Z"), new Date(chat_log.created + "Z"),
chat_log.onlineContext, chat_log.onlineContext,
chat_log.intent?.type, chat_log.intent?.type,
chat_log.intent?.["inferred-queries"]); chat_log.intent?.["inferred-queries"],
chat_log.intent?.query);
chatBody.appendChild(messageElement); chatBody.appendChild(messageElement);
// When the 4th oldest message is within viewing distance (~60% scroll up) // When the 4th oldest message is within viewing distance (~60% scroll up)
@@ -1249,7 +1314,8 @@ To get started, just start typing below. You can also type / to see a list of co
new Date(chat_log.created + "Z"), new Date(chat_log.created + "Z"),
chat_log.onlineContext, chat_log.onlineContext,
chat_log.intent?.type, chat_log.intent?.type,
chat_log.intent?.["inferred-queries"] chat_log.intent?.["inferred-queries"],
chat_log.intent?.query
); );
entry.target.replaceWith(messageElement); entry.target.replaceWith(messageElement);
@@ -1512,11 +1578,79 @@ To get started, just start typing below. You can also type / to see a list of co
conversationMenu.appendChild(editTitleButton); conversationMenu.appendChild(editTitleButton);
threeDotMenu.appendChild(conversationMenu); threeDotMenu.appendChild(conversationMenu);
let shareButton = document.createElement('button');
shareButton.innerHTML = "Share";
shareButton.type = "button";
shareButton.classList.add("share-conversation-button");
shareButton.classList.add("three-dot-menu-button-item");
shareButton.addEventListener('click', function(event) {
event.preventDefault();
let confirmation = confirm('Are you sure you want to share this chat session? This will make the conversation public.');
if (!confirmation) return;
let duplicateURL = `/api/chat/share?client=web&conversation_id=${incomingConversationId}`;
fetch(duplicateURL , { method: "POST" })
.then(response => response.ok ? response.json() : Promise.reject(response))
.then(data => {
if (data.status == "ok") {
flashStatusInChatInput("✅ Conversation shared successfully");
}
// Make a pop-up that shows data.url to share the conversation
let shareURL = data.url;
let shareModal = document.createElement('div');
shareModal.classList.add("modal");
shareModal.id = "share-conversation-modal";
let shareModalContent = document.createElement('div');
shareModalContent.classList.add("modal-content");
let shareModalHeader = document.createElement('div');
shareModalHeader.classList.add("modal-header");
let shareModalTitle = document.createElement('h2');
shareModalTitle.textContent = "Share Conversation";
let shareModalCloseButton = document.createElement('button');
shareModalCloseButton.classList.add("modal-close-button");
shareModalCloseButton.innerHTML = "&times;";
shareModalCloseButton.addEventListener('click', function() {
shareModal.remove();
});
shareModalHeader.appendChild(shareModalTitle);
shareModalHeader.appendChild(shareModalCloseButton);
shareModalContent.appendChild(shareModalHeader);
let shareModalBody = document.createElement('div');
shareModalBody.classList.add("modal-body");
let shareModalText = document.createElement('p');
shareModalText.textContent = "The link has been copied to your clipboard. Use it to share your conversation with others!";
let shareModalLink = document.createElement('input');
shareModalLink.setAttribute("value", shareURL);
shareModalLink.setAttribute("readonly", "");
shareModalLink.classList.add("share-link");
let copyButton = document.createElement('button');
copyButton.textContent = "Copy";
copyButton.addEventListener('click', function() {
shareModalLink.select();
document.execCommand('copy');
});
copyButton.id = "copy-share-url-button";
shareModalBody.appendChild(shareModalText);
shareModalBody.appendChild(shareModalLink);
shareModalBody.appendChild(copyButton);
shareModalContent.appendChild(shareModalBody);
shareModal.appendChild(shareModalContent);
document.body.appendChild(shareModal);
shareModalLink.select();
document.execCommand('copy');
})
.catch(err => {
return;
});
});
conversationMenu.appendChild(shareButton);
let deleteButton = document.createElement('button'); let deleteButton = document.createElement('button');
deleteButton.type = "button";
deleteButton.innerHTML = "Delete"; deleteButton.innerHTML = "Delete";
deleteButton.classList.add("delete-conversation-button"); deleteButton.classList.add("delete-conversation-button");
deleteButton.classList.add("three-dot-menu-button-item"); deleteButton.classList.add("three-dot-menu-button-item");
deleteButton.addEventListener('click', function() { deleteButton.addEventListener('click', function(event) {
event.preventDefault();
// Ask for confirmation before deleting chat session // Ask for confirmation before deleting chat session
let confirmation = confirm('Are you sure you want to delete this chat session?'); let confirmation = confirm('Are you sure you want to delete this chat session?');
if (!confirmation) return; if (!confirmation) return;
@@ -1864,9 +1998,10 @@ To get started, just start typing below. You can also type / to see a list of co
content: "▶"; content: "▶";
margin-right: 5px; margin-right: 5px;
display: inline-block; display: inline-block;
transition: transform 0.3s ease-in-out; transition: transform 0.1s ease-in-out;
} }
button.reference-button.expanded::before,
button.reference-button:active:before, button.reference-button:active:before,
button.reference-button[aria-expanded="true"]::before { button.reference-button[aria-expanded="true"]::before {
transform: rotate(90deg); transform: rotate(90deg);
@@ -2175,7 +2310,7 @@ To get started, just start typing below. You can also type / to see a list of co
} }
.side-panel-button { .side-panel-button {
background: var(--background-color); background: none;
border: none; border: none;
box-shadow: none; box-shadow: none;
font-size: 14px; font-size: 14px;
@@ -2274,6 +2409,32 @@ To get started, just start typing below. You can also type / to see a list of co
float: right; float: right;
} }
button.thumbs-up-button {
border-radius: 4px;
background-color: var(--background-color);
border: 1px solid var(--main-text-color);
text-align: center;
font-size: 16px;
transition: all 0.5s;
cursor: pointer;
padding: 4px;
float: right;
margin-right: 4px;
}
button.thumbs-down-button {
border-radius: 4px;
background-color: var(--background-color);
border: 1px solid var(--main-text-color);
text-align: center;
font-size: 16px;
transition: all 0.5s;
cursor: pointer;
padding: 4px;
float: right;
margin-right:4px;
}
button.copy-button span { button.copy-button span {
cursor: pointer; cursor: pointer;
display: inline-block; display: inline-block;
@@ -2282,8 +2443,18 @@ To get started, just start typing below. You can also type / to see a list of co
} }
img.copy-icon { img.copy-icon {
width: 16px; width: 18px;
height: 16px; height: 18px;
}
img.thumbs-up-icon {
width: 18px;
height: 18px;
}
img.thumbs-down-icon {
width: 18px;
height: 18px;
} }
button.copy-button:hover { button.copy-button:hover {
@@ -2291,6 +2462,16 @@ To get started, just start typing below. You can also type / to see a list of co
color: #f5f5f5; color: #f5f5f5;
} }
button.thumbs-up-button:hover {
background-color: var(--primary-hover);
color: #f5f5f5;
}
button.thumbs-down-button:hover {
background-color: var(--primary-hover);
color: #f5f5f5;
}
pre { pre {
text-wrap: unset; text-wrap: unset;
} }
@@ -2394,6 +2575,7 @@ To get started, just start typing below. You can also type / to see a list of co
margin-bottom: 8px; margin-bottom: 8px;
} }
button#copy-share-url-button,
button#new-conversation-button { button#new-conversation-button {
display: inline-flex; display: inline-flex;
align-items: center; align-items: center;
@@ -2414,14 +2596,12 @@ To get started, just start typing below. You can also type / to see a list of co
text-align: left; text-align: left;
display: flex; display: flex;
position: relative; position: relative;
margin-right: 8px;
} }
.three-dot-menu { .three-dot-menu {
display: block; display: block;
/* background: var(--background-color); */
/* border: 1px solid var(--main-text-color); */
border-radius: 5px; border-radius: 5px;
/* position: relative; */
position: absolute; position: absolute;
right: 4px; right: 4px;
top: 4px; top: 4px;
@@ -2603,13 +2783,6 @@ To get started, just start typing below. You can also type / to see a list of co
color: #333; color: #333;
} }
#agent-instructions {
font-size: 14px;
color: #666;
height: 50px;
overflow: auto;
}
#agent-owned-by-user { #agent-owned-by-user {
font-size: 12px; font-size: 12px;
color: #007BFF; color: #007BFF;
@@ -2631,7 +2804,7 @@ To get started, just start typing below. You can also type / to see a list of co
margin: 15% auto; /* 15% from the top and centered */ margin: 15% auto; /* 15% from the top and centered */
padding: 20px; padding: 20px;
border: 1px solid #888; border: 1px solid #888;
width: 250px; width: 300px;
text-align: left; text-align: left;
background: var(--background-color); background: var(--background-color);
border-radius: 5px; border-radius: 5px;
@@ -2705,6 +2878,28 @@ To get started, just start typing below. You can also type / to see a list of co
border: 1px solid var(--main-text-color); border: 1px solid var(--main-text-color);
} }
.share-link {
display: block;
width: 100%;
padding: 10px;
margin-top: 10px;
border: 1px solid #ccc;
border-radius: 4px;
background-color: #f9f9f9;
font-family: 'Courier New', monospace;
color: #333;
font-size: 16px;
box-sizing: border-box;
transition: all 0.3s ease;
}
.share-link:focus {
outline: none;
border-color: #007BFF;
box-shadow: 0 0 0 0.2rem rgba(0,123,255,.25);
}
button#copy-share-url-button,
button#new-conversation-submit-button { button#new-conversation-submit-button {
background: var(--summer-sun); background: var(--summer-sun);
transition: background 0.2s ease-in-out; transition: background 0.2s ease-in-out;
@@ -2715,6 +2910,7 @@ To get started, just start typing below. You can also type / to see a list of co
transition: background 0.2s ease-in-out; transition: background 0.2s ease-in-out;
} }
button#copy-share-url-button:hover,
button#new-conversation-submit-button:hover { button#new-conversation-submit-button:hover {
background: var(--primary); background: var(--primary);
} }

82
src/khoj/interface/web/config.html
View File

@@ -635,85 +635,6 @@
// List user's API keys on page load // List user's API keys on page load
listApiKeys(); listApiKeys();
function deleteAutomation(automationId) {
const AutomationList = document.getElementById("automations-list");
fetch(`/api/automation?automation_id=${automationId}`, {
method: 'DELETE',
})
.then(response => {
if (response.status == 200) {
const AutomationItem = document.getElementById(`automation-item-${automationId}`);
AutomationList.removeChild(AutomationItem);
}
});
}
function generateAutomationRow(automationObj) {
let automationId = automationObj.id;
let automationNextRun = `Next run at ${automationObj.next}`;
return `
<tr id="automation-item-${automationId}">
<td><b>${automationObj.subject}</b></td>
<td><b>${automationObj.scheduling_request}</b></td>
<td id="automation-query-to-run-${automationId}"><b>${automationObj.query_to_run}</b></td>
<td id="automation-${automationId}" title="${automationNextRun}">${automationObj.schedule}</td>
<td>
<img onclick="deleteAutomation('${automationId}')" class="automation-row-icon api-key-action enabled" src="/static/assets/icons/delete.svg" alt="Delete Automation" title="Delete Automation">
<img onclick="editAutomation('${automationId}')" class="automation-row-icon api-key-action enabled" src="/static/assets/icons/edit.svg" alt="Edit Automation" title="Edit Automation">
</td>
</tr>
`;
}
function listAutomations() {
const AutomationsList = document.getElementById("automations-list");
fetch('/api/automations')
.then(response => response.json())
.then(automations => {
if (!automations?.length > 0) return;
AutomationsList.innerHTML = automations.map(generateAutomationRow).join("");
});
}
async function createAutomation() {
const scheduling_request = window.prompt("Describe the automation you want to create");
if (!scheduling_request) return;
const ip_response = await fetch("https://ipapi.co/json");
const ip_data = await ip_response.json();
const query_string = `q=${scheduling_request}&city=${ip_data.city}&region=${ip_data.region}&country=${ip_data.country_name}&timezone=${ip_data.timezone}`;
const automation_response = await fetch(`/api/automation?${query_string}`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
});
if (!automation_response.ok) {
throw new Error(`Failed to create automation: ${automation_response.status}`);
}
listAutomations();
}
document.getElementById("create-automation").addEventListener("click", async () => { await createAutomation(); });
function editAutomation(automationId) {
const query_to_run = window.prompt("What is the query you want to run on this automation's schedule?");
if (!query_to_run) return;
fetch(`/api/automation?automation_id=${automationId}&query_to_run=${query_to_run}`, {
method: 'PATCH',
headers: {
'Content-Type': 'application/json',
},
}).then(response => {
if (response.ok) {
const automationQueryToRunColumn = document.getElementById(`automation-query-to-run-${automationId}`);
automationQueryToRunColumn.innerHTML = `<b>${query_to_run}</b>`;
}
});
}
function getIndexedDataSize() { function getIndexedDataSize() {
document.getElementById("indexed-data-size").innerHTML = "Calculating..."; document.getElementById("indexed-data-size").innerHTML = "Calculating...";
fetch('/api/config/index/size') fetch('/api/config/index/size')
@@ -723,9 +644,6 @@
}); });
} }
// List user's automations on page load
listAutomations();
function removeFile(path) { function removeFile(path) {
fetch('/api/config/data/file?filename=' + path, { fetch('/api/config/data/file?filename=' + path, {
method: 'DELETE', method: 'DELETE',

672
src/khoj/interface/web/config_automation.html
View File

@@ -6,17 +6,29 @@
<img class="card-icon" src="/static/assets/icons/automation.svg?v={{ khoj_version }}" alt="Automate"> <img class="card-icon" src="/static/assets/icons/automation.svg?v={{ khoj_version }}" alt="Automate">
<span class="card-title-text">Automate (Preview)</span> <span class="card-title-text">Automate (Preview)</span>
<div class="instructions"> <div class="instructions">
You can automate queries to run on a schedule using Khoj's automations for smart reminders. Results will be sent straight to your inbox. This is an experimental feature, so your results may vary. Report any issues to <a class="inline-link-light" href="mailto:team@khoj.dev">team@khoj.dev</a>. Automations allow you to schedule smart reminders using Khoj. This is an experimental feature, so your results may vary! Send any feedback to <a class="inline-link-light" href="mailto:team@khoj.dev">team@khoj.dev</a>.
</div>
<div class="instructions notice">
Sending automation results to <a class="inline-link-light" href="mailto:{{ username}}">{{ username }}</a>.
</div> </div>
</h2> </h2>
<div class="section-body"> <div class="section-body">
<button id="create-automation-button" type="button" class="positive-button"> <button id="create-automation-button" type="button" class="positive-button">
<img class="automation-action-icon" src="/static/assets/icons/new.svg" alt="Automations"> <img class="automation-action-icon" src="/static/assets/icons/new.svg" alt="Automations">
<span id="create-automation-button-text">Build</span> <span id="create-automation-button-text">Build Your Own</span>
</button> </button>
<div id="automations" class="section-cards"></div> <div id="automations" class="section-cards"></div>
<div id="suggested-automations">
<h2 class="section-title">
<span class="card-title-text">Suggested Automations</span>
</h2>
<div id="suggested-automations-list" class="section-cards"></div>
</div>
</div> </div>
</div> </div>
<div id="footer">
<a href="/">Back to Chat</a>
</div>
</div> </div>
<script src="/static/assets/natural-cron.min.js"></script> <script src="/static/assets/natural-cron.min.js"></script>
<style> <style>
@@ -27,21 +39,68 @@
width: 100%; width: 100%;
height: 100%; height: 100%;
grid-template-rows: none; grid-template-rows: none;
background-color: var(--frosted-background-color); background-color: white;
padding: 12px; border-radius: 20px;
padding: 20px;
box-shadow: rgba(3, 3, 3, 0.08) 0px 1px 12px;
transition: box-shadow 0.3s ease, transform 0.3s ease;
} }
div#footer {
width: auto;
padding: 10px;
background-color: var(--background-color);
border-top: 1px solid var(--main-text-color);
text-align: left;
margin-top: 12px;
margin-bottom: 12px;
}
div#footer a {
font-size: 18px;
font-weight: bold;
color: var(--primary-color);
}
img.automation-edit-cancel-icon,
img.automation-edit-icon {
width: 24px;
height: 24px;
object-fit: cover;
cursor: pointer;
margin: auto;
}
textarea.fake-input,
input.fake-input {
height: auto;
padding-top: 0px;
padding-bottom: 0px;
padding-left: 0px;
padding-right: 0px;
border: none;
}
#create-automation-button { #create-automation-button {
width: auto; width: auto;
} }
div.notice {
border-top: 1px solid black;
padding-top: 8px;
}
div#suggested-automations-list,
div#automations { div#automations {
margin-bottom: 12px; margin-bottom: 12px;
grid-template-columns: 1fr; grid-template-columns: 1fr 1fr;
} }
button.negative-button { button.negative-button {
background-color: gainsboro; background-color: gainsboro;
} }
.positive-button { .positive-button {
background-color: var(--primary-hover); background-color: var(--primary-hover)
} }
.positive-button:hover { .positive-button:hover {
background-color: var(--summer-sun); background-color: var(--summer-sun);
@@ -50,26 +109,44 @@
div.automation-buttons { div.automation-buttons {
display: grid; display: grid;
grid-gap: 8px; grid-gap: 8px;
grid-template-columns: 1fr 3fr; grid-template-columns: 1fr 1fr 1fr;
} }
button.save-automation-button { button.save-automation-button {
background-color: var(--summer-sun); background-color: var(--summer-sun);
} }
button.save-automation-button,
button.cancel-edit-automation-button,
button.send-preview-automation-button,
button.delete-automation-button {
padding: 8px;
margin-bottom: 0;
}
button.send-preview-automation-button {
border-color: var(--summer-sun);
}
button.save-automation-button:hover { button.save-automation-button:hover {
background-color: var(--primary-hover); background-color: var(--primary-hover);
} }
div.new-automation { div.new-automation {
background-color: var(--frosted-background-color); border-radius: 20px;
border-radius: 10px;
box-shadow: 0 4px 6px 0 hsla(0, 0%, 0%, 0.2); box-shadow: 0 4px 6px 0 hsla(0, 0%, 0%, 0.2);
margin-bottom: 20px; margin-bottom: 20px;
transition: box-shadow 0.3s ease, transform 0.3s ease; transition: box-shadow 0.3s ease, transform 0.3s ease;
position: absolute;
top: 50%;
left: 50%;
width: auto;
transform: translate(-50%, -50%);
z-index: 2;
height: auto;
} }
div.new-automation:hover { div.automation:not(.new-automation):hover {
box-shadow: 0 10px 15px 0 hsla(0, 0%, 0%, 0.1); box-shadow: 0 10px 15px 0 hsla(0, 0%, 0%, 0.1);
transform: translateY(-5px); transform: translateY(-5px);
} }
@@ -80,13 +157,42 @@
div.card-header { div.card-header {
display: grid; display: grid;
grid-template-columns: 1fr auto; grid-template-rows: auto 1fr;
grid-gap: 8px; grid-gap: 8px;
align-items: baseline; align-items: baseline;
padding: 8px; }
background-color: var(--frosted-background-color); input.schedule {
font-size: medium;
height: auto;
font-weight: lighter !important;
} }
h2.section-title {
font-size: larger;
}
div.card-header input {
font-weight: bold;
margin-bottom: 0 !important;
}
div.automation textarea {
width: 100%;
box-sizing: border-box;
border-radius: 4px;
resize: none;
}
img.promo-image {
width: 100%;
height: 100px;
object-fit: cover;
border-radius: 4px;
}
div.card-header textarea,
div.card-header input,
div.card-header:hover { div.card-header:hover {
cursor: pointer; cursor: pointer;
} }
@@ -96,6 +202,16 @@
height: 24px; height: 24px;
} }
div.subject-wrapper {
display: grid;
grid-template-columns: 1fr auto;
grid-gap: 8px;
}
div.subject-wrapper p {
margin: 0;
}
@keyframes confirmation { @keyframes confirmation {
0% { background-color: normal; transform: scale(1); } 0% { background-color: normal; transform: scale(1); }
50% { background-color: var(--primary); transform: scale(1.1); } 50% { background-color: var(--primary); transform: scale(1.1); }
@@ -106,6 +222,20 @@
animation: confirmation 1s; animation: confirmation 1s;
} }
@media screen and (max-width: 600px) {
div#automations,
div#suggested-automations-list {
grid-template-columns: 1fr;
}
div.automation-buttons {
grid-template-columns: 1fr;
}
div.new-automation {
width: 100%;
height: auto;
}
}
</style> </style>
<script> <script>
function deleteAutomation(automationId) { function deleteAutomation(automationId) {
@@ -140,63 +270,192 @@
queryEl.value = automation.query_to_run; queryEl.value = automation.query_to_run;
} }
function onClickAutomationCard(automationId) { function onClickEditAutomationCard(automationId) {
const automationIDElements = document.querySelectorAll(`.${automationId}`); const automationIDElements = document.querySelectorAll(`.${automationId}`);
automationIDElements.forEach(el => { automationIDElements.forEach(el => {
el.classList.toggle("hide-details"); if (el.classList.contains("automation-edit-icon")) {
el.classList.remove("automation-edit-icon");
el.classList.add("automation-edit-cancel-icon");
el.src = "/static/assets/icons/cancel.svg";
el.onclick = function(event) { clickCancelEdit(event, automationId); };
}
if (el.classList.contains("hide-details")) {
el.classList.add("hide-details-placeholder");
el.classList.remove("hide-details");
}
if (el.classList.contains("fake-input")) {
el.classList.add("fake-input-placeholder");
el.classList.remove("fake-input");
}
}); });
} }
function generateAutomationRow(automation) { function sendAPreviewAutomation(automationId) {
const notificationEl = document.getElementById(`automation-success-${automationId}`);
fetch(`/api/trigger/automation?automation_id=${automationId}`, { method: 'POST' })
.then(response =>
{
if (!response.ok) {
throw new Error('Network response was not ok');
}
return response;
})
.then(automations => {
notificationEl.style.display = 'block';
notificationEl.textContent = "Automation triggered. Check your inbox in a few minutes!";
})
.catch(error => {
notificationEl.style.display = 'block';
notificationEl.textContent = "Sorry, something went wrong. Try again later."
})
}
function clickCancelEdit(event, automationId) {
event.preventDefault();
event.stopPropagation();
const automationIDElements = document.querySelectorAll(`.${automationId}`);
automationIDElements.forEach(el => {
if (el.classList.contains("automation-edit-cancel-icon")) {
el.classList.remove("automation-edit-cancel-icon");
el.classList.add("automation-edit-icon");
el.src = "/static/assets/icons/pencil-edit.svg";
el.onclick = function() { onClickEditAutomationCard(automationId); };
}
if (el.classList.contains("hide-details-placeholder")) {
el.classList.remove("hide-details-placeholder");
el.classList.add("hide-details");
}
if (el.classList.contains("fake-input-placeholder")) {
el.classList.remove("fake-input-placeholder");
el.classList.add("fake-input");
}
})
}
function generateAutomationRow(automation, isSuggested=false) {
let automationId = automation.id; let automationId = automation.id;
let automationNextRun = `Next run at ${automation.next}\nCron: ${automation.crontime}`; let automationNextRun = `Next run at ${automation.next}\nCron: ${automation.crontime}`;
let automationEl = document.createElement("div"); let automationEl = document.createElement("div");
automationEl.innerHTML = ` automationEl.innerHTML = `
<div class="card automation" id="automation-card-${automationId}"> <div class="card automation" id="automation-card-${automationId}">
<div class="card-header" onclick="onClickAutomationCard('${automationId}')"> <div class="card-header" onclick="onClickEditAutomationCard('${automationId}')">
<input type="text" <div class="subject-wrapper">
id="automation-subject-${automationId}" <input type="text"
name="subject" id="automation-subject-${automationId}"
data-original="${automation.subject}" class="${automationId} fake-input"
value="${automation.subject}"> name="subject"
<div class="toggle-icon"> data-original="${automation.subject}"
<img src="/static/assets/icons/collapse.svg" alt="Toggle" class="toggle-icon"> value="${automation.subject}">
<img class="automation-edit-icon ${automationId}" src="/static/assets/icons/pencil-edit.svg" onclick="onClickEditAutomationCard('${automationId}')" alt="Automations">
</div> </div>
</div> <input type="text"
<label for="query-to-run" class="hide-details ${automationId}">Your automation</label>
<textarea id="automation-queryToRun-${automationId}"
class="hide-details ${automationId}"
data-original="${automation.query_to_run}"
name="query-to-run">${automation.query_to_run}</textarea>
<label for="schedule" class="hide-details">Schedule</label>
<input type="text"
class="hide-details ${automationId}"
id="automation-schedule-${automationId}" id="automation-schedule-${automationId}"
name="schedule" name="schedule"
class="schedule ${automationId} fake-input"
data-cron="${automation.crontime}" data-cron="${automation.crontime}"
data-original="${automation.schedule}" data-original="${automation.schedule}"
title="${automationNextRun}" title="${automationNextRun}"
value="${automation.schedule}"> value="${automation.schedule}">
<div class="hide-details automation-buttons ${automationId}"> <textarea id="automation-queryToRun-${automationId}"
<button type="button" class="automation-instructions ${automationId} fake-input"
class="delete-automation-button negative-button" data-original="${automation.query_to_run}"
id="delete-automation-button-${automationId}">Delete</button> name="query-to-run">${automation.query_to_run}</textarea>
<button type="button" ${isSuggested ?
class="save-automation-button positive-button" `<img class=promo-image src="${automation.promoImage}" alt="Promo Image">`:
id="save-automation-button-${automationId}">Save</button> ""
}
</div>
<div id="automation-buttons-wrapper">
<div class="automation-buttons">
${isSuggested ?
`<div id="empty-div"></div>
<div id="empty-div"></div>`:
`
<button type="button"
class="delete-automation-button negative-button"
id="delete-automation-button-${automationId}">Delete</button>
<button type="button"
class="send-preview-automation-button positive-button"
title="Immediately get a preview of this automation"
onclick="sendAPreviewAutomation('${automationId}')">Preview</button>
`
}
<button type="button"
class="save-automation-button positive-button"
id="save-automation-button-${automationId}">
${isSuggested ? "Add" : "Save"}
</button>
</div>
</div> </div>
<div id="automation-success-${automationId}" style="display: none;"></div> <div id="automation-success-${automationId}" style="display: none;"></div>
</div> </div>
`; `;
let automationButtonsSection = automationEl.querySelector(".automation-buttons");
if (!isSuggested) {
automationButtonsSection.classList.add("hide-details");
automationButtonsSection.classList.add(automationId);
}
let saveAutomationButtonEl = automationEl.querySelector(`#save-automation-button-${automation.id}`); let saveAutomationButtonEl = automationEl.querySelector(`#save-automation-button-${automation.id}`);
saveAutomationButtonEl.addEventListener("click", async () => { await saveAutomation(automation.id); }); saveAutomationButtonEl.addEventListener("click", async () => { await saveAutomation(automation.id, isSuggested); });
let deleteAutomationButtonEl = automationEl.querySelector(`#delete-automation-button-${automation.id}`); let deleteAutomationButtonEl = automationEl.querySelector(`#delete-automation-button-${automation.id}`);
deleteAutomationButtonEl.addEventListener("click", () => { deleteAutomation(automation.id); }); if (deleteAutomationButtonEl) {
deleteAutomationButtonEl.addEventListener("click", () => { deleteAutomation(automation.id); });
}
let cancelEditAutomationButtonEl = automationEl.querySelector(`#cancel-edit-automation-button-${automation.id}`);
if (cancelEditAutomationButtonEl) {
cancelEditAutomationButtonEl.addEventListener("click", (event) => { clickCancelEdit(event, automation.id); });
}
return automationEl.firstElementChild; return automationEl.firstElementChild;
} }
let timestamp = Math.floor(Date.now() / 1000);
let suggestedAutomationsMetadata = [
{
"subject": "Weekly Newsletter",
"query_to_run": "Compile a message including: 1. A recap of news from last week 2. A reminder to work out and stay hydrated 3. A quote to inspire me for the week ahead",
"schedule": "9AM every Monday",
"next": "Next run at 9AM on Monday",
"crontime": "0 9 * * 1",
"id": "suggested-automation" + timestamp,
"promoImage": "https://assets.khoj.dev/abstract_rectangles.webp",
},
{
"subject": "Daily Weather Update",
"query_to_run": "Get the weather forecast for today",
"schedule": "9AM every morning",
"next": "Next run at 9AM today",
"crontime": "0 9 * * *",
"id": "suggested-automation" + (timestamp + 1),
"promoImage": "https://assets.khoj.dev/blue_waves.webp",
},
{
"subject": "Front Page of Hacker News",
"query_to_run": "Summarize the top 5 posts from https://news.ycombinator.com/best and share them with me, including links",
"schedule": "9PM on every Wednesday",
"next": "Next run at 9PM on Wednesday",
"crontime": "0 21 * * 3",
"id": "suggested-automation" + (timestamp + 2),
"promoImage": "https://assets.khoj.dev/purple_triangles.webp",
},
{
"subject": "Market Summary",
"query_to_run": "Get the market summary for today and share it with me. Focus on tech stocks and the S&P 500.",
"schedule": "9AM on every weekday",
"next": "Next run at 9AM on Monday",
"crontime": "0 9 * * 1-5",
"id": "suggested-automation" + (timestamp + 3),
"promoImage": "https://assets.khoj.dev/blue_gears.webp",
}
];
function listAutomations() { function listAutomations() {
const AutomationsList = document.getElementById("automations"); const AutomationsList = document.getElementById("automations");
fetch('/api/automations') fetch('/api/automations')
@@ -204,11 +463,37 @@
.then(automations => { .then(automations => {
if (!automations?.length > 0) return; if (!automations?.length > 0) return;
AutomationsList.innerHTML = ''; // Clear existing content AutomationsList.innerHTML = ''; // Clear existing content
AutomationsList.append(...automations.map(automation => generateAutomationRow(automation))) AutomationsList.append(...automations.map(automation => generateAutomationRow(automation)));
// Check if any of the automations 'query-to-run' fields match the suggested automations
automations.forEach(automation => {
suggestedAutomationsMetadata.forEach(suggestedAutomation => {
if (automation.query_to_run === suggestedAutomation.query_to_run) {
let suggestedAutomationEl = document.getElementById(`automation-card-${suggestedAutomation.id}`);
suggestedAutomationEl.remove();
}
});
});
}); });
} }
listAutomations(); listAutomations();
if (suggestedAutomationsMetadata.length > 0) {
suggestedAutomationsMetadata.forEach(automation => {
automation.id = "suggested-automation" + timestamp;
timestamp++;
});
}
function listSuggestedAutomations() {
const SuggestedAutomationsList = document.getElementById("suggested-automations-list");
SuggestedAutomationsList.innerHTML = ''; // Clear existing content
SuggestedAutomationsList.append(...suggestedAutomationsMetadata.map(automation => generateAutomationRow(automation, true)));
}
listSuggestedAutomations();
function enableSaveOnlyWhenInputsChanged() { function enableSaveOnlyWhenInputsChanged() {
const inputs = document.querySelectorAll('input[name="schedule"], textarea[name="query-to-run"], input[name="subject"]'); const inputs = document.querySelectorAll('input[name="schedule"], textarea[name="query-to-run"], input[name="subject"]');
inputs.forEach(input => { inputs.forEach(input => {
@@ -231,35 +516,250 @@
}); });
} }
function createScheduleSelector(automationId) {
var scheduleContainer = document.createElement('div');
scheduleContainer.id = `schedule-container-${automationId}`;
var frequencyLabel = document.createElement('label');
frequencyLabel.for = `frequency-selector-${automationId}`;
frequencyLabel.textContent = 'Every';
var frequencySelector = document.createElement('select')
frequencySelector.id = `frequency-selector-${automationId}`;
var dayLabel = document.createElement('label');
dayLabel.id = `day-selector-label-${automationId}`;
dayLabel.for = `day-selector-${automationId}`;
dayLabel.textContent = 'on';
var daySelector = document.createElement('select');
daySelector.id = `day-selector-${automationId}`;
var dateLabel = document.createElement('label');
dateLabel.id = `date-label-${automationId}`;
dateLabel.for = `date-selector-${automationId}`;
dateLabel.textContent = 'on the';
var dateSelector = document.createElement('select');
dateSelector.id = `date-selector-${automationId}`;
var timeLabel = document.createElement('label');
timeLabel.for = `time-selector-${automationId}`;
timeLabel.textContent = 'at';
var timeSelector = document.createElement('select');
timeSelector.id = `time-selector-${automationId}`;
// Populate frequency selector with options for day, week, and month
var frequencies = ['day', 'week', 'month'];
for (var i = 0; i < frequencies.length; i++) {
var option = document.createElement('option');
option.value = frequencies[i];
option.text = frequencies[i];
frequencySelector.appendChild(option);
}
// Event listener for frequency selector change
frequencySelector.addEventListener('change', function() {
switch (this.value) {
case 'day':
daySelector.style.display = 'none';
dateSelector.style.display = 'none';
break;
case 'week':
daySelector.style.display = 'block';
dateSelector.style.display = 'none';
break;
case 'month':
daySelector.style.display = 'none';
dateSelector.style.display = 'block';
break;
}
});
// Populate the date selector with options for each day of the month
for (var i = 1; i <= 31; i++) {
var option = document.createElement('option');
option.value = i;
option.text = i;
dateSelector.appendChild(option);
}
// Populate the day selector with options for each day of the week
var days = ['Sunday', 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday'];
for (var i = 0; i < days.length; i++) {
var option = document.createElement('option');
option.value = i;
option.text = days[i];
daySelector.appendChild(option);
}
var timePeriods = ['AM', 'PM'];
// Populate the time selector with options for each hour of the day
for (var i = 0; i < timePeriods.length; i++) {
for (var hour = 0; hour < 12; hour++) {
for (var minute = 0; minute < 60; minute+=15) {
// Ensure all minutes are two digits
var paddedMinute = String(minute).padStart(2, '0');
var option = document.createElement('option');
var friendlyHour = hour === 0 ? 12 : hour;
option.value = `${friendlyHour}:${paddedMinute} ${timePeriods[i]}`;
option.text = `${friendlyHour}:${paddedMinute} ${timePeriods[i]}`;
timeSelector.appendChild(option);
}
}
}
// Populate date selector with options 1 through 31
for (var i = 1; i <= 31; i++) {
var option = document.createElement('option');
option.value = i;
option.text = i;
dateSelector.appendChild(option);
}
var hoursMinutesSelectorContainer = document.createElement('div');
hoursMinutesSelectorContainer.classList.add('hours-minutes-selector-container');
hoursMinutesSelectorContainer.appendChild(timeLabel);
hoursMinutesSelectorContainer.appendChild(timeSelector);
scheduleContainer.appendChild(frequencyLabel);
scheduleContainer.appendChild(frequencySelector);
scheduleContainer.appendChild(dayLabel);
scheduleContainer.appendChild(daySelector);
scheduleContainer.appendChild(dateLabel);
scheduleContainer.appendChild(dateSelector);
scheduleContainer.appendChild(hoursMinutesSelectorContainer);
return scheduleContainer;
}
function setupScheduleViewListener(cronString, automationId) {
// Parse the cron string
var cronParts = cronString.split(' ');
var minutes = cronParts[0];
var hours = cronParts[1];
var dayOfMonth = cronParts[2];
var month = cronParts[3];
var dayOfWeek = cronParts[4];
var timeSelector = document.getElementById(`time-selector-${automationId}`);
// Set the initial value of the time selector based on the cron string. Convert 24-hour time to 12-hour time
if (hours === '*' && minutes === '*') {
var currentTime = new Date();
hours = currentTime.getHours();
minutes = currentTime.getMinutes();
}
var hours = parseInt(hours);
var minutes = parseInt(minutes);
var timePeriod = hours >= 12 ? 'PM' : 'AM';
hours = hours % 12;
hours = hours ? hours : 12; // 0 should be 12
minutes = Math.round(minutes / 15) * 15;
minutes = String(minutes).padStart(2, '0');
// Resolve minutes to the nearest 15 minute interval
timeSelector.value = `${hours}:${minutes} ${timePeriod}`;
const frequencySelector = document.getElementById(`frequency-selector-${automationId}`);
const daySelector = document.getElementById(`day-selector-${automationId}`);
const daySelectorLabel = document.getElementById(`day-selector-label-${automationId}`);
const dateSelector = document.getElementById(`date-selector-${automationId}`);
const dateLabel = document.getElementById(`date-label-${automationId}`);
// Event listener for frequency selector change
frequencySelector.addEventListener('change', function() {
processFrequencySelector(frequencySelector, daySelector, daySelectorLabel, dateSelector, dateLabel);
});
// Set the initial value based on the frequency selector value
processFrequencySelector(frequencySelector, daySelector, daySelectorLabel, dateSelector, dateLabel);
}
function processFrequencySelector(frequencySelector, daySelector, daySelectorLabel, dateSelector, dateLabel) {
switch (frequencySelector.value) {
case 'day':
daySelector.style.display = 'none';
dateSelector.style.display = 'none';
daySelectorLabel.style.display = 'none';
dateLabel.style.display = 'none';
break;
case 'week':
daySelector.style.display = 'block';
dateSelector.style.display = 'none';
daySelectorLabel.style.display = 'block';
dateLabel.style.display = 'none';
break;
case 'month':
daySelector.style.display = 'none';
dateSelector.style.display = 'block';
daySelectorLabel.style.display = 'none';
dateLabel.style.display = 'block';
break;
}
}
function convertFrequencyToCron(automationId) {
var frequencySelector = document.getElementById(`frequency-selector-${automationId}`);
var daySelector = document.getElementById(`day-selector-${automationId}`);
var dateSelector = document.getElementById(`date-selector-${automationId}`);
var timeSelector = document.getElementById(`time-selector-${automationId}`);
var hours = timeSelector.value.split(':')[0];
var minutes = timeSelector.value.split(':')[1].split(' ')[0];
var cronString = '';
switch (frequencySelector.value) {
case 'day':
cronString = `${minutes} ${hours} * * *`;
break;
case 'week':
cronString = `${minutes} ${hours} * * ${daySelector.value}`;
break;
case 'month':
cronString = `${minutes} ${hours} ${dateSelector.value} * *`;
break;
}
return cronString;
}
async function saveAutomation(automationId, create=false) { async function saveAutomation(automationId, create=false) {
const scheduleEl = document.getElementById(`automation-schedule-${automationId}`); const scheduleEl = document.getElementById(`automation-schedule-${automationId}`);
const notificationEl = document.getElementById(`automation-success-${automationId}`); const notificationEl = document.getElementById(`automation-success-${automationId}`);
const saveButtonEl = document.getElementById(`save-automation-button-${automationId}`); const saveButtonEl = document.getElementById(`save-automation-button-${automationId}`);
const queryToRunEl = document.getElementById(`automation-queryToRun-${automationId}`); const queryToRunEl = document.getElementById(`automation-queryToRun-${automationId}`);
const queryToRun = encodeURIComponent(queryToRunEl.value); const queryToRun = encodeURIComponent(queryToRunEl.value);
const actOn = create ? "Create" : "Save"; const actOn = create ? "Creat" : "Sav";
var cronTime = null;
if (queryToRun == "" || scheduleEl.value == "") { if (queryToRun == "") {
notificationEl.textContent = `⚠️ Failed to automate. All input fields need to be filled.`; if (!create && scheduleEl.value == "") {
notificationEl.style.display = "block"; notificationEl.textContent = `⚠️ Failed to automate. All input fields need to be filled.`;
let originalQueryToRunElBorder = queryToRunEl.style.border; notificationEl.style.display = "block";
if (queryToRun === "") queryToRunEl.style.border = "2px solid red"; let originalQueryToRunElBorder = queryToRunEl.style.border;
let originalScheduleElBorder = scheduleEl.style.border; if (queryToRun === "") queryToRunEl.style.border = "2px solid red";
if (scheduleEl.value === "") scheduleEl.style.border = "2px solid red"; let originalScheduleElBorder = scheduleEl.style.border;
setTimeout(function() { if (scheduleEl.value === "") scheduleEl.style.border = "2px solid red";
if (queryToRun == "") queryToRunEl.style.border = originalQueryToRunElBorder; setTimeout(function() {
if (scheduleEl.value == "") scheduleEl.style.border = originalScheduleElBorder; if (queryToRun == "") queryToRunEl.style.border = originalQueryToRunElBorder;
}, 2000); if (scheduleEl.value == "") scheduleEl.style.border = originalScheduleElBorder;
}, 2000);
return;
}
return;
} }
// Get client location information from IP // Get client location information from IP
const ip_response = await fetch("https://ipapi.co/json") const ip_response = await fetch("https://ipapi.co/json");
const ip_data = await ip_response.json(); let ip_data = null;
if (ip_response.ok) {
ip_data = await ip_response.json();
}
// Get cron string from natural language user schedule, if changed // Get cron string from natural language user schedule, if changed
const crontime = scheduleEl.getAttribute('data-original') !== scheduleEl.value ? getCronString(scheduleEl.value) : scheduleEl.getAttribute('data-cron'); if (create && !scheduleEl) {
crontime = convertFrequencyToCron(automationId);
} else {
crontime = scheduleEl.getAttribute('data-original') !== scheduleEl.value ? getCronString(scheduleEl.value) : scheduleEl.getAttribute('data-cron');
}
if (crontime.startsWith("ERROR:")) { if (crontime.startsWith("ERROR:")) {
notificationEl.textContent = `⚠️ Failed to automate. Fix or simplify Schedule input field.`; notificationEl.textContent = `⚠️ Failed to automate. Fix or simplify Schedule input field.`;
notificationEl.style.display = "block"; notificationEl.style.display = "block";
@@ -274,7 +774,10 @@
const encodedCrontime = encodeURIComponent(crontime); const encodedCrontime = encodeURIComponent(crontime);
// Construct query string and select method for API call // Construct query string and select method for API call
let query_string = `q=${queryToRun}&crontime=${encodedCrontime}&city=${ip_data.city}&region=${ip_data.region}&country=${ip_data.country_name}&timezone=${ip_data.timezone}`; let query_string = `q=${queryToRun}&crontime=${encodedCrontime}`;
if (ip_data) {
query_string += `&city=${ip_data.city}&region=${ip_data.region}&country=${ip_data.country_name}&timezone=${ip_data.timezone}`;
}
let method = "POST"; let method = "POST";
if (!create) { if (!create) {
@@ -284,6 +787,11 @@
method = "PUT" method = "PUT"
} }
// Create a loading animation while waiting for the API response
// TODO add a more pleasant loading symbol here.
notificationEl.textContent = `⏳ ${actOn}ing automation...`;
notificationEl.style.display = "block";
fetch(`/api/automation?${query_string}`, { fetch(`/api/automation?${query_string}`, {
method: method, method: method,
headers: { headers: {
@@ -295,8 +803,16 @@
if (create) { if (create) {
const automationEl = document.getElementById(`automation-card-${automationId}`); const automationEl = document.getElementById(`automation-card-${automationId}`);
// Create a more interesting confirmation animation. // Create a more interesting confirmation animation.
automationEl.classList.add("confirmation") automationEl.classList.remove("new-automation");
setTimeout(function() { setTimeout(function() {
// Check if automationEl is a child of #automations or #suggested-automations-list
// If #suggested-automations-list, remove the element from the list and add it to #automations
let parentEl = automationEl.parentElement;
let isSuggested = parentEl.id === "suggested-automations-list";
if (isSuggested) {
parentEl.removeChild(automationEl);
document.getElementById("automations").prepend(automationEl);
}
automationEl.replaceWith(generateAutomationRow(automation)); automationEl.replaceWith(generateAutomationRow(automation));
}, 1000); }, 1000);
} else { } else {
@@ -304,17 +820,30 @@
} }
notificationEl.style.display = "none"; notificationEl.style.display = "none";
saveButtonEl.textContent = `✅ Automation ${actOn}d`; saveButtonEl.textContent = `✅ Automation ${actOn}ed`;
setTimeout(function() { setTimeout(function() {
const automationIDElements = document.querySelectorAll(`.${automationId}`);
automationIDElements.forEach(el => {
// If it has the class automation-buttons, turn on the hide-details class
if (el.classList.contains("automation-buttons"))
{
el.classList.add("hide-details");
}
// If it has the class automationId, turn on the fake-input class
else if (el.classList.contains(automationId))
{
el.classList.add("fake-input");
}
});
saveButtonEl.textContent = "Save"; saveButtonEl.textContent = "Save";
}, 2000); }, 2000);
}) })
.catch(error => { .catch(error => {
notificationEl.textContent = `⚠️ Failed to ${actOn.toLowerCase()} automations.`; notificationEl.textContent = `⚠️ Failed to ${actOn.toLowerCase()}e automation.`;
notificationEl.style.display = "block"; notificationEl.style.display = "block";
saveButtonEl.textContent = `⚠️ Failed to ${actOn.toLowerCase()} automations`; saveButtonEl.textContent = `⚠️ Failed to ${actOn.toLowerCase()}e automation`;
setTimeout(function() { setTimeout(function() {
saveButtonEl.textContent = actOn; saveButtonEl.textContent = `${actOn}e`;
}, 2000); }, 2000);
return; return;
}); });
@@ -329,14 +858,12 @@
automationEl.classList.add("new-automation") automationEl.classList.add("new-automation")
const placeholderId = Date.now(); const placeholderId = Date.now();
automationEl.id = "automation-card-" + placeholderId; automationEl.id = "automation-card-" + placeholderId;
var scheduleSelector = createScheduleSelector(placeholderId);
automationEl.innerHTML = ` automationEl.innerHTML = `
<label for="query-to-run">Your new automation</label> <label for="schedule">New Automation</label>
<textarea id="automation-queryToRun-${placeholderId}" placeholder="Share a Newsletter including: 1. Weather forecast for this Week. 2. A Book Highlight from my Notes. 3. Recap News from Last Week"></textarea> ${scheduleSelector.outerHTML}
<label for="schedule">Schedule</label> <label for="query-to-run">What would you like to receive in your automation?</label>
<input type="text" <textarea id="automation-queryToRun-${placeholderId}" placeholder="Provide me with a mindful moment, reminding me to be centered."></textarea>
id="automation-schedule-${placeholderId}"
name="schedule"
placeholder="9AM every morning">
<div class="automation-buttons"> <div class="automation-buttons">
<button type="button" <button type="button"
class="delete-automation-button negative-button" class="delete-automation-button negative-button"
@@ -350,6 +877,7 @@
<div id="automation-success-${placeholderId}" style="display: none;"></div> <div id="automation-success-${placeholderId}" style="display: none;"></div>
`; `;
document.getElementById("automations").insertBefore(automationEl, document.getElementById("automations").firstChild); document.getElementById("automations").insertBefore(automationEl, document.getElementById("automations").firstChild);
setupScheduleViewListener("* * * * *", placeholderId);
}) })
</script> </script>
{% endblock %} {% endblock %}

2
src/khoj/interface/web/login.html
View File

@@ -16,7 +16,7 @@
<!-- Login Modal --> <!-- Login Modal -->
<div id="login-modal"> <div id="login-modal">
<img class="khoj-logo" src="/static/assets/icons/favicon-128x128.png" alt="Khoj"></img> <img class="khoj-logo" src="/static/assets/icons/favicon-128x128.png" alt="Khoj"></img>
<div class="login-modal-title">Log in to Khoj</div> <div class="login-modal-title">Login to Khoj</div>
<!-- Sign Up/Login with Google OAuth --> <!-- Sign Up/Login with Google OAuth -->
<div <div
class="g_id_signin" class="g_id_signin"

71
src/khoj/interface/web/processor_conversation_input.html
View File

@@ -1,71 +0,0 @@
{% extends "base_config.html" %}
{% block content %}
<div class="page">
<div class="section">
<h2 class="section-title">
<img class="card-icon" src="/static/assets/icons/chat.svg" alt="Chat">
<span class="card-title-text">Chat</span>
</h2>
<form id="config-form">
<table>
<tr>
<td>
<label for="openai-api-key" title="Get your OpenAI key from https://platform.openai.com/account/api-keys">OpenAI API key</label>
</td>
<td>
<input type="text" id="openai-api-key" name="openai-api-key" value="{{ current_config['api_key'] }}">
</td>
</tr>
<tr>
<td>
<label for="chat-model">Chat Model</label>
</td>
<td>
<input type="text" id="chat-model" name="chat-model" value="{{ current_config['chat_model'] }}">
</td>
</tr>
</table>
<div class="section">
<div id="success" style="display: none;" ></div>
<button id="submit" type="submit">Save</button>
</div>
</form>
</div>
</div>
<script>
submit.addEventListener("click", function(event) {
event.preventDefault();
var openai_api_key = document.getElementById("openai-api-key").value;
var chat_model = document.getElementById("chat-model").value;
if (openai_api_key == "" || chat_model == "") {
document.getElementById("success").innerHTML = "⚠️ Please fill all the fields.";
document.getElementById("success").style.display = "block";
return;
}
const csrfToken = document.cookie.split('; ').find(row => row.startsWith('csrftoken'))?.split('=')[1];
fetch('/api/config/data/processor/conversation/openai', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-CSRFToken': csrfToken
},
body: JSON.stringify({
"api_key": openai_api_key,
"chat_model": chat_model
})
})
.then(response => response.json())
.then(data => {
if (data["status"] == "ok") {
document.getElementById("success").innerHTML = "✅ Successfully updated. Go to your <a href='/config'>settings page</a> to complete setup.";
document.getElementById("success").style.display = "block";
} else {
document.getElementById("success").innerHTML = "⚠️ Failed to update settings.";
document.getElementById("success").style.display = "block";
}
})
});
</script>
{% endblock %}

1916
src/khoj/interface/web/public_conversation.html Normal file
View File

File diff suppressed because it is too large Load Diff

1
src/khoj/interface/web/utils.html
View File

@@ -37,6 +37,7 @@
<div id="khoj-nav-menu" class="khoj-nav-dropdown-content"> <div id="khoj-nav-menu" class="khoj-nav-dropdown-content">
<div class="khoj-nav-username"> {{ username }} </div> <div class="khoj-nav-username"> {{ username }} </div>
<a id="settings-nav" class="khoj-nav" href="/config">Settings</a> <a id="settings-nav" class="khoj-nav" href="/config">Settings</a>
<a id="github-nav" class="khoj-nav" href="https://github.com/khoj-ai/khoj">GitHub</a>
<a id="help-nav" class="khoj-nav" href="https://docs.khoj.dev" target="_blank">Help</a> <a id="help-nav" class="khoj-nav" href="https://docs.khoj.dev" target="_blank">Help</a>
<a class="khoj-nav" href="/auth/logout">Logout</a> <a class="khoj-nav" href="/auth/logout">Logout</a>
</div> </div>

2
src/khoj/main.py
View File

@@ -96,7 +96,7 @@ from khoj.utils.initialization import initialization
def shutdown_scheduler(): def shutdown_scheduler():
logger.info("🌑 Shutting down Khoj") logger.info("🌑 Shutting down Khoj")
state.scheduler.shutdown() # state.scheduler.shutdown()
def run(should_start_server=True): def run(should_start_server=True):

0
src/khoj/processor/conversation/anthropic/__init__.py Normal file
View File

202
src/khoj/processor/conversation/anthropic/anthropic_chat.py Normal file
View File

@@ -0,0 +1,202 @@
import json
import logging
import re
from datetime import datetime, timedelta
from typing import Dict, Optional
from langchain.schema import ChatMessage
from khoj.database.models import Agent
from khoj.processor.conversation import prompts
from khoj.processor.conversation.anthropic.utils import (
anthropic_chat_completion_with_backoff,
anthropic_completion_with_backoff,
)
from khoj.processor.conversation.utils import generate_chatml_messages_with_context
from khoj.utils.helpers import ConversationCommand, is_none_or_empty
from khoj.utils.rawconfig import LocationData
logger = logging.getLogger(__name__)
def extract_questions_anthropic(
text,
model: Optional[str] = "claude-instant-1.2",
conversation_log={},
api_key=None,
temperature=0,
location_data: LocationData = None,
):
"""
Infer search queries to retrieve relevant notes to answer user query
"""
# Extract Past User Message and Inferred Questions from Conversation Log
location = f"{location_data.city}, {location_data.region}, {location_data.country}" if location_data else "Unknown"
# Extract Past User Message and Inferred Questions from Conversation Log
chat_history = "".join(
[
f'Q: {chat["intent"]["query"]}\nKhoj: {{"queries": {chat["intent"].get("inferred-queries") or list([chat["intent"]["query"]])}}}\nA: {chat["message"]}\n\n'
for chat in conversation_log.get("chat", [])[-4:]
if chat["by"] == "khoj" and "text-to-image" not in chat["intent"].get("type")
]
)
# Get dates relative to today for prompt creation
today = datetime.today()
current_new_year = today.replace(month=1, day=1)
last_new_year = current_new_year.replace(year=today.year - 1)
system_prompt = prompts.extract_questions_anthropic_system_prompt.format(
current_date=today.strftime("%Y-%m-%d"),
day_of_week=today.strftime("%A"),
last_new_year=last_new_year.strftime("%Y"),
last_new_year_date=last_new_year.strftime("%Y-%m-%d"),
current_new_year_date=current_new_year.strftime("%Y-%m-%d"),
yesterday_date=(today - timedelta(days=1)).strftime("%Y-%m-%d"),
location=location,
)
prompt = prompts.extract_questions_anthropic_user_message.format(
chat_history=chat_history,
text=text,
)
messages = [ChatMessage(content=prompt, role="user")]
response = anthropic_completion_with_backoff(
messages=messages,
system_prompt=system_prompt,
model_name=model,
temperature=temperature,
api_key=api_key,
)
# Extract, Clean Message from Claude's Response
try:
response = response.strip()
match = re.search(r"\{.*?\}", response)
if match:
response = match.group()
response = json.loads(response)
response = [q.strip() for q in response["queries"] if q.strip()]
if not isinstance(response, list) or not response:
logger.error(f"Invalid response for constructing subqueries: {response}")
return [text]
return response
except:
logger.warning(f"Claude returned invalid JSON. Falling back to using user message as search query.\n{response}")
questions = [text]
logger.debug(f"Extracted Questions by Claude: {questions}")
return questions
def anthropic_send_message_to_model(messages, api_key, model):
"""
Send message to model
"""
# Anthropic requires the first message to be a 'user' message, and the system prompt is not to be sent in the messages parameter
system_prompt = None
if len(messages) == 1:
messages[0].role = "user"
else:
system_prompt = ""
for message in messages.copy():
if message.role == "system":
system_prompt += message.content
messages.remove(message)
# Get Response from GPT. Don't use response_type because Anthropic doesn't support it.
return anthropic_completion_with_backoff(
messages=messages,
system_prompt=system_prompt,
model_name=model,
api_key=api_key,
)
def converse_anthropic(
references,
user_query,
online_results: Optional[Dict[str, Dict]] = None,
conversation_log={},
model: Optional[str] = "claude-instant-1.2",
api_key: Optional[str] = None,
completion_func=None,
conversation_commands=[ConversationCommand.Default],
max_prompt_size=None,
tokenizer_name=None,
location_data: LocationData = None,
user_name: str = None,
agent: Agent = None,
):
"""
Converse with user using Anthropic's Claude
"""
# Initialize Variables
current_date = datetime.now().strftime("%Y-%m-%d")
compiled_references = "\n\n".join({f"# {item}" for item in references})
conversation_primer = prompts.query_prompt.format(query=user_query)
if agent and agent.personality:
system_prompt = prompts.custom_personality.format(
name=agent.name, bio=agent.personality, current_date=current_date
)
else:
system_prompt = prompts.personality.format(current_date=current_date)
if location_data:
location = f"{location_data.city}, {location_data.region}, {location_data.country}"
location_prompt = prompts.user_location.format(location=location)
system_prompt = f"{system_prompt}\n{location_prompt}"
if user_name:
user_name_prompt = prompts.user_name.format(name=user_name)
system_prompt = f"{system_prompt}\n{user_name_prompt}"
# Get Conversation Primer appropriate to Conversation Type
if conversation_commands == [ConversationCommand.Notes] and is_none_or_empty(compiled_references):
completion_func(chat_response=prompts.no_notes_found.format())
return iter([prompts.no_notes_found.format()])
elif conversation_commands == [ConversationCommand.Online] and is_none_or_empty(online_results):
completion_func(chat_response=prompts.no_online_results_found.format())
return iter([prompts.no_online_results_found.format()])
if ConversationCommand.Online in conversation_commands or ConversationCommand.Webpage in conversation_commands:
conversation_primer = (
f"{prompts.online_search_conversation.format(online_results=str(online_results))}\n{conversation_primer}"
)
if not is_none_or_empty(compiled_references):
conversation_primer = f"{prompts.notes_conversation.format(query=user_query, references=compiled_references)}\n\n{conversation_primer}"
# Setup Prompt with Primer or Conversation History
messages = generate_chatml_messages_with_context(
conversation_primer,
conversation_log=conversation_log,
model_name=model,
max_prompt_size=max_prompt_size,
tokenizer_name=tokenizer_name,
)
for message in messages.copy():
if message.role == "system":
system_prompt += message.content
messages.remove(message)
truncated_messages = "\n".join({f"{message.content[:40]}..." for message in messages})
logger.debug(f"Conversation Context for Claude: {truncated_messages}")
# Get Response from Claude
return anthropic_chat_completion_with_backoff(
messages=messages,
compiled_references=references,
online_results=online_results,
model_name=model,
temperature=0,
api_key=api_key,
system_prompt=system_prompt,
completion_func=completion_func,
max_prompt_size=max_prompt_size,
)

116
src/khoj/processor/conversation/anthropic/utils.py Normal file
View File

@@ -0,0 +1,116 @@
import logging
from threading import Thread
from typing import Dict, List
import anthropic
from tenacity import (
before_sleep_log,
retry,
stop_after_attempt,
wait_exponential,
wait_random_exponential,
)
from khoj.processor.conversation.utils import ThreadedGenerator
logger = logging.getLogger(__name__)
anthropic_clients: Dict[str, anthropic.Anthropic] = {}
DEFAULT_MAX_TOKENS_ANTHROPIC = 3000
@retry(
wait=wait_random_exponential(min=1, max=10),
stop=stop_after_attempt(2),
before_sleep=before_sleep_log(logger, logging.DEBUG),
reraise=True,
)
def anthropic_completion_with_backoff(
messages, system_prompt, model_name, temperature=0, api_key=None, model_kwargs=None, max_tokens=None
) -> str:
if api_key not in anthropic_clients:
client: anthropic.Anthropic = anthropic.Anthropic(api_key=api_key)
anthropic_clients[api_key] = client
else:
client = anthropic_clients[api_key]
formatted_messages = [{"role": message.role, "content": message.content} for message in messages]
aggregated_response = ""
max_tokens = max_tokens or DEFAULT_MAX_TOKENS_ANTHROPIC
model_kwargs = model_kwargs or dict()
if system_prompt:
model_kwargs["system"] = system_prompt
with client.messages.stream(
messages=formatted_messages,
model=model_name, # type: ignore
temperature=temperature,
timeout=20,
max_tokens=max_tokens,
**(model_kwargs),
) as stream:
for text in stream.text_stream:
aggregated_response += text
return aggregated_response
@retry(
wait=wait_exponential(multiplier=1, min=4, max=10),
stop=stop_after_attempt(2),
before_sleep=before_sleep_log(logger, logging.DEBUG),
reraise=True,
)
def anthropic_chat_completion_with_backoff(
messages,
compiled_references,
online_results,
model_name,
temperature,
api_key,
system_prompt,
max_prompt_size=None,
completion_func=None,
model_kwargs=None,
):
g = ThreadedGenerator(compiled_references, online_results, completion_func=completion_func)
t = Thread(
target=anthropic_llm_thread,
args=(g, messages, system_prompt, model_name, temperature, api_key, max_prompt_size, model_kwargs),
)
t.start()
return g
def anthropic_llm_thread(
g, messages, system_prompt, model_name, temperature, api_key, max_prompt_size=None, model_kwargs=None
):
if api_key not in anthropic_clients:
client: anthropic.Anthropic = anthropic.Anthropic(api_key=api_key)
anthropic_clients[api_key] = client
else:
client: anthropic.Anthropic = anthropic_clients[api_key]
formatted_messages: List[anthropic.types.MessageParam] = [
anthropic.types.MessageParam(role=message.role, content=message.content) for message in messages
]
max_prompt_size = max_prompt_size or DEFAULT_MAX_TOKENS_ANTHROPIC
with client.messages.stream(
messages=formatted_messages,
model=model_name, # type: ignore
temperature=temperature,
system=system_prompt,
timeout=20,
max_tokens=max_prompt_size,
**(model_kwargs or dict()),
) as stream:
for text in stream.text_stream:
g.send(text)
g.close()

1
src/khoj/processor/conversation/openai/gpt.py
View File

@@ -67,7 +67,6 @@ def extract_questions(
messages=messages, messages=messages,
model=model, model=model,
temperature=temperature, temperature=temperature,
max_tokens=max_tokens,
api_base_url=api_base_url, api_base_url=api_base_url,
model_kwargs={"response_format": {"type": "json_object"}}, model_kwargs={"response_format": {"type": "json_object"}},
openai_api_key=api_key, openai_api_key=api_key,

3
src/khoj/processor/conversation/openai/utils.py
View File

@@ -34,7 +34,7 @@ openai_clients: Dict[str, openai.OpenAI] = {}
reraise=True, reraise=True,
) )
def completion_with_backoff( def completion_with_backoff(
messages, model, temperature=0, openai_api_key=None, api_base_url=None, model_kwargs=None, max_tokens=None messages, model, temperature=0, openai_api_key=None, api_base_url=None, model_kwargs=None
) -> str: ) -> str:
client_key = f"{openai_api_key}--{api_base_url}" client_key = f"{openai_api_key}--{api_base_url}"
client: openai.OpenAI = openai_clients.get(client_key) client: openai.OpenAI = openai_clients.get(client_key)
@@ -53,7 +53,6 @@ def completion_with_backoff(
model=model, # type: ignore model=model, # type: ignore
temperature=temperature, temperature=temperature,
timeout=20, timeout=20,
max_tokens=max_tokens,
**(model_kwargs or dict()), **(model_kwargs or dict()),
) )
aggregated_response = "" aggregated_response = ""

42
src/khoj/processor/conversation/prompts.py
View File

@@ -261,6 +261,45 @@ Khoj:
""".strip() """.strip()
) )
extract_questions_anthropic_system_prompt = PromptTemplate.from_template(
"""
You are Khoj, an extremely smart and helpful document search assistant with only the ability to retrieve information from the user's notes. Disregard online search requests. Construct search queries to retrieve relevant information to answer the user's question.
- You will be provided past questions(Q) and answers(A) for context.
- Add as much context from the previous questions and answers as required into your search queries.
- Break messages into multiple search queries when required to retrieve the relevant information.
- Add date filters to your search queries from questions and answers when required to retrieve the relevant information.
What searches will you perform to answer the users question? Respond with a JSON object with the key "queries" mapping to a list of searches you would perform on the user's knowledge base. Just return the queries and nothing else.
Current Date: {day_of_week}, {current_date}
User's Location: {location}
Here are some examples of how you can construct search queries to answer the user's question:
User: How was my trip to Cambodia?
Assistant: {{"queries": ["How was my trip to Cambodia?"]}}
User: What national parks did I go to last year?
Assistant: {{"queries": ["National park I visited in {last_new_year} dt>='{last_new_year_date}' dt<'{current_new_year_date}'"]}}
User: How can you help me?
Assistant: {{"queries": ["Social relationships", "Physical and mental health", "Education and career", "Personal life goals and habits"]}}
User: Who all did I meet here yesterday?
Assistant: {{"queries": ["Met in {location} on {yesterday_date} dt>='{yesterday_date}' dt<'{current_date}'"]}}
""".strip()
)
extract_questions_anthropic_user_message = PromptTemplate.from_template(
"""
Here's our most recent chat history:
{chat_history}
User: {text}
Assistant:
""".strip()
)
system_prompt_extract_relevant_information = """As a professional analyst, create a comprehensive report of the most relevant information from a web page in response to a user's query. The text provided is directly from within the web page. The report you create should be multiple paragraphs, and it should represent the content of the website. Tell the user exactly what the website says in response to their query, while adhering to these guidelines: system_prompt_extract_relevant_information = """As a professional analyst, create a comprehensive report of the most relevant information from a web page in response to a user's query. The text provided is directly from within the web page. The report you create should be multiple paragraphs, and it should represent the content of the website. Tell the user exactly what the website says in response to their query, while adhering to these guidelines:
1. Answer the user's query as specifically as possible. Include many supporting details from the website. 1. Answer the user's query as specifically as possible. Include many supporting details from the website.
@@ -454,7 +493,7 @@ You are Khoj, an advanced google search assistant. You are tasked with construct
- Break messages into multiple search queries when required to retrieve the relevant information. - Break messages into multiple search queries when required to retrieve the relevant information.
- Use site: google search operators when appropriate - Use site: google search operators when appropriate
- You have access to the the whole internet to retrieve information. - You have access to the the whole internet to retrieve information.
- Official, up-to-date information about you, Khoj, is available at site:khoj.dev, github or pypi. - Official, up-to-date information about you, Khoj, is available at site:khoj.dev
What Google searches, if any, will you need to perform to answer the user's question? What Google searches, if any, will you need to perform to answer the user's question?
Provide search queries as a list of strings in a JSON object. Provide search queries as a list of strings in a JSON object.
@@ -510,6 +549,7 @@ Q: How many oranges would fit in NASA's Saturn V rocket?
Khoj: {{"queries": ["volume of an orange", "volume of saturn v rocket"]}} Khoj: {{"queries": ["volume of an orange", "volume of saturn v rocket"]}}
Now it's your turn to construct Google search queries to answer the user's question. Provide them as a list of strings in a JSON object. Do not say anything else. Now it's your turn to construct Google search queries to answer the user's question. Provide them as a list of strings in a JSON object. Do not say anything else.
Now it's your turn to construct a search query for Google to answer the user's question.
History: History:
{chat_history} {chat_history}

67
src/khoj/routers/api.py
View File

@@ -3,8 +3,10 @@ import json
import logging import logging
import math import math
import os import os
import threading
import time import time
import uuid import uuid
from random import random
from typing import Any, Callable, List, Optional, Union from typing import Any, Callable, List, Optional, Union
import cron_descriptor import cron_descriptor
@@ -26,6 +28,9 @@ from khoj.database.adapters import (
get_user_search_model_or_default, get_user_search_model_or_default,
) )
from khoj.database.models import ChatModelOptions, KhojUser, SpeechToTextModelOptions from khoj.database.models import ChatModelOptions, KhojUser, SpeechToTextModelOptions
from khoj.processor.conversation.anthropic.anthropic_chat import (
extract_questions_anthropic,
)
from khoj.processor.conversation.offline.chat_model import extract_questions_offline from khoj.processor.conversation.offline.chat_model import extract_questions_offline
from khoj.processor.conversation.offline.whisper import transcribe_audio_offline from khoj.processor.conversation.offline.whisper import transcribe_audio_offline
from khoj.processor.conversation.openai.gpt import extract_questions from khoj.processor.conversation.openai.gpt import extract_questions
@@ -338,6 +343,17 @@ async def extract_references_and_questions(
api_key=api_key, api_key=api_key,
conversation_log=meta_log, conversation_log=meta_log,
location_data=location_data, location_data=location_data,
max_tokens=conversation_config.max_prompt_size,
)
elif conversation_config.model_type == ChatModelOptions.ModelType.ANTHROPIC:
api_key = conversation_config.openai_config.api_key
chat_model = conversation_config.chat_model
inferred_queries = extract_questions_anthropic(
defiltered_query,
model=chat_model,
api_key=api_key,
conversation_log=meta_log,
location_data=location_data,
) )
# Collate search results as context for GPT # Collate search results as context for GPT
@@ -454,14 +470,28 @@ async def post_automation(
crontime = " ".join(crontime.split(" ")[:5]) crontime = " ".join(crontime.split(" ")[:5])
# Convert crontime to standard unix crontime # Convert crontime to standard unix crontime
crontime = crontime.replace("?", "*") crontime = crontime.replace("?", "*")
# Disallow minute level automation recurrence
minute_value = crontime.split(" ")[0]
if not minute_value.isdigit():
return Response(
content="Recurrence of every X minutes is unsupported. Please create a less frequent schedule.",
status_code=400,
)
subject = await acreate_title_from_query(q) subject = await acreate_title_from_query(q)
# Create new Conversation Session associated with this new task
conversation = await ConversationAdapters.acreate_conversation_session(user, request.user.client_app)
calling_url = request.url.replace(query=f"{request.url.query}&conversation_id={conversation.id}")
# Schedule automation with query_to_run, timezone, subject directly provided by user # Schedule automation with query_to_run, timezone, subject directly provided by user
try: try:
# Use the query to run as the scheduling request if the scheduling request is unset # Use the query to run as the scheduling request if the scheduling request is unset
automation = await schedule_automation(query_to_run, subject, crontime, timezone, q, user, request.url) automation = await schedule_automation(query_to_run, subject, crontime, timezone, q, user, calling_url)
except Exception as e: except Exception as e:
logger.error(f"Error creating automation {q} for {user.email}: {e}") logger.error(f"Error creating automation {q} for {user.email}: {e}", exc_info=True)
return Response( return Response(
content=f"Unable to create automation. Ensure the automation doesn't already exist.", content=f"Unable to create automation. Ensure the automation doesn't already exist.",
media_type="text/plain", media_type="text/plain",
@@ -475,6 +505,31 @@ async def post_automation(
return Response(content=json.dumps(automation_info), media_type="application/json", status_code=200) return Response(content=json.dumps(automation_info), media_type="application/json", status_code=200)
@api.post("/trigger/automation", response_class=Response)
@requires(["authenticated"])
def trigger_manual_job(
request: Request,
automation_id: str,
):
user: KhojUser = request.user.object
# Check, get automation to edit
try:
automation: Job = AutomationAdapters.get_automation(user, automation_id)
except ValueError as e:
logger.error(f"Error triggering automation {automation_id} for {user.email}: {e}", exc_info=True)
return Response(content="Invalid automation", status_code=403)
# Trigger the job without waiting for the result.
scheduled_chat_func = automation.func
# Run the function in a separate thread
thread = threading.Thread(target=scheduled_chat_func, args=automation.args, kwargs=automation.kwargs)
thread.start()
return Response(content="Automation triggered", status_code=200)
@api.put("/automation", response_class=Response) @api.put("/automation", response_class=Response)
@requires(["authenticated"]) @requires(["authenticated"])
def edit_job( def edit_job(
@@ -515,6 +570,14 @@ def edit_job(
# Convert crontime to standard unix crontime # Convert crontime to standard unix crontime
crontime = crontime.replace("?", "*") crontime = crontime.replace("?", "*")
# Disallow minute level automation recurrence
minute_value = crontime.split(" ")[0]
if not minute_value.isdigit():
return Response(
content="Recurrence of every X minutes is unsupported. Please create a less frequent schedule.",
status_code=400,
)
# Construct updated automation metadata # Construct updated automation metadata
automation_metadata = json.loads(automation.name) automation_metadata = json.loads(automation.name)
automation_metadata["scheduling_request"] = q automation_metadata["scheduling_request"] = q

174
src/khoj/routers/api_chat.py
View File

@@ -13,7 +13,12 @@ from starlette.authentication import requires
from starlette.websockets import WebSocketDisconnect from starlette.websockets import WebSocketDisconnect
from websockets import ConnectionClosedOK from websockets import ConnectionClosedOK
from khoj.database.adapters import ConversationAdapters, EntryAdapters, aget_user_name from khoj.database.adapters import (
ConversationAdapters,
EntryAdapters,
PublicConversationAdapters,
aget_user_name,
)
from khoj.database.models import KhojUser from khoj.database.models import KhojUser
from khoj.processor.conversation.prompts import ( from khoj.processor.conversation.prompts import (
help_message, help_message,
@@ -38,6 +43,7 @@ from khoj.routers.helpers import (
construct_automation_created_message, construct_automation_created_message,
create_automation, create_automation,
get_conversation_command, get_conversation_command,
is_query_empty,
is_ready_to_chat, is_ready_to_chat,
text_to_image, text_to_image,
update_telemetry_state, update_telemetry_state,
@@ -62,6 +68,23 @@ conversation_command_rate_limiter = ConversationCommandRateLimiter(
api_chat = APIRouter() api_chat = APIRouter()
from pydantic import BaseModel
from khoj.routers.email import send_query_feedback
class FeedbackData(BaseModel):
uquery: str
kquery: str
sentiment: str
@api_chat.post("/feedback")
@requires(["authenticated"])
async def sendfeedback(request: Request, data: FeedbackData):
user: KhojUser = request.user.object
await send_query_feedback(data.uquery, data.kquery, data.sentiment, user.email)
@api_chat.get("/starters", response_class=Response) @api_chat.get("/starters", response_class=Response)
@requires(["authenticated"]) @requires(["authenticated"])
@@ -132,6 +155,60 @@ def chat_history(
return {"status": "ok", "response": meta_log} return {"status": "ok", "response": meta_log}
@api_chat.get("/share/history")
def get_shared_chat(
request: Request,
common: CommonQueryParams,
public_conversation_slug: str,
n: Optional[int] = None,
):
user = request.user.object if request.user.is_authenticated else None
# Load Conversation History
conversation = PublicConversationAdapters.get_public_conversation_by_slug(public_conversation_slug)
if conversation is None:
return Response(
content=json.dumps({"status": "error", "message": f"Conversation: {public_conversation_slug} not found"}),
status_code=404,
)
agent_metadata = None
if conversation.agent:
agent_metadata = {
"slug": conversation.agent.slug,
"name": conversation.agent.name,
"avatar": conversation.agent.avatar,
"isCreator": conversation.agent.creator == user,
}
meta_log = conversation.conversation_log
meta_log.update(
{
"conversation_id": conversation.id,
"slug": conversation.title if conversation.title else conversation.slug,
"agent": agent_metadata,
}
)
if n:
# Get latest N messages if N > 0
if n > 0 and meta_log.get("chat"):
meta_log["chat"] = meta_log["chat"][-n:]
# Else return all messages except latest N
elif n < 0 and meta_log.get("chat"):
meta_log["chat"] = meta_log["chat"][:n]
update_telemetry_state(
request=request,
telemetry_type="api",
api="public_conversation_history",
**common.__dict__,
)
return {"status": "ok", "response": meta_log}
@api_chat.delete("/history") @api_chat.delete("/history")
@requires(["authenticated"]) @requires(["authenticated"])
async def clear_chat_history( async def clear_chat_history(
@@ -154,6 +231,69 @@ async def clear_chat_history(
return {"status": "ok", "message": "Conversation history cleared"} return {"status": "ok", "message": "Conversation history cleared"}
@api_chat.post("/share/fork")
@requires(["authenticated"])
def fork_public_conversation(
request: Request,
common: CommonQueryParams,
public_conversation_slug: str,
):
user = request.user.object
# Load Conversation History
public_conversation = PublicConversationAdapters.get_public_conversation_by_slug(public_conversation_slug)
# Duplicate Public Conversation to User's Private Conversation
ConversationAdapters.create_conversation_from_public_conversation(
user, public_conversation, request.user.client_app
)
chat_metadata = {"forked_conversation": public_conversation.slug}
update_telemetry_state(
request=request,
telemetry_type="api",
api="fork_public_conversation",
**common.__dict__,
metadata=chat_metadata,
)
redirect_uri = str(request.app.url_path_for("chat_page"))
return Response(status_code=200, content=json.dumps({"status": "ok", "next_url": redirect_uri}))
@api_chat.post("/share")
@requires(["authenticated"])
def duplicate_chat_history_public_conversation(
request: Request,
common: CommonQueryParams,
conversation_id: int,
):
user = request.user.object
# Duplicate Conversation History to Public Conversation
conversation = ConversationAdapters.get_conversation_by_user(user, request.user.client_app, conversation_id)
public_conversation = ConversationAdapters.make_public_conversation_copy(conversation)
public_conversation_url = PublicConversationAdapters.get_public_conversation_url(public_conversation)
domain = request.headers.get("host")
scheme = request.url.scheme
update_telemetry_state(
request=request,
telemetry_type="api",
api="post_chat_share",
**common.__dict__,
)
return Response(
status_code=200, content=json.dumps({"status": "ok", "url": f"{scheme}://{domain}{public_conversation_url}"})
)
@api_chat.get("/sessions") @api_chat.get("/sessions")
@requires(["authenticated"]) @requires(["authenticated"])
def chat_sessions( def chat_sessions(
@@ -347,6 +487,10 @@ async def websocket_endpoint(
if conversation: if conversation:
await sync_to_async(conversation.refresh_from_db)(fields=["conversation_log"]) await sync_to_async(conversation.refresh_from_db)(fields=["conversation_log"])
q = await websocket.receive_text() q = await websocket.receive_text()
# Refresh these because the connection to the database might have been closed
await conversation.arefresh_from_db()
except WebSocketDisconnect: except WebSocketDisconnect:
logger.debug(f"User {user} disconnected web socket") logger.debug(f"User {user} disconnected web socket")
break break
@@ -358,6 +502,14 @@ async def websocket_endpoint(
await send_rate_limit_message(e.detail) await send_rate_limit_message(e.detail)
break break
if is_query_empty(q):
await send_message("start_llm_response")
await send_message(
"It seems like your query is incomplete. Could you please provide more details or specify what you need help with?"
)
await send_message("end_llm_response")
continue
user_message_time = datetime.now().strftime("%Y-%m-%d %H:%M:%S") user_message_time = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
conversation_commands = [get_conversation_command(query=q, any_references=True)] conversation_commands = [get_conversation_command(query=q, any_references=True)]
@@ -465,11 +617,17 @@ async def websocket_endpoint(
if ConversationCommand.Webpage in conversation_commands: if ConversationCommand.Webpage in conversation_commands:
try: try:
online_results = await read_webpages(defiltered_query, meta_log, location, send_status_update) direct_web_pages = await read_webpages(defiltered_query, meta_log, location, send_status_update)
webpages = [] webpages = []
for query in online_results: for query in direct_web_pages:
for webpage in online_results[query]["webpages"]: if online_results.get(query):
online_results[query]["webpages"] = direct_web_pages[query]["webpages"]
else:
online_results[query] = {"webpages": direct_web_pages[query]["webpages"]}
for webpage in direct_web_pages[query]["webpages"]:
webpages.append(webpage["link"]) webpages.append(webpage["link"])
await send_status_update(f"**📚 Read web pages**: {webpages}") await send_status_update(f"**📚 Read web pages**: {webpages}")
except ValueError as e: except ValueError as e:
logger.warning( logger.warning(
@@ -585,6 +743,12 @@ async def chat(
) -> Response: ) -> Response:
user: KhojUser = request.user.object user: KhojUser = request.user.object
q = unquote(q) q = unquote(q)
if is_query_empty(q):
return Response(
content="It seems like your query is incomplete. Could you please provide more details or specify what you need help with?",
media_type="text/plain",
status_code=400,
)
user_message_time = datetime.now().strftime("%Y-%m-%d %H:%M:%S") user_message_time = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
logger.info(f"Chat request by {user.username}: {q}") logger.info(f"Chat request by {user.username}: {q}")
@@ -634,7 +798,7 @@ async def chat(
q, timezone, user, request.url, meta_log q, timezone, user, request.url, meta_log
) )
except Exception as e: except Exception as e:
logger.error(f"Error creating automation {q} for {user.email}: {e}") logger.error(f"Error creating automation {q} for {user.email}: {e}", exc_info=True)
return Response( return Response(
content=f"Unable to create automation. Ensure the automation doesn't already exist.", content=f"Unable to create automation. Ensure the automation doesn't already exist.",
media_type="text/plain", media_type="text/plain",

31
src/khoj/routers/email.py
View File

@@ -42,7 +42,7 @@ async def send_welcome_email(name, email):
r = resend.Emails.send( r = resend.Emails.send(
{ {
"from": "team@khoj.dev", "sender": "team@khoj.dev",
"to": email, "to": email,
"subject": f"{name}, four ways to use Khoj" if name else "Four ways to use Khoj", "subject": f"{name}, four ways to use Khoj" if name else "Four ways to use Khoj",
"html": html_content, "html": html_content,
@@ -50,6 +50,33 @@ async def send_welcome_email(name, email):
) )
async def send_query_feedback(uquery, kquery, sentiment, user_email):
if not is_resend_enabled():
logger.debug(f"Sentiment: {sentiment}, Query: {uquery}, Khoj Response: {kquery}")
return
logger.info(f"Sending feedback email for query {uquery}")
# rendering feedback email using feedback.html as template
template = env.get_template("feedback.html")
html_content = template.render(
uquery=uquery if not is_none_or_empty(uquery) else "N/A",
kquery=kquery if not is_none_or_empty(kquery) else "N/A",
sentiment=sentiment if not is_none_or_empty(sentiment) else "N/A",
user_email=user_email if not is_none_or_empty(user_email) else "N/A",
)
# send feedback from two fixed accounts
r = resend.Emails.send(
{
"sender": "saba@khoj.dev",
"to": "team@khoj.dev",
"subject": f"User Feedback",
"html": html_content,
}
)
return {"message": "Sent Email"}
def send_task_email(name, email, query, result, subject): def send_task_email(name, email, query, result, subject):
if not is_resend_enabled(): if not is_resend_enabled():
logger.debug("Email sending disabled") logger.debug("Email sending disabled")
@@ -64,7 +91,7 @@ def send_task_email(name, email, query, result, subject):
r = resend.Emails.send( r = resend.Emails.send(
{ {
"from": "Khoj <khoj@khoj.dev>", "sender": "Khoj <khoj@khoj.dev>",
"to": email, "to": email,
"subject": f"✨ {subject}", "subject": f"✨ {subject}",
"html": html_content, "html": html_content,

82
src/khoj/routers/helpers.py
View File

@@ -4,10 +4,12 @@ import hashlib
import io import io
import json import json
import logging import logging
import math
import re import re
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timedelta, timezone from datetime import datetime, timedelta, timezone
from functools import partial from functools import partial
from random import random
from typing import ( from typing import (
Annotated, Annotated,
Any, Any,
@@ -53,6 +55,10 @@ from khoj.database.models import (
UserRequests, UserRequests,
) )
from khoj.processor.conversation import prompts from khoj.processor.conversation import prompts
from khoj.processor.conversation.anthropic.anthropic_chat import (
anthropic_send_message_to_model,
converse_anthropic,
)
from khoj.processor.conversation.offline.chat_model import ( from khoj.processor.conversation.offline.chat_model import (
converse_offline, converse_offline,
send_message_to_model_offline, send_message_to_model_offline,
@@ -84,6 +90,10 @@ logger = logging.getLogger(__name__)
executor = ThreadPoolExecutor(max_workers=1) executor = ThreadPoolExecutor(max_workers=1)
def is_query_empty(query: str) -> bool:
return is_none_or_empty(query.strip())
def validate_conversation_config(): def validate_conversation_config():
default_config = ConversationAdapters.get_default_conversation_config() default_config = ConversationAdapters.get_default_conversation_config()
@@ -109,7 +119,7 @@ async def is_ready_to_chat(user: KhojUser):
if ( if (
user_conversation_config user_conversation_config
and user_conversation_config.model_type == "openai" and (user_conversation_config.model_type == "openai" or user_conversation_config.model_type == "anthropic")
and user_conversation_config.openai_config and user_conversation_config.openai_config
): ):
return True return True
@@ -388,9 +398,13 @@ async def extract_relevant_info(q: str, corpus: str) -> Union[str, None]:
corpus=corpus.strip(), corpus=corpus.strip(),
) )
summarizer_model: ChatModelOptions = await ConversationAdapters.aget_summarizer_conversation_config()
with timer("Chat actor: Extract relevant information from data", logger): with timer("Chat actor: Extract relevant information from data", logger):
response = await send_message_to_model_wrapper( response = await send_message_to_model_wrapper(
extract_relevant_information, prompts.system_prompt_extract_relevant_information extract_relevant_information,
prompts.system_prompt_extract_relevant_information,
chat_model_option=summarizer_model,
) )
return response.strip() return response.strip()
@@ -445,8 +459,11 @@ async def send_message_to_model_wrapper(
message: str, message: str,
system_message: str = "", system_message: str = "",
response_type: str = "text", response_type: str = "text",
chat_model_option: ChatModelOptions = None,
): ):
conversation_config: ChatModelOptions = await ConversationAdapters.aget_default_conversation_config() conversation_config: ChatModelOptions = (
chat_model_option or await ConversationAdapters.aget_default_conversation_config()
)
if conversation_config is None: if conversation_config is None:
raise HTTPException(status_code=500, detail="Contact the server administrator to set a default chat model.") raise HTTPException(status_code=500, detail="Contact the server administrator to set a default chat model.")
@@ -497,6 +514,21 @@ async def send_message_to_model_wrapper(
) )
return openai_response return openai_response
elif conversation_config.model_type == "anthropic":
api_key = conversation_config.openai_config.api_key
truncated_messages = generate_chatml_messages_with_context(
user_message=message,
system_message=system_message,
model_name=chat_model,
max_prompt_size=max_tokens,
tokenizer_name=tokenizer,
)
return anthropic_send_message_to_model(
messages=truncated_messages,
api_key=api_key,
model=chat_model,
)
else: else:
raise HTTPException(status_code=500, detail="Invalid conversation config") raise HTTPException(status_code=500, detail="Invalid conversation config")
@@ -531,8 +563,7 @@ def send_message_to_model_wrapper_sync(
) )
elif conversation_config.model_type == "openai": elif conversation_config.model_type == "openai":
openai_chat_config = ConversationAdapters.get_openai_conversation_config() api_key = conversation_config.openai_config.api_key
api_key = openai_chat_config.api_key
truncated_messages = generate_chatml_messages_with_context( truncated_messages = generate_chatml_messages_with_context(
user_message=message, system_message=system_message, model_name=chat_model user_message=message, system_message=system_message, model_name=chat_model
) )
@@ -542,6 +573,21 @@ def send_message_to_model_wrapper_sync(
) )
return openai_response return openai_response
elif conversation_config.model_type == "anthropic":
api_key = conversation_config.openai_config.api_key
truncated_messages = generate_chatml_messages_with_context(
user_message=message,
system_message=system_message,
model_name=chat_model,
max_prompt_size=max_tokens,
)
return anthropic_send_message_to_model(
messages=truncated_messages,
api_key=api_key,
model=chat_model,
)
else: else:
raise HTTPException(status_code=500, detail="Invalid conversation config") raise HTTPException(status_code=500, detail="Invalid conversation config")
@@ -620,6 +666,24 @@ def generate_chat_response(
agent=agent, agent=agent,
) )
elif conversation_config.model_type == "anthropic":
api_key = conversation_config.openai_config.api_key
chat_response = converse_anthropic(
compiled_references,
q,
online_results,
meta_log,
model=conversation_config.chat_model,
api_key=api_key,
completion_func=partial_completion,
conversation_commands=conversation_commands,
max_prompt_size=conversation_config.max_prompt_size,
tokenizer_name=conversation_config.tokenizer,
location_data=location_data,
user_name=user_name,
agent=agent,
)
metadata.update({"chat_model": conversation_config.chat_model}) metadata.update({"chat_model": conversation_config.chat_model})
except Exception as e: except Exception as e:
@@ -931,7 +995,7 @@ def scheduled_chat(
# Stop if the chat API call was not successful # Stop if the chat API call was not successful
if raw_response.status_code != 200: if raw_response.status_code != 200:
logger.error(f"Failed to run schedule chat: {raw_response.text}") logger.error(f"Failed to run schedule chat: {raw_response.text}, user: {user}, query: {query_to_run}")
return None return None
# Extract the AI response from the chat API response # Extract the AI response from the chat API response
@@ -965,6 +1029,12 @@ async def schedule_automation(
user: KhojUser, user: KhojUser,
calling_url: URL, calling_url: URL,
): ):
# Disable minute level automation recurrence
minute_value = crontime.split(" ")[0]
if not minute_value.isdigit():
# Run automation at some random minute (to distribute request load) instead of running every X minutes
crontime = " ".join([str(math.floor(random() * 60))] + crontime.split(" ")[1:])
user_timezone = pytz.timezone(timezone) user_timezone = pytz.timezone(timezone)
trigger = CronTrigger.from_crontab(crontime, user_timezone) trigger = CronTrigger.from_crontab(crontime, user_timezone)
trigger.jitter = 60 trigger.jitter = 60

61
src/khoj/routers/web_client.py
View File

@@ -11,9 +11,9 @@ from starlette.authentication import has_required_scope, requires
from khoj.database import adapters from khoj.database import adapters
from khoj.database.adapters import ( from khoj.database.adapters import (
AgentAdapters, AgentAdapters,
AutomationAdapters,
ConversationAdapters, ConversationAdapters,
EntryAdapters, EntryAdapters,
PublicConversationAdapters,
get_user_github_config, get_user_github_config,
get_user_name, get_user_name,
get_user_notion_config, get_user_notion_config,
@@ -350,9 +350,9 @@ def notion_config_page(request: Request):
@web_client.get("/config/content-source/computer", response_class=HTMLResponse) @web_client.get("/config/content-source/computer", response_class=HTMLResponse)
@requires(["authenticated"], redirect="login_page") @requires(["authenticated"], redirect="login_page")
def computer_config_page(request: Request): def computer_config_page(request: Request):
user = request.user.object user = request.user.object if request.user.is_authenticated else None
user_picture = request.session.get("user", {}).get("picture") user_picture = request.session.get("user", {}).get("picture") if user else None
has_documents = EntryAdapters.user_has_entries(user=user) has_documents = EntryAdapters.user_has_entries(user=user) if user else False
return templates.TemplateResponse( return templates.TemplateResponse(
"content_source_computer_input.html", "content_source_computer_input.html",
@@ -367,6 +367,59 @@ def computer_config_page(request: Request):
) )
@web_client.get("/share/chat/{public_conversation_slug}", response_class=HTMLResponse)
def view_public_conversation(request: Request):
public_conversation_slug = request.path_params.get("public_conversation_slug")
public_conversation = PublicConversationAdapters.get_public_conversation_by_slug(public_conversation_slug)
if not public_conversation:
return templates.TemplateResponse(
"404.html",
context={
"request": request,
"khoj_version": state.khoj_version,
},
)
user = request.user.object if request.user.is_authenticated else None
user_picture = request.session.get("user", {}).get("picture") if user else None
has_documents = EntryAdapters.user_has_entries(user=user) if user else False
all_agents = AgentAdapters.get_all_accessible_agents(request.user.object if request.user.is_authenticated else None)
# Filter out the current agent
all_agents = [agent for agent in all_agents if agent != public_conversation.agent]
agents_packet = []
for agent in all_agents:
agents_packet.append(
{
"slug": agent.slug,
"avatar": agent.avatar,
"name": agent.name,
}
)
google_client_id = os.environ.get("GOOGLE_CLIENT_ID")
redirect_uri = str(request.app.url_path_for("auth"))
next_url = str(
request.app.url_path_for("view_public_conversation", public_conversation_slug=public_conversation_slug)
)
return templates.TemplateResponse(
"public_conversation.html",
context={
"request": request,
"username": user.username if user else None,
"user_photo": user_picture,
"is_active": has_required_scope(request, ["premium"]),
"has_documents": has_documents,
"khoj_version": state.khoj_version,
"public_conversation_slug": public_conversation_slug,
"agents": agents_packet,
"google_client_id": google_client_id,
"redirect_uri": f"{redirect_uri}?next={next_url}",
},
)
@web_client.get("/automations", response_class=HTMLResponse) @web_client.get("/automations", response_class=HTMLResponse)
@requires(["authenticated"], redirect="login_page") @requires(["authenticated"], redirect="login_page")
def automations_config_page(request: Request): def automations_config_page(request: Request):

3
versions.json
View File

@@ -48,5 +48,6 @@
"1.11.0": "0.15.0", "1.11.0": "0.15.0",
"1.11.1": "0.15.0", "1.11.1": "0.15.0",
"1.11.2": "0.15.0", "1.11.2": "0.15.0",
"1.12.0": "0.15.0" "1.12.0": "0.15.0",
"1.12.1": "0.15.0"
} }