klbr/khoj
1
0
Fork 0
mirror of https://github.com/khoaliber/khoj.git synced 2026-03-06 13:22:12 +00:00
Code Issues Packages Projects Releases Wiki Activity

Refactor Config API and Settings pages for Reuse and Consistency (#852)

Browse Source 
### Major
- Reuse get config data logic across config pages on web client
- Make config api endpoint urls and response fields consistent
- Rename API path /api/config to /api/configure
- Move Web, Desktop client settings page to be under `/settings` from the previous `/config` url path

### Minor
- Pass isMobileWidth prop to SidePanel via chat share interface
- Turn prettier off instead of throwing error for now
- Do no explicitly add line-clamp plugin as it's in Tailwind by default
This commit is contained in:
Debanjum
2024-07-17 01:03:06 -07:00
committed by GitHub
parent 1a5405e24c a1c362a4f7
commit bf815e4463
42 changed files with 453 additions and 557 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
documentation/docs/advanced/litellm.md
View File

@@ -34,4 +34,4 @@ Using LiteLLM with Khoj makes it possible to turn any LLM behind an API into you
5. Create a new [Server Chat Setting](http://localhost:42110/server/admin/database/serverchatsettings/add/) on your Khoj admin panel 5. Create a new [Server Chat Setting](http://localhost:42110/server/admin/database/serverchatsettings/add/) on your Khoj admin panel
- Default model: `<name of chat model option you created in step 4>` - Default model: `<name of chat model option you created in step 4>`
- Summarizer model: `<name of chat model option you created in step 4>` - Summarizer model: `<name of chat model option you created in step 4>`
6. Go to [your config](http://localhost:42110/config) and select the model you just created in the chat model dropdown. 6. Go to [your config](http://localhost:42110/settings) and select the model you just created in the chat model dropdown.

2
documentation/docs/advanced/lmstudio.md
View File

@@ -27,4 +27,4 @@ LM Studio can expose an [OpenAI API compatible server](https://lmstudio.ai/docs/
5. Create a new [Server Chat Setting](http://localhost:42110/server/admin/database/serverchatsettings/add/) on your Khoj admin panel 5. Create a new [Server Chat Setting](http://localhost:42110/server/admin/database/serverchatsettings/add/) on your Khoj admin panel
- Default model: `<name of chat model option you created in step 4>` - Default model: `<name of chat model option you created in step 4>`
- Summarizer model: `<name of chat model option you created in step 4>` - Summarizer model: `<name of chat model option you created in step 4>`
6. Go to [your config](http://localhost:42110/config) and select the model you just created in the chat model dropdown. 6. Go to [your config](http://localhost:42110/settings) and select the model you just created in the chat model dropdown.

2
documentation/docs/advanced/ollama.md
View File

@@ -31,6 +31,6 @@ Ollama exposes a local [OpenAI API compatible server](https://github.com/ollama/
5. Create a new [Server Chat Setting](http://localhost:42110/server/admin/database/serverchatsettings/add/) on your Khoj admin panel 5. Create a new [Server Chat Setting](http://localhost:42110/server/admin/database/serverchatsettings/add/) on your Khoj admin panel
- Default model: `<name of chat model option you created in step 4>` - Default model: `<name of chat model option you created in step 4>`
- Summarizer model: `<name of chat model option you created in step 4>` - Summarizer model: `<name of chat model option you created in step 4>`
6. Go to [your config](http://localhost:42110/config) and select the model you just created in the chat model dropdown. 6. Go to [your config](http://localhost:42110/settings) and select the model you just created in the chat model dropdown.
That's it! You should now be able to chat with your Ollama model from Khoj. If you want to add additional models running on Ollama, repeat step 6 for each model. That's it! You should now be able to chat with your Ollama model from Khoj. If you want to add additional models running on Ollama, repeat step 6 for each model.

2
documentation/docs/advanced/use-openai-proxy.md
View File

@@ -34,4 +34,4 @@ For specific integrations, see our [Ollama](/advanced/ollama), [LMStudio](/advan
5. Create a new [Server Chat Setting](http://localhost:42110/server/admin/database/serverchatsettings/add/) on your Khoj admin panel 5. Create a new [Server Chat Setting](http://localhost:42110/server/admin/database/serverchatsettings/add/) on your Khoj admin panel
- Default model: `<name of chat model option you created in step 4>` - Default model: `<name of chat model option you created in step 4>`
- Summarizer model: `<name of chat model option you created in step 4>` - Summarizer model: `<name of chat model option you created in step 4>`
6. Go to [your config](http://localhost:42110/config) and select the model you just created in the chat model dropdown. 6. Go to [your config](http://localhost:42110/settings) and select the model you just created in the chat model dropdown.

2
documentation/docs/clients/desktop.md
View File

@@ -23,7 +23,7 @@ Khoj will keep these files in sync to provide contextual responses when you sear
## Setup ## Setup
1. Install the [Khoj Desktop app](https://khoj.dev/downloads) for your OS 1. Install the [Khoj Desktop app](https://khoj.dev/downloads) for your OS
2. Generate an API key on the [Khoj Web App](https://app.khoj.dev/config#clients) 2. Generate an API key on the [Khoj Web App](https://app.khoj.dev/settings#clients)
3. Set your Khoj API Key on the *Settings* page of the Khoj Desktop app 3. Set your Khoj API Key on the *Settings* page of the Khoj Desktop app
4. [Optional] Add any files, folders you'd like Khoj to be aware of on the *Settings* page and Click *Save* 4. [Optional] Add any files, folders you'd like Khoj to be aware of on the *Settings* page and Click *Save*
These files and folders will be automatically kept in sync for you These files and folders will be automatically kept in sync for you

2
documentation/docs/clients/emacs.md
View File

@@ -30,7 +30,7 @@ sidebar_position: 2
| ![khoj search on emacs](/img/khoj_search_on_emacs.png) | ![khoj chat on emacs](/img/khoj_chat_on_emacs.png) | | ![khoj search on emacs](/img/khoj_search_on_emacs.png) | ![khoj chat on emacs](/img/khoj_chat_on_emacs.png) |
## Setup ## Setup
1. Generate an API key on the [Khoj Web App](https://app.khoj.dev/config#clients) 1. Generate an API key on the [Khoj Web App](https://app.khoj.dev/settings#clients)
2. Add below snippet to your Emacs config file, usually at `~/.emacs.d/init.el` 2. Add below snippet to your Emacs config file, usually at `~/.emacs.d/init.el`

2
documentation/docs/clients/obsidian.md
View File

@@ -23,7 +23,7 @@ sidebar_position: 3
1. Open [Khoj](https://obsidian.md/plugins?id=khoj) from the *Community plugins* tab in Obsidian settings panel 1. Open [Khoj](https://obsidian.md/plugins?id=khoj) from the *Community plugins* tab in Obsidian settings panel
2. Click *Install*, then *Enable* on the Khoj plugin page in Obsidian 2. Click *Install*, then *Enable* on the Khoj plugin page in Obsidian
3. Generate an API key on the [Khoj Web App](https://app.khoj.dev/config#clients) 3. Generate an API key on the [Khoj Web App](https://app.khoj.dev/settings#clients)
4. Set your Khoj API Key in the Khoj plugin settings in Obsidian 4. Set your Khoj API Key in the Khoj plugin settings in Obsidian
See the official [Obsidian Plugin Docs](https://help.obsidian.md/Extending+Obsidian/Community+plugins) for more details on installing Obsidian plugins. See the official [Obsidian Plugin Docs](https://help.obsidian.md/Extending+Obsidian/Community+plugins) for more details on installing Obsidian plugins.

2
documentation/docs/clients/whatsapp.md
View File

@@ -10,7 +10,7 @@ Text [+1 (848) 800 4242](https://wa.me/18488004242) or scan [this QR code](https
Without any desktop clients, you can start chatting with Khoj on WhatsApp. Bear in mind you do need one of the desktop clients in order to share and sync your data with Khoj. The WhatsApp AI bot will work right away for answering generic queries and using Khoj in default mode. Without any desktop clients, you can start chatting with Khoj on WhatsApp. Bear in mind you do need one of the desktop clients in order to share and sync your data with Khoj. The WhatsApp AI bot will work right away for answering generic queries and using Khoj in default mode.
In order to use Khoj on WhatsApp with your own data, you need to setup a Khoj Cloud account and connect your WhatsApp account to it. This is a one time setup and you can do it from the [Khoj Cloud config page](https://app.khoj.dev/config). In order to use Khoj on WhatsApp with your own data, you need to setup a Khoj Cloud account and connect your WhatsApp account to it. This is a one time setup and you can do it from the [Khoj Cloud config page](https://app.khoj.dev/settings).
If you hit usage limits for the WhatsApp bot, upgrade to [a paid plan](https://khoj.dev/pricing) on Khoj Cloud. If you hit usage limits for the WhatsApp bot, upgrade to [a paid plan](https://khoj.dev/pricing) on Khoj Cloud.

4
documentation/docs/data-sources/github_integration.md
View File

@@ -4,11 +4,11 @@ The Github integration allows you to index as many repositories as you want. It'
# Configure your settings # Configure your settings
1. Go to [https://app.khoj.dev/config](https://app.khoj.dev/config) and enter in settings for the data sources you want to index. You'll have to specify the file paths. 1. Go to [https://app.khoj.dev/settings](https://app.khoj.dev/settings) and enter in settings for the data sources you want to index. You'll have to specify the file paths.
## Use the Github plugin ## Use the Github plugin
1. Generate a [classic PAT (personal access token)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) from [Github](https://github.com/settings/tokens) with `repo` and `admin:org` scopes at least. 1. Generate a [classic PAT (personal access token)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) from [Github](https://github.com/settings/tokens) with `repo` and `admin:org` scopes at least.
2. Navigate to [https://app.khoj.dev/config/content-source/github](https://app.khoj.dev/config/content-source/github) to configure your Github settings. Enter in your PAT, along with details for each repository you want to index. 2. Navigate to [https://app.khoj.dev/settings/content/github](https://app.khoj.dev/settings/content/github) to configure your Github settings. Enter in your PAT, along with details for each repository you want to index.
3. Click `Save`. Go back to the settings page and click `Configure`. 3. Click `Save`. Go back to the settings page and click `Configure`.
4. Go to [https://app.khoj.dev/](https://app.khoj.dev/) and start searching! 4. Go to [https://app.khoj.dev/](https://app.khoj.dev/) and start searching!

6
documentation/docs/data-sources/notion_integration.md
View File

@@ -2,7 +2,7 @@
The Notion integration allows you to search/chat with your Notion workspaces. [Notion](https://notion.so/) is a platform people use for taking notes, especially for collaboration. The Notion integration allows you to search/chat with your Notion workspaces. [Notion](https://notion.so/) is a platform people use for taking notes, especially for collaboration.
Go to https://app.khoj.dev/config to connect your Notion workspace(s) to Khoj. Go to https://app.khoj.dev/settings to connect your Notion workspace(s) to Khoj.
![notion_integration](https://assets.khoj.dev/notion_integration.gif) ![notion_integration](https://assets.khoj.dev/notion_integration.gif)
@@ -13,7 +13,7 @@ Go to https://app.khoj.dev/config to connect your Notion workspace(s) to Khoj.
![setup_new_integration](https://github.com/khoj-ai/khoj/assets/65192171/b056e057-d4dc-47dc-aad3-57b59a22c68b) ![setup_new_integration](https://github.com/khoj-ai/khoj/assets/65192171/b056e057-d4dc-47dc-aad3-57b59a22c68b)
3. Share all the workspaces that you want to integrate with the Khoj integration you just made in the previous step 3. Share all the workspaces that you want to integrate with the Khoj integration you just made in the previous step
![enable_workspace](https://github.com/khoj-ai/khoj/assets/65192171/98290303-b5b8-4cb0-b32c-f68c6923a3d0) ![enable_workspace](https://github.com/khoj-ai/khoj/assets/65192171/98290303-b5b8-4cb0-b32c-f68c6923a3d0)
4. In the first step, you generated an API key. Use the newly generated API Key in your Khoj settings, by default at http://localhost:42110/config/content-source/notion. Click `Save`. 4. In the first step, you generated an API key. Use the newly generated API Key in your Khoj settings, by default at http://localhost:42110/settings/content/notion. Click `Save`.
5. Click `Configure` in http://localhost:42110/config to index your Notion workspace(s). 5. Click `Configure` in http://localhost:42110/settings to index your Notion workspace(s).
That's it! You should be ready to start searching and chatting. Make sure you've configured your [chat settings](/get-started/setup#2-configure). That's it! You should be ready to start searching and chatting. Make sure you've configured your [chat settings](/get-started/setup#2-configure).

2
src/interface/desktop/main.js
View File

@@ -253,7 +253,7 @@ function pushDataToKhoj (regenerate = false) {
console.error(error); console.error(error);
state["completed"] = false; state["completed"] = false;
if (error?.response?.status === 429 && (BrowserWindow.getAllWindows().find(win => win.webContents.getURL().includes('config')))) { if (error?.response?.status === 429 && (BrowserWindow.getAllWindows().find(win => win.webContents.getURL().includes('config')))) {
state["error"] = `Looks like you're out of space to sync your files. <a href="https://app.khoj.dev/config">Upgrade your plan</a> to unlock more space.`; state["error"] = `Looks like you're out of space to sync your files. <a href="https://app.khoj.dev/settings">Upgrade your plan</a> to unlock more space.`;
const win = BrowserWindow.getAllWindows().find(win => win.webContents.getURL().includes('config')); const win = BrowserWindow.getAllWindows().find(win => win.webContents.getURL().includes('config'));
if (win) win.webContents.send('needsSubscription', true); if (win) win.webContents.send('needsSubscription', true);
} else if (error?.code === 'ECONNREFUSED') { } else if (error?.code === 'ECONNREFUSED') {

2
src/interface/desktop/renderer.js
View File

@@ -182,7 +182,7 @@ window.updateStateAPI.onUpdateState((event, state) => {
window.needsSubscriptionAPI.onNeedsSubscription((event, needsSubscription) => { window.needsSubscriptionAPI.onNeedsSubscription((event, needsSubscription) => {
console.log("needs subscription", needsSubscription); console.log("needs subscription", needsSubscription);
if (needsSubscription) { if (needsSubscription) {
window.alert("Looks like you're out of space to sync your files. Upgrade your plan to unlock more space here: https://app.khoj.dev/config"); window.alert("Looks like you're out of space to sync your files. Upgrade your plan to unlock more space here: https://app.khoj.dev/settings");
needsSubscriptionElement.style.display = 'block'; needsSubscriptionElement.style.display = 'block';
} }
}); });

4
src/interface/desktop/search.html
View File

@@ -212,12 +212,12 @@
const headers = { 'Authorization': `Bearer ${khojToken}` }; const headers = { 'Authorization': `Bearer ${khojToken}` };
// Populate type dropdown field with enabled content types only // Populate type dropdown field with enabled content types only
fetch(`${hostURL}/api/config/types`, { headers }) fetch(`${hostURL}/api/configure/types`, { headers })
.then(response => response.json()) .then(response => response.json())
.then(enabled_types => { .then(enabled_types => {
// Show warning if no content types are enabled // Show warning if no content types are enabled
if (enabled_types.detail) { if (enabled_types.detail) {
document.getElementById("results").innerHTML = "<div id='results-error'>To use Khoj search, setup your content plugins on the Khoj <a class='inline-chat-link' href='/config'>settings page</a>.</div>"; document.getElementById("results").innerHTML = "<div id='results-error'>To use Khoj search, setup your content plugins on the Khoj <a class='inline-chat-link' href='/settings'>settings page</a>.</div>";
document.getElementById("query").setAttribute("disabled", "disabled"); document.getElementById("query").setAttribute("disabled", "disabled");
document.getElementById("query").setAttribute("placeholder", "Configure Khoj to enable search"); document.getElementById("query").setAttribute("placeholder", "Configure Khoj to enable search");
return []; return [];

0
src/interface/desktop/config.html → src/interface/desktop/settings.html
View File

2
src/interface/desktop/utils.js
View File

@@ -85,7 +85,7 @@ async function populateHeaderPane() {
<div id="khoj-nav-menu" class="khoj-nav-dropdown-content"> <div id="khoj-nav-menu" class="khoj-nav-dropdown-content">
<div class="khoj-nav-username"> ${username} </div> <div class="khoj-nav-username"> ${username} </div>
<a id="github-nav" class="khoj-nav" href="https://github.com/khoj-ai/khoj">GitHub</a> <a id="github-nav" class="khoj-nav" href="https://github.com/khoj-ai/khoj">GitHub</a>
<a id="settings-nav" class="khoj-nav" href="./config.html">⚙️ Settings</a> <a id="settings-nav" class="khoj-nav" href="./settings.html">⚙️ Settings</a>
</div> </div>
</div> </div>
` : ''} ` : ''}

4
src/interface/emacs/khoj.el
View File

@@ -99,7 +99,7 @@
:type 'boolean) :type 'boolean)
(defcustom khoj-api-key nil (defcustom khoj-api-key nil
"API Key to your Khoj. Default at https://app.khoj.dev/config#clients." "API Key to your Khoj. Default at https://app.khoj.dev/settings#clients."
:group 'khoj :group 'khoj
:type 'string) :type 'string)
@@ -697,7 +697,7 @@ Optionally apply CALLBACK with JSON parsed response and CBARGS."
(defun khoj--get-enabled-content-types () (defun khoj--get-enabled-content-types ()
"Get content types enabled for search from API." "Get content types enabled for search from API."
(khoj--call-api "/api/config/types" "GET" nil `(lambda (item) (mapcar #'intern item)))) (khoj--call-api "/api/configure/types" "GET" nil `(lambda (item) (mapcar #'intern item))))
(defun khoj--query-search-api-and-render-results (query content-type buffer-name &optional rerank is-find-similar) (defun khoj--query-search-api-and-render-results (query content-type buffer-name &optional rerank is-find-similar)
"Query Khoj Search API with QUERY, CONTENT-TYPE and RERANK as query params. "Query Khoj Search API with QUERY, CONTENT-TYPE and RERANK as query params.

4
src/interface/obsidian/src/utils.ts
View File

@@ -201,12 +201,12 @@ export function getBackendStatusMessage(
): string { ): string {
// Welcome message with default settings. Khoj cloud always expects an API key. // Welcome message with default settings. Khoj cloud always expects an API key.
if (!khojApiKey && khojUrl === 'https://app.khoj.dev') if (!khojApiKey && khojUrl === 'https://app.khoj.dev')
return `🌈 Welcome to Khoj! Get your API key from ${khojUrl}/config#clients and set it in the Khoj plugin settings on Obsidian`; return `🌈 Welcome to Khoj! Get your API key from ${khojUrl}/settings#clients and set it in the Khoj plugin settings on Obsidian`;
if (!connectedToServer) if (!connectedToServer)
return `❗️Could not connect to Khoj at ${khojUrl}. Ensure your can access it`; return `❗️Could not connect to Khoj at ${khojUrl}. Ensure your can access it`;
else if (!userEmail) else if (!userEmail)
return `✅ Connected to Khoj. ❗️Get a valid API key from ${khojUrl}/config#clients to log in`; return `✅ Connected to Khoj. ❗️Get a valid API key from ${khojUrl}/settings#clients to log in`;
else if (userEmail === 'default@example.com') else if (userEmail === 'default@example.com')
// Logged in as default user in anonymous mode // Logged in as default user in anonymous mode
return `✅ Signed in to Khoj`; return `✅ Signed in to Khoj`;

2
src/interface/web/.eslintrc.json
View File

@@ -6,6 +6,6 @@
], ],
"plugins": ["prettier"], "plugins": ["prettier"],
"rules": { "rules": {
"prettier/prettier": "error" "prettier/prettier": "off"
} }
} }

6
src/interface/web/app/components/modelPicker/modelPicker.tsx
View File

@@ -68,8 +68,8 @@ interface ModelPickerProps {
} }
export const ModelPicker: React.FC<any> = (props: ModelPickerProps) => { export const ModelPicker: React.FC<any> = (props: ModelPickerProps) => {
const { data: models } = useOptionsRequest('/api/config/data/conversation/model/options'); const { data: models } = useOptionsRequest('/api/configure/chat/model/options');
const { data: selectedModel } = useSelectedModel('/api/config/data/conversation/model'); const { data: selectedModel } = useSelectedModel('/api/configure/chat/model');
const [openLoginDialog, setOpenLoginDialog] = React.useState(false); const [openLoginDialog, setOpenLoginDialog] = React.useState(false);
let userData = useAuthenticatedData(); let userData = useAuthenticatedData();
@@ -94,7 +94,7 @@ export const ModelPicker: React.FC<any> = (props: ModelPickerProps) => {
props.setModelUsed(model); props.setModelUsed(model);
} }
fetch('/api/config/data/conversation/model' + '?id=' + String(model.id), { method: 'POST', body: JSON.stringify(model) }) fetch('/api/configure/chat/model' + '?id=' + String(model.id), { method: 'POST', body: JSON.stringify(model) })
.then((response) => { .then((response) => {
if (!response.ok) { if (!response.ok) {
throw new Error('Failed to select model'); throw new Error('Failed to select model');

4
src/interface/web/app/components/navMenu/navMenu.tsx
View File

@@ -128,7 +128,7 @@ export default function NavMenu(props: NavMenuProps) {
<DropdownMenuSeparator /> <DropdownMenuSeparator />
<DropdownMenuLabel>Profile</DropdownMenuLabel> <DropdownMenuLabel>Profile</DropdownMenuLabel>
<DropdownMenuItem> <DropdownMenuItem>
<Link href="/config">Settings</Link> <Link href="/settings">Settings</Link>
</DropdownMenuItem> </DropdownMenuItem>
<DropdownMenuItem> <DropdownMenuItem>
<Link href="https://docs.khoj.dev">Help</Link> <Link href="https://docs.khoj.dev">Help</Link>
@@ -172,7 +172,7 @@ export default function NavMenu(props: NavMenuProps) {
{userData && {userData &&
<> <>
<MenubarItem> <MenubarItem>
<Link href="/config"> <Link href="/settings">
Settings Settings
</Link> </Link>
</MenubarItem> </MenubarItem>

4
src/interface/web/app/components/sidePanel/chatHistorySidePanel.tsx
View File

@@ -148,7 +148,7 @@ interface FilesMenuProps {
function FilesMenu(props: FilesMenuProps) { function FilesMenu(props: FilesMenuProps) {
// Use SWR to fetch files // Use SWR to fetch files
const { data: files, error } = useSWR<string[]>(props.conversationId ? '/api/config/data/computer' : null, fetcher); const { data: files, error } = useSWR<string[]>(props.conversationId ? '/api/configure/content/computer' : null, fetcher);
const { data: selectedFiles, error: selectedFilesError } = useSWR(props.conversationId ? `/api/chat/conversation/file-filters/${props.conversationId}` : null, fetcher); const { data: selectedFiles, error: selectedFilesError } = useSWR(props.conversationId ? `/api/chat/conversation/file-filters/${props.conversationId}` : null, fetcher);
const [isOpen, setIsOpen] = useState(false); const [isOpen, setIsOpen] = useState(false);
const [unfilteredFiles, setUnfilteredFiles] = useState<string[]>([]); const [unfilteredFiles, setUnfilteredFiles] = useState<string[]>([]);
@@ -604,7 +604,7 @@ function UserProfileComponent(props: UserProfileProps) {
return ( return (
<div className={styles.profile}> <div className={styles.profile}>
<Link href="/config" target="_blank" rel="noopener noreferrer"> <Link href="/settings">
<Avatar> <Avatar>
<AvatarImage src={props.userProfile.photo} alt="user profile" /> <AvatarImage src={props.userProfile.photo} alt="user profile" />
<AvatarFallback> <AvatarFallback>

2
src/interface/web/app/factchecker/page.tsx
View File

@@ -533,7 +533,7 @@ export default function FactChecker() {
<Button disabled={clickedVerify} onClick={() => onClickVerify()}>Verify</Button> <Button disabled={clickedVerify} onClick={() => onClickVerify()}>Verify</Button>
</div> </div>
<h3 className={`mt-4 mb-4`}> <h3 className={`mt-4 mb-4`}>
Try with a particular model. You must be <a href="/config" className="font-medium text-blue-600 dark:text-blue-500 hover:underline">subscribed</a> to configure the model. Try with a particular model. You must be <a href="/settings" className="font-medium text-blue-600 dark:text-blue-500 hover:underline">subscribed</a> to configure the model.
</h3> </h3>
</div> </div>
} }

4
src/interface/web/app/share/chat/page.tsx
View File

@@ -299,7 +299,9 @@ export default function SharedChat() {
<SidePanel <SidePanel
webSocketConnected={!!conversationId ? (chatWS != null) : true} webSocketConnected={!!conversationId ? (chatWS != null) : true}
conversationId={conversationId ?? null} conversationId={conversationId ?? null}
uploadedFiles={uploadedFiles} /> uploadedFiles={uploadedFiles}
isMobileWidth={isMobileWidth}
/>
</div> </div>
<div className={styles.chatBox}> <div className={styles.chatBox}>

1
src/interface/web/package.json
View File

@@ -35,7 +35,6 @@
"@radix-ui/react-slot": "^1.1.0", "@radix-ui/react-slot": "^1.1.0",
"@radix-ui/react-toast": "^1.2.1", "@radix-ui/react-toast": "^1.2.1",
"@radix-ui/react-toggle": "^1.1.0", "@radix-ui/react-toggle": "^1.1.0",
"@tailwindcss/line-clamp": "^0.4.4",
"@types/dompurify": "^3.0.5", "@types/dompurify": "^3.0.5",
"@types/katex": "^0.16.7", "@types/katex": "^0.16.7",
"@types/markdown-it": "^14.1.1", "@types/markdown-it": "^14.1.1",

1
src/interface/web/tailwind.config.ts
View File

@@ -76,7 +76,6 @@ const config = {
}, },
plugins: [ plugins: [
require("tailwindcss-animate"), require("tailwindcss-animate"),
require('@tailwindcss/line-clamp'),
], ],
} satisfies Config } satisfies Config

5
src/interface/web/yarn.lock
View File

@@ -1090,11 +1090,6 @@
"@swc/counter" "^0.1.3" "@swc/counter" "^0.1.3"
tslib "^2.4.0" tslib "^2.4.0"
"@tailwindcss/line-clamp@^0.4.4":
version "0.4.4"
resolved "https://registry.yarnpkg.com/@tailwindcss/line-clamp/-/line-clamp-0.4.4.tgz#767cf8e5d528a5d90c9740ca66eb079f5e87d423"
integrity sha512-5U6SY5z8N42VtrCrKlsTAA35gy2VSyYtHWCsg1H87NU1SXnEfekTVlrga9fzUDrrHcGi2Lb5KenUWb4lRQT5/g==
"@ts-morph/common@~0.19.0": "@ts-morph/common@~0.19.0":
version "0.19.0" version "0.19.0"
resolved "https://registry.yarnpkg.com/@ts-morph/common/-/common-0.19.0.tgz#927fcd81d1bbc09c89c4a310a84577fb55f3694e" resolved "https://registry.yarnpkg.com/@ts-morph/common/-/common-0.19.0.tgz#927fcd81d1bbc09c89c4a310a84577fb55f3694e"

4
src/khoj/configure.py
View File

@@ -316,7 +316,7 @@ def configure_routes(app):
app.include_router(api, prefix="/api") app.include_router(api, prefix="/api")
app.include_router(api_chat, prefix="/api/chat") app.include_router(api_chat, prefix="/api/chat")
app.include_router(api_agents, prefix="/api/agents") app.include_router(api_agents, prefix="/api/agents")
app.include_router(api_config, prefix="/api/config") app.include_router(api_config, prefix="/api/configure")
app.include_router(indexer, prefix="/api/v1/index") app.include_router(indexer, prefix="/api/v1/index")
app.include_router(notion_router, prefix="/api/notion") app.include_router(notion_router, prefix="/api/notion")
app.include_router(web_client) app.include_router(web_client)
@@ -336,7 +336,7 @@ def configure_routes(app):
if is_twilio_enabled(): if is_twilio_enabled():
from khoj.routers.api_phone import api_phone from khoj.routers.api_phone import api_phone
app.include_router(api_phone, prefix="/api/config/phone") app.include_router(api_phone, prefix="/api/configure/phone")
logger.info("📞 Enabled Twilio") logger.info("📞 Enabled Twilio")

6
src/khoj/interface/web/chat.html
View File

@@ -44,7 +44,7 @@ Hi, I am Khoj, your open, personal AI 👋🏽. I can:
- 📚 Understand files you drag & drop here - 📚 Understand files you drag & drop here
- 👩🏾‍🚀 Be tuned to your conversation needs via [agents](./agents) - 👩🏾‍🚀 Be tuned to your conversation needs via [agents](./agents)
Get the Khoj [Desktop](https://khoj.dev/downloads), [Obsidian](https://docs.khoj.dev/clients/obsidian#setup), [Emacs](https://docs.khoj.dev/clients/emacs#setup) apps to search, chat with your 🖥️ computer docs. You can manage all the files you've shared with me at any time by going to [your settings](/config/content-source/computer/). Get the Khoj [Desktop](https://khoj.dev/downloads), [Obsidian](https://docs.khoj.dev/clients/obsidian#setup), [Emacs](https://docs.khoj.dev/clients/emacs#setup) apps to search, chat with your 🖥️ computer docs. You can manage all the files you've shared with me at any time by going to [your settings](/settings/content/computer/).
To get started, just start typing below. You can also type / to see a list of commands. To get started, just start typing below. You can also type / to see a list of commands.
`.trim() `.trim()
@@ -1333,7 +1333,7 @@ To get started, just start typing below. You can also type / to see a list of co
- 📚 Understand files you drag & drop here - 📚 Understand files you drag & drop here
- 👩🏾‍🚀 Be tuned to your conversation needs via [agents](./agents) - 👩🏾‍🚀 Be tuned to your conversation needs via [agents](./agents)
Get the Khoj [Desktop](https://khoj.dev/downloads), [Obsidian](https://docs.khoj.dev/clients/obsidian#setup), or [Emacs](https://docs.khoj.dev/clients/emacs#setup) app to keep your files in sync. You can manage all the files you've shared with me at any time by going to [your settings](/config/content-source/computer/). Get the Khoj [Desktop](https://khoj.dev/downloads), [Obsidian](https://docs.khoj.dev/clients/obsidian#setup), or [Emacs](https://docs.khoj.dev/clients/emacs#setup) app to keep your files in sync. You can manage all the files you've shared with me at any time by going to [your settings](/settings/content/computer/).
To get started, just start typing below. You can also type / to see a list of commands. To get started, just start typing below. You can also type / to see a list of commands.
@@ -1954,7 +1954,7 @@ To get started, just start typing below. You can also type / to see a list of co
} }
var allFiles; var allFiles;
function renderAllFiles() { function renderAllFiles() {
fetch('/api/config/data/computer') fetch('/api/configure/content/computer')
.then(response => response.json()) .then(response => response.json())
.then(data => { .then(data => {
var indexedFiles = document.getElementsByClassName("indexed-files")[0]; var indexedFiles = document.getElementsByClassName("indexed-files")[0];

6
src/khoj/interface/web/content_source_computer_input.html
View File

@@ -32,7 +32,7 @@
</style> </style>
<script> <script>
function removeFile(path) { function removeFile(path) {
fetch('/api/config/data/file?filename=' + path, { fetch('/api/configure/content/file?filename=' + path, {
method: 'DELETE', method: 'DELETE',
headers: { headers: {
'Content-Type': 'application/json', 'Content-Type': 'application/json',
@@ -48,7 +48,7 @@
// Get all currently indexed files // Get all currently indexed files
function getAllComputerFilenames() { function getAllComputerFilenames() {
fetch('/api/config/data/computer') fetch('/api/configure/content/computer')
.then(response => response.json()) .then(response => response.json())
.then(data => { .then(data => {
var indexedFiles = document.getElementsByClassName("indexed-files")[0]; var indexedFiles = document.getElementsByClassName("indexed-files")[0];
@@ -122,7 +122,7 @@
deleteAllComputerFilesButton.textContent = "🗑️ Deleting..."; deleteAllComputerFilesButton.textContent = "🗑️ Deleting...";
deleteAllComputerFilesButton.disabled = true; deleteAllComputerFilesButton.disabled = true;
fetch('/api/config/data/content-source/computer', { fetch('/api/configure/content/computer', {
method: 'DELETE', method: 'DELETE',
headers: { headers: {
'Content-Type': 'application/json', 'Content-Type': 'application/json',

2
src/khoj/interface/web/content_source_github_input.html
View File

@@ -165,7 +165,7 @@
// Save Github config on server // Save Github config on server
const csrfToken = document.cookie.split('; ').find(row => row.startsWith('csrftoken'))?.split('=')[1]; const csrfToken = document.cookie.split('; ').find(row => row.startsWith('csrftoken'))?.split('=')[1];
fetch('/api/config/data/content-source/github', { fetch('/api/configure/content/github', {
method: 'POST', method: 'POST',
headers: { headers: {
'Content-Type': 'application/json', 'Content-Type': 'application/json',

2
src/khoj/interface/web/content_source_notion_input.html
View File

@@ -45,7 +45,7 @@
// Save Notion config on server // Save Notion config on server
const csrfToken = document.cookie.split('; ').find(row => row.startsWith('csrftoken'))?.split('=')[1]; const csrfToken = document.cookie.split('; ').find(row => row.startsWith('csrftoken'))?.split('=')[1];
fetch('/api/config/data/content-source/notion', { fetch('/api/configure/content/notion', {
method: 'POST', method: 'POST',
headers: { headers: {
'Content-Type': 'application/json', 'Content-Type': 'application/json',

2
src/khoj/interface/web/public_conversation.html
View File

@@ -34,7 +34,7 @@ Hi, I am Khoj, your open, personal AI 👋🏽. I can:
- 📚 Understand files you drag & drop here - 📚 Understand files you drag & drop here
- 👩🏾‍🚀 Be tuned to your conversation needs via [agents](./agents) - 👩🏾‍🚀 Be tuned to your conversation needs via [agents](./agents)
Get the Khoj [Desktop](https://khoj.dev/downloads), [Obsidian](https://docs.khoj.dev/clients/obsidian#setup), [Emacs](https://docs.khoj.dev/clients/emacs#setup) apps to search, chat with your 🖥️ computer docs. You can manage all the files you've shared with me at any time by going to [your settings](/config/content-source/computer/). Get the Khoj [Desktop](https://khoj.dev/downloads), [Obsidian](https://docs.khoj.dev/clients/obsidian#setup), [Emacs](https://docs.khoj.dev/clients/emacs#setup) apps to search, chat with your 🖥️ computer docs. You can manage all the files you've shared with me at any time by going to [your settings](/settings/content/computer/).
To get started, just start typing below. You can also type / to see a list of commands. To get started, just start typing below. You can also type / to see a list of commands.
`.trim() `.trim()

4
src/khoj/interface/web/search.html
View File

@@ -209,12 +209,12 @@
function populate_type_dropdown() { function populate_type_dropdown() {
// Populate type dropdown field with enabled content types only // Populate type dropdown field with enabled content types only
fetch("/api/config/types") fetch("/api/configure/types")
.then(response => response.json()) .then(response => response.json())
.then(enabled_types => { .then(enabled_types => {
// Show warning if no content types are enabled, or just one ("all") // Show warning if no content types are enabled, or just one ("all")
if (enabled_types[0] === "all" && enabled_types.length === 1) { if (enabled_types[0] === "all" && enabled_types.length === 1) {
document.getElementById("results").innerHTML = "<div id='results-error'>To use Khoj search, setup your content plugins on the Khoj <a class='inline-chat-link' href='/config'>settings page</a>.</div>"; document.getElementById("results").innerHTML = "<div id='results-error'>To use Khoj search, setup your content plugins on the Khoj <a class='inline-chat-link' href='/settings'>settings page</a>.</div>";
document.getElementById("query").setAttribute("disabled", "disabled"); document.getElementById("query").setAttribute("disabled", "disabled");
document.getElementById("query").setAttribute("placeholder", "Configure Khoj to enable search"); document.getElementById("query").setAttribute("placeholder", "Configure Khoj to enable search");
return []; return [];

56
src/khoj/interface/web/config.html → src/khoj/interface/web/settings.html
View File

@@ -34,7 +34,7 @@
<h3 id="card-title-computer" class="card-title"> <h3 id="card-title-computer" class="card-title">
<span>Files</span> <span>Files</span>
<img id="configured-icon-computer" <img id="configured-icon-computer"
style="display: {% if not current_model_state.computer %}none{% endif %}" style="display: {% if not enabled_content_source.computer %}none{% endif %}"
class="configured-icon" class="configured-icon"
src="/static/assets/icons/confirm-icon.svg" src="/static/assets/icons/confirm-icon.svg"
alt="Configured"> alt="Configured">
@@ -44,8 +44,8 @@
<p class="card-description">Manage files from your computer</p> <p class="card-description">Manage files from your computer</p>
</div> </div>
<div class="card-action-row"> <div class="card-action-row">
<a class="card-button" href="/config/content-source/computer"> <a class="card-button" href="/settings/content/computer">
{% if current_model_state.computer %} {% if enabled_content_source.computer %}
Update Update
{% else %} {% else %}
Setup Setup
@@ -53,7 +53,7 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M12 5l7 7-7 7"></path></svg> <svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M12 5l7 7-7 7"></path></svg>
</a> </a>
<div id="clear-computer" class="card-action-row" <div id="clear-computer" class="card-action-row"
style="display: {% if not current_model_state.computer %}none{% endif %}"> style="display: {% if not enabled_content_source.computer %}none{% endif %}">
<button class="card-button" onclick="clearContentType('computer')"> <button class="card-button" onclick="clearContentType('computer')">
Disable Disable
</button> </button>
@@ -69,15 +69,15 @@
class="configured-icon" class="configured-icon"
src="/static/assets/icons/confirm-icon.svg" src="/static/assets/icons/confirm-icon.svg"
alt="Configured" alt="Configured"
style="display: {% if not current_model_state.github %}none{% endif %}"> style="display: {% if not enabled_content_source.github %}none{% endif %}">
</h3> </h3>
</div> </div>
<div class="card-description-row"> <div class="card-description-row">
<p class="card-description">Set repositories to index</p> <p class="card-description">Set repositories to index</p>
</div> </div>
<div class="card-action-row"> <div class="card-action-row">
<a class="card-button" href="/config/content-source/github"> <a class="card-button" href="/settings/content/github">
{% if current_model_state.github %} {% if enabled_content_source.github %}
Update Update
{% else %} {% else %}
Setup Setup
@@ -86,7 +86,7 @@
</a> </a>
<div id="clear-github" <div id="clear-github"
class="card-action-row" class="card-action-row"
style="display: {% if not current_model_state.github %}none{% endif %}"> style="display: {% if not enabled_content_source.github %}none{% endif %}">
<button class="card-button" onclick="clearContentType('github')"> <button class="card-button" onclick="clearContentType('github')">
Disable Disable
</button> </button>
@@ -102,15 +102,15 @@
class="configured-icon" class="configured-icon"
src="/static/assets/icons/confirm-icon.svg" src="/static/assets/icons/confirm-icon.svg"
alt="Configured" alt="Configured"
style="display: {% if not current_model_state.notion %}none{% endif %}"> style="display: {% if not enabled_content_source.notion %}none{% endif %}">
</h3> </h3>
</div> </div>
<div class="card-description-row"> <div class="card-description-row">
<p class="card-description">Sync your Notion pages</p> <p class="card-description">Sync your Notion pages</p>
</div> </div>
<div class="card-action-row"> <div class="card-action-row">
{% if current_model_state.notion %} {% if enabled_content_source.notion %}
<a class="card-button" href="/config/content-source/notion"> <a class="card-button" href="/settings/content/notion">
Update Update
<svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M12 5l7 7-7 7"></path></svg> <svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M12 5l7 7-7 7"></path></svg>
</a> </a>
@@ -120,7 +120,7 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M12 5l7 7-7 7"></path></svg> <svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M12 5l7 7-7 7"></path></svg>
</a> </a>
{% else %} {% else %}
<a class="card-button" href="/config/content-source/notion"> <a class="card-button" href="/settings/content/notion">
Setup Setup
<svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M12 5l7 7-7 7"></path></svg> <svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M12 5l7 7-7 7"></path></svg>
</a> </a>
@@ -128,7 +128,7 @@
<div id="clear-notion" <div id="clear-notion"
class="card-action-row" class="card-action-row"
style="display: {% if not current_model_state.notion %}none{% endif %}"> style="display: {% if not enabled_content_source.notion %}none{% endif %}">
<button class="card-button" onclick="clearContentType('notion')"> <button class="card-button" onclick="clearContentType('notion')">
Disable Disable
</button> </button>
@@ -181,8 +181,8 @@
</div> </div>
<div class="card-description-row"> <div class="card-description-row">
<select id="chat-models"> <select id="chat-models">
{% for option in conversation_options %} {% for option in chat_model_options %}
<option value="{{ option.id }}" {% if option.id == selected_conversation_config %}selected{% endif %}>{{ option.chat_model }}</option> <option value="{{ option.id }}" {% if option.id == selected_chat_model_config %}selected{% endif %}>{{ option.name }}</option>
{% endfor %} {% endfor %}
</select> </select>
</div> </div>
@@ -208,7 +208,7 @@
<div class="card-description-row"> <div class="card-description-row">
<select id="paint-models"> <select id="paint-models">
{% for option in paint_model_options %} {% for option in paint_model_options %}
<option value="{{ option.id }}" {% if option.id == selected_paint_model_config %}selected{% endif %}>{{ option.model_name }}</option> <option value="{{ option.id }}" {% if option.id == selected_paint_model_config %}selected{% endif %}>{{ option.name }}</option>
{% endfor %} {% endfor %}
</select> </select>
</div> </div>
@@ -394,7 +394,7 @@
function saveProfileGivenName() { function saveProfileGivenName() {
const givenName = document.getElementById("profile_given_name").value; const givenName = document.getElementById("profile_given_name").value;
fetch('/api/config/user/name?name=' + givenName, { fetch('/api/configure/user/name?name=' + givenName, {
method: 'POST', method: 'POST',
headers: { headers: {
'Content-Type': 'application/json', 'Content-Type': 'application/json',
@@ -421,7 +421,7 @@
saveVoiceModelButton.disabled = true; saveVoiceModelButton.disabled = true;
saveVoiceModelButton.textContent = "Saving..."; saveVoiceModelButton.textContent = "Saving...";
fetch('/api/config/data/voice/model?id=' + voiceModel, { fetch('/api/configure/voice/model?id=' + voiceModel, {
method: 'POST', method: 'POST',
headers: { headers: {
'Content-Type': 'application/json', 'Content-Type': 'application/json',
@@ -455,7 +455,7 @@
saveModelButton.innerHTML = ""; saveModelButton.innerHTML = "";
saveModelButton.textContent = "Saving..."; saveModelButton.textContent = "Saving...";
fetch('/api/config/data/conversation/model?id=' + chatModel, { fetch('/api/configure/chat/model?id=' + chatModel, {
method: 'POST', method: 'POST',
headers: { headers: {
'Content-Type': 'application/json', 'Content-Type': 'application/json',
@@ -494,7 +494,7 @@
saveSearchModelButton.disabled = true; saveSearchModelButton.disabled = true;
saveSearchModelButton.textContent = "Saving..."; saveSearchModelButton.textContent = "Saving...";
fetch('/api/config/data/search/model?id=' + searchModel, { fetch('/api/configure/search/model?id=' + searchModel, {
method: 'POST', method: 'POST',
headers: { headers: {
'Content-Type': 'application/json', 'Content-Type': 'application/json',
@@ -526,7 +526,7 @@
saveModelButton.disabled = true; saveModelButton.disabled = true;
saveModelButton.innerHTML = "Saving..."; saveModelButton.innerHTML = "Saving...";
fetch('/api/config/data/paint/model?id=' + paintModel, { fetch('/api/configure/paint/model?id=' + paintModel, {
method: 'POST', method: 'POST',
headers: { headers: {
'Content-Type': 'application/json', 'Content-Type': 'application/json',
@@ -553,7 +553,7 @@
}; };
function clearContentType(content_source) { function clearContentType(content_source) {
fetch('/api/config/data/content-source/' + content_source, { fetch('/api/configure/content/' + content_source, {
method: 'DELETE', method: 'DELETE',
headers: { headers: {
'Content-Type': 'application/json', 'Content-Type': 'application/json',
@@ -676,7 +676,7 @@
content_sources = ["computer", "github", "notion"]; content_sources = ["computer", "github", "notion"];
content_sources.forEach(content_source => { content_sources.forEach(content_source => {
fetch(`/api/config/data/${content_source}`, { fetch(`/api/configure/content/${content_source}`, {
method: 'GET', method: 'GET',
headers: { headers: {
'Content-Type': 'application/json', 'Content-Type': 'application/json',
@@ -807,7 +807,7 @@
function getIndexedDataSize() { function getIndexedDataSize() {
document.getElementById("indexed-data-size").textContent = "Calculating..."; document.getElementById("indexed-data-size").textContent = "Calculating...";
fetch('/api/config/index/size') fetch('/api/configure/content/size')
.then(response => response.json()) .then(response => response.json())
.then(data => { .then(data => {
document.getElementById("indexed-data-size").textContent = data.indexed_data_size_in_mb + " MB used"; document.getElementById("indexed-data-size").textContent = data.indexed_data_size_in_mb + " MB used";
@@ -815,7 +815,7 @@
} }
function removeFile(path) { function removeFile(path) {
fetch('/api/config/data/file?filename=' + path, { fetch('/api/configure/content/file?filename=' + path, {
method: 'DELETE', method: 'DELETE',
headers: { headers: {
'Content-Type': 'application/json', 'Content-Type': 'application/json',
@@ -890,7 +890,7 @@
}) })
phonenumberRemoveButton.addEventListener("click", () => { phonenumberRemoveButton.addEventListener("click", () => {
fetch('/api/config/phone', { fetch('/api/configure/phone', {
method: 'DELETE', method: 'DELETE',
headers: { headers: {
'Content-Type': 'application/json', 'Content-Type': 'application/json',
@@ -917,7 +917,7 @@
}, 5000); }, 5000);
} else { } else {
const mobileNumber = iti.getNumber(); const mobileNumber = iti.getNumber();
fetch('/api/config/phone?phone_number=' + mobileNumber, { fetch('/api/configure/phone?phone_number=' + mobileNumber, {
method: 'POST', method: 'POST',
headers: { headers: {
'Content-Type': 'application/json', 'Content-Type': 'application/json',
@@ -970,7 +970,7 @@
return; return;
} }
fetch('/api/config/phone/verify?code=' + otp, { fetch('/api/configure/phone/verify?code=' + otp, {
method: 'POST', method: 'POST',
headers: { headers: {
'Content-Type': 'application/json', 'Content-Type': 'application/json',

2
src/khoj/interface/web/utils.html
View File

@@ -36,7 +36,7 @@
{% endif %} {% endif %}
<div id="khoj-nav-menu" class="khoj-nav-dropdown-content"> <div id="khoj-nav-menu" class="khoj-nav-dropdown-content">
<div class="khoj-nav-username"> {{ username }} </div> <div class="khoj-nav-username"> {{ username }} </div>
<a id="settings-nav" class="khoj-nav" href="/config">Settings</a> <a id="settings-nav" class="khoj-nav" href="/settings">Settings</a>
<a id="github-nav" class="khoj-nav" href="https://github.com/khoj-ai/khoj">GitHub</a> <a id="github-nav" class="khoj-nav" href="https://github.com/khoj-ai/khoj">GitHub</a>
<a id="help-nav" class="khoj-nav" href="https://docs.khoj.dev" target="_blank">Help</a> <a id="help-nav" class="khoj-nav" href="https://docs.khoj.dev" target="_blank">Help</a>
<a class="khoj-nav" href="/auth/logout">Logout</a> <a class="khoj-nav" href="/auth/logout">Logout</a>

3
src/khoj/routers/api.py
View File

@@ -6,7 +6,6 @@ import os
import threading import threading
import time import time
import uuid import uuid
from random import random
from typing import Any, Callable, List, Optional, Union from typing import Any, Callable, List, Optional, Union
import cron_descriptor import cron_descriptor
@@ -190,7 +189,7 @@ def update(
): ):
user = request.user.object user = request.user.object
if not state.config: if not state.config:
error_msg = f"🚨 Khoj is not configured.\nConfigure it via http://localhost:42110/config, plugins or by editing {state.config_file}." error_msg = f"🚨 Khoj is not configured.\nConfigure it via http://localhost:42110/settings, plugins or by editing {state.config_file}."
logger.warning(error_msg) logger.warning(error_msg)
raise HTTPException(status_code=500, detail=error_msg) raise HTTPException(status_code=500, detail=error_msg)
try: try:

36
src/khoj/routers/api_config.py
View File

@@ -98,9 +98,9 @@ def _initialize_config():
state.config.search_type = SearchConfig.model_validate(constants.default_config["search-type"]) state.config.search_type = SearchConfig.model_validate(constants.default_config["search-type"])
@api_config.post("/data/content-source/github", status_code=200) @api_config.post("/content/github", status_code=200)
@requires(["authenticated"]) @requires(["authenticated"])
async def set_content_config_github_data( async def set_content_github(
request: Request, request: Request,
updated_config: Union[GithubContentConfig, None], updated_config: Union[GithubContentConfig, None],
client: Optional[str] = None, client: Optional[str] = None,
@@ -130,9 +130,9 @@ async def set_content_config_github_data(
return {"status": "ok"} return {"status": "ok"}
@api_config.post("/data/content-source/notion", status_code=200) @api_config.post("/content/notion", status_code=200)
@requires(["authenticated"]) @requires(["authenticated"])
async def set_content_config_notion_data( async def set_content_notion(
request: Request, request: Request,
updated_config: Union[NotionContentConfig, None], updated_config: Union[NotionContentConfig, None],
client: Optional[str] = None, client: Optional[str] = None,
@@ -161,9 +161,9 @@ async def set_content_config_notion_data(
return {"status": "ok"} return {"status": "ok"}
@api_config.delete("/data/content-source/{content_source}", status_code=200) @api_config.delete("/content/{content_source}", status_code=200)
@requires(["authenticated"]) @requires(["authenticated"])
async def remove_content_source_data( async def delete_content_source(
request: Request, request: Request,
content_source: str, content_source: str,
client: Optional[str] = None, client: Optional[str] = None,
@@ -189,9 +189,9 @@ async def remove_content_source_data(
return {"status": "ok"} return {"status": "ok"}
@api_config.delete("/data/file", status_code=200) @api_config.delete("/content/file", status_code=201)
@requires(["authenticated"]) @requires(["authenticated"])
async def remove_file_data( async def delete_content_file(
request: Request, request: Request,
filename: str, filename: str,
client: Optional[str] = None, client: Optional[str] = None,
@@ -210,9 +210,9 @@ async def remove_file_data(
return {"status": "ok"} return {"status": "ok"}
@api_config.get("/data/{content_source}", response_model=List[str]) @api_config.get("/content/{content_source}", response_model=List[str])
@requires(["authenticated"]) @requires(["authenticated"])
async def get_all_filenames( async def get_content_source(
request: Request, request: Request,
content_source: str, content_source: str,
client: Optional[str] = None, client: Optional[str] = None,
@@ -229,7 +229,7 @@ async def get_all_filenames(
return await sync_to_async(list)(EntryAdapters.get_all_filenames_by_source(user, content_source)) # type: ignore[call-arg] return await sync_to_async(list)(EntryAdapters.get_all_filenames_by_source(user, content_source)) # type: ignore[call-arg]
@api_config.get("/data/conversation/model/options", response_model=Dict[str, Union[str, int]]) @api_config.get("/chat/model/options", response_model=Dict[str, Union[str, int]])
def get_chat_model_options( def get_chat_model_options(
request: Request, request: Request,
client: Optional[str] = None, client: Optional[str] = None,
@@ -243,7 +243,7 @@ def get_chat_model_options(
return Response(content=json.dumps(all_conversation_options), media_type="application/json", status_code=200) return Response(content=json.dumps(all_conversation_options), media_type="application/json", status_code=200)
@api_config.get("/data/conversation/model") @api_config.get("/chat/model")
@requires(["authenticated"]) @requires(["authenticated"])
def get_user_chat_model( def get_user_chat_model(
request: Request, request: Request,
@@ -259,7 +259,7 @@ def get_user_chat_model(
return Response(status_code=200, content=json.dumps({"id": chat_model.id, "chat_model": chat_model.chat_model})) return Response(status_code=200, content=json.dumps({"id": chat_model.id, "chat_model": chat_model.chat_model}))
@api_config.post("/data/conversation/model", status_code=200) @api_config.post("/chat/model", status_code=200)
@requires(["authenticated", "premium"]) @requires(["authenticated", "premium"])
async def update_chat_model( async def update_chat_model(
request: Request, request: Request,
@@ -284,7 +284,7 @@ async def update_chat_model(
return {"status": "ok"} return {"status": "ok"}
@api_config.post("/data/voice/model", status_code=200) @api_config.post("/voice/model", status_code=200)
@requires(["authenticated", "premium"]) @requires(["authenticated", "premium"])
async def update_voice_model( async def update_voice_model(
request: Request, request: Request,
@@ -308,7 +308,7 @@ async def update_voice_model(
return Response(status_code=202, content=json.dumps({"status": "ok"})) return Response(status_code=202, content=json.dumps({"status": "ok"}))
@api_config.post("/data/search/model", status_code=200) @api_config.post("/search/model", status_code=200)
@requires(["authenticated"]) @requires(["authenticated"])
async def update_search_model( async def update_search_model(
request: Request, request: Request,
@@ -341,7 +341,7 @@ async def update_search_model(
return {"status": "ok"} return {"status": "ok"}
@api_config.post("/data/paint/model", status_code=200) @api_config.post("/paint/model", status_code=200)
@requires(["authenticated"]) @requires(["authenticated"])
async def update_paint_model( async def update_paint_model(
request: Request, request: Request,
@@ -370,9 +370,9 @@ async def update_paint_model(
return {"status": "ok"} return {"status": "ok"}
@api_config.get("/index/size", response_model=Dict[str, int]) @api_config.get("/content/size", response_model=Dict[str, int])
@requires(["authenticated"]) @requires(["authenticated"])
async def get_indexed_data_size(request: Request, common: CommonQueryParams): async def get_content_size(request: Request, common: CommonQueryParams):
user = request.user.object user = request.user.object
indexed_data_size_in_mb = await sync_to_async(EntryAdapters.get_size_of_indexed_data_in_mb)(user) indexed_data_size_in_mb = await sync_to_async(EntryAdapters.get_size_of_indexed_data_in_mb)(user)
return Response( return Response(

308
src/khoj/routers/helpers.py
View File

@@ -5,6 +5,7 @@ import io
import json import json
import logging import logging
import math import math
import os
import re import re
from concurrent.futures import ThreadPoolExecutor from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timedelta, timezone from datetime import datetime, timedelta, timezone
@@ -35,6 +36,7 @@ from PIL import Image
from starlette.authentication import has_required_scope from starlette.authentication import has_required_scope
from starlette.requests import URL from starlette.requests import URL
from khoj.database import adapters
from khoj.database.adapters import ( from khoj.database.adapters import (
AgentAdapters, AgentAdapters,
AutomationAdapters, AutomationAdapters,
@@ -42,18 +44,30 @@ from khoj.database.adapters import (
EntryAdapters, EntryAdapters,
create_khoj_token, create_khoj_token,
get_khoj_tokens, get_khoj_tokens,
get_user_name,
get_user_subscription_state,
run_with_process_lock, run_with_process_lock,
) )
from khoj.database.models import ( from khoj.database.models import (
ChatModelOptions, ChatModelOptions,
ClientApplication, ClientApplication,
Conversation, Conversation,
GithubConfig,
KhojUser, KhojUser,
NotionConfig,
ProcessLock, ProcessLock,
Subscription, Subscription,
TextToImageModelConfig, TextToImageModelConfig,
UserRequests, UserRequests,
) )
from khoj.processor.content.docx.docx_to_entries import DocxToEntries
from khoj.processor.content.github.github_to_entries import GithubToEntries
from khoj.processor.content.images.image_to_entries import ImageToEntries
from khoj.processor.content.markdown.markdown_to_entries import MarkdownToEntries
from khoj.processor.content.notion.notion_to_entries import NotionToEntries
from khoj.processor.content.org_mode.org_to_entries import OrgToEntries
from khoj.processor.content.pdf.pdf_to_entries import PdfToEntries
from khoj.processor.content.plaintext.plaintext_to_entries import PlaintextToEntries
from khoj.processor.conversation import prompts from khoj.processor.conversation import prompts
from khoj.processor.conversation.anthropic.anthropic_chat import ( from khoj.processor.conversation.anthropic.anthropic_chat import (
anthropic_send_message_to_model, anthropic_send_message_to_model,
@@ -69,11 +83,15 @@ from khoj.processor.conversation.utils import (
generate_chatml_messages_with_context, generate_chatml_messages_with_context,
save_to_conversation_log, save_to_conversation_log,
) )
from khoj.processor.speech.text_to_speech import is_eleven_labs_enabled
from khoj.routers.email import is_resend_enabled, send_task_email from khoj.routers.email import is_resend_enabled, send_task_email
from khoj.routers.storage import upload_image from khoj.routers.storage import upload_image
from khoj.routers.twilio import is_twilio_enabled
from khoj.search_type import text_search
from khoj.utils import state from khoj.utils import state
from khoj.utils.config import OfflineChatProcessorModel from khoj.utils.config import OfflineChatProcessorModel
from khoj.utils.helpers import ( from khoj.utils.helpers import (
LRU,
ConversationCommand, ConversationCommand,
ImageIntentType, ImageIntentType,
is_none_or_empty, is_none_or_empty,
@@ -90,6 +108,11 @@ logger = logging.getLogger(__name__)
executor = ThreadPoolExecutor(max_workers=1) executor = ThreadPoolExecutor(max_workers=1)
NOTION_OAUTH_CLIENT_ID = os.getenv("NOTION_OAUTH_CLIENT_ID")
NOTION_OAUTH_CLIENT_SECRET = os.getenv("NOTION_OAUTH_CLIENT_SECRET")
NOTION_REDIRECT_URI = os.getenv("NOTION_REDIRECT_URI")
def is_query_empty(query: str) -> bool: def is_query_empty(query: str) -> bool:
return is_none_or_empty(query.strip()) return is_none_or_empty(query.strip())
@@ -902,7 +925,7 @@ class ApiUserRateLimiter:
) )
raise HTTPException( raise HTTPException(
status_code=429, status_code=429,
detail="We're glad you're enjoying Khoj! You've exceeded your usage limit for today. Come back tomorrow or subscribe to increase your usage limit via [your settings](https://app.khoj.dev/config).", detail="We're glad you're enjoying Khoj! You've exceeded your usage limit for today. Come back tomorrow or subscribe to increase your usage limit via [your settings](https://app.khoj.dev/settings).",
) )
# Add the current request to the cache # Add the current request to the cache
@@ -941,7 +964,7 @@ class ConversationCommandRateLimiter:
if not subscribed and count_requests >= self.trial_rate_limit: if not subscribed and count_requests >= self.trial_rate_limit:
raise HTTPException( raise HTTPException(
status_code=429, status_code=429,
detail=f"We're glad you're enjoying Khoj! You've exceeded your `/{conversation_command.value}` command usage limit for today. Subscribe to increase your usage limit via [your settings](https://app.khoj.dev/config).", detail=f"We're glad you're enjoying Khoj! You've exceeded your `/{conversation_command.value}` command usage limit for today. Subscribe to increase your usage limit via [your settings](https://app.khoj.dev/settings).",
) )
await UserRequests.objects.acreate(user=user, slug=command_slug) await UserRequests.objects.acreate(user=user, slug=command_slug)
return return
@@ -1186,3 +1209,284 @@ def construct_automation_created_message(automation: Job, crontime: str, query_t
Manage your automations [here](/automations). Manage your automations [here](/automations).
""".strip() """.strip()
def get_user_config(user: KhojUser, request: Request, is_detailed: bool = False):
user_picture = request.session.get("user", {}).get("picture")
is_active = has_required_scope(request, ["premium"])
has_documents = EntryAdapters.user_has_entries(user=user)
if not is_detailed:
return {
"request": request,
"username": user.username if user else None,
"user_photo": user_picture,
"is_active": is_active,
"has_documents": has_documents,
"khoj_version": state.khoj_version,
}
user_subscription_state = get_user_subscription_state(user.email)
user_subscription = adapters.get_user_subscription(user.email)
subscription_renewal_date = (
user_subscription.renewal_date.strftime("%d %b %Y")
if user_subscription and user_subscription.renewal_date
else (user_subscription.created_at + timedelta(days=7)).strftime("%d %b %Y")
)
given_name = get_user_name(user)
enabled_content_sources_set = set(EntryAdapters.get_unique_file_sources(user))
enabled_content_sources = {
"computer": ("computer" in enabled_content_sources_set),
"github": ("github" in enabled_content_sources_set),
"notion": ("notion" in enabled_content_sources_set),
}
selected_chat_model_config = ConversationAdapters.get_conversation_config(user)
chat_models = ConversationAdapters.get_conversation_processor_options().all()
chat_model_options = list()
for chat_model in chat_models:
chat_model_options.append({"name": chat_model.chat_model, "id": chat_model.id})
search_model_options = adapters.get_or_create_search_models().all()
all_search_model_options = list()
for search_model_option in search_model_options:
all_search_model_options.append({"name": search_model_option.name, "id": search_model_option.id})
current_search_model_option = adapters.get_user_search_model_or_default(user)
selected_paint_model_config = ConversationAdapters.get_user_text_to_image_model_config(user)
paint_model_options = ConversationAdapters.get_text_to_image_model_options().all()
all_paint_model_options = list()
for paint_model in paint_model_options:
all_paint_model_options.append({"name": paint_model.model_name, "id": paint_model.id})
notion_oauth_url = get_notion_auth_url(user)
eleven_labs_enabled = is_eleven_labs_enabled()
voice_models = ConversationAdapters.get_voice_model_options()
voice_model_options = list()
for voice_model in voice_models:
voice_model_options.append({"name": voice_model.name, "id": voice_model.model_id})
if len(voice_model_options) == 0:
eleven_labs_enabled = False
selected_voice_config = ConversationAdapters.get_voice_model_config(user)
return {
"request": request,
# user info
"username": user.username if user else None,
"user_photo": user_picture,
"is_active": is_active,
"given_name": given_name,
"phone_number": user.phone_number,
"is_phone_number_verified": user.verified_phone_number,
# user content, model settings
"enabled_content_source": enabled_content_sources,
"has_documents": has_documents,
"search_model_options": all_search_model_options,
"selected_search_model_config": current_search_model_option.id,
"chat_model_options": chat_model_options,
"selected_chat_model_config": selected_chat_model_config.id if selected_chat_model_config else None,
"paint_model_options": all_paint_model_options,
"selected_paint_model_config": selected_paint_model_config.id if selected_paint_model_config else None,
"voice_model_options": voice_model_options,
"selected_voice_config": selected_voice_config.model_id if selected_voice_config else None,
# user billing info
"subscription_state": user_subscription_state,
"subscription_renewal_date": subscription_renewal_date,
# server settings
"khoj_cloud_subscription_url": os.getenv("KHOJ_CLOUD_SUBSCRIPTION_URL"),
"billing_enabled": state.billing_enabled,
"is_eleven_labs_enabled": eleven_labs_enabled,
"is_twilio_enabled": is_twilio_enabled(),
"khoj_version": state.khoj_version,
"anonymous_mode": state.anonymous_mode,
"notion_oauth_url": notion_oauth_url,
}
def configure_content(
files: Optional[dict[str, dict[str, str]]],
regenerate: bool = False,
t: Optional[state.SearchType] = state.SearchType.All,
full_corpus: bool = True,
user: KhojUser = None,
) -> bool:
success = True
if t == None:
t = state.SearchType.All
if t is not None and t in [type.value for type in state.SearchType]:
t = state.SearchType(t)
if t is not None and not t.value in [type.value for type in state.SearchType]:
logger.warning(f"🚨 Invalid search type: {t}")
return False
search_type = t.value if t else None
no_documents = all([not files.get(file_type) for file_type in files])
if files is None:
logger.warning(f"🚨 No files to process for {search_type} search.")
return True
try:
# Initialize Org Notes Search
if (search_type == state.SearchType.All.value or search_type == state.SearchType.Org.value) and files["org"]:
logger.info("🦄 Setting up search for orgmode notes")
# Extract Entries, Generate Notes Embeddings
text_search.setup(
OrgToEntries,
files.get("org"),
regenerate=regenerate,
full_corpus=full_corpus,
user=user,
)
except Exception as e:
logger.error(f"🚨 Failed to setup org: {e}", exc_info=True)
success = False
try:
# Initialize Markdown Search
if (search_type == state.SearchType.All.value or search_type == state.SearchType.Markdown.value) and files[
"markdown"
]:
logger.info("💎 Setting up search for markdown notes")
# Extract Entries, Generate Markdown Embeddings
text_search.setup(
MarkdownToEntries,
files.get("markdown"),
regenerate=regenerate,
full_corpus=full_corpus,
user=user,
)
except Exception as e:
logger.error(f"🚨 Failed to setup markdown: {e}", exc_info=True)
success = False
try:
# Initialize PDF Search
if (search_type == state.SearchType.All.value or search_type == state.SearchType.Pdf.value) and files["pdf"]:
logger.info("🖨️ Setting up search for pdf")
# Extract Entries, Generate PDF Embeddings
text_search.setup(
PdfToEntries,
files.get("pdf"),
regenerate=regenerate,
full_corpus=full_corpus,
user=user,
)
except Exception as e:
logger.error(f"🚨 Failed to setup PDF: {e}", exc_info=True)
success = False
try:
# Initialize Plaintext Search
if (search_type == state.SearchType.All.value or search_type == state.SearchType.Plaintext.value) and files[
"plaintext"
]:
logger.info("📄 Setting up search for plaintext")
# Extract Entries, Generate Plaintext Embeddings
text_search.setup(
PlaintextToEntries,
files.get("plaintext"),
regenerate=regenerate,
full_corpus=full_corpus,
user=user,
)
except Exception as e:
logger.error(f"🚨 Failed to setup plaintext: {e}", exc_info=True)
success = False
try:
if no_documents:
github_config = GithubConfig.objects.filter(user=user).prefetch_related("githubrepoconfig").first()
if (
search_type == state.SearchType.All.value or search_type == state.SearchType.Github.value
) and github_config is not None:
logger.info("🐙 Setting up search for github")
# Extract Entries, Generate Github Embeddings
text_search.setup(
GithubToEntries,
None,
regenerate=regenerate,
full_corpus=full_corpus,
user=user,
config=github_config,
)
except Exception as e:
logger.error(f"🚨 Failed to setup GitHub: {e}", exc_info=True)
success = False
try:
if no_documents:
# Initialize Notion Search
notion_config = NotionConfig.objects.filter(user=user).first()
if (
search_type == state.SearchType.All.value or search_type == state.SearchType.Notion.value
) and notion_config:
logger.info("🔌 Setting up search for notion")
text_search.setup(
NotionToEntries,
None,
regenerate=regenerate,
full_corpus=full_corpus,
user=user,
config=notion_config,
)
except Exception as e:
logger.error(f"🚨 Failed to setup Notion: {e}", exc_info=True)
success = False
try:
# Initialize Image Search
if (search_type == state.SearchType.All.value or search_type == state.SearchType.Image.value) and files[
"image"
]:
logger.info("🖼️ Setting up search for images")
# Extract Entries, Generate Image Embeddings
text_search.setup(
ImageToEntries,
files.get("image"),
regenerate=regenerate,
full_corpus=full_corpus,
user=user,
)
except Exception as e:
logger.error(f"🚨 Failed to setup images: {e}", exc_info=True)
success = False
try:
if (search_type == state.SearchType.All.value or search_type == state.SearchType.Docx.value) and files["docx"]:
logger.info("📄 Setting up search for docx")
text_search.setup(
DocxToEntries,
files.get("docx"),
regenerate=regenerate,
full_corpus=full_corpus,
user=user,
)
except Exception as e:
logger.error(f"🚨 Failed to setup docx: {e}", exc_info=True)
success = False
# Invalidate Query Cache
if user:
state.query_cache[user.uuid] = LRU()
return success
def get_notion_auth_url(user: KhojUser):
if not NOTION_OAUTH_CLIENT_ID or not NOTION_OAUTH_CLIENT_SECRET or not NOTION_REDIRECT_URI:
return None
return f"https://api.notion.com/v1/oauth/authorize?client_id={NOTION_OAUTH_CLIENT_ID}&redirect_uri={NOTION_REDIRECT_URI}&response_type=code&state={user.uuid}"

195
src/khoj/routers/indexer.py
View File

@@ -6,20 +6,14 @@ from fastapi import APIRouter, Depends, Header, Request, Response, UploadFile
from pydantic import BaseModel from pydantic import BaseModel
from starlette.authentication import requires from starlette.authentication import requires
from khoj.database.models import GithubConfig, KhojUser, NotionConfig from khoj.routers.helpers import (
from khoj.processor.content.docx.docx_to_entries import DocxToEntries ApiIndexedDataLimiter,
from khoj.processor.content.github.github_to_entries import GithubToEntries configure_content,
from khoj.processor.content.images.image_to_entries import ImageToEntries update_telemetry_state,
from khoj.processor.content.markdown.markdown_to_entries import MarkdownToEntries )
from khoj.processor.content.notion.notion_to_entries import NotionToEntries
from khoj.processor.content.org_mode.org_to_entries import OrgToEntries
from khoj.processor.content.pdf.pdf_to_entries import PdfToEntries
from khoj.processor.content.plaintext.plaintext_to_entries import PlaintextToEntries
from khoj.routers.helpers import ApiIndexedDataLimiter, update_telemetry_state
from khoj.search_type import text_search
from khoj.utils import constants, state from khoj.utils import constants, state
from khoj.utils.config import SearchModels from khoj.utils.config import SearchModels
from khoj.utils.helpers import LRU, get_file_type from khoj.utils.helpers import get_file_type
from khoj.utils.rawconfig import ContentConfig, FullConfig, SearchConfig from khoj.utils.rawconfig import ContentConfig, FullConfig, SearchConfig
from khoj.utils.yaml import save_config_to_file_updated_state from khoj.utils.yaml import save_config_to_file_updated_state
@@ -170,180 +164,3 @@ def configure_search(search_models: SearchModels, search_config: Optional[Search
search_models = SearchModels() search_models = SearchModels()
return search_models return search_models
def configure_content(
files: Optional[dict[str, dict[str, str]]],
regenerate: bool = False,
t: Optional[state.SearchType] = state.SearchType.All,
full_corpus: bool = True,
user: KhojUser = None,
) -> bool:
success = True
if t == None:
t = state.SearchType.All
if t is not None and t in [type.value for type in state.SearchType]:
t = state.SearchType(t)
if t is not None and not t.value in [type.value for type in state.SearchType]:
logger.warning(f"🚨 Invalid search type: {t}")
return False
search_type = t.value if t else None
no_documents = all([not files.get(file_type) for file_type in files])
if files is None:
logger.warning(f"🚨 No files to process for {search_type} search.")
return True
try:
# Initialize Org Notes Search
if (search_type == state.SearchType.All.value or search_type == state.SearchType.Org.value) and files["org"]:
logger.info("🦄 Setting up search for orgmode notes")
# Extract Entries, Generate Notes Embeddings
text_search.setup(
OrgToEntries,
files.get("org"),
regenerate=regenerate,
full_corpus=full_corpus,
user=user,
)
except Exception as e:
logger.error(f"🚨 Failed to setup org: {e}", exc_info=True)
success = False
try:
# Initialize Markdown Search
if (search_type == state.SearchType.All.value or search_type == state.SearchType.Markdown.value) and files[
"markdown"
]:
logger.info("💎 Setting up search for markdown notes")
# Extract Entries, Generate Markdown Embeddings
text_search.setup(
MarkdownToEntries,
files.get("markdown"),
regenerate=regenerate,
full_corpus=full_corpus,
user=user,
)
except Exception as e:
logger.error(f"🚨 Failed to setup markdown: {e}", exc_info=True)
success = False
try:
# Initialize PDF Search
if (search_type == state.SearchType.All.value or search_type == state.SearchType.Pdf.value) and files["pdf"]:
logger.info("🖨️ Setting up search for pdf")
# Extract Entries, Generate PDF Embeddings
text_search.setup(
PdfToEntries,
files.get("pdf"),
regenerate=regenerate,
full_corpus=full_corpus,
user=user,
)
except Exception as e:
logger.error(f"🚨 Failed to setup PDF: {e}", exc_info=True)
success = False
try:
# Initialize Plaintext Search
if (search_type == state.SearchType.All.value or search_type == state.SearchType.Plaintext.value) and files[
"plaintext"
]:
logger.info("📄 Setting up search for plaintext")
# Extract Entries, Generate Plaintext Embeddings
text_search.setup(
PlaintextToEntries,
files.get("plaintext"),
regenerate=regenerate,
full_corpus=full_corpus,
user=user,
)
except Exception as e:
logger.error(f"🚨 Failed to setup plaintext: {e}", exc_info=True)
success = False
try:
if no_documents:
github_config = GithubConfig.objects.filter(user=user).prefetch_related("githubrepoconfig").first()
if (
search_type == state.SearchType.All.value or search_type == state.SearchType.Github.value
) and github_config is not None:
logger.info("🐙 Setting up search for github")
# Extract Entries, Generate Github Embeddings
text_search.setup(
GithubToEntries,
None,
regenerate=regenerate,
full_corpus=full_corpus,
user=user,
config=github_config,
)
except Exception as e:
logger.error(f"🚨 Failed to setup GitHub: {e}", exc_info=True)
success = False
try:
if no_documents:
# Initialize Notion Search
notion_config = NotionConfig.objects.filter(user=user).first()
if (
search_type == state.SearchType.All.value or search_type == state.SearchType.Notion.value
) and notion_config:
logger.info("🔌 Setting up search for notion")
text_search.setup(
NotionToEntries,
None,
regenerate=regenerate,
full_corpus=full_corpus,
user=user,
config=notion_config,
)
except Exception as e:
logger.error(f"🚨 Failed to setup Notion: {e}", exc_info=True)
success = False
try:
# Initialize Image Search
if (search_type == state.SearchType.All.value or search_type == state.SearchType.Image.value) and files[
"image"
]:
logger.info("🖼️ Setting up search for images")
# Extract Entries, Generate Image Embeddings
text_search.setup(
ImageToEntries,
files.get("image"),
regenerate=regenerate,
full_corpus=full_corpus,
user=user,
)
except Exception as e:
logger.error(f"🚨 Failed to setup images: {e}", exc_info=True)
success = False
try:
if (search_type == state.SearchType.All.value or search_type == state.SearchType.Docx.value) and files["docx"]:
logger.info("📄 Setting up search for docx")
text_search.setup(
DocxToEntries,
files.get("docx"),
regenerate=regenerate,
full_corpus=full_corpus,
user=user,
)
except Exception as e:
logger.error(f"🚨 Failed to setup docx: {e}", exc_info=True)
success = False
# Invalidate Query Cache
if user:
state.query_cache[user.uuid] = LRU()
return success

8
src/khoj/routers/notion.py
View File

@@ -11,7 +11,7 @@ from starlette.responses import RedirectResponse
from khoj.database.adapters import aget_user_by_uuid from khoj.database.adapters import aget_user_by_uuid
from khoj.database.models import KhojUser, NotionConfig from khoj.database.models import KhojUser, NotionConfig
from khoj.routers.indexer import configure_content from khoj.routers.helpers import configure_content
from khoj.utils.state import SearchType from khoj.utils.state import SearchType
NOTION_OAUTH_CLIENT_ID = os.getenv("NOTION_OAUTH_CLIENT_ID") NOTION_OAUTH_CLIENT_ID = os.getenv("NOTION_OAUTH_CLIENT_ID")
@@ -25,12 +25,6 @@ executor = ThreadPoolExecutor()
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
def get_notion_auth_url(user: KhojUser):
if not NOTION_OAUTH_CLIENT_ID or not NOTION_OAUTH_CLIENT_SECRET or not NOTION_REDIRECT_URI:
return None
return f"https://api.notion.com/v1/oauth/authorize?client_id={NOTION_OAUTH_CLIENT_ID}&redirect_uri={NOTION_REDIRECT_URI}&response_type=code&state={user.uuid}"
async def run_in_executor(func, *args): async def run_in_executor(func, *args):
loop = asyncio.get_event_loop() loop = asyncio.get_event_loop()
return await loop.run_in_executor(executor, func, *args) return await loop.run_in_executor(executor, func, *args)

299
src/khoj/routers/web_client.py
View File

@@ -1,30 +1,21 @@
# System Packages # System Packages
import json import json
import os import os
from datetime import timedelta
from typing import Optional from typing import Optional
from fastapi import APIRouter, Request from fastapi import APIRouter, Request
from fastapi.responses import FileResponse, HTMLResponse, RedirectResponse from fastapi.responses import FileResponse, HTMLResponse, RedirectResponse
from fastapi.templating import Jinja2Templates from fastapi.templating import Jinja2Templates
from starlette.authentication import has_required_scope, requires from starlette.authentication import requires
from khoj.database import adapters
from khoj.database.adapters import ( from khoj.database.adapters import (
AgentAdapters, AgentAdapters,
ConversationAdapters,
EntryAdapters,
PublicConversationAdapters, PublicConversationAdapters,
get_user_github_config, get_user_github_config,
get_user_name,
get_user_notion_config, get_user_notion_config,
get_user_subscription_state,
) )
from khoj.database.models import KhojUser from khoj.database.models import KhojUser
from khoj.processor.speech.text_to_speech import is_eleven_labs_enabled from khoj.routers.helpers import get_next_url, get_user_config
from khoj.routers.helpers import get_next_url
from khoj.routers.notion import get_notion_auth_url
from khoj.routers.twilio import is_twilio_enabled
from khoj.utils import constants, state from khoj.utils import constants, state
from khoj.utils.rawconfig import ( from khoj.utils.rawconfig import (
GithubContentConfig, GithubContentConfig,
@@ -42,80 +33,36 @@ templates = Jinja2Templates([constants.web_directory, constants.next_js_director
@requires(["authenticated"], redirect="login_page") @requires(["authenticated"], redirect="login_page")
def index(request: Request): def index(request: Request):
user = request.user.object user = request.user.object
user_picture = request.session.get("user", {}).get("picture") user_config = get_user_config(user, request)
has_documents = EntryAdapters.user_has_entries(user=user)
return templates.TemplateResponse( return templates.TemplateResponse("chat.html", context=user_config)
"chat.html",
context={
"request": request,
"username": user.username,
"user_photo": user_picture,
"is_active": has_required_scope(request, ["premium"]),
"has_documents": has_documents,
"khoj_version": state.khoj_version,
},
)
@web_client.post("/", response_class=FileResponse) @web_client.post("/", response_class=FileResponse)
@requires(["authenticated"], redirect="login_page") @requires(["authenticated"], redirect="login_page")
def index_post(request: Request): def index_post(request: Request):
user = request.user.object user = request.user.object
user_picture = request.session.get("user", {}).get("picture") user_config = get_user_config(user, request)
has_documents = EntryAdapters.user_has_entries(user=user)
return templates.TemplateResponse( return templates.TemplateResponse("chat.html", context=user_config)
"chat.html",
context={
"request": request,
"username": user.username,
"user_photo": user_picture,
"is_active": has_required_scope(request, ["premium"]),
"has_documents": has_documents,
"khoj_version": state.khoj_version,
},
)
@web_client.get("/search", response_class=FileResponse) @web_client.get("/search", response_class=FileResponse)
@requires(["authenticated"], redirect="login_page") @requires(["authenticated"], redirect="login_page")
def search_page(request: Request): def search_page(request: Request):
user = request.user.object user = request.user.object
user_picture = request.session.get("user", {}).get("picture") user_config = get_user_config(user, request)
has_documents = EntryAdapters.user_has_entries(user=user)
return templates.TemplateResponse( return templates.TemplateResponse("search.html", context=user_config)
"search.html",
context={
"request": request,
"username": user.username,
"user_photo": user_picture,
"is_active": has_required_scope(request, ["premium"]),
"has_documents": has_documents,
"khoj_version": state.khoj_version,
},
)
@web_client.get("/chat", response_class=FileResponse) @web_client.get("/chat", response_class=FileResponse)
@requires(["authenticated"], redirect="login_page") @requires(["authenticated"], redirect="login_page")
def chat_page(request: Request): def chat_page(request: Request):
user = request.user.object user = request.user.object
user_picture = request.session.get("user", {}).get("picture") user_config = get_user_config(user, request)
has_documents = EntryAdapters.user_has_entries(user=user)
return templates.TemplateResponse( return templates.TemplateResponse("chat.html", context=user_config)
"chat.html",
context={
"request": request,
"username": user.username,
"user_photo": user_picture,
"is_active": has_required_scope(request, ["premium"]),
"has_documents": has_documents,
"khoj_version": state.khoj_version,
},
)
@web_client.get("/experimental", response_class=FileResponse) @web_client.get("/experimental", response_class=FileResponse)
@@ -169,25 +116,14 @@ def agents_page(request: Request):
@web_client.get("/agent/{agent_slug}", response_class=HTMLResponse) @web_client.get("/agent/{agent_slug}", response_class=HTMLResponse)
def agent_page(request: Request, agent_slug: str): def agent_page(request: Request, agent_slug: str):
user: KhojUser = request.user.object if request.user.is_authenticated else None user: KhojUser = request.user.object if request.user.is_authenticated else None
user_picture = request.session.get("user", {}).get("picture") if user else None user_config = get_user_config(user, request)
agent = AgentAdapters.get_agent_by_slug(agent_slug) agent = AgentAdapters.get_agent_by_slug(agent_slug)
has_documents = EntryAdapters.user_has_entries(user=user)
if agent == None: if agent == None:
return templates.TemplateResponse( user_config["has_documents"] = False
"404.html", return templates.TemplateResponse("404.html", context=user_config)
context={
"request": request,
"khoj_version": state.khoj_version,
"username": user.username if user else None,
"has_documents": False,
"is_active": has_required_scope(request, ["premium"]),
"user_photo": user_picture,
},
)
agent_metadata = { user_config["agent"] = {
"slug": agent.slug, "slug": agent.slug,
"avatar": agent.avatar, "avatar": agent.avatar,
"name": agent.name, "name": agent.name,
@@ -199,115 +135,23 @@ def agent_page(request: Request, agent_slug: str):
"creator_not_self": agent.creator != user, "creator_not_self": agent.creator != user,
} }
return templates.TemplateResponse( return templates.TemplateResponse("agent.html", context=user_config)
"agent.html",
context={
"request": request,
"agent": agent_metadata,
"khoj_version": state.khoj_version,
"username": user.username if user else None,
"has_documents": has_documents,
"is_active": has_required_scope(request, ["premium"]),
"user_photo": user_picture,
},
)
@web_client.get("/config", response_class=HTMLResponse) @web_client.get("/settings", response_class=HTMLResponse)
@requires(["authenticated"], redirect="login_page") @requires(["authenticated"], redirect="login_page")
def config_page(request: Request): def config_page(request: Request):
user: KhojUser = request.user.object user: KhojUser = request.user.object
user_picture = request.session.get("user", {}).get("picture") user_config = get_user_config(user, request, is_detailed=True)
has_documents = EntryAdapters.user_has_entries(user=user)
user_subscription_state = get_user_subscription_state(user.email) return templates.TemplateResponse("settings.html", context=user_config)
user_subscription = adapters.get_user_subscription(user.email)
subscription_renewal_date = (
user_subscription.renewal_date.strftime("%d %b %Y")
if user_subscription and user_subscription.renewal_date
else (user_subscription.created_at + timedelta(days=7)).strftime("%d %b %Y")
)
given_name = get_user_name(user)
enabled_content_source = set(EntryAdapters.get_unique_file_sources(user))
successfully_configured = {
"computer": ("computer" in enabled_content_source),
"github": ("github" in enabled_content_source),
"notion": ("notion" in enabled_content_source),
}
selected_conversation_config = ConversationAdapters.get_conversation_config(user)
conversation_options = ConversationAdapters.get_conversation_processor_options().all()
all_conversation_options = list()
for conversation_option in conversation_options:
all_conversation_options.append({"chat_model": conversation_option.chat_model, "id": conversation_option.id})
search_model_options = adapters.get_or_create_search_models().all()
all_search_model_options = list()
for search_model_option in search_model_options:
all_search_model_options.append({"name": search_model_option.name, "id": search_model_option.id})
current_search_model_option = adapters.get_user_search_model_or_default(user)
selected_paint_model_config = ConversationAdapters.get_user_text_to_image_model_config(user)
paint_model_options = ConversationAdapters.get_text_to_image_model_options().all()
all_paint_model_options = list()
for paint_model in paint_model_options:
all_paint_model_options.append({"model_name": paint_model.model_name, "id": paint_model.id})
notion_oauth_url = get_notion_auth_url(user)
eleven_labs_enabled = is_eleven_labs_enabled()
voice_models = ConversationAdapters.get_voice_model_options()
voice_model_options = list()
for voice_model in voice_models:
voice_model_options.append({"name": voice_model.name, "id": voice_model.model_id})
if len(voice_model_options) == 0:
eleven_labs_enabled = False
selected_voice_config = ConversationAdapters.get_voice_model_config(user)
return templates.TemplateResponse(
"config.html",
context={
"request": request,
"current_model_state": successfully_configured,
"anonymous_mode": state.anonymous_mode,
"username": user.username,
"given_name": given_name,
"search_model_options": all_search_model_options,
"selected_search_model_config": current_search_model_option.id,
"conversation_options": all_conversation_options,
"selected_conversation_config": selected_conversation_config.id if selected_conversation_config else None,
"paint_model_options": all_paint_model_options,
"selected_paint_model_config": selected_paint_model_config.id if selected_paint_model_config else None,
"user_photo": user_picture,
"billing_enabled": state.billing_enabled,
"subscription_state": user_subscription_state,
"subscription_renewal_date": subscription_renewal_date,
"khoj_cloud_subscription_url": os.getenv("KHOJ_CLOUD_SUBSCRIPTION_URL"),
"is_active": has_required_scope(request, ["premium"]),
"has_documents": has_documents,
"is_twilio_enabled": is_twilio_enabled(),
"is_eleven_labs_enabled": eleven_labs_enabled,
"voice_model_options": voice_model_options,
"selected_voice_config": selected_voice_config.model_id if selected_voice_config else None,
"phone_number": user.phone_number,
"is_phone_number_verified": user.verified_phone_number,
"khoj_version": state.khoj_version,
"notion_oauth_url": notion_oauth_url,
},
)
@web_client.get("/config/content-source/github", response_class=HTMLResponse) @web_client.get("/settings/content/github", response_class=HTMLResponse)
@requires(["authenticated"], redirect="login_page") @requires(["authenticated"], redirect="login_page")
def github_config_page(request: Request): def github_config_page(request: Request):
user = request.user.object user = request.user.object
user_picture = request.session.get("user", {}).get("picture") user_config = get_user_config(user, request)
has_documents = EntryAdapters.user_has_entries(user=user)
current_github_config = get_user_github_config(user) current_github_config = get_user_github_config(user)
if current_github_config: if current_github_config:
@@ -329,66 +173,32 @@ def github_config_page(request: Request):
else: else:
current_config = {} # type: ignore current_config = {} # type: ignore
return templates.TemplateResponse( user_config["current_config"] = current_config
"content_source_github_input.html", return templates.TemplateResponse("content_source_github_input.html", context=user_config)
context={
"request": request,
"current_config": current_config,
"username": user.username,
"user_photo": user_picture,
"is_active": has_required_scope(request, ["premium"]),
"has_documents": has_documents,
"khoj_version": state.khoj_version,
},
)
@web_client.get("/config/content-source/notion", response_class=HTMLResponse) @web_client.get("/settings/content/notion", response_class=HTMLResponse)
@requires(["authenticated"], redirect="login_page") @requires(["authenticated"], redirect="login_page")
def notion_config_page(request: Request): def notion_config_page(request: Request):
user = request.user.object user = request.user.object
user_picture = request.session.get("user", {}).get("picture") user_config = get_user_config(user, request)
has_documents = EntryAdapters.user_has_entries(user=user)
current_notion_config = get_user_notion_config(user) current_notion_config = get_user_notion_config(user)
token = current_notion_config.token if current_notion_config else ""
current_config = NotionContentConfig( current_config = NotionContentConfig(token=token)
token=current_notion_config.token if current_notion_config else "",
)
current_config = json.loads(current_config.model_dump_json()) current_config = json.loads(current_config.model_dump_json())
return templates.TemplateResponse( user_config["current_config"] = current_config
"content_source_notion_input.html", return templates.TemplateResponse("content_source_notion_input.html", context=user_config)
context={
"request": request,
"current_config": current_config,
"username": user.username,
"user_photo": user_picture,
"is_active": has_required_scope(request, ["premium"]),
"has_documents": has_documents,
"khoj_version": state.khoj_version,
},
)
@web_client.get("/config/content-source/computer", response_class=HTMLResponse) @web_client.get("/settings/content/computer", response_class=HTMLResponse)
@requires(["authenticated"], redirect="login_page") @requires(["authenticated"], redirect="login_page")
def computer_config_page(request: Request): def computer_config_page(request: Request):
user = request.user.object if request.user.is_authenticated else None user = request.user.object if request.user.is_authenticated else None
user_picture = request.session.get("user", {}).get("picture") if user else None user_config = get_user_config(user, request)
has_documents = EntryAdapters.user_has_entries(user=user) if user else False
return templates.TemplateResponse( return templates.TemplateResponse("content_source_computer_input.html", context=user_config)
"content_source_computer_input.html",
context={
"request": request,
"username": user.username,
"user_photo": user_picture,
"is_active": has_required_scope(request, ["premium"]),
"has_documents": has_documents,
"khoj_version": state.khoj_version,
},
)
@web_client.get("/share/chat/{public_conversation_slug}", response_class=HTMLResponse) @web_client.get("/share/chat/{public_conversation_slug}", response_class=HTMLResponse)
@@ -404,8 +214,9 @@ def view_public_conversation(request: Request):
}, },
) )
user = request.user.object if request.user.is_authenticated else None user = request.user.object if request.user.is_authenticated else None
user_picture = request.session.get("user", {}).get("picture") if user else None user_config = get_user_config(user, request)
has_documents = EntryAdapters.user_has_entries(user=user) if user else False user_config["public_conversation_slug"] = public_conversation_slug
user_config["google_client_id"] = os.environ.get("GOOGLE_CLIENT_ID")
all_agents = AgentAdapters.get_all_accessible_agents(request.user.object if request.user.is_authenticated else None) all_agents = AgentAdapters.get_all_accessible_agents(request.user.object if request.user.is_authenticated else None)
@@ -420,28 +231,15 @@ def view_public_conversation(request: Request):
"name": agent.name, "name": agent.name,
} }
) )
user_config["agents"] = agents_packet
google_client_id = os.environ.get("GOOGLE_CLIENT_ID")
redirect_uri = str(request.app.url_path_for("auth")) redirect_uri = str(request.app.url_path_for("auth"))
next_url = str( next_url = str(
request.app.url_path_for("view_public_conversation", public_conversation_slug=public_conversation_slug) request.app.url_path_for("view_public_conversation", public_conversation_slug=public_conversation_slug)
) )
user_config["redirect_uri"] = f"{redirect_uri}?next={next_url}"
return templates.TemplateResponse( return templates.TemplateResponse("public_conversation.html", context=user_config)
"public_conversation.html",
context={
"request": request,
"username": user.username if user else None,
"user_photo": user_picture,
"is_active": has_required_scope(request, ["premium"]),
"has_documents": has_documents,
"khoj_version": state.khoj_version,
"public_conversation_slug": public_conversation_slug,
"agents": agents_packet,
"google_client_id": google_client_id,
"redirect_uri": f"{redirect_uri}?next={next_url}",
},
)
@web_client.get("/automations", response_class=HTMLResponse) @web_client.get("/automations", response_class=HTMLResponse)
@@ -452,20 +250,9 @@ def automations_config_page(
queryToRun: Optional[str] = None, queryToRun: Optional[str] = None,
): ):
user = request.user.object if request.user.is_authenticated else None user = request.user.object if request.user.is_authenticated else None
user_picture = request.session.get("user", {}).get("picture") user_config = get_user_config(user, request)
has_documents = EntryAdapters.user_has_entries(user=user) if user else False user_config["subject"] = subject if subject else ""
user_config["crontime"] = crontime if crontime else ""
user_config["queryToRun"] = queryToRun if queryToRun else ""
return templates.TemplateResponse( return templates.TemplateResponse("config_automation.html", context=user_config)
"config_automation.html",
context={
"request": request,
"username": user.username if user else None,
"user_photo": user_picture,
"is_active": has_required_scope(request, ["premium"]),
"has_documents": has_documents,
"khoj_version": state.khoj_version,
"subject": subject if subject else "",
"crontime": crontime if crontime else "",
"queryToRun": queryToRun if queryToRun else "",
},
)

4
tests/test_client.py
View File

@@ -269,7 +269,7 @@ def test_get_api_config_types(client, sample_org_data, default_user: KhojUser):
text_search.setup(OrgToEntries, sample_org_data, regenerate=False, user=default_user) text_search.setup(OrgToEntries, sample_org_data, regenerate=False, user=default_user)
# Act # Act
response = client.get(f"/api/config/types", headers=headers) response = client.get(f"/api/configure/types", headers=headers)
# Assert # Assert
assert response.status_code == 200 assert response.status_code == 200
@@ -289,7 +289,7 @@ def test_get_configured_types_with_no_content_config(fastapi_app: FastAPI):
client = TestClient(fastapi_app) client = TestClient(fastapi_app)
# Act # Act
response = client.get(f"/api/config/types") response = client.get(f"/api/configure/types")
# Assert # Assert
assert response.status_code == 200 assert response.status_code == 200