Clean non useful slash commands to make chat API more maintanable.
- App version, chat model via /help is visible in other parts of the
UX. Asking help questions with site:docs.khoj.dev filter isn't used
or known to folks
- /summarize is esoterically tuned. Should be rewritten if add back.
It wasn't being used by /research already
- Automations can be configured via UX. It wasn't being shown in UX
already
The chat actor (and director) tests haven't been looked into in a long
while. They'd gone stale in how they were calling thee functions. And
what was required to run them. Now the online chat actor tests work
again.
Using model specific extract questions was an artifact from older
times, with less guidable models.
New changes collate and reuse logic
- Rely on send_message_to_model_wrapper for model specific formatting.
- Use same prompt, context for all LLMs as can handle prompt variation.
- Use response schema enforcer to ensure response consistency across models.
Extract questions (because of its age) was the only tool directly within
each provider code. Put it into helpers to have all the (mini) tools
in one place.
- Rename GET /api/automations to GET /api/automation
- Rename POST /api/trigger/automation to POST /api/automation/trigger
- Update calls to the automations API from the web app.
- Add context based on information provided rather than conversation
commands. Let caller handle passing appropriate context to ai
provider converse methods
Increase timeout to 180 (from 120s previous) and graceful timeout to
90 (from 30s default) to reduce
Increase default gunicorn workers and make it configurable to better
utilize (v)CPUs. This is manually configured (instead of using
multiprocessing.cpu_count()) as VMs/containers may read cpu count of
host machine instead of their VMs/containers.
The chat dictionary is an artifact from earlier non-db chat history
storage. We've been ensuring new chat messages have valid type before
being written to DB for more than 6 months now.
Move to using the deeply typed chat history helps avoids null refs,
makes code more readable and easier to reason about.
Next Steps:
The current update entangles chat_history written to DB
with any virtual chat history message generated for intermediate
steps. The chat message type written to DB should be decoupled from
type that can be passed to AI model APIs (maybe?).
For now we've made the ChatMessage.message type looser to allow
for list[dict] type (apart from string). But later maybe a good idea
to decouple the chat_history recieved by send_message_to_model from
the chat_history saved to DB (which can then have its stricter type check)
- Converts response schema into a anthropic tool call definition.
- Works with simple enums without needing to rely on $defs, $refs as
unsupported by Anthropic API
- Do not force specific tool use as not supported with deep thought
This puts anthropic models on parity with openai, gemini models for
response schema following. Reduces need for complex json response
parsing on khoj end.
There seems to be a more standard mechanism of specifying launch.json
params for devcontainers. Previous mechanism to write launch.json to
.vscode/launch.json in post creation step does not work.
Improve default launch.json to include khoj admin username, password
with placeholder values to get started with local development faster.
Define dockerfile for devcontainer to pre-built server, web app
dependencies during dev container image creation stage. So install on
dev container startup is sped up as no need to install dependencies.
## Description
This PR introduces significant improvements to the Obsidian Khoj
plugin's chat interface and editing capabilities, enhancing the overall
user experience and content management functionality.
## Features
### 🔍 Enhanced Communication Mode
I've implemented radio buttons below the chat window for easier
communication mode selection. The modes are now displayed as emojis in
the conversation for a cleaner interface, replacing the previous
text-based system (e.g., /default, /research). I've also documented the
search mode functionality in the help command.
#### Screenshots
- Radio buttons for mode selection
- Emoji display in conversations
### 💬 Revamped Message Interaction
I've redesigned the message buttons with improved spacing and color
coding for better visual differentiation. The new edit button allows
quick message modifications - clicking it removes the conversation up to
that point and copies the message to the input field for easy editing or
retrying questions.
#### Screenshots
- New message styling and color scheme
- Edit button functionality
### 🤖 Advanced Agent Selection System
I've added a new chat creation button with agent selection capability.
Users can now choose from their available agents when starting a new
chat. While agents can't be switched mid-conversation to maintain
context, users can easily start fresh conversations with different
agents.
#### Screenshots
- Agent selection dropdown
### 👁️ Real-Time Context Awareness
I've added a button that gives Khoj access to read Obsidian opened tabs.
This allows Khoj to read open notes and track changes in real-time,
maintaining a history of previous versions to provide more contextual
assistance.
#### Screenshots
- Window access toggle
### ✏️ Smart Document Editing
Inspired by Cursor IDE's intelligent editing and ChatGPT's Canvas
functionality, I've implemented a first version of a content creation
system we've been discussing. Using a JSON-based modification system,
Khoj can now make precise changes to specific parts of files, with
changes previewed in yellow highlighting before application.
Modification code blocks are neatly organized in collapsible sections
with clear action summaries. While this is just a first step, it's
working remarkably well and I have several ideas for expanding this
functionality to make Khoj an even more powerful content creation
assistant.
#### Screenshots
- JSON modification preview
- Change highlighting system
- Collapsible code blocks
- Accept/cancel controls
---------
Co-authored-by: Debanjum <debanjum@gmail.com>
## Summary
- Enable Khoj to operate computers: Add experimental computer operator
functionality that allows Khoj to interact with desktop environments,
browsers, and terminals to accomplish complex tasks
- Multi-environment support: Implement computer environments with GUI,
file system, and terminal access. Can control host computer or Docker
container computer
## Key Features
### Computer Operation Capabilities
- Desktop control (screenshots, clicking, typing, keyboard shortcuts)
- File editing and management
- Terminal/bash command execution
- Web browser automation
- Visual feedback via train-of-thought video playback
### Infrastructure & Architecture:
- Docker container (ghcr.io/khoj-ai/computer:latest) with Ubuntu 24.04,
XFCE desktop, VNC access
- Local computer environment support with pyautogui
- Modular operator agent system supporting multiple environment types
- Trajectory compression and context management for long-running tasks
### Model Integration:
- Anthropic models only (Claude Sonnet 4, Claude 3.7 Sonnet, Claude Opus
4)
- OpenAI and binary operator agents temporarily disabled
- Enhanced caching and context management for operator conversations
### User Experience:
- `/operator` command or just ask Khoj to use operator tool to invoke
computer operation
- Integrate with research mode for extended 30+ minute task execution
- Video of computer operation in train of thought for transparency
### Configuration
- Set `KHOJ_OPERATOR_ENABLED=True` in `docker-compose.yml`
- Requires Anthropic API key
- Computer container runs on port 5900 (VNC)
