We'd reversed the formatting of openai messages to drop invalid
messages without affecting the other messages being appended . But we
need to reverse the final formatted list to return in the right order.
Previously
- message with invalid content were getting dropped in normal order
which would change the item index being iterated for gemini and
anthropic models
- messages with empty content weren't getting dropped for openai
compatible api models. While openai api is resilient to this, it's
better to drop these invalid messages as other openai compatible
APIs may not handle this.
We see messages with empty or no content when chat gets interrupted
due to disconnections, interrupt messages or explicit aborts by user.
This changes should now drop invalid messages and not mess formatting
of the other messages in a conversation. It should allow continuing
interrupted conversations with any ai model.
Inspired by my previous turnstyle ux explorations.
But basically user message becomes section title and khoj message
becomes section body with the timestamp being used a section title,
body divider.
Previous organic results enumerator only handled the scenario where
organic key wasn't present in online search results.
It did not handle the case where there were no organic online search
results.
- Methods calling send_message_to_model_wrapper_sync handn't been
update to handle the function returning new ResponseWithThought
- Store, load request.url to DB as, from string to avoid serialization
issues
For files not synced after the previous release, context uri is unset.
This results in failure to save chat messages that retrieve documents
as the uri field cannot be unset so pre save validation fails.
We'd use a db migration to handle this but this is a quick mitigation
for now.
For files not synced after the previous release, context uri is unset.
This results in failure to save chat messages that retrieve documents
as the uri field cannot be unset so pre save validation fails
It'll work similar to the master branch but with pre-1x and latest-1x
tagged series of docker images.
This should ease deployment changes from 1.x vs 2.x series
## Overview
Show deep link URI and raw document context to provide deeper, richer
context to Khoj. This should allow it better combine semantic search
with other new document retrieval tools like line range based file
viewer and regex tools added in #1205
## Details
- Attach line number based deeplinks to each indexed document entry
Document URI follows URL fragment based schema of form
`file:///path/to/file.txt#line=123`
- Show raw indexed document entries with deep links to LLM when it uses
the semantic search tool
- Reduce structural changes to raw org-mode entries for easier deep
linking.
Use url fragment schema for deep link URIs, borrowing from URL/PDF
schemas. E.g file:///path/to/file.txt#line=<line_no>&#page=<page_no>
Compute line number during (recursive) markdown entry chunking.
Test line number in URI maps to line number of chunk in actual md file.
This deeplink URI with line number is passed to llm as context to
better combine with line range based view file tool.
Grep tool already passed matching line number. This change passes
line number in URIs of markdown entries matched by the semantic search
tool.
Use url fragment schema for deep link URIs, borrowing from URL/PDF
schemas. E.g file:///path/to/file.txt#line=<line_no>&#page=<page_no>
Compute line number during (recursive) org-mode entry chunking.
Thoroughly test line number in URI maps to line number of chunk in
actual org mode file.
This deeplink URI with line number is passed to llm as context to
better combine with line range based view file tool.
Grep tool already passed matching line number. This change passes
line number in URIs of org entries matched by the semantic search tool
Only embedding models see, operate on compiled text.
LLMs should see raw entry to improve combining it with other document
traversal tools for better regex and line matching.
Users see raw entry for better matching with their actual notes.
Reduce structural changes to raw entry allows better deep-linking and
re-annotation. Currently done via line number in new uri field.
Only add properties drawer to raw entry if entry has properties
Previously line and source properties were inserted into raw entries.
This isn't done anymore. Line, source are deprecated for use in khoj.el.
## Why
Move to function calling paradigm to give models tool call -> tool
result in formats they're fine-tuned to understand. Previously we were
giving them results in our specific format (as function calling paradigm
wasn't well-established yet).
And improve prompt cache hits by caching tool definitions.
This is a **breaking change**. AI Models and APIs that do not support
function calling will not work with Khoj in research mode. Function
calling is supported by:
- Standard commercial AI Models and APIs like Anthropic, Gemini, OpenAI,
OpenRouter
- Standard open-source AI APIs like llama.cpp server, Ollama
- Standard open source models like Qwen, DeepSeek, Gemma, Llama, Mistral
## What
### Use Function Calling for Tool Use
- Add Function Calling support to Anthropic, Gemini, OpenAI AI Model
APIs
- Move Existing Research Mode Tools to Use Function Calling
### Get More Comprehensive Results from your Knowledge Base (KB)
- Give Research Agent better Document Retrieval Tools
- Add grep files tool to enable researcher to find documents by regex
- Add list files tool to enable researcher to find documents by path
- Add file viewer tool to enable researcher to read documents
### Miscellaneous
- Improve Research Prompt, Truncation, Retry and Caching
- Show reasoning model thoughts in Khoj train of thought for
intermediate steps as well
- Cache last anthropic message. Given research mode now uses function
calling paradigm and not the old research mode structure.
- Cache tool definitions passed to anthropic models
- Stop dropping first message if by assistant as seems like Anthropic
API doesn't complain about it any more.
- Drop tool result when tool call is truncated as invalid state
- Do not truncate tool use message content, just drop the whole tool
use message.
AI model APIs need tool use assistant message content in specific
form (e.g with thinking etc.). So dropping content items breaks
expected tool use message content format.
Handle tool use scenarios where iteration query isn't set for retry
- Deepcopy messages before formatting message for Anthropic to allow
idempotency so retry on failure behaves as expected
- Handle failed calls to pick next tools to pass failure warning and
continue next research iteration. Previously if API call to pick
next failed, the research run would crash
- Add null response check for when Gemini models fail to respond
Previously if anthropic models were using tools, the models text
response accompanying the tool use wouldn't be shown as they were
overwritten in aggregated response with the tool call json.
This changes appends the text response to the thoughts portion on tool
use to still show model's thinking. Thinking and text response are
delineated by italics vs normal text for such cases.
This should avoid the need to reformat the Khoj standardized tool call
for cache hits and satisfying ai model api requirements.
Previously multi-turn tool use calls to anthropic reasoning models
would fail as needed their thoughts to be passed back. Other AI model
providers can have other requirements.
Passing back the raw response as is should satisfy the default case.
Tracking raw response should make it easy to apply any formatting
required before sending previous response back, if any ai model
provider requires that.
Details
---
- Raw response content is passed back in ResponseWithThoughts.
- Research iteration stores this and puts it into model response
ChatMessageModel when constructing iteration history when it is
present.
Fallback to using parsed tool call when raw response isn't present.
- No need to format tool call messages for anthropic models as we're
passing the raw response as is.
Researcher is expanding into accomplish task behavior, especially with
tool use from the previous collect information to answer user query
behavior.
Update the researcher's system prompt to reflect the new objective better.
Encourage model to not stop working on task until achieve objective
