klbr/khoj
1
0
Fork 0
mirror of https://github.com/khoaliber/khoj.git synced 2026-03-02 21:19:12 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fix engaging anthropic api cache for operator trajectories.

Browse Source 
It had become broken at some point due to refactoring. The cache
control was getting added and removed right after in add_action_results

What we actually wanted to do is clear the old cache breakpoint and
put a new one at the latest operator tool result message.

This should improve operator speed and lower costs with anthropic
models.
This commit is contained in:
Debanjum
2025-05-27 17:34:23 -07:00
parent faecbdb7d8
commit 253656b634
1 changed files with 6 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
src/khoj/processor/operator/operator_agent_anthropic.py
View File

@@ -206,11 +206,6 @@ class AnthropicOperatorAgent(OperatorAgent):
if env_step.error:
action_result["is_error"] = True
# Append tool results to the message history
self.messages += [AgentMessage(role="environment", content=agent_action.action_results)]
# Mark the final tool result as a cache break point
agent_action.action_results[-1]["cache_control"] = {"type": "ephemeral"}
# Remove previous cache controls
for msg in self.messages:
if msg.role == "environment" and isinstance(msg.content, list):
@@ -218,6 +213,12 @@ class AnthropicOperatorAgent(OperatorAgent):
if isinstance(block, dict) and "cache_control" in block:
del block["cache_control"]
# Mark the final tool result as a cache break point
agent_action.action_results[-1]["cache_control"] = {"type": "ephemeral"}
# Append tool results to the message history
self.messages += [AgentMessage(role="environment", content=agent_action.action_results)]
def _format_message_for_api(self, messages: list[AgentMessage]) -> list[dict]:
"""Format Anthropic response into a single string."""
formatted_messages = []