klbr/khoj
1
0
Fork 0
mirror of https://github.com/khoaliber/khoj.git synced 2026-03-02 21:19:12 +00:00
Code Issues Packages Projects Releases Wiki Activity

Show debug log, not warning when use default tokenizer for context stuffing

Browse Source
This commit is contained in:
Debanjum Singh Solanky
2024-09-12 20:20:14 -07:00
parent f173188dcf
commit 178b78f87b
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
src/khoj/processor/conversation/utils.py
View File

@@ -238,7 +238,7 @@ def truncate_messages(
else:
encoder = AutoTokenizer.from_pretrained(default_tokenizer)
state.pretrained_tokenizers[default_tokenizer] = encoder
logger.warning(
logger.debug(
f"Fallback to default chat model tokenizer: {tokenizer_name}.\nConfigure tokenizer for unsupported model: {model_name} in Khoj settings to improve context stuffing."
)