Debanjum Singh Solanky 982ac1859c Parse markdown file as single entry if it fits with max token limits ...

These changes improve entry context available to the search model
Specifically this should improve entry context from short knowledge trees,
that is knowledge bases with small files

Previously we split all markdown files by their headings,
even if the file was small enough to fit entirely within the max token
limits of the search model. This used to reduce the context available
to select the appropriate entries for a given query for the search model,
especially from short knowledge trees

2024-04-04 02:41:55 +05:30

5.6 KiB

Raw Blame History

View Raw