Debanjum Singh Solanky db2581459f Parse markdown parent entries as single entry if fit within max tokens
These changes improve context available to the search model.
Specifically this should improve entry context from short knowledge trees,
that is knowledge bases with sparse, short heading/entry trees

Previously we'd always split markdown files by headings, even if a
parent entry was small enough to fit entirely within the max token
limits of the search model. This used to reduce the context available
to the search model to select appropriate entries for a query,
especially from short entry trees

Revert back to using regex to parse through markdown file instead of
using MarkdownHeaderTextSplitter. It was easier to implement the
logical split using regexes rather than bend MarkdowHeaderTextSplitter
to implement it.
- DFS traverse the markdown knowledge tree, prefix ancestry to each entry
2024-04-04 02:41:55 +05:30
2024-03-31 00:06:15 +05:30
2024-04-02 16:16:06 +05:30
2024-03-10 17:48:30 +05:30
2024-03-31 00:06:15 +05:30

Khoj Logo

test dockerize pypi Discord

An AI personal assistant for your digital brain


Khoj is an AI application to search and chat with your notes and documents.
It is open-source, self-hostable and accessible on Desktop, Emacs, Obsidian, Web and Whatsapp.
It works with pdf, markdown, org-mode, notion files and github repositories.
It can paint, search the internet and understand speech.


🔎 Search 💬 Chat
Quickly retrieve relevant documents using natural language Get answers and create content from your existing knowledge base
Does not need internet Can be configured to work without internet

Contributors

Cheers to our awesome contributors! 🎉

Made with contrib.rocks.

Description
No description provided
Readme AGPL-3.0 116 MiB
Languages
Python 51%
TypeScript 36.1%
CSS 4.1%
HTML 3.2%
Emacs Lisp 2.4%
Other 3.1%