Debanjum 11ce3e2268 Update Text Chunking Strategy to Improve Search Context (#645) ...

## Major
- Parse markdown, org parent entries as single entry if fit within max tokens
- Parse a file as single entry if it fits with max token limits
- Add parent heading ancestry to extracted markdown entries for context
- Chunk text in preference order of para, sentence, word, character

## Minor
- Create wrapper function to get entries from org, md, pdf & text files
- Remove unused Entry to Jsonl converter from text to entry class, tests
- Dedupe code by using single func to process an org file into entries

Resolves #620

2024-04-08 13:56:38 +05:30

18 KiB

Raw Blame History

View Raw