Debanjum Singh Solanky 86575b2946 Chunk text in preference order of para, sentence, word, character ...

- Previous simplistic chunking strategy of splitting text by space
  didn't capture notes with newlines, no spaces. For e.g in #620

- New strategy will try chunk the text at more natural points like
  paragraph, sentence, word first. If none of those work it'll split
  at character to fit within max token limit

- Drop long words while preserving original delimiters

Resolves #620

2024-04-04 02:41:55 +05:30

15 KiB

Raw Blame History

View Raw