Debanjum Singh Solanky e057c8e208 Add method to split entries by specified max tokens limit ...

- Issue
   ML Models truncate entries exceeding some max token limit.
   This lowers the quality of search results

- Fix
  Split entries by max tokens before indexing.
  This should improve searching for content in longer entries.

- Miscellaneous
  - Test method to split entries by max tokens

2022-12-23 16:24:04 -03:00

4.9 KiB

Raw Blame History

View Raw