Debanjum Singh Solanky e057c8e208 Add method to split entries by specified max tokens limit ...

- Issue
   ML Models truncate entries exceeding some max token limit.
   This lowers the quality of search results

- Fix
  Split entries by max tokens before indexing.
  This should improve searching for content in longer entries.

- Miscellaneous
  - Test method to split entries by max tokens

2022-12-23 16:24:04 -03:00

..

conversation

Convert search_filter, conversation dir to proper modules

2022-08-02 20:23:42 +03:00

ledger

Use new Text Entry class to track text entries in Intermediate Format

2022-10-08 12:06:05 +03:00

markdown

Use new Text Entry class to track text entries in Intermediate Format

2022-10-08 12:06:05 +03:00

org_mode

Use new Text Entry class to track text entries in Intermediate Format

2022-10-08 12:06:05 +03:00

__init__.py

Move application files under src directory. Update Readmes

2021-08-17 04:11:03 -07:00

text_to_jsonl.py

Add method to split entries by specified max tokens limit

2022-12-23 16:24:04 -03:00