Debanjum Singh Solanky e057c8e208 Add method to split entries by specified max tokens limit ...

- Issue
   ML Models truncate entries exceeding some max token limit.
   This lowers the quality of search results

- Fix
  Split entries by max tokens before indexing.
  This should improve searching for content in longer entries.

- Miscellaneous
  - Test method to split entries by max tokens

2022-12-23 16:24:04 -03:00

..

data

Do not version API. Premature given current state of the codebase

2022-10-08 16:32:46 +03:00

__init__.py

Move tests out to project root. Use absolute import in project

2021-09-30 04:12:14 -07:00

conftest.py

Use Base TextToJsonl class to standardize <text>_to_jsonl processors

2022-09-16 00:53:11 +03:00

test_beancount_to_jsonl.py

Use Base TextToJsonl class to standardize <text>_to_jsonl processors

2022-09-16 00:53:11 +03:00

test_chatbot.py

Fix the user intent extraction prompt for GPT. Clean up chatbot test

2022-01-12 10:36:01 -05:00

test_cli.py

Clean-up generated file after image search test run

2022-09-10 21:43:31 +03:00

test_client.py

Do not version API. Premature given current state of the codebase

2022-10-08 16:32:46 +03:00

test_date_filter.py

Remove unused imports, `embeddings' variable from text search tests

2022-10-08 12:06:05 +03:00

test_file_filter.py

Remove unused imports, `embeddings' variable from text search tests

2022-10-08 12:06:05 +03:00

test_helpers.py

Create LRU helper class for caching

2022-09-04 16:31:46 +03:00

test_image_search.py

Remove unused imports, `embeddings' variable from text search tests

2022-10-08 12:06:05 +03:00

test_markdown_to_jsonl.py

Use Base TextToJsonl class to standardize <text>_to_jsonl processors

2022-09-16 00:53:11 +03:00

test_org_to_jsonl.py

Add method to split entries by specified max tokens limit

2022-12-23 16:24:04 -03:00

test_orgnode.py

Fix OrgNode render of entries with property drawers and empty body

2022-09-11 16:09:19 +03:00

test_text_search.py

Use new Text Entry class to track text entries in Intermediate Format

2022-10-08 12:06:05 +03:00

test_word_filter.py

Remove unused imports, `embeddings' variable from text search tests

2022-10-08 12:06:05 +03:00