Debanjum Singh Solanky 86575b2946 Chunk text in preference order of para, sentence, word, character ...

- Previous simplistic chunking strategy of splitting text by space
  didn't capture notes with newlines, no spaces. For e.g in #620

- New strategy will try chunk the text at more natural points like
  paragraph, sentence, word first. If none of those work it'll split
  at character to fit within max token limit

- Drop long words while preserving original delimiters

Resolves #620

2024-04-04 02:41:55 +05:30

..

data

Update text search test since indexing ancestor hierarchy added

2023-11-17 15:26:55 -08:00

__init__.py

Move tests out to project root. Use absolute import in project

2021-09-30 04:12:14 -07:00

conftest.py

Part 1: Server-side changes to support agents integrated with Conversations (#671)

2024-03-23 22:09:38 +05:30

helpers.py

Use llama.cpp for offline chat models

2024-03-26 22:33:01 +05:30

test_cli.py

Add isort to the pre-commit configuration and apply it to the whole project (#595)

2023-12-28 18:04:02 +05:30

test_client.py

Short-circuit API rate limiter for unauthenticated users (#607)

2024-01-17 00:59:52 +05:30

test_conversation_utils.py

Handle msg truncation when question is larger than max prompt size

2024-03-31 15:50:06 +05:30

test_date_filter.py

Improve date filter regexes to extract structured, natural, partial dates

2024-03-30 00:07:19 +05:30

test_file_filter.py

[Multi-User Part 1]: Enable storage of settings for plaintext files based on user account (#498)

2023-10-26 09:42:29 -07:00

test_helpers.py

Part 2: Add web UI updates for basic agent interactions (#675)

2024-03-26 18:13:24 +05:30

test_image_search.py

Add isort to the pre-commit configuration and apply it to the whole project (#595)

2023-12-28 18:04:02 +05:30

test_markdown_to_entries.py

Remove unused Entry to Jsonl converter from text to entry class, tests

2024-04-04 02:41:55 +05:30

test_multiple_users.py

Add isort to the pre-commit configuration and apply it to the whole project (#595)

2023-12-28 18:04:02 +05:30

test_offline_chat_actors.py

Merge branch 'master' into migrate-to-llama-cpp-for-offline-chat

2024-03-31 00:59:20 +05:30

test_offline_chat_director.py

Merge branch 'master' into migrate-to-llama-cpp-for-offline-chat

2024-03-31 00:59:20 +05:30

test_openai_chat_actors.py

Part 2: Add web UI updates for basic agent interactions (#675)

2024-03-26 18:13:24 +05:30

test_openai_chat_director.py

Part 1: Server-side changes to support agents integrated with Conversations (#671)

2024-03-23 22:09:38 +05:30

test_org_to_entries.py

Chunk text in preference order of para, sentence, word, character

2024-04-04 02:41:55 +05:30

test_orgnode.py

Add isort to the pre-commit configuration and apply it to the whole project (#595)

2023-12-28 18:04:02 +05:30

test_pdf_to_entries.py

Remove unused Entry to Jsonl converter from text to entry class, tests

2024-04-04 02:41:55 +05:30

test_plaintext_to_entries.py

Remove unused Entry to Jsonl converter from text to entry class, tests

2024-04-04 02:41:55 +05:30

test_rawconfig.py

Add isort to the pre-commit configuration and apply it to the whole project (#595)

2023-12-28 18:04:02 +05:30

test_text_search.py

Chunk text in preference order of para, sentence, word, character

2024-04-04 02:41:55 +05:30

test_word_filter.py

Fix test word filter

2023-11-19 13:14:58 -08:00