mirror of
https://github.com/khoaliber/khoj.git
synced 2026-03-04 05:39:06 +00:00
04a9a6d62f
- Provides mechanism to update notes from within application
- Instead of having to pass the same arguments multiple times
Pass it once (or rely on defaults when possible) and let app keep
state and location of intermediary files
- Allows user to not have to deal with the internals of the application
- E.g user doesn't have to specify the jsonl.gz or embeddings file path
The app will still put those files in a default location
- The user doesn't have to run the generation from the commandline
as a separate step
Semantic Search
Provide natural language search on user personal content like notes, images using ML models
All data is processed locally. User can interface with semantic-search app via Emacs, API or Commandline
Dependencies
- Python3
- Miniconda
Install
git clone https://github.com/debanjum/semantic-search && cd semantic-search
conda env create -f environment.yml
conda activate semantic-search
Setup
Generate compressed JSONL from specified org-mode files
python3 processor/org-mode/org-to-jsonl.py \
--input-files ~/Notes/Schedule.org ~/Notes/Incoming.org \
--output-file .notes.jsonl.gz \
--verbose
Run
Load ML model, generate embeddings and expose API interface to run user queries on above org-mode files
python3 main.py \
--compressed-jsonl .notes.jsonl.gz \
--embeddings .notes_embeddings.pt \
--verbose
Use
-
Semantic Search via Emacs
- Install semantic-search.el
- Run
M-x semantic-search "<user-query>"or CallC-c C-s
-
Call Semantic Search via API
-
Call Semantic Search via Python Script Directly
python3 search_types/asymmetric.py \ --compressed-jsonl .notes.jsonl.gz \ --embeddings .notes_embeddings.pt \ --results-count 5 \ --verbose \ --interactive
Acknowledgments
- MiniLM Model for Asymmetric Text Search. See SBert Documentation
- OpenAI CLIP Model for Image Search. See SBert Documentation
- Charles Cave for OrgNode Parser