mirror of
https://github.com/khoaliber/khoj.git
synced 2026-03-07 21:29:13 +00:00
Update README and default folders in docker_sample_config.yml
- Add instruction to using Docker with README - Use the ./tests/data folder in docker_sample_conifg.yml so it can work right away for users
This commit is contained in:
88
README.org
88
README.org
@@ -5,32 +5,52 @@
|
|||||||
|
|
||||||
All data is processed locally. User can interface with semantic-search app via [[./src/interface/emacs/semantic-search.el][Emacs]], API or Commandline
|
All data is processed locally. User can interface with semantic-search app via [[./src/interface/emacs/semantic-search.el][Emacs]], API or Commandline
|
||||||
|
|
||||||
** Dependencies
|
** Setup
|
||||||
- Python3
|
|
||||||
- [[https://docs.conda.io/en/latest/miniconda.html#latest-miniconda-installer-links][Miniconda]]
|
|
||||||
|
|
||||||
** Install
|
*** Setup using Docker
|
||||||
#+begin_src shell
|
|
||||||
git clone https://github.com/debanjum/semantic-search && cd semantic-search
|
|
||||||
conda env create -f environment.yml
|
|
||||||
conda activate semantic-search
|
|
||||||
#+end_src
|
|
||||||
|
|
||||||
*** Install Environmental Dependencies
|
**** 1. Clone Repository
|
||||||
#+begin_src shell
|
#+begin_src shell
|
||||||
sudo apt-get -y install libimage-exiftool-perl
|
git clone https://github.com/debanjum/semantic-search && cd semantic-search
|
||||||
#+end_src
|
#+end_src
|
||||||
|
|
||||||
** Configure
|
**** 2. Configure
|
||||||
Configure application search types and their underlying data source/files in ~sample_config.yml~
|
Add Content Directories for Semantic Search to Docker-Compose
|
||||||
Use the ~sample_config.yml~ as reference
|
Update [[./docker-compose.yml][docker-compose.yml]] to mount your images, org-mode notes, ledger/beancount directories
|
||||||
|
If required, edit config settings in [[./docker_sample_config.yml][docker_sample_config.yml]].
|
||||||
|
|
||||||
** Run
|
**** 3. Run
|
||||||
Load ML model, generate embeddings and expose API to query notes, images, transactions etc specified in config YAML
|
#+begin_src shell
|
||||||
|
docker-compose up -d
|
||||||
|
#+end_src
|
||||||
|
|
||||||
#+begin_src shell
|
*** Setup on Local Machine
|
||||||
python3 -m src.main -c=sample_config.yml -vv
|
|
||||||
#+end_src
|
**** 1. Install Dependencies
|
||||||
|
1. Install Python3 [Required[
|
||||||
|
2. [[https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html][Install Conda]] [Required]
|
||||||
|
3. Install Exiftool [Optional]
|
||||||
|
#+begin_src shell
|
||||||
|
sudo apt-get -y install libimage-exiftool-perl
|
||||||
|
#+end_src
|
||||||
|
|
||||||
|
**** 2. Install Semantic Search
|
||||||
|
#+begin_src shell
|
||||||
|
git clone https://github.com/debanjum/semantic-search && cd semantic-search
|
||||||
|
conda env create -f environment.yml
|
||||||
|
conda activate semantic-search
|
||||||
|
#+end_src
|
||||||
|
|
||||||
|
**** 3. Configure
|
||||||
|
Configure application search types and their underlying data source/files in ~sample_config.yml~
|
||||||
|
Use the ~sample_config.yml~ as reference
|
||||||
|
|
||||||
|
**** 4. Run
|
||||||
|
Load ML model, generate embeddings and expose API to query notes, images, transactions etc specified in config YAML
|
||||||
|
|
||||||
|
#+begin_src shell
|
||||||
|
python3 -m src.main -c=sample_config.yml -vv
|
||||||
|
#+end_src
|
||||||
|
|
||||||
** Use
|
** Use
|
||||||
- *Semantic Search via Emacs*
|
- *Semantic Search via Emacs*
|
||||||
@@ -39,19 +59,29 @@
|
|||||||
|
|
||||||
- *Semantic Search via API*
|
- *Semantic Search via API*
|
||||||
- Query: ~GET~ [[http://localhost:8000/search?q=%22what%20is%20the%20meaning%20of%20life%22][http://localhost:8000/search?q="What is the meaning of life"&t=notes]]
|
- Query: ~GET~ [[http://localhost:8000/search?q=%22what%20is%20the%20meaning%20of%20life%22][http://localhost:8000/search?q="What is the meaning of life"&t=notes]]
|
||||||
- Regenerate Embeddings: ~GET~ [[http://localhost:8000/regenerate][http://localhost:8000/regenerate?t=image]]
|
- Regenerate Embeddings: ~GET~ [[http://localhost:8000/regenerate][http://localhost:8000/regenerate]]
|
||||||
- [[http://localhost:8000/docs][Semantic Search API Docs]]
|
- [[http://localhost:8000/docs][Semantic Search API Docs]]
|
||||||
|
|
||||||
|
- *UI to Edit Config*
|
||||||
|
- [[https://localhost:8000/ui][Config UI]]
|
||||||
|
|
||||||
** Upgrade
|
** Upgrade
|
||||||
#+begin_src shell
|
|
||||||
cd semantic-search
|
*** Using Docker
|
||||||
git pull origin master
|
#+begin_src shell
|
||||||
conda env update -f environment.yml
|
docker-compose up
|
||||||
conda activate semantic-search
|
#+end_src
|
||||||
#+end_src
|
|
||||||
|
*** On Local Machine
|
||||||
|
#+begin_src shell
|
||||||
|
cd semantic-search
|
||||||
|
git pull origin master
|
||||||
|
conda env update -f environment.yml
|
||||||
|
conda activate semantic-search
|
||||||
|
#+end_src
|
||||||
|
|
||||||
** Acknowledgments
|
** Acknowledgments
|
||||||
- [[https://huggingface.co/sentence-transformers/msmarco-MiniLM-L-6-v3][MiniLM Model]] for Asymmetric Text Search. See [[https://www.sbert.net/examples/applications/retrieve_rerank/README.html][SBert Documentation]]
|
- [[https://huggingface.co/sentence-transformers/msmarco-MiniLM-L-6-v3][MiniLM Model]] for Asymmetric Text Search. See [[https://www.sbert.net/examples/applications/retrieve_rerank/README.html][SBert Documentation]]
|
||||||
- [[https://github.com/openai/CLIP][OpenAI CLIP Model]] for Image Search. See [[https://www.sbert.net/examples/applications/image-search/README.html][SBert Documentation]]
|
- [[https://github.com/openai/CLIP][OpenAI CLIP Model]] for Image Search. See [[https://www.sbert.net/examples/applications/image-search/README.html][SBert Documentation]]
|
||||||
- Charles Cave for [[http://members.optusnet.com.au/~charles57/GTD/orgnode.html][OrgNode Parser]]
|
- Charles Cave for [[http://members.optusnet.com.au/~charles57/GTD/orgnode.html][OrgNode Parser]]
|
||||||
- Sven Marnach for [[https://github.com/smarnach/pyexiftool/blob/master/exiftool.py][PyExifTool]]
|
- Sven Marnach for [[https://github.com/smarnach/pyexiftool/blob/master/exiftool.py][PyExifTool]]
|
||||||
@@ -20,13 +20,13 @@ services:
|
|||||||
# points to the files you want to index.
|
# points to the files you want to index.
|
||||||
# The path of the mounted directory (right hand side),
|
# The path of the mounted directory (right hand side),
|
||||||
# must match the path prefix in your config file.
|
# must match the path prefix in your config file.
|
||||||
- /path/to/notes/:/data/notes/
|
- ./tests/data/:/data/notes/
|
||||||
- /path/to/photos/:/data/images/
|
- ./tests/data/:/data/images/
|
||||||
- /path/to/ledger/:/data/ledger/
|
- ./tests/data/:/data/ledger/
|
||||||
- /path/to/music/:/data/music/
|
- ./tests/data/:/data/music/
|
||||||
# It's ok if you don't have existing embeddings.
|
# It's ok if you don't have existing embeddings.
|
||||||
# Leave the line as is - an empty volume will be created if it doesn't exist.
|
# You can set this volume to point to an empty folder.
|
||||||
- /path/to/embeddings/:/data/generated/
|
- ./tests/data/:/data/generated/
|
||||||
|
|
||||||
# Use 0.0.0.0 to explicitly set the host ip for the service on the container. https://pythonspeed.com/articles/docker-connection-refused/
|
# Use 0.0.0.0 to explicitly set the host ip for the service on the container. https://pythonspeed.com/articles/docker-connection-refused/
|
||||||
command: --host="0.0.0.0" --port=8000 -c=docker_sample_config.yml
|
command: --host="0.0.0.0" --port=8000 -c=docker_sample_config.yml -vv
|
||||||
|
|||||||
Reference in New Issue
Block a user