Debanjum c015eeb5dd Improve Online Search: Parallelize Search, Use Jina Reader API by default (#832)
- Overview
  Khoj wil be able to do online search out of the box, even for self-hosted users
  - Default to Jina search, reader API when no Serper.dev, Olostep API keys
  - Run online searches in parallel to process multiple queries faster

- Details
  - Jina provides a [reader API](https://github.com/jina-ai/reader) for online search and web page reading
     It requires no API key. This provides a good default to enable 
     online search for self-hosted readers requiring no additional setup. 

  - Jina search API also returns webpage contents with the results, so
     just use those directly when Jina Search API used instead of
     trying to read webpages separately. The extract relevant content from
     webpage step using a chat model is still used from the
    `read_webpage_and_extract_content' func in this case.

  - Parse search results from Jina search API into same format as
     Serper.dev for accurate rendering of online references by clients

  - Run online searches in parallel with AsyncIO to process multiple queries faster
2024-07-02 17:44:51 +05:30
2024-06-26 10:38:16 +05:30
2024-06-26 10:38:16 +05:30

Khoj Logo

test dockerize pypi Discord

The open-source, personal AI for your digital brain

🤖 Read Docs   •   🏮 Khoj Cloud   •   💬 Get Involved   •   📚 Read Blog


Khoj is an application that creates always-available, personal AI agents for you to extend your capabilities.

  • You can share your notes and documents to extend your digital brain.
  • Your AI agents have access to the internet, allowing you to incorporate realtime information.
  • Khoj is accessible on Desktop, Emacs, Obsidian, Web and Whatsapp.
  • You can share pdf, markdown, org-mode, notion files and github repositories.
  • You'll get fast, accurate semantic search on top of your docs.
  • Your agents can create deeply personal images and understand your speech.
  • Khoj is open-source, self-hostable. Always.

See it in action

Khoj Demo

Go to https://app.khoj.dev to see Khoj live.

Full feature list

You can see the full feature list here.

Self-Host

To get started with self-hosting Khoj, read the docs.

Contributors

Cheers to our awesome contributors! 🎉

Made with contrib.rocks.

Interested in Contributing?

We are always looking for contributors to help us build new features, improve the project documentation, or fix bugs. If you're interested, please see our Contributing Guidelines and check out our Contributors Project Board.

Sponsors

Shout out to our brilliant sponsors! 🌈

Description
No description provided
Readme AGPL-3.0 116 MiB
Languages
Python 51%
TypeScript 36.1%
CSS 4.1%
HTML 3.2%
Emacs Lisp 2.4%
Other 3.1%