Add Vision Support (#889)

# Summary of Changes
* New UI to show preview of image uploads
* ChatML message changes to support gpt-4o vision based responses on images
* AWS S3 image uploads for persistent image context in conversations
* Database changes to have `vision_enabled` option in server admin panel while configuring models
* Render previously uploaded images in the chat history, show uploaded images for pending msgs
* Pass the uploaded_image_url through to subqueries
* Allow image to render upon first message from the homepage
* Add rendering support for images to shared chat as well
* Fix some UI/functionality bugs in the share page
* Convert user attached images for chat to webp format before upload
* Use placeholder to attached image for data source, response mode actors
* Update all clients to call /api/chat as a POST instead of GET request
* Fix copying chat messages with images to clipboard

TLDR; Add vision support for openai models on Khoj via the web UI!

---------

Co-authored-by: sabaimran <narmiabas@gmail.com>
Co-authored-by: Debanjum Singh Solanky <debanjum@gmail.com>
This commit is contained in:
Raghav Tirumale
2024-09-09 17:22:18 -05:00
committed by GitHub
parent b553bba1d8
commit 549686a7a4
33 changed files with 740 additions and 417 deletions

View File

@@ -12,15 +12,11 @@ div.main {
div.inputBox {
border: 1px solid var(--border-color);
border-radius: 16px;
margin-bottom: 20px;
gap: 12px;
padding-left: 20px;
padding-right: 20px;
align-content: center;
}
input.inputBox {
border: none;
}
@@ -31,7 +27,7 @@ input.inputBox:focus {
}
div.inputBox:focus {
box-shadow: 0px 8px 16px 0px rgba(0,0,0,0.2);
box-shadow: 0px 8px 16px 0px rgba(0, 0, 0, 0.2);
}
div.chatBodyFull {
@@ -94,7 +90,6 @@ div.agentIndicator {
padding: 10px;
}
@media (max-width: 768px) {
div.chatBody {
grid-template-columns: 0fr 1fr;
@@ -124,5 +119,4 @@ div.agentIndicator {
gap: 0;
grid-template-columns: 1fr;
}
}