Use multi-part form to receive files to index on server

- This uses existing HTTP affordance to process files
  - Better handling of binary file formats as removes need to url encode/decode
  - Less memory utilization than streaming json as files get
    automatically written to disk once memory utilization exceeds preset limits
  - No manual parsing of raw files streams required
This commit is contained in:
Debanjum Singh Solanky
2023-10-11 17:14:15 -07:00
parent 9ba173bc2d
commit 60e9a61647
3 changed files with 21 additions and 35 deletions

View File

@@ -40,6 +40,7 @@ dependencies = [
"dateparser >= 1.1.1",
"defusedxml == 0.7.1",
"fastapi == 0.77.1",
"python-multipart >= 0.0.5",
"jinja2 == 3.1.2",
"openai >= 0.27.0, < 1.0.0",
"tiktoken >= 0.3.2",