Miscellaneous bugs and fixes for chat sessions (#646)

* Display given_name field only if it is not None

* Add default slugs in the migration script

* Ensure that updated_at is saved appropriately, make sure most recent chat is returned for default history

* Remove the bin button from the chat interface, given deletion is handled in the drop-down menus

* Refresh the side panel when a new chat is created

* Improveme tool retrieval prompt, don't let /online fail, and improve parsing of extract questions

* Fix ending chat response by offline chat on hitting a stop phrase

Previously the whole phrase wouldn't be in the same response chunk, so
chat response wouldn't stop on hitting a stop phrase

Now use a queue to keep track of last 3 chunks, and to stop responding
when hit a stop phrase

* Make chat on Obsidian backward compatible post chat session API updates

- Make chat on Obsidian get chat history from
  `responseJson.response.chat' when available (i.e when using new api)
- Else fallback to loading chat history from
  responseJson.response (i.e when using old api)

* Fix detecting success of indexing update in khoj.el

When khoj.el attempts to index on a Khoj server served behind an https
endpoint, the success reponse status contains plist with certs. This
doesn't mean the update failed.

Look for :errors key in status instead to determine if indexing API
call failed. This fixes detecting indexing API call success on the
Khoj Emacs client, even for Khoj servers running behind SSL/HTTPS

* Fix the mechanism for populating notes references in the conversation primer for both offline and online chat

* Return conversation.default when empty list for dynamic prompt selection, send all cmds in telemetry

* Fix making chat on Obsidian backward compatible post chat session API updates

New API always has conversation_id set, not `chat' which can be unset
when chat session is empty.

So use conversation_id to decide whether to get chat logs from
`responseJson.response.chat' or `responseJson.response' instead

---------

Co-authored-by: Debanjum Singh Solanky <debanjum@gmail.com>

src/khoj/processor/conversation/openai/gpt.py

@@ -1,3 +1,4 @@
+import json
 import logging
 from datetime import datetime, timedelta
 from typing import Optional
@@ -29,10 +30,6 @@ def extract_questions(
     """
     Infer search queries to retrieve relevant notes to answer user query
     """
-
-    def _valid_question(question: str):
-        return not is_none_or_empty(question) and question != "[]"
-
     location = f"{location_data.city}, {location_data.region}, {location_data.country}" if location_data else "Unknown"
 
     # Extract Past User Message and Inferred Questions from Conversation Log
@@ -75,23 +72,13 @@ def extract_questions(
 
     # Extract, Clean Message from GPT's Response
     try:
-        split_questions = (
-            response.content.strip(empty_escape_sequences)
-            .replace("['", '["')
-            .replace("']", '"]')
-            .replace("', '", '", "')
-            .replace('["', "")
-            .replace('"]', "")
-            .split('", "')
-        )
-        questions = []
-
-        for question in split_questions:
-            if question not in questions and _valid_question(question):
-                questions.append(question)
-
-        if is_none_or_empty(questions):
-            raise ValueError("GPT returned empty JSON")
+        response = response.strip()
+        response = json.loads(response)
+        response = [q.strip() for q in response if q.strip()]
+        if not isinstance(response, list) or not response or len(response) == 0:
+            logger.error(f"Invalid response for constructing subqueries: {response}")
+            return [text]
+        return response
     except:
         logger.warning(f"GPT returned invalid JSON. Falling back to using user message as search query.\n{response}")
         questions = [text]
@@ -165,7 +152,7 @@ def converse(
                 simplified_online_results[result] = online_results[result]["extracted_content"]
 
         conversation_primer = f"{prompts.online_search_conversation.format(online_results=str(simplified_online_results))}\n{conversation_primer}"
-    if ConversationCommand.Notes in conversation_commands:
+    if not is_none_or_empty(compiled_references):
         conversation_primer = f"{prompts.notes_conversation.format(query=user_query, references=compiled_references)}\n{conversation_primer}"
 
     # Setup Prompt with Primer or Conversation History