khoj/documentation/docs/advanced/gcp-vertex.md

Google Vertex AI

:::info This is only helpful for self-hosted users. If you're using Khoj Cloud, you can directly use any of the pre-configured AI models. :::

Khoj can use Google's Gemini and Anthropic's Claude family of AI models from Vertex AI on Google Cloud. Explore Anthropic and Gemini AI models available on Vertex AI's Model Garden.

Setup

1. Follow these instructions to use models on GCP Vertex AI.
2. Create Service Account credentials.
   • Download the credentials keyfile in json format.
   • Base64 encode the credentials json keyfile. For example by running the following command from your terminal: base64 -i <service_account_credentials_keyfile.json>
3. Create a new API Model API on your Khoj admin panel.
   • Name: Google Vertex (or whatever friendly name you prefer).
   • Api Key: base64 encoded json keyfile from step 2.
   • Api Base Url: https://{MODEL_GCP_REGION}-aiplatform.googleapis.com/v1/projects/{YOUR_GCP_PROJECT_ID}
     • MODEL_GCP_REGION: A region the AI model is available in. For example us-east5 works for Claude.
     • YOUR_GCP_PROJECT_ID: Get your project id from the Google cloud dashboard
4. Create a new Chat Model on your Khoj admin panel.
   • Name: claude-3-7-sonnet@20250219. Any Claude or Gemini model on Vertex's Model Garden should work.
   • Model Type: Anthropic or Google
   • Ai Model API: the Google Vertex Ai Model API you created in step 3
   • Max prompt size: 60000 (replace with the max prompt size of your model)
   • Tokenizer: Do not set
5. Select the chat model on your settings page and start a conversation.

Troubleshooting & gcp AI Tips

• Permission Denied?
  Ensure your service account has the Vertex AI User role and that the API is enabled in your GCP project.

• Region Errors?
  Double-check that the model you're trying to use is supported in your selected region. Some Claude or Gemini models are restricted to specific zones like us-east5 or us-central1.

• Prompt Size Limitations
  The "Max prompt size" should align with the limits defined in the model documentation. Exceeding it can silently fail or truncate inputs.

• Testing the API Key
  Before adding it to Khoj, you can verify that your key works by making a simple curl request to Vertex AI. This helps debug auth issues early.

• Use Environment Variables
  For better security, consider using environment variables to manage sensitive keys and inject them at runtime during base64 encoding.

If you encounter any issues, the Khoj Discord is a great place to ask for help!