If you've ever spent an hour staring at a Could not automatically determine credentials error while trying to get the Vision API working, you're not alone.
The Google Cloud Vision API is genuinely powerful — drop in an image, get back labels, text, faces, landmarks — but the initial setup has a few friction points that documentation doesn't explain cleanly. Throw in Antigravity's project configuration layer and it's easy to get lost.
This guide walks through the full Python integration: service account setup, managing credentials the right way inside Antigravity, writing API calls with proper error handling, and wiring it all into a FastAPI endpoint. Everything here is based on code that actually runs.
What Google Cloud Vision API Can Do
The API accepts an image (either as raw bytes or a public URL) and returns structured analysis results:
- Label detection — classifies the contents ("dog", "park", "overcast sky")
- OCR / text detection — reads printed and handwritten text
- Face detection — locates faces and scores emotions (joy, surprise, etc.)
- Landmark detection — identifies famous locations
- Safe search — flags potentially inappropriate content
For most personal projects, label detection and OCR are the workhorses. Automatically categorizing user-uploaded photos, digitizing business cards, parsing receipts — these are the use cases Vision API handles well. The free tier covers 1,000 units per month, which is plenty for prototyping.
Setting Up Antigravity for a Vision API Project
Define Your Context with .rules
Before writing any code, I add a .rules file at the project root. This tells Antigravity's AI assistant what kind of project it's working on, which significantly improves the quality of generated code:
# .rules (project root)
You are working on a Python project using Google Cloud Vision API.
- Always use google-cloud-vision >= 3.7 package
- Store API credentials in .env file, never hardcode
- Use GOOGLE_APPLICATION_CREDENTIALS for service account auth
- Error handling is required for all API calls
- Use Python 3.11+With this in place, when you ask Antigravity to "write a function that detects labels in an image," it will automatically generate code that reads credentials from .env and includes proper error handling — without you having to specify those requirements every time. It's a small investment that compounds quickly.
Managing Credentials Safely
The most common mistake I see with Vision API projects: hardcoding the path to the service account JSON file.
# ❌ Don't do this
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/Users/masaki/keys/service-account.json"The right approach is to keep it in a .env file that stays out of version control:
# .env (add this to .gitignore immediately)
GOOGLE_APPLICATION_CREDENTIALS=./credentials/service-account.json
GOOGLE_CLOUD_PROJECT=your-project-id# In your application code
from dotenv import load_dotenv
load_dotenv()
# The google-cloud library picks up GOOGLE_APPLICATION_CREDENTIALS automaticallyThe google-cloud library will find the credentials file automatically from the environment variable — you don't need to pass it explicitly anywhere in your code. For broader guidance on secrets management in Antigravity, see Managing Environment Variables and Secrets in Antigravity.
Writing the API Calls
Label Detection from a File
Here's the core implementation, with error handling that covers the edge case most tutorials skip:
# vision_client.py
import os
from google.cloud import vision
from dotenv import load_dotenv
load_dotenv()
def analyze_image_labels(image_path: str) -> list[dict]:
"""
Analyze a local image file and return detected labels.
Returns:
list[dict]: [{"description": "Dog", "score": 0.98, "topicality": 0.97}, ...]
Raises:
RuntimeError: If the Vision API returns an error in the response body
FileNotFoundError: If the specified file doesn't exist
"""
client = vision.ImageAnnotatorClient()
with open(image_path, "rb") as f:
content = f.read()
image = vision.Image(content=content)
response = client.label_detection(image=image)
# Critical: Vision API can return HTTP 200 with an error in the response body
if response.error.message:
raise RuntimeError(
f"Vision API error: {response.error.message}\n"
"Reference: https://cloud.google.com/apis/design/errors"
)
return [
{
"description": label.description,
"score": round(label.score, 4),
"topicality": round(label.topicality, 4),
}
for label in response.label_annotations
]
if __name__ == "__main__":
labels = analyze_image_labels("sample.jpg")
for label in labels:
print(f"{label['description']}: {label['score']:.1%}")
# Expected output:
# Dog: 98.6%
# Puppy: 95.2%
# Carnivore: 93.8%The response.error.message check is the part that catches people off guard. Unlike most APIs, the Vision API can return a successful HTTP 200 response while carrying an error in the response body. Skip this check and you'll get silent failures that are hard to trace.
Label Detection from a URL
When your app accepts image URLs rather than file uploads, this version is more convenient:
def analyze_image_from_url(image_url: str) -> list[dict]:
"""
Analyze an image at a public URL.
Note:
Only works with publicly accessible URLs.
Authenticated endpoints or internal-only URLs will fail — Google's servers
make the request directly.
Raises:
RuntimeError: If Vision API returns an error response
"""
client = vision.ImageAnnotatorClient()
image = vision.Image()
image.source.image_uri = image_url
response = client.label_detection(image=image)
if response.error.message:
raise RuntimeError(f"Vision API error: {response.error.message}")
return [
{"description": label.description, "score": round(label.score, 4)}
for label in response.label_annotations
]The key constraint: the URL must be publicly accessible. Google's servers fetch the image directly, so any URL that requires authentication or is behind a firewall won't work.
Building a FastAPI Endpoint
Once the Vision API calls are working, wrapping them in a FastAPI endpoint is straightforward. In Antigravity, I use the inline chat (Cmd+I) with a specific prompt:
Integrate this vision_client.py into a FastAPI app.
POST /analyze should accept a JSON body with an image URL and return
the labels list. Use proper HTTP status codes for error cases.
Because of the .rules file, Antigravity knows to use environment variables for credentials and to include error handling. The generated code typically comes out looking like this:
# main.py
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, HttpUrl
from vision_client import analyze_image_from_url
app = FastAPI(title="Vision API Service")
class ImageAnalysisRequest(BaseModel):
url: HttpUrl
class LabelResult(BaseModel):
description: str
score: float
@app.post("/analyze", response_model=list[LabelResult])
async def analyze_image(request: ImageAnalysisRequest):
"""Accepts an image URL and returns detected labels."""
try:
labels = analyze_image_from_url(str(request.url))
return labels
except RuntimeError as e:
# Vision API errors are a downstream failure — use 502
raise HTTPException(status_code=502, detail=f"Vision API error: {e}")
except Exception as e:
raise HTTPException(status_code=500, detail=f"Internal error: {e}")
@app.get("/health")
async def health_check():
return {"status": "ok"}For a deeper look at FastAPI project structure and deployment patterns with Antigravity, see Antigravity × FastAPI Backend Development Guide.
Errors That Trip Up Most Developers
"Could not automatically determine credentials"
Either the path in GOOGLE_APPLICATION_CREDENTIALS is wrong, or the environment variable isn't loaded in the current shell session. Run these in Antigravity's terminal to diagnose:
echo $GOOGLE_APPLICATION_CREDENTIALS
python -c "from google.auth import default; creds, project = default(); print(f'Project: {project}')"One subtle issue: even with load_dotenv() in your code, if you've opened a new terminal session in Antigravity, the .env file may not be loaded yet. Open a fresh terminal, cd into your project directory, and try again.
"Request had insufficient authentication scopes"
Your service account exists but lacks the right IAM role. In Google Cloud Console, navigate to IAM & Admin → Service Accounts, find your account, and add the Cloud Vision API User role.
Quota exceeded during development
The 1,000 unit free tier goes faster than expected when you're running tests with new images. A practical workaround: reuse the same image URL in tests, and add a simple counter guard during development:
# Development-only rate limiting guard
_request_count = 0
_MAX_DEV_REQUESTS = 50
def analyze_with_dev_limit(image_url: str) -> list[dict]:
global _request_count
if _request_count >= _MAX_DEV_REQUESTS:
raise RuntimeError("Development request limit reached — check your Vision API quota")
_request_count += 1
return analyze_image_from_url(image_url)Where to Go Next
Once label detection is working, OCR is the natural next step. Switch client.label_detection() to client.text_detection() and the same patterns apply — it's a quick win that opens up document parsing, receipt digitization, and similar use cases.
For more sophisticated pipelines, Vision API pairs naturally with Gemini: extract structured information from images with Vision, then use Gemini to reason over or summarize that information in natural language. The Antigravity × Gemini API Advanced Integration Guide covers that combination in detail.
Start with just the analyze_image_labels() function — get one successful API response back, and the rest will follow naturally from there.