Gemini API guide: GenerateContent, Live, and Interactions

Separate Google's recommendation from gateway compatibility

Google made Interactions API generally available in June 2026 and recommends it for all new projects. The original generateContent remains supported but is now Legacy. Endpoint support on this site still depends on gateway implementation; Google's publication of /v1beta/interactions does not make it callable here. The gateway supports GenerateContent plus the native gemini-3.8-live WebSocket below; they use HTTP and WebSocket respectively.

Call GenerateContent through this site

This site recommends Authorization: Bearer; for Gemini-native client compatibility, /v1beta/models/... routes also accept the site key in x-goog-api-key. The model identifier is part of the URL and must come from the model marketplace. contents[].parts can combine text and supported multimodal input; do not send content types that the official model page does not list.

Copy an exact model ID whose marketplace entry explicitly lists the GenerateContent route:

Copyexport BASE_URL="https://api.tu-zi.com"
export API_KEY="your site API key"
export GEMINI_MODEL_NAME="exact model ID that supports GenerateContent"

Copycurl "$BASE_URL/v1beta/models/$GEMINI_MODEL_NAME:generateContent" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"contents":[{"role":"user","parts":[{"text":"Reply only: connected"}]}]}'

Direct Google Interactions shape

Direct Google calls use x-goog-api-key. Interactions posts to /v1beta/interactions with model and input in JSON. Its response contains id, status, usage, and typed steps[]. SDK output_text is convenient for simple final text, but clients must traverse steps by type when reasoning, tools, images, or audio are interleaved.

Multi-turn calls can use previous_interaction_id, but tools, system instructions, and generation configuration are not inherited and must be sent again. store=false supports stateless handling but cannot be combined with continuation from stored interactions.

The response and stream parsers are different

GenerateContent returns candidates, finishReason, promptFeedback, and safety information. Interactions streams typed events such as interaction.created, step.start/delta/stop, and interaction.completed. Function calls are typed steps in the new protocol rather than a function-response part. Errors can occur after HTTP 200, so tolerate unknown events and wait for an explicit completion event.

Errors and retry

Check authentication on 401, permission on 403, model or resource on 404, and rate or quota on 429. Read the error status and message before handling any 5xx: overload or a transient service fault can merit bounded backoff, but an oversized GenerateContent input or context can also surface as 500 or 504 and must be shortened instead of retried. Parameter, safety, and quota problems likewise need a request or account change. Preserve request IDs, error codes, status, and the final step; see API troubleshooting.

Migration boundaries to verify

Interactions does not yet cover every GenerateContent feature, including selected video metadata, Batch, explicit caching, and custom safety settings. Before migration, verify the model, /v1 or /v1beta version, files, tools, structured output, storage policy, and data controls, with separate contract tests for both formats. Do not change only the URL.

gemini-3.8-live native WebSocket

For a Gemini channel, set Base URL to the upstream root and the channel model to gemini-3.8-live; enable this model in the token group too. This gateway upgrades GET /v1beta/live?model=gemini-3.8-live to WebSocket, for example wss://api.tu-zi.com/v1beta/live?model=gemini-3.8-live. Authenticate with this site's Authorization: Bearer key. It uses native Gemini Live messages, not GenerateContent bodies, OpenAI Realtime events, or HTTP POST/SSE. The gateway does not expose the model-less Google SDK WebSocket path.

The first message must be setup with model: models/gemini-3.8-live. This model supports only responseModalities: ["AUDIO"]; set outputAudioTranscription: {} to also read serverContent.outputTranscription.text. Wait for setupComplete before sending input.

Minimal Python example: text input, WAV, and transcript

Use Python 3.11+ on a trusted server. Install python3 -m pip install "websockets>=14,<16". Set BASE_URL to this site's HTTPS root and API_KEY to a site key:

Copyexport BASE_URL="https://api.tu-zi.com"
export API_KEY="your site API key"

The script incrementally writes reply.wav (24 kHz, PCM16, mono) and reply.txt, without buffering the full audio or transcript in memory. Each message is limited to 2 MiB, the receive queue to 4 messages, and the entire operation to 120 seconds. An exception can leave partial files, which the caller should mark incomplete.

Copyimport asyncio, base64, json, os, wave
from urllib.parse import urlencode, urlsplit, urlunsplit
from websockets.asyncio.client import connect

async def main():
    base = urlsplit(os.environ["BASE_URL"].rstrip("/"))
    if base.scheme not in ("https", "http", "wss", "ws"):
        raise ValueError("BASE_URL must be an HTTP(S) or WS(S) origin")
    scheme = "wss" if base.scheme in ("https", "wss") else "ws"
    url = urlunsplit((scheme, base.netloc, "/v1beta/live", urlencode({"model": "gemini-3.8-live"}), ""))
    headers = {"Authorization": "Bearer " + os.environ["API_KEY"]}
    async with asyncio.timeout(120):
        async with connect(url, additional_headers=headers, max_size=2*1024*1024,
                           max_queue=4, compression=None, open_timeout=10,
                           close_timeout=5, ping_interval=20, ping_timeout=20) as ws:
            await ws.send(json.dumps({"setup": {"model": "models/gemini-3.8-live",
                "generationConfig": {"responseModalities": ["AUDIO"]},
                "outputAudioTranscription": {}}}))
            while True:
                event = json.loads(await asyncio.wait_for(ws.recv(), timeout=30))
                if "error" in event:
                    raise RuntimeError(event["error"])
                if "setupComplete" in event:
                    break
            await ws.send(json.dumps({"clientContent": {"turns": [{"role": "user",
                "parts": [{"text": "Say hello briefly."}]}], "turnComplete": True}}))
            with wave.open("reply.wav", "wb") as audio, open("reply.txt", "w", encoding="utf-8") as transcript:
                audio.setnchannels(1)
                audio.setsampwidth(2)
                audio.setframerate(24000)
                while True:
                    event = json.loads(await asyncio.wait_for(ws.recv(), timeout=30))
                    if "error" in event:
                        raise RuntimeError(event["error"])
                    content = event.get("serverContent", {})
                    for part in content.get("modelTurn", {}).get("parts", []):
                        inline = part.get("inlineData", {})
                        if inline.get("mimeType", "").startswith("audio/pcm"):
                            audio.writeframesraw(base64.b64decode(inline["data"], validate=True))
                    text = content.get("outputTranscription", {}).get("text", "")
                    if text:
                        transcript.write(text)
                        transcript.flush()
                    if content.get("turnComplete"):
                        print(json.dumps(event.get("usageMetadata", {}), ensure_ascii=False))
                        break

asyncio.run(main())

Image, realtime audio, and video-frame input

Combine text and JPEG image in clientContent.turns[].parts:

Copy{"clientContent":{"turns":[{"role":"user","parts":[{"text":"Describe this image"},{"inlineData":{"mimeType":"image/jpeg","data":"<JPEG_BASE64>"}}]}],"turnComplete":true}}

An established session also accepts realtimeInput. Audio is 16 kHz PCM16 mono, Base64-encoded in small chunks. Send video as JPEG frames with bounded frame rate, resolution, and frame size. Send audioStreamEnd when audio input ends. Keep consuming model output and do not load an entire recording or video into memory.

Copy{"realtimeInput":{"audio":{"mimeType":"audio/pcm;rate=16000","data":"<PCM16_BASE64>"}}}
{"realtimeInput":{"video":{"mimeType":"image/jpeg","data":"<JPEG_BASE64>"}}}
{"realtimeInput":{"audioStreamEnd":true}}

MOV video: chronological narration

Install ffmpeg, add this function to the Python example, replace its text-input await ws.send(...) with await send_video(ws, os.environ["VIDEO_PATH"]), and increase the outer deadline to asyncio.timeout(600). Wait for setupComplete first and keep the same incremental WAV/transcript receive loop. Keep the connection open until turnComplete.

Copyexport VIDEO_PATH="/path/to/movie.mov"

Copyasync def send_video(ws, video_path):
    import pathlib, tempfile
    # Bound video input to 400 JPEG frames, 640 px wide, at 5-second intervals.
    with tempfile.TemporaryDirectory(prefix="gemini-live-") as directory:
        ffmpeg = await asyncio.create_subprocess_exec(
            "ffmpeg", "-v", "error", "-threads", "2", "-i", video_path,
            "-vf", "fps=1/5:start_time=0,scale=640:-2", "-frames:v", "400",
            "-threads", "2", "-q:v", "5", directory + "/frame_%05d.jpg",
            stdout=asyncio.subprocess.DEVNULL, stderr=asyncio.subprocess.DEVNULL)
        try:
            returncode = await asyncio.wait_for(ffmpeg.wait(), timeout=120)
        finally:
            if ffmpeg.returncode is None:
                ffmpeg.kill()
                await ffmpeg.wait()
        if returncode:
            raise RuntimeError("ffmpeg extraction failed")
        frames = sorted(pathlib.Path(directory).glob("frame_*.jpg"))
        if not frames:
            raise ValueError("No video frames extracted")
        for index, filename in enumerate(frames):
            # Only one bounded image and one JSON message are resident at a time.
            with filename.open("rb") as image:
                jpeg = image.read(1024 * 1024 + 1)
            if len(jpeg) > 1024 * 1024:
                raise ValueError("JPEG exceeds 1 MiB; reduce resolution")
            seconds = index * 5
            timestamp = f"{seconds // 3600:02d}:{seconds % 3600 // 60:02d}:{seconds % 60:02d}"
            await ws.send(json.dumps({"clientContent": {"turns": [{
                "role": "user", "parts": [{"text": f"[{timestamp}]"},
                    {"inlineData": {"mimeType": "image/jpeg",
                     "data": base64.b64encode(jpeg).decode("ascii")}}]}],
                "turnComplete": False}}))
        await ws.send(json.dumps({"clientContent": {"turns": [{"role": "user",
            "parts": [{"text": "以上是一段视频按时间顺序每 5 秒截取的画面，方括号里是该画面在视频中的时间。请按时间顺序解说这段视频。"}]}],
            "turnComplete": True}}))

The example samples the video every 5 seconds, up to 400 frames (about 33 minutes; split longer videos or increase the interval). Intermediate messages use turnComplete: false; only the final prompt ends the turn. It analyzes sampled images without sending the original soundtrack. Send separate 16 kHz PCM16 chunks when dialogue matters. The returned narration can be brief; request an explicit segment length when detailed commentary is required.

Per-turn usage and billing

One WebSocket connection counts as one request. A turn is settled only when serverContent.turnComplete=true, using that turn's usageMetadata. A disconnected, incomplete turn is not charged; completed turns remain charged. Each turn's input includes prior session context, so that context is counted again on later turns. Usage fields include promptTokenCount, promptTokensDetails, responseTokenCount, and responseTokensDetails, with possible thoughtsTokenCount. Reconcile by modality; these per-turn fields are not cumulative snapshots and must not be added twice.

Aggregate counts can differ from modality details: unclassified input remainder is billed as text, and thoughtsTokenCount is separately included in text output even when omitted from totalTokenCount. Reconcile the five normalized token categories against frozen pricing and the applied group multiplier.

The table gives upstream reference prices in USD per million tokens. Actual site prices, group multipliers, and discounts come from the admin console and usage logs. The existing built-in billing expression can be used in model billing configuration:

Copytier("live", p*1.35 + c*8.1 + img*1.8 + ai*5.4 + ao*21.6)

When versioned authoritative billing is enabled, create, save, and publish a billing version for this model in the admin console before calling it. The expression is only a reference and does not create or publish a price automatically.

Errors and closure

For handshake failures, check the site key, balance, model visibility, and channel availability. Handle in-session errors and close codes: 1008 commonly indicates quota or invalid setup/model, 1011 an upstream failure, and 1001 shutdown. Stop input and close resources on disconnect. Reconnection creates a new request; do not automatically replay completed turns.