Get Started- Complete documentation overview

Stream AI responses in real-time using Server-Sent Events (SSE) for a better user experience.

Overview

Streaming allows users to see AI responses as they're generated, rather than waiting for the complete response. This provides instant feedback and a more engaging experience.

How It Works

The streaming system uses Server-Sent Events (SSE) to send chunks of the AI response as they're generated:

Client Request → Server → AI Provider (streaming) ↓ SSE Stream ↓ Client (receives chunks in real-time)

API Endpoint

/api/ai/stream

Stream AI responses using SSE.

Request:

POST /api/ai/stream
Content-Type: application/json
Authorization: Bearer <token>

{
  "message": "Explain quantum computing",
  "messages": [...], // Optional conversation history
  "systemPrompt": "...", // Optional
  "temperature": 0.7,
  "maxTokens": 2000,
  "provider": "openai" // Optional
}

Response: Server-Sent Events stream

data: {"content":"Quantum","done":false} data: {"content":" computing","done":false} data: {"content":" is...","done":false} data: {"content":"","done":true}

Client-Side Usage

Using EventSource (Browser)

const eventSource = new EventSource("/api/ai/stream", {
  method: "POST",
  body: JSON.stringify({
    message: "Explain quantum computing",
  }),
});

eventSource.onmessage = (event) => {
  const chunk = JSON.parse(event.data);
  
  if (chunk.done) {
    eventSource.close();
  } else {
    // Append chunk.content to UI
    appendToChat(chunk.content);
  }
};

eventSource.onerror = (error) => {
  console.error("Stream error:", error);
  eventSource.close();
};

Using Fetch with ReadableStream

const response = await fetch("/api/ai/stream", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    message: "Explain quantum computing",
  }),
});

const reader = response.body?.getReader();
const decoder = new TextDecoder();

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  
  const chunk = decoder.decode(value);
  const lines = chunk.split("\n");
  
  for (const line of lines) {
    if (line.startsWith("data: ")) {
      const data = JSON.parse(line.slice(6));
      if (!data.done) {
        appendToChat(data.content);
      }
    }
  }
}

Server-Side Implementation

The streaming endpoint (src/app/api/ai/stream/route.ts) handles:

  1. Authentication - Verifies user is logged in
  2. Subscription Check - Ensures user has active subscription
  3. Content Moderation - Validates input
  4. Streaming - Streams response chunks
  5. Response Streaming - Streams response chunks to client

Stream Chunk Format

Each chunk in the stream follows this format:

interface StreamChunk {
  content: string;      // Text chunk
  done: boolean;        // true when stream is complete
  provider: "openai" | "anthropic";
  model?: string;       // Model used
}

Example: React Component

"use client";

import { useState } from "react";

export function StreamingChat() {
  const [message, setMessage] = useState("");
  const [response, setResponse] = useState("");
  const [isStreaming, setIsStreaming] = useState(false);

  const handleStream = async () => {
    setIsStreaming(true);
    setResponse("");

    const eventSource = new EventSource(
      `/api/ai/stream?message=${encodeURIComponent(message)}`
    );

    eventSource.onmessage = (event) => {
      const chunk = JSON.parse(event.data);
      
      if (chunk.done) {
        eventSource.close();
        setIsStreaming(false);
      } else {
        setResponse((prev) => prev + chunk.content);
      }
    };

    eventSource.onerror = () => {
      eventSource.close();
      setIsStreaming(false);
    };
  };

  return (
    <div>
      <input
        value={message}
        onChange={(e) => setMessage(e.target.value)}
        placeholder="Ask a question..."
      />
      <button onClick={handleStream} disabled={isStreaming}>
        {isStreaming ? "Streaming..." : "Send"}
      </button>
      <div>{response}</div>
    </div>
  );
}

Benefits of Streaming

  1. Better UX - Users see responses immediately
  2. Perceived Performance - Feels faster even if total time is similar
  3. Progressive Loading - Can show partial results
  4. Error Recovery - Can handle errors mid-stream

Best Practices

  1. Show typing indicators while streaming
  2. Handle connection errors gracefully
  3. Allow cancellation of in-progress streams
  4. Debounce rapid requests to prevent abuse
  5. Show progress for long responses

Limitations

  • SSE only works in browsers - Use WebSockets for Node.js clients
  • One-way communication - Client can't send data mid-stream
  • Connection limits - Browsers limit concurrent SSE connections

Next Steps