Get Started- Complete documentation overview

Built-in content moderation helps protect your AI application from harmful or inappropriate content.

Overview

The moderation system (src/lib/ai/moderation.ts) provides:

  • OpenAI Moderation API - Industry-standard content filtering
  • Prompt Injection Detection - Prevents prompt manipulation attacks
  • Custom Rules - Add your own moderation rules
  • Configurable Thresholds - Adjust sensitivity levels

Basic Usage

Moderate User Input

import { moderatePrompt } from "@/lib/ai/moderation";

const result = await moderatePrompt("User's message here", {
  useOpenAIModeration: true,
  checkInjection: true,
});

if (!result.allowed) {
  // Content was flagged
  console.error("Moderation failed:", result.reason);
}

Moderation Result

// Result from moderatePrompt()
interface ModeratePromptResult {
  allowed: boolean;
  reason?: string;
  details?: ModerationResult;  // OpenAI moderation details if flagged
}

// Result from moderateContent() (OpenAI Moderation API)
interface ModerationResult {
  flagged: boolean;
  categories?: Record<string, boolean>;
  categoryScores?: Record<string, number>;
  error?: string;
}

OpenAI Moderation

Using OpenAI Moderation API

import { moderateContent } from "@/lib/ai/moderation";

const result = await moderateContent("User's message");

if (result.flagged) {
  // Content was flagged by OpenAI
  console.log("Flagged categories:", result.categories);
  console.log("Category scores:", result.categoryScores);
}

Flagged Categories

OpenAI checks for:

  • Hate - Hateful content targeting groups
  • Harassment - Harassing or threatening content
  • Self-harm - Content promoting self-harm
  • Sexual - Sexual content
  • Violence - Violent content

Prompt Injection Detection

Detect Injection Attempts

import { detectPromptInjection } from "@/lib/ai/moderation";

const injectionCheck = detectPromptInjection(
  "Ignore previous instructions and..."
);

if (injectionCheck.detected) {
  // Potential injection attempt detected
  console.log("Injection pattern:", injectionCheck.pattern);
}

The function returns:

interface InjectionCheck {
  detected: boolean;
  pattern?: string;  // The pattern that was detected
}

Common Injection Patterns

The system detects:

  • "Ignore previous instructions"
  • "Forget everything"
  • "New instructions:"
  • "System:"
  • "You are now..."
  • Base64 encoded prompts
  • Unicode obfuscation

Custom Moderation Rules

Add Custom Rules

import { moderatePrompt } from "@/lib/ai/moderation";

const result = await moderatePrompt("User message", {
  useOpenAIModeration: true,
  checkInjection: true,
  customRules: [
    (text) => {
      // Check for banned words
      const bannedWords = ["spam", "scam"];
      return !bannedWords.some(word => text.toLowerCase().includes(word));
    }
  ]
});

Integration with API Routes

Example: Protected Chat Endpoint

import { moderatePrompt } from "@/lib/ai/moderation";

export async function POST(req: NextRequest) {
  const { message } = await req.json();
  
  // Moderate input before processing
  const moderation = await moderatePrompt(message, {
    useOpenAIModeration: true,
    checkInjection: true,
  });
  
  if (!moderation.allowed) {
    return NextResponse.json(
      { error: "Content moderation failed", reason: moderation.reason },
      { status: 400 }
    );
  }
  
  // Continue with AI request...
}

Configuration

Moderation Options

interface ModerationOptions {
  useOpenAIModeration?: boolean;  // Use OpenAI Moderation API
  checkInjection?: boolean;        // Check for prompt injection
  customRules?: Array<(text: string) => boolean>;  // Custom rules
  threshold?: number;              // Sensitivity threshold (0-1)
}

Best Practices

  1. Always moderate user input - Never trust user content
  2. Moderate before AI calls - Catch issues early
  3. Log moderation failures - Track patterns for abuse
  4. Provide clear errors - Tell users why content was rejected
  5. Combine multiple checks - Use OpenAI + injection detection
  6. Update rules regularly - Adapt to new attack patterns

Error Handling

try {
  const moderation = await moderatePrompt(userInput);
  
  if (!moderation.allowed) {
    // Handle moderation failure
    return {
      error: "Content not allowed",
      reason: moderation.reason
    };
  }
} catch (error) {
  // Handle moderation API errors
  console.error("Moderation error:", error);
  // Fallback: reject or use alternative moderation
}

Security Considerations

  • Rate limit moderation calls - Prevent abuse of moderation API
  • Cache results - Reduce API calls for repeated content
  • Monitor false positives - Adjust thresholds as needed
  • Keep rules updated - Stay current with attack patterns
  • Log everything - Audit trail for security incidents

Next Steps