Add AI Comment Moderation to a React and Next.js App

Open comment boxes attract insults, harassment and link spam within days. Keyword blocklists miss creative spelling and flag innocent words. In this tutorial you will add AI comment moderation to a React and Next.js app: every comment is checked on the server before it is saved, and users get instant, polite feedback in the UI.

Why keyword filters fail

A blocklist is the first thing most teams try, and it breaks in two directions at once. It misses abuse that is spelled creatively (id10t, spaced-out letters, emoji substitutions) and it over-blocks innocent text, such as a comment about the town of Scunthorpe or a user complaining that a product is "the worst". Neither problem gets better by adding more words to the list.

Language-model moderation reads the whole comment in context. It can tell harsh but legitimate criticism from harassment, and a polite sentence with a suspicious link from a genuine recommendation. That distinction is the difference between a community people trust and one that either fills with abuse or silences honest feedback.

1. Keep the API key on the server

Your RSFlowHub key is a secret. Anything in client-side React code can be read by visitors, so the browser should talk to your backend, and only your backend talks to the moderation API:

1. React form sends the comment to /api/comments.
2. The Next.js route handler calls POST https://rsflowhub.com/api/v1/ai/comment-moderate with the secret key.
3. Safe comments are saved; unsafe ones are rejected or held for review.

Add the key to .env.local without the NEXT_PUBLIC_ prefix so it is never bundled for the browser:

RSFLOWHUB_KEY=rfh_your_api_key

2. Create the route handler

import { NextResponse } from 'next/server';

type Moderation = {
  is_safe: boolean;
  categories: string[];
  severity: 'none' | 'low' | 'medium' | 'high';
  recommended_action: string;
  reason: string;
  confidence: number;
};

async function moderate(comment: string, context?: string): Promise<Moderation | null> {
  const res = await fetch('https://rsflowhub.com/api/v1/ai/comment-moderate', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'X-API-Key': process.env.RSFLOWHUB_KEY!,
    },
    body: JSON.stringify({ comment, platform: 'blog', context }),
    signal: AbortSignal.timeout(30_000),
  });

  if (!res.ok) return null;
  const json = await res.json();
  return json.data.result as Moderation;
}

export async function POST(request: Request) {
  const { comment, postTitle } = await request.json();

  if (typeof comment !== 'string' || comment.trim().length < 2) {
    return NextResponse.json({ error: 'Comment is empty.' }, { status: 422 });
  }

  const result = await moderate(comment, postTitle ? `Comment on the article: ${postTitle}` : undefined);

  // If moderation is unavailable, hold the comment instead of publishing it.
  if (!result) {
    // await db.comment.create({ data: { body: comment, status: 'pending' } });
    return NextResponse.json({ status: 'pending' });
  }

  if (!result.is_safe && result.severity !== 'low') {
    return NextResponse.json(
      { status: 'rejected', reason: 'Please keep the conversation respectful.' },
      { status: 400 },
    );
  }

  // await db.comment.create({ data: { body: comment, status: result.is_safe ? 'published' : 'pending' } });
  return NextResponse.json({ status: result.is_safe ? 'published' : 'pending' });
}

The moderation result tells you what was found (categories such as harassment or spam), how bad it is (severity) and what to do (recommended_action), plus a short human-readable reason you can log for your moderators.

Two optional fields improve accuracy. platform tells the model what kind of space it is moderating, and context is a short description of what the comment is replying to. A comment such as "this is a scam" means something very different under a phishing warning than under a product review, so even one sentence of context helps.

3. Build the React comment form

'use client';

import { useState } from 'react';

export default function CommentForm({ postTitle }: { postTitle?: string }) {
  const [comment, setComment] = useState('');
  const [message, setMessage] = useState<string | null>(null);
  const [sending, setSending] = useState(false);

  async function submit(e: React.FormEvent) {
    e.preventDefault();
    setSending(true);
    setMessage(null);

    const res = await fetch('/api/comments', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ comment, postTitle }),
    });
    const data = await res.json();

    if (data.status === 'published') {
      setComment('');
      setMessage('Thanks! Your comment is live.');
    } else if (data.status === 'pending') {
      setComment('');
      setMessage('Thanks! Your comment will appear after review.');
    } else {
      setMessage(data.reason ?? 'Your comment could not be posted.');
    }
    setSending(false);
  }

  return (
    <form onSubmit={submit}>
      <textarea value={comment} onChange={(e) => setComment(e.target.value)} rows={4} required />
      <button type="submit" disabled={sending}>{sending ? 'Checking...' : 'Post comment'}</button>
      {message && <p role="status">{message}</p>}
    </form>
  );
}

4. Understand the moderation result

A typical response for a spam comment looks like this, and each field maps to a decision in your code:

{
  "success": true,
  "data": {
    "result": {
      "is_safe": false,
      "categories": ["spam", "promotional"],
      "severity": "high",
      "recommended_action": "flag",
      "reason": "Comment contains suspicious promotional URLs attempting to sell fake engagement services.",
      "confidence": 0.99
    }
  }
}

5. Choose your moderation rules

• Strict communities (kids, education): publish only when is_safe is true; hold everything else.
• General communities: reject medium and high severity, hold low severity for review, publish the rest.
• Spam: the categories array includes spam, so promotional comments with links can be rejected even when they are polite.
• Fail closed: if the API times out, save the comment as pending instead of publishing it.

Moderation calls take about 1 to 2 seconds, which is acceptable while the button shows a loading state. For very busy sites, save comments as pending immediately and moderate them in a background queue instead.

6. Add a review queue for held comments

Held comments need somewhere to go. A minimal queue is a comments table with a status column (published, pending, rejected) plus the fields from the moderation result, so a moderator sees why the comment was held without re-running anything:

model Comment {
  id          Int      @id @default(autoincrement())
  postId      Int
  body        String
  status      String   @default("pending") // published | pending | rejected
  severity    String?
  categories  String[]
  reason      String?
  confidence  Float?
  createdAt   DateTime @default(now())
}

Saving the severity, categories, reason and confidence next to each comment also gives you data for tuning. After a few weeks you can see how many held comments moderators approved. If most are approved, your rules are too strict; if moderators keep rejecting comments that were published, they are too loose.

7. Server Actions and the Pages Router

You do not have to use a route handler. In the App Router you can call the same moderate function from a Server Action, which removes the need for a separate fetch from the form:

'use server';

export async function postComment(formData: FormData) {
  const comment = String(formData.get('comment') ?? '').trim();
  if (comment.length < 2) return { status: 'error', reason: 'Comment is empty.' };

  const res = await fetch('https://rsflowhub.com/api/v1/ai/comment-moderate', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json', 'X-API-Key': process.env.RSFLOWHUB_KEY! },
    body: JSON.stringify({ comment, platform: 'blog' }),
  });

  if (!res.ok) return { status: 'pending' }; // fail closed

  const { data } = await res.json();
  const result = data.result;
  if (!result.is_safe && result.severity !== 'low') {
    return { status: 'rejected', reason: 'Please keep the conversation respectful.' };
  }
  return { status: result.is_safe ? 'published' : 'pending' };
}

With the Pages Router the logic is identical; only the wrapper changes. Export a default handler from pages/api/comments.ts, read req.body, and answer with res.status(...).json(...). The request to RSFlowHub and the decision rules stay exactly the same.

8. Test and tune the rules

Before launch, run a mixed set of comments through the route and compare what happens to what you expected. Include the borderline cases, not only obvious abuse:

• A normal, friendly comment (should publish).
• A harsh but legitimate criticism (should publish, not be rejected).
• A polite comment containing a promotional link (should be held or rejected as spam).
• A direct insult aimed at another commenter (should be rejected).
• A comment in another language your audience uses.

Record what the API returns for each and adjust the severity thresholds in the route handler, not the model. If you find the line between "hold" and "reject" in the wrong place, moving it is a one-line change in the condition that checks severity.

9. Common mistakes to avoid

• Moderating only on the client. Client checks can be skipped by calling your endpoint directly. The server check is the one that counts.
• Publishing when the API fails. A timeout should hold the comment as pending, not let it through.
• No feedback for users. Rejecting silently makes people retry. Show a short, polite message, but do not reveal exactly which rule triggered, or abusers will learn to work around it.
• No rate limiting. Without a limit per user or IP, one bot can run up your request count. Add a simple limit before the moderation call.
• Sending the entire thread. Moderate the new comment, plus a short context string. Sending more costs more and rarely helps.

Frequently Asked Questions