User Profile Moderation

Moderate user profiles with AI-powered scanning, admin reset tools, profile blocklists, and webhook automation.

SDK v7.x · Last verified March 2026 · iOS · Android · Web · Flutter

Speed run — just the code

// 1. Flag a user profile
await UserRepository.flagUser('userId', 'inappropriate_profile');

// 2. Unflag a user profile
await UserRepository.unflagUser('userId');

// 3. Update a user profile (triggers AI scan)
await UserRepository.updateUser('userId', {
  displayName: 'New Display Name',
  description: 'Updated bio text',
});

Full walkthrough below ↓

Platform note — code samples below use TypeScript. Every method has an equivalent in the iOS (Swift), Android (Kotlin), and Flutter (Dart) SDKs — see the linked SDK reference in each step.

Content moderation catches bad posts — but what about bad profiles? Offensive display names, NSFW avatars, and scam-laden bios erode trust before a user even posts. This guide covers the full user profile moderation pipeline: AI-powered scanning, the admin review workflow, profile reset tools, and blocklists.

Prerequisites: SDK installed and authenticated, Admin Console access for moderation configuration.

Also recommended: Complete Content Moderation Pipeline first — user profile moderation extends the same AI and console infrastructure.

After completing this guide you’ll have:

• User profile flagging via SDK (flag / unflag users)
• AI scanning of display names, descriptions, and avatars
• Admin review workflow with reset, clear, and ban actions
• Profile blocklist configured for display names and descriptions

---

How Profile Moderation Differs from Content Moderation

User profile moderation is flag-only — unlike posts and messages, flagged profiles are never auto-deleted. The block confidence threshold that auto-removes content does not apply to profiles. Admins must explicitly decide what to do.

Behavior	Content (Posts / Messages)	User Profiles
Pre-moderation	✅ Images blocked before upload	✅ Avatar images blocked before upload
Post-moderation	✅ AI scans after publish	✅ AI scans after profile save
Auto-delete at block confidence	✅ Content removed automatically	❌ Never — flag only
Admin reset	N/A	✅ Reset display name, avatar, description

Flag-only for profiles: Flagged profiles always require manual admin review. This prevents automated systems from silently wiping user accounts.

---

Step-by-Step Implementation

1. Flag a user profile from the SDK

   Let users report profiles they find inappropriate. Flagged profiles enter the moderation queue alongside AI-flagged profiles.

   TypeScript

   import { UserRepository } from '@amityco/ts-sdk';
   
   // Flag a user with a reason
   await UserRepository.flagUser('user-456');
   
   // Unflag if the report was a mistake
   await UserRepository.unflagUser('user-456');
   

   Full reference → Content Flagging

2. AI scanning — what gets checked

   When a user saves profile changes, AI scans three fields automatically:

   Field	Scan Type
   Display Name	Text
   Description	Text
   Avatar (uploaded file)	Image

   Text is scanned for: harassment, sexual content, violence, hate speech, scam promotion, self-harm, and PII (URLs, person types).

   Avatars are scanned for: nudity, suggestive imagery, violence, hate symbols, and substance-related content.

   External avatar URL gap: Avatars set via avatarCustomUrl bypass the upload pipeline — no pre-moderation image scan runs. Only post-moderation scanning applies.

3. Review flagged profiles in the Admin Console

   Flagged profiles appear in Admin Console → Moderation → Moderation Feed → Users tab.

   Each flagged profile shows:

   • AI moderation labels with category counts (e.g., “Hate (2)”, “Violence (1)”)
   • Which field triggered the flag — display name or description
   • User report count from community members
   • Last flagged timestamp

   Available actions:

   • Reset profile — restore flagged fields to safe defaults
   • Ban globally — ban the user across all communities
   • Clear flag — dismiss the flag and approve the profile

   → AI User Profile Moderation

4. Reset a flagged profile

   When resetting a profile, fields are restored to safe defaults:

   Field	Reset behavior
   Display Name	Set to {prefix}{UUID} (e.g., User_a1b2c3d4e5f6)
   Description	Cleared to empty
   Avatar	Removed — reverts to default; original file deleted from storage

   The user receives a USER_PROFILE_RESET notification through the notification tray informing them which fields were reset.

   Avatar reset is irreversible — the original image file is deleted from storage. Only the reset event is logged; the original image cannot be recovered.

5. Configure the profile blocklist

   Block specific words and phrases from appearing in display names and descriptions.

   In Admin Console → Moderation → Blocklist, add terms under the User Profile category. This blocklist is independent from the content blocklist — you can maintain different blocked terms for profiles versus posts.

   Blocklist Category	Applies to
   Content	Posts, comments, messages
   User Profile	Display names, descriptions

---

Admin Console: Moderation Feed — Users Tab

The Users tab in the Moderation Feed is the central hub for profile moderation.

To Review

Displays all profiles requiring moderator attention — both AI-flagged and user-reported.

• AI moderation labels with detected categories and occurrence counts
• (Profile description is flagged) label when the description triggered the flag
• User report counts from other community members
• Actions: Reset profile, Ban globally, Clear flag

Reviewed

Displays profiles that have already been moderated, with the action taken:

• Flag cleared by [moderator]
• Globally banned by [moderator]
• Profile reset by [moderator]

Use the Select moderator filter to view actions by a specific team member.

---

Confidence Thresholds

Profile moderation reuses the same confidence thresholds configured for content moderation — but only the flag threshold applies.

Threshold	Applies to profiles?	Behavior
Flag Confidence (default: 40)	✅ Yes	Profile flagged for admin review
Block Confidence (default: 80)	❌ No	Does not auto-delete profile fields

Start with medium flag confidence (40–60) and adjust based on false positive rates. Since profiles are never auto-deleted, a lower threshold is safer — it just means more items in the review queue.

---

Webhook Automation

Profile moderation events trigger webhooks for downstream automation — sending notifications, syncing with CRM systems, or logging incidents.

Node.js

const express = require('express');
const crypto = require('crypto');
const app = express();

app.use(express.json());

app.post('/webhook', (req, res) => {
  const signature = req.headers['x-amity-signature'];
  const payload = JSON.stringify(req.body);
  const secret = process.env.WEBHOOK_SECRET;

  const expectedSig = crypto
    .createHmac('sha256', secret)
    .update(payload)
    .digest('hex');

  if (!crypto.timingSafeEqual(
    Buffer.from(signature, 'hex'),
    Buffer.from(expectedSig, 'hex')
  )) {
    return res.status(401).json({ error: 'Invalid signature' });
  }

  const { event, data } = req.body;

  switch (event) {
    case 'user.didFlag':
      notifyModerationTeam(data);
      break;
    case 'user.didBan':
      revokeAccessAcrossSystems(data.userId);
      break;
  }

  res.status(200).json({ received: true });
});

→ Webhook Events Reference

---

Common Mistakes

Relying only on AI scanning without a blocklist — AI moderation catches broad categories but may miss brand-specific terms, competitor names, or community-specific slurs. Always pair AI scanning with a curated blocklist for display names.

Not handling avatarCustomUrl profiles — Avatars set via external URLs bypass the upload pipeline and skip pre-moderation. Ensure post-moderation is enabled to catch these, and consider restricting avatarCustomUrl in sensitive communities.

Auto-banning on first flag — A single AI flag or user report may be a false positive. Use the reset action for first offenses and escalate to bans for repeat violations. Check user history for patterns before banning.

Best Practices

Graduated enforcement

Apply escalating actions based on violation history:

1. First offense — Reset the flagged field and notify the user
2. Second offense — Reset + temporary mute
3. Repeat violations — Global ban

Check the User History page for past moderation events before deciding on an action.

Blocklist maintenance

• Review and update the profile blocklist monthly — slang and evasion tactics evolve
• Add leetspeak and Unicode variants of blocked terms (e.g., h4te, ⓗⓐⓣⓔ)
• Keep the profile blocklist separate from the content blocklist — profile terms are often different (impersonation names, brand abuse, etc.)

Team workflow

• Assign profile moderation to a dedicated moderator or rotation — profile flags require different judgment than content flags
• Use the Select moderator filter in the Reviewed tab for audit and load-balancing
• Document your reset-vs-ban criteria so the team is consistent

User communication

• The USER_PROFILE_RESET notification tells users what happened — but consider also linking to your community guidelines
• Allow users to update their profile again after a reset — the new version will be re-scanned
• Provide an appeal path for users who believe the reset was incorrect

---

Dive deeper: AI User Profile Moderation has the full console reference including detection categories, threshold configuration, and admin reset details.

Next Steps

Your next step → Roles, Permissions & Governance

Profile moderation is live — now configure who gets moderator access and what they can do.

Or explore related guides:

Content Moderation Pipeline

The full content moderation loop — AI screening, user flagging, admin review, and webhook automation

User Profiles & Social Graph

Build the profile pages and social graph that this guide moderates

Notifications & Engagement

Configure the notification tray that delivers profile reset alerts