Autonomous AI clipping for modern video channels.

A production-grade pipeline that converts long-form horizontal videos into viral 9:16 vertical clips with active speaker detection, zero-jitter camera stabilization, adaptive split layouts, kinetic subtitles, and multi-platform SEO packaging.

2.4 min
10-min podcast to finished clips
0.0 px
Dialogue jitter (cluster locking)
100%
Autonomous (URL to SEO pack)
Experience the Output ↓ Inspect Repository ↗
Autonomous 9:16 crop • Adaptive vertical split • Seam-anchored typography
The Problem

The short-form editing bottleneck.

Short-form vertical video is the dominant growth channel for creators, but converting horizontal podcasts into short clips is a grueling 4-hour manual process.

⏱

Manual Hook Hunting.

Editors spend hours scanning transcripts to find self-contained conversational hooks with strong retention potential.

🤢

Nauseating Camera Drift.

Naive facial tracking causes continuous micro-panning and horizontal wobble that triggers viewer drop-off.

👥

Multi-Speaker Chaos.

Traditional tools crop one person at random, cutting out listener reactions or flipping erratically mid-word during banter.

System Differentiation

Why ClippedAI is fundamentally different.

Standard clipping tools and baseline scripts perform simple center crops and unaligned Whisper subtitles. ClippedAI is engineered as an end-to-end computer vision and speech intelligence engine.

Pipeline Area Baseline Approach ClippedAI Production Engine
Speaker Tracking Static center crop or standard face detector (frequently tracks silent listeners). Fast-ASD (TalkNet + S3FD) lip synchrony cross-attention. Only tracks faces whose mouth motion correlates with speech.
Camera Motion Unfiltered face tracking with continuous horizontal wander and camera jitter. Stationary Cluster Locking (signal_helpers.py). Spatial clustering locks coordinates with 0.0px jitter during stationary dialogue.
Multi-Speaker Banter Single-speaker focus; cuts off secondary speakers and reactions. Adaptive Split Layouts. Stacks 2 speakers vertically (1080×960), renders 3-speaker top/bottom panels, or 4-speaker grids.
Scene Transitions Delayed panning across camera shot changes and angle shifts. Instant Snap-Cuts. PySceneDetect enforces instant hard cuts on scene transitions, snapping focus immediately.
Aspect Ratio Safety Fixed bounding crop without vertical headroom or aspect ratio compensation. 2D AR-Safe Headroom Normalization. Preserves exact 1.776× scaling on both axes with natural eye-line headroom ($cy = 0.42$).
Subtitle Placement Static bottom subtitles that overlap speaker faces in split layouts. Layout-Aware Anchoring. Dynamically anchors subtitles to the central dividing seam (\an5\pos(540,960)) in split mode.
LLM Reliability Single-provider dependency (susceptible to rate limits and API downtime). 4-Tier Resilient Cascade. Gemini 2.5 Flash ➔ Groq Llama 3.3 ➔ OpenRouter ➔ Heuristic Fallback with deterministic cache reuse.
Deliverable Cleanliness Unorganized directory with mixed temporary files and loose scripts. Isolated Creator Structure. Root folder contains purely clean .mp4 clips; thumbnails and SEO copy isolated in assets/.

Experience the generated clips.

The actual output produced by the ClippedAI pipeline from real full-length podcast interviews.

90/100 Viral Score • Relatable Struggle Hook

Stepping Off the Conveyor Belt.

Source: YouTube URL (How I bootstrapped and sold an app worth millions at 18).
Processed autonomously through dynamic Active Speaker Detection, smooth velocity-dampened motion tracking, and Hormozi neon kinetic subtitles.

1
Smooth Dynamic Motion Tracking: The active speaker is actively walking and gesturing across the stage while ClippedAI smoothly tracks their motion in real time.
2
Velocity-Dampened Panning: Continuous EMA smoothing prevents nauseating camera micro-wander while panning fluidly with the speaker's movement.
3
AR-Safe Headroom Balance: Retains natural eye-line headroom ($cy = 0.42$) in 1080×1920 portrait format even during rapid postural shifts.
80/100 Viral Score • Hypothetical Question

Fake Dating to Find Love? No Way!

Source: YouTube URL (Off-Campus Cast Ask Rapid).
Processed autonomously from raw URL through multi-speaker Active Speaker Detection (Fast-ASD), adaptive split-framing, seam-anchored subtitles, and SEO packaging.

1
Adaptive Split Layout: Seamlessly stacks both interview subjects vertically (1080×960 each) to preserve dialogue chemistry and reactions.
2
Seam-Anchored Neon Typography: Subtitles dynamically anchor on the center dividing seam (\an5\pos(540,960)), preventing overlap with either speaker's face.
3
Lip-Sync Active Speaker Detection: Fast-ASD cross-modal attention identifies the active speaker in real time with zero horizontal camera jitter.
End-to-End Deliverables

The complete creator distribution pack.

ClippedAI is not just a video cropper—it is an autonomous content engine. Each run produces a fully isolated bundle containing ready-to-publish vertical clips, CTR hook thumbnails, and multi-platform SEO packaging.

Generated Hook Thumbnail
Auto-Generated Hook Thumbnail: High-contrast typography, centered hook phrasing, and gradient dark vignettes engineered for maximum click-through rates.
Directory Output Structure
clips/How_I_bootstrapped.../
├── clip_1.mp4
├── clip_2.mp4  # Stepping Off
├── clip_3.mp4
└── assets/
    ├── clip_2_thumb.jpg
    ├── clip_2_metadata.json
    └── seo_summary.txt
System Architecture

The 6-stage clipping pipeline.

Every video ingested passes through six deterministic processing stages engineered for speed and reliability.

01

Video Ingestion & Lightweight Audio Extraction.

Downloads up to 1080p source video via yt-dlp, then extracts a lightweight 48kbps mono Opus/AAC audio track to minimize speech transcription payload size.

yt-dlp • FFmpeg
02

Sub-Word Diarization & 4-Tier Virality Cascade.

Transcribes speech with exact millisecond word timestamps. Evaluates psychological retention hooks (Curiosity Gap, Contrarian, Insight) scored out of 100 with breath pause snapping.

AssemblyAI • Gemini • Groq
03

Fast-ASD Lip-Sync & Diarization-Visual Fusion.

Cross-attends mouth region bounding boxes with audio phonemes to track active speakers, fusing coordinates with diarization speaker IDs to eliminate crosstalk flips.

PyTorch • TalkNet • S3FD
04

Stationary Cluster Locking & Hard Cuts.

Computes spatial clusters. If movement is below threshold (σ < 0.06), coordinates lock to cluster medians. Enforces instant snap-cuts on scene transitions.

signal_helpers.py • SciPy
05

Adaptive Split Compositing & Subtitles.

Renders single portrait crops or stacked 2/3/4 speaker panels with AR-safe scaling. Compiles kinetic karaoke subtitles (.ass) anchored dynamically to the central seam in split mode.

OpenCV • Libass • Pillow
06

Creator Distribution Pack & Isolated Storage.

Saves vertical MP4 clips in root folder clips/<Title>/clip_X.mp4. Generates centered hook thumbnails and multi-platform SEO packages cleanly isolated into assets/.

cli.py • export_pack.py
Engineering Deep Dive

Solved hard problems.

Production video processing requires solving fundamental computer vision and audio-visual synchronization edge cases.

Stationary Cluster Locking.

Face bounding boxes naturally jitter by 0.02–0.06 normalized coordinates every frame. Spatial gap analysis clusters positions; when σ < 0.06, camera coordinates lock to the median with 0.0px micro-jitter.

backend/src/signal_helpers.py
largest_cluster = max(clusters, key=len)
is_stationary = float(np.std(largest_cluster)) < 0.06

if is_stationary:
    # Locked Mode: Assign frames to cluster center
    cluster_centers = [float(np.median(c)) for c in clusters]
    out = np.full(seg_len, default)
    # Camera stays locked until real scene cut occurs

2D AR-Safe Headroom Normalization.

Cropping horizontal video into vertical 9:16 split panels (1080×960) typically causes severe distortion. Equal 608×541 source crops guarantee identical 1.776× scaling on both axes with natural eye-line headroom (cy = 0.42).

backend/src/video_processing.py
# AR-Safe 9:16 crop dimensions (1080p source)
CROP_W_1 = 608  # universal 9:16 width
h_half = OUT_H // 2
CROP_H_HALF = int(round(CROP_W_1 * h_half / OUT_W))  # 541px
# Scale mismatch (H vs V) is < 0.1%
Quickstart

Local installation & dual interfaces.

Run ClippedAI locally on macOS (Apple Silicon MPS) or Linux (CPU/CUDA) via the interactive terminal CLI or as a background FastAPI server.

1. Environment Setup
git clone https://github.com/ETS2K7/clipped-ai.git && cd clipped-ai
python3 -m venv venv && source venv/bin/activate
pip install -r backend/requirements.txt
cp .env.example backend/.env  # configure API keys

Option A: Interactive Pipeline CLI

Terminal workflow with colored stage trackers, automatic video extension resolution, and direct output packaging.

Terminal — CLI
# Run interactive CLI
./cli.py

# Or pass a URL directly with options
./cli.py --url "https://youtu.be/..." --aspect-ratio original --no-clip

Option B: FastAPI REST & SSE Server

Production server with OpenAPI docs, non-blocking asynchronous task execution, and real-time Server-Sent Events (SSE) streaming.

Terminal — Backend Server
# Start FastAPI server
./backend/run.sh

# Interactive Docs: http://localhost:8000/docs
# Health Check:     GET  /health
# Submit Task:      POST /api/process
# SSE Stream:       GET  /api/progress/{task_id}