AI Video Pipeline

End-to-end cinematic video production — from a single text prompt to a fully rendered, social-media-ready video. The pipeline combines LLM scene planning, image keyframe generation, and video synthesis into one orchestrated flow.

Pipeline Overview

User Prompt
    ↓
Prompt Expander (LLM)         ← enriches brief input with cinematic details
    ↓
Scene Planner (LLM)           ← breaks concept into 3–6 scenes with camera moves
    ↓
Keyframe Generator (Image)    ← generates hero image per scene (e.g. DALL·E, Nano Banana)
    ↓
Video Generator (Video)       ← animates keyframes (e.g. Kling 2.5, Runway, Sora)
    ↓
Scene Stitcher (FFmpeg)       ← concatenates clips with transitions + music
    ↓
Social Packager (LLM)         ← caption, hashtags, thumbnail, multi-format export
    ↓
Final Video

Each scene = 2–5 second clip. Draft mode generates shorter 2s previews for cost optimization.

Quick Start

package controllers

import (
    "github.com/CodeSyncr/nimbus/http"
    "github.com/CodeSyncr/nimbus/plugins/ai"
)

func (ctrl *Video) Generate(c *http.Context) error {
    pipeline := ai.NewVideoPipeline().
        ImageModel("dall-e-3").
        VideoModel("kling-2.5").
        GlobalStyle("cinematic, warm lighting, film grain, shallow depth of field").
        OutputFmt(ai.FormatWide)

    project, err := pipeline.Generate(c.Request().Context(),
        c.Request.FormValue("prompt"),
    )
    if err != nil {
        return err
    }

    return c.JSON(200, project)
}

Configuration

Method	Default	Description
`ImageModel(m)`	dall-e-3	Image generator for keyframes
`VideoModel(m)`	kling-2.5	Video synthesizer for animation
`PlannerModel(m)`	(default)	LLM for scene planning
`ExpanderModel(m)`	(default)	LLM for prompt enrichment
`GlobalStyle(s)`	cinematic…	Style appended to every scene prompt
`MaxScenes(n)`	6	Maximum scenes in the plan
`DefaultDuration(sec)`	3	Per-scene clip duration
`DraftMode(bool)`	false	2s preview clips to save cost
`OutputFmt(f)`	FormatWide	Aspect ratio and resolution
`ConcurrentScenes(n)`	3	Parallel scene rendering
`SeedVariants(n)`	1	Keyframe variants per scene
`OnScene(fn)`	nil	Progress callback per stage

Scene Planning

The scene planner uses an LLM to decompose a user's brief prompt into structured cinematic scenes. Each scene gets a detailed visual prompt, camera move, and duration.

// Input prompt:
"cinematic pizza advertisement in italian cafe"

// Scene planner output (auto-generated JSON ← LLM):
{
  "scenes": [
    {
      "prompt": "chef placing neapolitan pizza into wood fired oven, warm cinematic lighting, rustic italian kitchen",
      "camera": "dolly",
      "duration": 3
    },
    {
      "prompt": "cheese melting in slow motion, pizza slice being lifted with stretching mozzarella",
      "camera": "macro",
      "duration": 3
    },
    {
      "prompt": "pizza served on rustic table with fresh basil, olive oil drizzle, soft natural light",
      "camera": "crane",
      "duration": 2
    }
  ]
}

Camera Controls

Camera moves are prompt modifiers that control the video synthesis. They translate to professional cinematography terms in the keyframe and video prompts.

Camera	Constant	Prompt Effect
Dolly	`ai.CameraDolly`	Slow cinematic dolly in
Orbit	`ai.CameraOrbit`	Smooth orbit around subject
Pan	`ai.CameraPan`	Slow horizontal pan
Drone	`ai.CameraDrone`	Aerial drone flyover
Handheld	`ai.CameraHandheld`	Handheld with subtle shake
Macro	`ai.CameraMacro`	Extreme macro close-up with rack focus
Zoom In	`ai.CameraZoomIn`	Slow cinematic zoom in
Zoom Out	`ai.CameraZoomOut`	Slow cinematic zoom out
Static	`ai.CameraStatic`	Locked off static shot
Crane	`ai.CameraCrane`	Crane up reveal
Tracking	`ai.CameraTracking`	Tracking shot following subject

Style Consistency

The GlobalStyle string is automatically appended to every scene prompt — both for keyframe generation and video synthesis. This ensures consistent mood, lighting, and color grading across all scenes.

pipeline := ai.NewVideoPipeline().
    GlobalStyle("cinematic food commercial, warm golden hour lighting, " +
        "film grain, soft shadows, shallow depth of field, " +
        "anamorphic lens flare, professional color grading")

Tip: Treat the global style as a "visual DNA" — it keeps every scene feeling like part of the same production.

Keyframe Generation

For each scene, the pipeline generates a hero image using the configured image model. The keyframe serves as the visual anchor for video synthesis.

// Under the hood, each scene's prompt becomes:
// "{scene_prompt}, {global_style}, 35mm lens, professional photography"
//
// For macro camera: adds "macro lens, extreme close-up"
// For drone camera: adds "aerial view, birds eye perspective"
// For crane camera: adds "high angle, looking down"

// The keyframe URL is stored on the scene:
for _, scene := range project.Scenes {
    fmt.Println(scene.KeyframeURL) // URL of generated image
}

Video Synthesis

Each keyframe is fed to the video model (image-to-video) along with the camera move prompt and duration. This is the core of the pipeline — turning static images into animated footage.

// Direct video generation (without the full pipeline):
video, err := ai.Video().
    Model("kling-2.5").
    Prompt("slow cinematic dolly in").
    FromImage("https://storage.example.com/scene_1.png").
    Duration(3).
    Size("1920x1080").
    Generate(ctx)

fmt.Println(video.URL)

Scene Stitching (FFmpeg)

After rendering, stitch scene clips into a final video using the FFmpeg helpers. The pipeline generates the commands — your backend executes them.

// Generate the FFmpeg concat file content.
concatContent := ai.GenerateConcatFileContent(project.Scenes)
// Writes:
//   file 'scene_0.mp4'
//   file 'scene_1.mp4'
//   file 'scene_2.mp4'
os.WriteFile("/tmp/concat.txt", []byte(concatContent), 0644)

// Generate the FFmpeg command.
cmd := ai.GenerateStitchCommand(project.Scenes, ai.StitchConfig{
    OutputPath:  "/tmp/final.mp4",
    ConcatFile:  "/tmp/concat.txt",
    FadeSeconds: 0.5,
    MusicPath:   "/assets/background.mp3",
})
// cmd = ["ffmpeg", "-y", "-f", "concat", "-safe", "0", "-i", "/tmp/concat.txt",
//        "-i", "/assets/background.mp3", "-shortest", "-c", "copy",
//        "-movflags", "+faststart", "/tmp/final.mp4"]

exec.Command(cmd[0], cmd[1:]...).Run()

Draft Mode (Cost Optimization)

Video generation is expensive. Use draft mode to generate 2-second preview clips before committing to a full render.

// Draft: quick 2s previews for storyboard approval.
draft := ai.NewVideoPipeline().
    DraftMode(true).
    VideoModel("kling-2.5").
    Generate(ctx, "luxury car commercial in desert")

// Preview the storyboard.
for _, scene := range draft.Scenes {
    fmt.Printf("Scene %d: %s → %s\n", scene.Index, scene.Prompt, scene.VideoURL)
}

// User approves → re-render at full quality.
final := ai.NewVideoPipeline().
    DraftMode(false).
    DefaultDuration(5).
    VideoModel("kling-2.5").
    Generate(ctx, "luxury car commercial in desert")

Social Media Packaging

Automatically generate social-ready assets: captions, hashtags, thumbnails, and multi-format exports. Uses an LLM to create engaging copy.

project, err := ai.NewVideoPipeline().
    VideoModel("kling-2.5").
    OutputFmt(ai.FormatTikTok).
    GenerateWithSocial(ctx, "cinematic pizza ad in italian cafe")

social := project.Social
fmt.Println(social.Caption)
// → "Watch this authentic Italian pizza come to life 🍕 From wood-fired oven to table."
fmt.Println(social.Hashtags)
// → ["#pizza", "#italianfood", "#pizzalovers", "#foodporn", "#cinematic"]

Multi-Format Export

Re-scale the final video for every platform in one call:

// Auto-generate all social format commands.
formats := map[string]ai.OutputFormat{
    "tiktok":    ai.FormatTikTok,    // 1080×1920 (9:16)
    "instagram": ai.FormatInstagram, // 1080×1920 (9:16)
    "youtube":   ai.FormatYouTube,   // 1080×1920 (9:16)
    "square":    ai.FormatSquare,    // 1080×1080 (1:1)
    "wide":      ai.FormatWide,      // 1920×1080 (16:9)
    "cinematic": ai.FormatCinematic, // 1920×816  (2.35:1)
}

for name, format := range formats {
    cmd := ai.GenerateFFmpegResizeCommand("/tmp/final.mp4", format,
        fmt.Sprintf("/tmp/final_%s.mp4", name))
    exec.Command(cmd[0], cmd[1:]...).Run()
}
// Outputs: final_tiktok.mp4, final_instagram.mp4, final_square.mp4, etc.

Progress Tracking

Monitor real-time progress of each scene through the pipeline with the OnScene callback. Fires at: keyframe_start, keyframe_done, video_start, video_done.

pipeline := ai.NewVideoPipeline().
    OnScene(func(scene ai.Scene, stage string) {
        // Push to WebSocket / SSE for real-time UI updates.
        ws.Broadcast("video-progress", map[string]any{
            "scene": scene.Index,
            "stage": stage,
            "prompt": scene.Prompt,
        })
        log.Printf("[video] scene=%d stage=%s", scene.Index, stage)
    })

Full Example: Video API with Queue

Production-ready pattern: accept the request, queue the render job, and return immediately. The worker processes scenes in the background.

1. API Controller

package controllers

import (
    "github.com/CodeSyncr/nimbus/http"
    "github.com/CodeSyncr/nimbus/queue"
    "nimbus-starter/app/jobs"
)

type Video struct{}

func (ctrl *Video) Create(c *http.Context) error {
    prompt := c.Request.FormValue("prompt")
    format := c.Request.FormValue("format") // "tiktok", "wide", etc.
    draft  := c.Request.FormValue("draft") == "true"

    job := &jobs.RenderVideo{
        Prompt: prompt,
        Format: format,
        Draft:  draft,
        UserID: c.Session().GetString("user_id"),
    }

    if err := queue.Dispatch(job).Dispatch(c.Request().Context()); err != nil {
        return c.JSON(500, map[string]string{"error": "failed to queue render"})
    }

    return c.JSON(202, map[string]string{
        "status":  "queued",
        "message": "Video rendering started. You'll be notified when complete.",
    })
}

2. Background Job

package jobs

import (
    "context"
    "fmt"
    "github.com/CodeSyncr/nimbus/plugins/ai"
)

type RenderVideo struct {
    Prompt string
    Format string
    Draft  bool
    UserID string
}

func (j *RenderVideo) Handle(ctx context.Context) error {
    // Pick output format.
    format := ai.FormatWide
    if f, ok := ai.ExportFormats[j.Format]; ok {
        format = f
    }

    pipeline := ai.NewVideoPipeline().
        ImageModel("dall-e-3").
        VideoModel("kling-2.5").
        OutputFmt(format).
        DraftMode(j.Draft).
        GlobalStyle("cinematic, professional color grading, shallow DOF").
        ConcurrentScenes(3).
        OnScene(func(scene ai.Scene, stage string) {
            // Update job progress in DB or push to WebSocket.
            fmt.Printf("[render:%s] scene=%d stage=%s\n", j.UserID, scene.Index, stage)
        })

    project, err := pipeline.GenerateWithSocial(ctx, j.Prompt)
    if err != nil {
        return err
    }

    // Save results to storage.
    fmt.Printf("Rendered %d scenes in %s\n", len(project.Scenes), project.RenderDuration)
    fmt.Printf("Caption: %s\n", project.Social.Caption)

    // TODO: Upload to S3/R2 and notify user.
    return nil
}

func (j *RenderVideo) Queue() string { return "video" }
func (j *RenderVideo) Retries() int  { return 2 }

3. Route

videoCtrl := &controllers.Video{}
app.Router.Post("/api/video/generate", videoCtrl.Create)

Workflow Integration

Use the AI Workflow engine for advanced orchestration — sequential chains, parallel branches, and conditional logic.

wf := ai.NewWorkflow("video-production").
    Step("plan", func(wc *ai.WorkflowContext) error {
        // Use LLM to create scene plan.
        resp, _ := ai.Generate(wc, "Plan 4 cinematic scenes for: "+wc.GetString("prompt"),
            ai.WithSystem("Return JSON with scenes array"))
        wc.Set("plan", resp.Text)
        return nil
    }).
    Parallel("render",
        ai.StepFunc("scene_0", renderSceneFunc(0)),
        ai.StepFunc("scene_1", renderSceneFunc(1)),
        ai.StepFunc("scene_2", renderSceneFunc(2)),
        ai.StepFunc("scene_3", renderSceneFunc(3)),
    ).
    Step("stitch", func(wc *ai.WorkflowContext) error {
        // FFmpeg concatenation.
        return nil
    }).
    Step("social", func(wc *ai.WorkflowContext) error {
        // Generate caption + hashtags.
        return nil
    })

result, err := wf.Run(ctx, ai.WorkflowInput{"prompt": "luxury watch commercial"})

Direct Image & Video Generation

You can also use the lower-level image and video builders directly, outside the full pipeline.

Image Generation

img, err := ai.Image().
    Model("dall-e-3").
    Prompt("neapolitan pizza in wood-fired oven, cinematic lighting, 35mm lens").
    Size("1792x1024").
    Style("natural").
    Generate(ctx)

fmt.Println(img.Images[0].URL)

Video Generation (Image-to-Video)

// Animate an existing image:
video, err := ai.Video().
    Model("kling-2.5").
    Prompt("slow cinematic dolly in, soft camera movement").
    FromImage(img.Images[0].URL).
    Duration(5).
    Size("1920x1080").
    FPS(24).
    Generate(ctx)

fmt.Println(video.URL)

Text-to-Video

// Generate video directly from text (no keyframe step):
video, err := ai.Video().
    Model("kling-2.5").
    Prompt("a sunset over the mediterranean sea, drone shot, golden hour").
    Duration(5).
    Generate(ctx)

Storage Structure

Recommended structure for projects using S3, Cloudflare R2, or MinIO:

projects/
  {project_id}/
    plan.json              ← scene plan + metadata
    scene_0_keyframe.png   ← keyframe images
    scene_0.mp4            ← rendered clips
    scene_1_keyframe.png
    scene_1.mp4
    scene_2_keyframe.png
    scene_2.mp4
    final.mp4              ← stitched output
    final_tiktok.mp4       ← social format exports
    final_instagram.mp4
    final_square.mp4
    social.json            ← caption, hashtags, thumbnail

Output Formats

Format	Constant	Resolution	Aspect
TikTok	`ai.FormatTikTok`	1080×1920	9:16
Instagram Reel	`ai.FormatInstagram`	1080×1920	9:16
YouTube Short	`ai.FormatYouTube`	1080×1920	9:16
Square	`ai.FormatSquare`	1080×1080	1:1
Wide (16:9)	`ai.FormatWide`	1920×1080	16:9
Cinematic	`ai.FormatCinematic`	1920×816	2.35:1

Architecture

The video pipeline is designed for a worker-based backend architecture:

API Server
 ├── /api/video/generate       ← accepts prompt, queues job
 ├── /api/video/:id/status     ← poll render progress
 └── /api/video/:id/result     ← download final video

Workers (Queue)
 ├── prompt-expand-worker      ← LLM prompt enrichment
 ├── scene-plan-worker         ← LLM scene decomposition
 ├── image-worker              ← keyframe generation (DALL·E, Nano Banana)
 ├── video-worker              ← video synthesis (Kling 2.5, Runway)
 └── render-worker             ← FFmpeg stitch + multi-format export

Storage (S3 / R2 / MinIO)
 └── projects/{id}/            ← keyframes, clips, final video, metadata

AI SDK — Text generation, agents, RAG, embeddings, tracing, cost tracking
Queue — Background job processing for render workers
Storage — S3/R2/local file storage for video assets
WebSockets — Real-time progress updates to the UI