Pictures and video

On the main list

The tools a client sees when it connects.

find_media

Find media for a line

ONE search for a shot, across everywhere media lives: the project shelf (pinned and kept first), the account library (uploads, imports, generated; matched on its captions), scenes in the account's other projects, skills (styles and recipes), and stock + archives (Pexels, Pixabay, Wikimedia, NASA…). Pass the narration line or what the shot should show. Returns numbered picks — kind, size, length, licence, a title or caption, a cut window (in/len) when you pass need_sec, and use: the url to place (already on Typecut storage), the import_asset call for an outside file, or the copy_scenes call for a scene — plus ONE contact sheet with every pick's picture under its number, so you choose by looking, in this call. Scenes and project covers are also found by what they LOOK like ("stickmen with lightsabers"), where the account has them indexed; each such pick carries a stable id (sc_… scene, pj_… project) you can quote and pass to resolve_handles. Free. Use it instead of search_assets + list_media + get_shelf.

generate_image

Generate or edit a picture

Images into the library. prompts: ≤10; reference_mode identity keeps the subject; grid: many in one (guide "Grids"). SPENDS CREDITS.

generate_video

Generate a video clip

≤8 s MP4 from image, text or scene (place: lands in it). SPENDS CREDITS/s.

decompose_image

Take a picture apart in one call

Every layer you need from one image in ONE pass — reach for this before segment_image / depth_map / plate / cutout when you want more than one of them (one locate, one encode, one charge per op). Name parts; end a name with * for EVERY instance ("windows*" → windows.1, windows.2 … in reading order, and "windows" then means the whole group). Add depth (2–6 bands: each part also gets its z), plates (each entry a list of parts/groups to remove → a clean plate), cutouts (parts/groups on transparency, cropped to their box). Returns a compact manifest — per part box [x,y,w,h], c, area, z, a mask PNG url (alpha, use as CSS mask-image) and its color — plus groups, depth, plates, cutouts, missing, and preview: the picture with every part tinted in its colour, attached as an image so you can CHECK the masks before you build on them, and checks: {ok, findings[]} from the tool itself (a speck, a whole-scene mask, two names on one object, a smeared plate, a halo on a cutout, a flat depth map), each with its fix. Outlines only with detail:"outline". Costs the sum of the ops it runs (quote with dry_run). SPENDS CREDITS. Waits up to ~50s; if it comes back status: "generating", poll get_generation with the generation_id (the same result arrives there). All coordinates are the source image's pixels: place things with box/w and box/h as percentages so they track the image at any size.

Callable by name

Not listed on connect, to keep the list short. Call them directly by name, or find them with tools.

add_to_shelf

Put found material on the asset shelf

Put material YOU found on the project's asset shelf, so a person can keep, pin or reject it and the plan can place it: a clip from a post, a figure from a blog, a search_assets pick you imported. Up to 50 per call. Free.

Import first (import_asset / ingest_video_url) and pass media_asset_id: the file is then hosted and cannot expire. Say WHAT it is in caption (what is on screen, and who posted or published it — a video's library caption is read from one poster frame, so your words are what identifies it), what it is about in entities, which beat or story it serves in for_beat, and why in rationale. Credit the source in attribution.

Licence: say what you know. Third-party posts and press clips are \{tier: "unknown"\}: they are placed only after a person PINS them, which is the review step. Re-adding the same url refreshes the description and never changes a decision a person made.

attach_media

Set a scene background

Set a scene's background from a URL (e.g. a search_assets result) or an asset already in the project library. Free.

clip_candidates

Find the best clips in a video

The best SELF-CONTAINED reels in an analysed long video (a podcast, a talk, a film): ranked, with clean cut points. Reads the analysis you already have (analyze_video, ready, with speech) and returns up to count clips of min_sec–max_sec seconds (default 25–60), each {start, end, score 0–100, hook, cold_open, text, signals, reasons}. start/end are already snapped: they begin on a sentence start with a short head and end on a sentence end with a 0.45 s TAIL past the last word, so the cut does not feel abrupt. cold_open is set when the strongest line is not the first: play it first, then the clip. Place a clip with put_scene_composition (speech: {media_id, from: start, to: end}) and a <video data-tc-shot="at:0 in:<start> len:<end-start> dur:<source length>">. The ranking is a transparent heuristic (hook strength, numbers, self-containment, pace, energy, payoff), not a prediction of views; the transcript may misread numbers, so check any figure you put on screen. Free; CPU, seconds.

create_upload

Send a file without pasting it

A short-lived link to upload files from your own disk, so their contents never pass through this conversation (a pasted 30 KB scene is re-read on every later turn).

1. create_upload(file_name: "scene-3.html") → upload_url, public_url and a ready curl line;
2. run the curl from your shell;
3. put_scene_composition(scene_id, src_url: public_url). Text (.html .js .mjs .css .json) and fonts (load with @font-face) give a permanent public_url; code several scenes share belongs in put_module. Audio, Rive (.riv), 3D (.glb, license required), images and video (pass bytes; project_id files it there) go straight to storage: run the returned curl as is (PUT, then POST complete_url), which prints asset_id (audio also track_id for set_music; kind:"sfx" for an effect). files: [...] uploads up to 25 in one call. Limits, link lifetimes, licences and storage_full: typecut://studio/guide "Send files, not pastes". Free.

cutout

Cut a part out of a picture

The named parts (default: the main subject) on transparency, as url — an alpha PNG the size of the source, so it stacks exactly over the original with the same CSS; crop:true trims it to box instead. SPENDS CREDITS (quote with dry_run). Waits up to ~50s; if it comes back status: "generating", poll get_generation with the generation_id (the same result arrives there). All coordinates are the source image's pixels: place things with box/w and box/h as percentages so they track the image at any size.

decompose_images

Take a set of pictures apart in one call

decompose_image for 1-8 pictures at once (a cast, a storyboard's stills), answered as a TABLE and ONE contact sheet instead of eight manifests. Give items (each an asset_id or url, an optional label) and the shared spec in defaults (parts, depth, plates, cutouts, detail); any item can override it (an item with its own parts uses its own plates and cutouts, since those name parts). Every picture is an ordinary decompose (same price, charge, refund and checks) and the set is quoted as a whole first: dry_run returns the total, and nothing is charged unless all of it can be paid for. Returns per picture: parts found of asked, missing, ok (its own checks passed and every part was found), the worst finding with its fix, credits and generation_id; plus summary (one line per problem) and sheet.url, attached as an image: every preview in one grid, numbered, bordered green (fine), amber (look at it) or red (nothing found or failed). A picture's manifest is one get_generation away; include:"results" returns them all. Waits up to ~50s; pictures still running come back generating. SPENDS CREDITS (the sum of its pictures).

depth_map

Depth map of a picture

Relative depth of every pixel: url is a 16-bit greyscale PNG (white = near), and bands cut it into parallax layers — band 0 is the whole frame (farthest), each later band is everything at least that near (cumulative, so moving a nearer layer never opens a hole), with z (0..1, 1 = near), range, box, c, area and a mask PNG url (alpha). Name parts to also get each part's own z (median depth under its mask) in parts. Drive parallax by z: nearer moves more. Pair with plate to fill behind what moves. SPENDS CREDITS (quote with dry_run). Waits up to ~50s; if it comes back status: "generating", poll get_generation with the generation_id (the same result arrives there). All coordinates are the source image's pixels: place things with box/w and box/h as percentages so they track the image at any size.

extract_html_scene

Lift an imported HTML slice into a scene of its own

import_html tier 3. A scene import_html made is a SLICE: it plays its window of the shared file, pixel-identical, but its document is only a wrapper. This lifts it: an LLM pass writes the window's DOM, CSS and tweens as a standalone composition on local time (t = 0 at the window start). It LANDS ONLY IF it passes a frame diff against the slice — both rendered in the scene on the production renderer at the same times, every frame at or above 36 dB PSNR (the HyperFrames conformance bar). Otherwise the slice stays and nothing is lost; candidate_url holds the attempt either way. Do it on the scenes you are about to edit, not all of them. It takes a minute or two, so it runs in the background: the first call starts it (status running); call again with the same scene_id to read progress, then the result; retry: true starts a new attempt. dry_run diffs without landing. Staff and authoring pilot only; free for now (the model call is booked, not charged).

find_motion

Find body motion by words

Search the clean motion-capture library (CMU optical mocap, free for commercial use, licence on every clip) for whole-body moves by words: "left hook", "duck under a punch", "stumble and fall", "sad walk", "jump kick", "two people quarrel". Each result is a clip with its pack module, length, travel, and its DETECTED MOMENTS (punch L/R head|body, kick, air, crouch, down, turn — seconds on the clip clock), plus a ready segment around the first matching moment. Two-person takes captured together carry duo (place them with Moves.pair).

Use the clips in a __tcThree scene: pin the returned pin modules (anim:moves first), then Moves.seq([segment, {clip, from, to, speed, mirror, hold} ...]) stitches them with blends, root alignment and foot locking; Moves.bind(THREE, model).apply(seq.frame(t), {look, reach}) plays them on any Biped or Mixamo actor, with hands or feet reaching targets and the head looking at a point. Free.

generate_grid

Generate a storyboard or sprite sheet

MANY pictures from ONE image call, each cut out as its own library asset. SPENDS CREDITS: one sheet at the model's per-call price (+ the upscale price per cell with upscale); dry_run:true quotes.

WHEN TO USE IT instead of generate_image: (1) CONSECUTIVE BEATS on one set with one cast — mode "panels" draws them on one storyboard sheet, so the set, the characters and the story state (what was broken, who holds what) stay consistent from panel to panel, which separate calls do not do, and it costs one call. Put what every panel shares (set, cast, style) in panels.shared and one beat per panel in panels.prompts, row by row. (2) A SPRITE SHEET of props or icons in one style — mode "items", ≤ 12 labelled items, each returned on transparency (background "transparent") ready to place in your HyperFrames. One picture: use generate_image.

Returns result {mode, model, contact_sheet_url, cells: [{index, label?, url, asset_id, width, height, box}]}. Waits ~50 s; if status is "generating", call wait_for_generation with the generation_id. "not available yet" means the sheet job is still rolling out: use generate_image with prompts meanwhile.

generate_thumbnail

Generate thumbnail options

Design 1–3 YouTube thumbnail OPTIONS for a project — each a different concept and hook, styled after a frame of the video itself unless you pass reference_url (a thumbnail to learn from). SPENDS CREDITS (the thumbnail price per option; failed options are refunded). Asynchronous: poll list_thumbnails until pending is empty. It never replaces a thumbnail the project already has — show the user the options and set_thumbnail the one they pick.

get_reference

What a project is meant to match

The source a project is a replica of: its video analysis (state, duration, shots, size), its audio analysis when it has one, and the alignment of replica scenes to spans of the source. Read it first when you are handed a project to supervise: it says what the work is measured against. Pictures of it: get_frames with the project_id.

get_scene_reference

Resolve a scene reference

Resolve a typecut://scene/<project_id>/<scene_id> or typecut://project/<project_id>?scenes=a,b,c reference — pasted from the editor — into what you need to reuse a style or scene: per scene its title, chapter, duration, brief and narration, entry/exit state, the AUTHORED source (the truth for the look; over a size cap you get a sha256 and the call that fetches it, never a truncation), the tokens in use, the skill ref, and 3 stills. With a skill: load it with get_skill for the rules and treat the source as an example to adapt. Without: the source is the only style source. intent "style" = extract palette, type and motion character and say what to keep; "scene" = reuse structure and timing; default both. Read-only, free.

get_shelf

Read the project's asset shelf

The ASSET SHELF: everything this film could use — the user's own files and what discovery and stock found for the topic — each with what it is, its licence and credit line, a relevance score, and a DECISION: auto (nobody decided; the best items per beat are used), kept, pinned (must be placed; the note says how) or rejected. Decided items come first. Free.

Show the user what was found and let them keep, pin or reject it with set_shelf_decision — but never block on it: an unreviewed shelf still makes a film.

Hosts that render apps show the shelf as a thumbnail grid the person can keep, pin or reject from directly; those decisions are recorded as they are made, so do not repeat them with set_shelf_decision.

get_video_analysis

Read a video analysis

The state of an analysis started with analyze_video: "queued"/"processing" (progress: stage, pct, message — poll every 20–30 s; a short takes about a minute, 30–60 minutes takes 10–25), "ready" (result: condensed — the whole video as one timeline, read it first — outline when there is a blueprint, then shots[], sound, transcript, contact_sheets[], blueprint, audio, notes), or "failed" (error says why).

LONG VIDEOS (over 10 minutes or 120 shots) come back as an OVERVIEW — condensed, outline, format, the sound summary, the chapter list — plus pages. Read one page per call: chapter=<index> (that chapter's shots, words, sound, scenes with their frames, and its beat sheet) or t0/t1 in seconds. Each page is a few thousand tokens; full: true returns everything at once (large). words: true adds the word-level transcript timings. result.motion.pace = the film's pace (real cuts, camera movement, punches, subject speed, cadence, cuts vs beat, a recipe); every shot carries its own measured motion (shots[i].motion: camera line + camera_path keys, subject speed, held share). result.motion also has recommended_probes: windows worth a look.

A WINDOW (t0/t1 or chapter) is a zoom-in: only that stretch — its shots, words, sound, scenes and a window_timeline of its shots with their motion (the whole-video condensed / outline / summary are not repeated). detail "events" re-senses the window and adds window_read.motion (the stretch as one camera path) and window_read.track (camera pan/zoom/roll and subject speed every 0.25 s); "strips" adds labelled dense frames (window at most 30 s). CPU, seconds.

compare_to: pass the REFERENCE's analysis id while reading an analysis of YOUR RENDER — result.comparison scores how closely the replica MOVES like the reference (0–100), lists film-wide pace diffs (cuts, camera, punches, subject speed, stillness, cadence) and the worst stretches with the fix for each, quoting the reference's measured camera for that stretch. Free.

async: true answers at once with a job_id; jobs {id} reads the result (no 60 s timeout).

image_parts

Parts of a picture

Named parts of an image as files and JSON for your HTML. op: segment, depth, plate, cutout (guide "Media you can read from HTML"). SPENDS CREDITS.

import_asset

Import an external asset

Import an EXTERNAL url (a search_assets pick, a file on the web) into the media library: downloads it, re-hosts it on Typecut storage, records provenance and dimensions, captions it on ingest. Free. Returns the library url to use with attach_media / create_scene / generate_scene / reference_image_url. Always import before building on an external url — provider links expire. Do NOT import a url that is already on Typecut storage (a list_media / generate_image result): use it directly.

ingest_video_url

Bring a video link into the library

ADMIN-ONLY. Download a public video from a LINK — X/Twitter, TikTok, Reddit, Instagram, a direct .mp4, or a file someone already put on our storage — into the media library as a hosted mp4 with its poster, subtitles (when the platform has them), duration, size and source metadata (title, uploader, description, chapters). Free.

Use it to study or replicate a reference video: ingest, then run analyze_video on the returned asset_id for shots and contact sheets, and transcribe_media for the words.

YOUTUBE: downloads only when the YouTube pool is enabled and healthy; otherwise error youtube_unsupported (words: pass it as a project source; frames: download locally, put the file on storage and ingest that storage URL). When enabled the asset is 480p, carries the Gemini transcript + oEmbed in metadata.ingest.youtube, and a pool that cannot serve fails with an explicit youtube_pool_exhausted / cookies_expired code (the transcript is still attached).

ASYNCHRONOUS: returns status: "pending" with an asset_id. Call again with the SAME url to poll — it returns the same asset, then ready (video_url, poster_url, subtitles) or failed with an error code (auth_required = needs a login, blocked, rate_limited, too_long = over 20 min, unsupported, stalled / timed_out = the job was lost or hung — a pending ingest never stays pending past ~15 min unstarted or ~25 min running). Calling again after failed retries. Public content only.

link_transposed_references

Link transposed designs

After transpose_project has planned: pair every scene of the NEW project with the source's scene at the same chapter index and position and set that composition as its design REFERENCE (free; one undoable changeset per 40 scenes). The new scenes then generate as siblings of the originals. dry_run:true shows the pairing. Then resolve_project on the new project (dry_run first for the quote).

list_media

List the media library

The media library (uploads, imports, generated images), newest first, optionally for one project. Each asset carries a caption — annotation.description + annotation.tags, written on ingest — so pick assets by MEANING from the captions rather than guessing from file names. Use before attach_media / create_scene / generate_scene when the user refers to library content ("the mountain photo", "that clip I uploaded"). Free.

list_thumbnails

List thumbnail options

The project's current thumbnail, every generated option (url, on-image text, concept), and any thumbnail generation still running. Free.

map_data

Map geometry for a scene, in one call

Everything a map scene needs, projected to YOUR canvas, so you never fetch datasets or hand-roll d3-geo. Ask for regions (modern countries by ISO-3 or name, states/provinces by ISO 3166-2 or name, or HISTORICAL polities at a year — by polity name, or every holding of a ruler, e.g. {ruler:"Spain", year:1802}; 53 border snapshots from 123000 BC to 2010, nearest earlier one used), rivers by name (Natural Earth 10m), lakes, places (names from a 7k-city gazetteer, "Paris, France", or lng/lat), and routes (known sea corridors via:"suez"|"cape", your own waypoints, else a great circle flagged exact:false). It fits one projection to fit (bbox or ids; default: everything asked for) inside width×height, clips to a margin around the frame so a camera can roam, and hosts the result as the module map:<name>@N next to kit:atlas@N, which draws it.

Returns pins (the two <script data-tc-module> tags, kit first), usage (a working scene snippet), and every id with its pixel bbox/centre so you can author against it. In the scene: var map = TcAtlas.mount(root, name); map.fill(id, color); map.drawOn(tl, riverOrRouteId, at, dur); map.marker(placeId); map.label(regionId); map.play(tl, [{t:0, at:regionId, zoom:"fit"}, {t:4, at:placeId, zoom:5}]) — a seek-safe camera; map.project([lng,lat]) / map.screen(id) for your own elements, map.pin(el, target) to make one follow the camera. Reuse the SAME map in consecutive scenes and end one camera where the next starts: the film is one continuous map (check_continuity reads the camera as the scene's set). The same request lands on the same version. Nothing unknown is invented: it is left out and named in warnings. dry_run:true builds and reports without hosting. Free.

mine_motion

Learn gestures and expressions from a video

Watch a person in a video and cut what they do into named TAKES a character can play: gestures in kit:acting's vocabulary (wave, point, offer, explain, shrug, stop, think, facepalm, despair, cross, hips, plead, little, beat, fist, shush …) and facial expressions (smile, grin, wow, doubt, worried, angry, tired …), each with its stroke time, side, energy and tone tags, measured from MediaPipe pose, hands and the face's 52 blendshapes. Returns url (the hosted library JSON: every take's frames, body-relative, the same format anim:mocap@2 plays) and the takes without their frames.

Most scenes do not need this: anim:mocap@2 already holds 206 takes from free stock footage, and kit:acting@9 picks one per gesture beat on its own. Mine when a character should move like a SPECIFIC person or style, or a gesture the library lacks.

Source: analysis_id (ingest_video_url) or media_asset_id (a library video). from/to: seconds, at most 40 s apart; medium shots where the hands stay in frame mine best. LICENCE IS REQUIRED and is recorded on every take: only mine footage you may use (free stock such as Pexels, licensed, or the user's own). Only the motion is kept, not the person's likeness. Free (CPU), about 3-4x the window's length.

plate

Clean plate (background behind parts)

The same picture with the named parts inpainted away — what is BEHIND them — as url (a PNG the size of the source). Lay it under the original and move a part's cutout or masked copy: the hole it leaves is already filled. hole has the filled region's box and mask; parts the geometry found. Pixels outside the hole are the source's own. SPENDS CREDITS (quote with dry_run). Waits up to ~50s; if it comes back status: "generating", poll get_generation with the generation_id (the same result arrives there). All coordinates are the source image's pixels: place things with box/w and box/h as percentages so they track the image at any size.

search_assets

Search for assets

Search for real imagery and FOOTAGE, with licensing surfaced. Covers two kinds of source at once: entity-grounded archives (Wikimedia, NASA, Smithsonian, Europeana) for material OF a specific place, person, artefact or event; and stock (Pexels, Pixabay) for a mood or an action shot to be used. "Voyager 1" is the first kind, "slow aerial over pine forest" the second. Use when a scene needs real material rather than generated motion design. PASS kind:"video" WHEN YOU WANT CLIPS — the default is stills, and archives hold almost none, so an unqualified search reads as "there is no footage". Video results carry width, height and duration; keep the duration, because a shot cut from a clip needs it to be trimmed correctly. Pass the full-resolution url to import_asset. For historic pictures, pass year_from / year_to and picture_kind: that asks only the archives that can date and classify (The Met, Art Institute of Chicago, Wellcome) and leaves stock out. Results come as a numbered list: pick by number, then import the url.

segment_image

Find the named parts of a picture

Where each named part of an image is, as pixel truth: per part a mask PNG url (white, the part in ALPHA — use it directly as CSS mask-image or a canvas clip), box [x,y,w,h], polygon [[x,y]…], d (the same outline as SVG path data), centroid c [x,y], q (4-corner quad TL,TR,BR,BL), plus w,h of the image and missing (names not found). SPENDS CREDITS (quote with dry_run). Waits up to ~50s; if it comes back status: "generating", poll get_generation with the generation_id (the same result arrives there). All coordinates are the source image's pixels: place things with box/w and box/h as percentages so they track the image at any size.

set_reference

Record what a project is meant to match

Record the source a project replicates, so a builder and a supervisor measure against the same thing without being handed an id. analysis_id = the video (analyze_video first), audio_analysis_id = a song or voice (analyze_audio), either or both. alignment = [{scene_id, ref_t0, ref_t1}]: the span of the source (seconds) each replica scene should match; leave it out and the two clocks line up proportionally. The body REPLACES the whole reference; both ids null clears it. expected_version = the project's edit_version, to refuse a stale write. Free.

set_shelf_decision

Keep, pin or reject shelf items

Record what the user decided about items on the asset shelf (ids from get_shelf). Up to 50 per call. Free.

pinned = the user wants it IN the film — put how in note ("use for the intro"). kept = may be used. rejected = never used. auto = back to undecided. Record the user's choices, not your own taste.

set_thumbnail

Set the thumbnail

Make one image the project's thumbnail: an option from list_thumbnails, or a file already in the library (import_asset first for anything else). Free.

track_video

Read how the camera moves

The camera path of a source video at every frame of a window, cut into moves (push-in / pull-out, pan with a direction, roll), each with its size, timing and the GSAP ease it fits, e.g. "camera pushes in 1.00→1.60x over 1.20 s (2.10→3.30 s) (fits power2.inOut)". Pass analysis_id OR project_id (its recorded reference). from/to: seconds on the SOURCE clock, at most 20 s apart — narrow it to the move. region: [x, y, w, h] fractions of the frame to track one thing (a title, a product) with the camera taken out. Free (CPU, seconds).

A move marked parallax is a 3D camera (the picture does not move as one plane): trust its direction and timing, not its size. Copy the move into the replica with the same ease and timing, then check the replica with look show:["motion"].

upscale_image

Upscale a picture

One picture at 2× or 4× the pixels, re-hosted as a PNG library asset — for a close-up camera move, a crop into a detail, or a 4K export. SPENDS CREDITS (dry_run:true quotes). model picks one upscaler from list_catalog("models", modality: "upscale"); omit it to try the deployment's upscalers in order. Returns result {assetId, url, width, height, model}; usually within the ~50 s wait, else poll get_generation with the generation_id.

video_frame

Take a frame out of a video

ONE frame of any video as a hosted picture — by default the LAST frame, the picture a clip ends on. The link that closes a media chain: generate_image → generate_video {image_url: $img.url} → video_frame {url: $clip.url} → generate_video {image_url: $last.url} (the next shot starts where this one ended) or generate_image {reference_image_urls: [$last.url]}. Also: look at what a generated clip actually did (at 0, the middle, the end) before placing it. url = any public https video (a generate_video result, a find_media hit) or asset_id = a library video. Returns {url, at, t, width, duration_sec}. Free; seconds. The JPEG is named by content, so asking again is free and the url is stable.