Frame Sampling for Multimodal Video Reasoning: Timestamps, Scene Coverage, and Temporal Consistency

"Take one image every 30 frames" approximates one frame per second only when the video is around 30 fps, the frame rate is constant, and the timestamps are valid. Phone screen recordings, stitched surveillance footage, and network transcodes are often variable-frame-rate, so sampling must follow PTS timestamps; equal gaps between frame numbers can represent different spans of time. Sampling only keyframes also biases the selection toward the encoder's compression points rather than business events.

This article does not repeat the image Q&A covered in Multimodal Vision Applications. Instead, it converts video into a finite set of frames with temporal evidence. Packing a long video directly into the context also crowds out the text and output budgets. The "locate first, then read" principle from Long Context vs. RAG applies here as well.

First Establish a Trustworthy Media Timeline

Keep the original video read-only. The probe manifest records the container, stream index, time base, starting and ending PTS values, average and actual frame rates, duration, rotation, resolution, audio tracks, and decoder version. Duration must not be trusted from the container header alone; negative timestamps, timestamp regressions, huge gaps, and mismatched audio and video lengths must all be flagged explicitly. When anomalies exceed policy limits, reject the input or route it to offline repair. Never silently treat an invalid timeline as a normal video.

# Run only in the digest-pinned image specified by the manifest; the runner separately enforces CPU/memory limits, a 30s wall-clock limit, and a 256 MiB temporary disk.
MAX_FRAMES=96
ffmpeg -version > toolchain.txt
timeout --signal=TERM --kill-after=5s 30s \
ffprobe -v error -show_format -show_streams \
  -of json input.mp4 > media.json
timeout --signal=TERM --kill-after=5s 30s \
ffprobe -v error -select_streams v:0 -read_intervals '%+120' -show_frames \
  -show_entries frame=stream_index,best_effort_timestamp_time,key_frame \
  -of json input.mp4 > video-timeline.json

timeout --signal=TERM --kill-after=5s 30s \
ffmpeg -t 120 -i input.mp4 \
  -vf "select='gt(scene,0.35)',scale=1280:-2" \
  -fps_mode vfr -frames:v "$MAX_FRAMES" scene-%06d.png

These commands run only in the controlled offline runner described above and must not run in the upload request process. timeout is a GNU/Linux guardrail, while the runner enforces the CPU, memory, and temporary-disk limits. -read_intervals, -t, and -frames:v 96 limit the probed media interval, decode interval, and number of PNGs, respectively. In production, after the baseline, scene, and event candidates are merged, the result must still be deterministically capped at 96 frames before anything is written to disk. Actual tasks do not reconstruct time from file sequence numbers; they preserve the original PTS, canonical milliseconds, and hash. The run report records the image digest, ffmpeg -version, and command hash. Normalize rotation and color first according to the Multimodal Input Normalization Contract.

Layered Sampling Is More Reliable Than a Single FPS

The first layer establishes a uniform temporal baseline, such as one frame every two seconds, ensuring coverage across the full duration. The second layer adds scene-change points, but the scene threshold can only provide candidates: flashes, subtitles, and camera shake can create false cuts, while gradual movement may not receive a high score. The third layer increases local sampling density based on business signals, such as ASR keywords, operation logs, shot detection, or intervals identified by the first-pass model, adding several frames before and after each interval.

Set per-request limits for total frames, frames per minute, frames per scene, and local densification. When candidates exceed the budget, first preserve the beginning, the end, the uniform baseline, and high-priority events, then rank the remaining candidates by a deterministic score. Do not truncate randomly. Use perceptual hashes to help deduplicate frames, but do not mistakenly merge consecutive frames with minor subtitle changes because a single changed digit may be the task's exact target.

sampler: temporal-v5
runtimeImage: registry.example/media-tools@sha256:6d2f...
ffmpegVersion: 7.1.1
baselineIntervalMs: 2000
sceneThreshold: 0.35
eventWindowMs: 3000
eventIntervalMs: 250
maxFrames: 96
maxFramesPerScene: 12
timestampRoundingMs: 1

Send model inputs in ascending time order, with a stable frame_id and timestamp attached before each frame. Never rely on the incidental order of the uploaded array. Multi-segment videos must also include the segment ID and original time range. If the model API samples frames internally, record the provider's strategy or convert the video into explicit frame inputs. Otherwise, it is impossible to explain why the same file produced different evidence selections in two runs.

Audio and Video Must Share the Same Alignment Reference

ASR segments use the same canonical timeline as the video and record word- or sentence-level start and end times, the ASR version, and revision status. Do not place the complete transcript before the frame list without timestamps; the model may associate speech from the second half with imagery from the first half. When audio-video offset occurs, first mark the uncertain interval or repair the timeline. Do not ask the model to guess the missing alignment.

Assertions for temporal questions differ from those for static images. Evaluation samples should include whether an event occurred, its ordering, its duration range, changes in the state of people or objects, and the evidence-frame interval. Slice out critical brief events separately and calculate event recall, temporal-boundary error, and ordering accuracy; then use blind review for open-ended descriptions. Offline frame extraction at scale can be placed in LLM Batch Inference, but every task must still be bound to a sampler digest.

Calibrate with Controlled Defects Before a Gradual Rollout

The gold set covers constant and variable frame rates, duplicate frames, brief flash events, gradual scene changes, subtitle updates, rotation, corrupt PTS, silence, and audio-video offset. Intentionally remove a critical sample, swap two frames, or shift an ASR timestamp, and verify that the corresponding assertion always fails. Gates may require 100% recall for high-risk brief events, no decline in ordering accuracy, average frame counts and visual tokens within budget, and no timeline error ever reaching the online model.

First run the new strategy alongside the old one on the same original hash, comparing frame IDs, temporal coverage, task quality, frame-extraction CPU usage, and visual cost. Roll it out by video type at 5%, 25%, and 100%; do not combine short surveillance clips and long meetings into a single average. The release process can reuse the stop gates from Canary Rollouts for Model Serving.

If brief-event recall declines, the frame budget overflows, audio-video alignment becomes abnormal, or provider costs deviate, immediately restore the sampler alias to the previous digest. The cache key includes the original video, normalization, sampling, and model versions. After rollback, do not read the new frame manifest. In-flight tasks must either finish using their already bound strategy or be rescheduled in full; they cannot use old sampling for one portion and new sampling for another.

The goal of video sampling is not to select the "best-looking screenshots," but to preserve temporal coverage, critical changes, and verifiable ordering within a limited budget. Only when the mapping between frames and the original timeline can be reconstructed do the model's conclusions have evidence that can be reviewed afterward.

Further Reading

• Multimodal Vision Applications
• Multimodal Input Normalization Contract
• Long Context vs. RAG
• LLM Batch Inference
• Canary Rollouts for Model Serving