Video Generation: Jointly Modeling Spatial Appearance, Temporal Motion, and Cross-Shot State

The most intuitive first reaction to video generation is: since an image model can draw one frame, wouldn't stitching together 24 individual frames make one second of video? This path doesn't work. If each frame is independently resampled from text or noise, then there is no shared random state between the previous frame and the next frame. Pedestrians' face shapes, clothing colors, background signs, and the number of objects in the scene would randomly jump from frame to frame, producing not motion but a flickering slideshow.

Motion itself must rely on cross-frame information. Where an object is at this moment and which direction it should move in the next instant can only be determined by observing the states of several frames before and after at the same time; no single frame viewed in isolation is sufficient to constrain the direction and speed of motion. Therefore, video cannot be understood as 'a collection of several images,' but as a whole unfolding along the time axis.

Technically, the model no longer faces a single H×W image, but a T×H×W spatiotemporal volume: T is the number of frames along the time dimension, and H and W are the height and width of each frame. The network must simultaneously accomplish two things—learn spatial features such as texture, edges, and object appearance within each frame, while keeping feature correspondence for the same object and same identity across adjacent frames. The former corresponds to existing image generation capabilities, and the latter is a video-specific challenge; both must be trained simultaneously in the same parameter set.

This extra time axis also changes the cost structure of errors. In a still image, an erroneous pixel only affects that single image; in video, an isolated pixel noise often only flashes for one or two frames and is almost visually negligible. What is truly fatal is structural error: once a person's identity, an object's geometric shape, or background layout shifts in a frame, subsequent frames treat this error as an established fact and continue to extrapolate from it, and the error accumulates and amplifies along the time axis, becoming increasingly difficult to recover from. This means video quality is not determined by the average level of all frames, but by the worst time segment and the transitions between shots—viewers tend to remember the moment of a continuity error, not the average performance of other normal frames.

The raw representation of video is astonishingly large. Take a 4-second, 24 fps, 512×512 RGB video as an example: the number of frames T = 4×24 = 96, each frame is 512×512×3 scalars, and the entire video has 96×512×512×3 ≈ 75,497,472 values, about 75.5 million. Performing diffusion sampling directly on a tensor of this magnitude requires reading, writing, and processing nearly 100 million scalars at each step, and the memory and computational costs will quickly overwhelm any single GPU.

Image generation has already shown a way out: first generate in a compressed latent space, and finally decode back to pixels. Video follows the same idea, but compression can act on the spatial dimension and can further act on the temporal dimension. If a VAE is used to downsample the spatial dimension by 8 times and map the color channels to 4 latent channels, then the original 512×512 space is compressed to 64×64, and the entire video becomes 96×64×64×4 ≈ 1,572,864 scalars, about 1.57 million. Compared with 75.5 million, this is a reduction of about 48 times, and the cost of two-stage diffusion in latent space drops by orders of magnitude accordingly.

The temporal axis can also be compressed as well: let several adjacent frames share the same set of latent features, so the tensor becomes further shorter in the T dimension. This can push the cost of a single sampling step even lower and allow the same memory budget to cover longer videos, but at the cost of losing fast motion more easily. Temporal compression essentially assumes that adjacent frames are highly similar; once there is rapid waving, high-speed crossing of objects, or violent deformation in the picture, the merged-away intermediate states cannot be recovered at the decoding end, and details become blurry or show trailing artifacts.

Therefore, the degree of latent compression and the spatiotemporal scope of the attention mechanism are two sides of the same coin: the smaller the latent space and the shorter the spatiotemporal window covered by the attention mechanism, the cheaper each step is and the longer the duration that can be accommodated; conversely, more spatiotemporal details are retained. Together, they determine memory usage, generation speed, and how long a video a model can coherently model. This trade-off runs through every design of video generation systems.

The sampling object of video diffusion is no longer a single image, but an entire segment of noisy latent video. A natural question is: if the same noise is fed to all frames, wouldn't denoising automatically yield consistent motion? The answer is no. Consistency does not mean replicating the same noise; it requires objects to maintain continuity of identity, geometry, and appearance throughout the motion, and this information must be explicitly established at every denoising step through cross-frame interactions, rather than being maintained by the crude coincidence of identical noise.

Specifically, the model works on noisy spatiotemporal latents, and the network can be structured as 3D convolutions, temporal attention, or a decomposed structure that processes spatial and temporal layers separately. Regardless of the structure, the core is to let features from different frames communicate with each other: 3D convolutions directly mix information within a spatiotemporal neighborhood, temporal attention lets a frame reference previous and subsequent frames at arbitrary distances, and a decomposed structure first computes the spatial features of each frame and then exchanges them along the temporal axis. Conditioning signals can be text, the first frame, motion trajectories, or camera parameters; they tell the model “which motion the denoising direction should point to.” Shared network parameters and these cross-frame interactions create correlations between the appearances of the same object across different frames, which is precisely the basis for coherent motion.

But the strength of sharing must lie in an intermediate zone. If sharing is too strong, the features of all frames are compressed into almost the same thing, motion is frozen, and the picture becomes a static image; if sharing is too weak, each frame is denoised independently, falling back into the flicker of independently generated single frames. The entire art of the model lies in tuning the interval between these two failures so that adjacent frames are similar enough to maintain continuity while retaining enough difference to carry real motion.

When sequences become longer, the computational cost of having each frame attend to all other frames grows quadratically with the number of frames and quickly becomes prohibitive. Therefore, long-sequence generation usually switches to local temporal windows, sparse attention, or hierarchical keyframe structures: fine-grained cross-frame interactions are performed within short windows, while long-range consistency is passed through sparse connections or a number of keyframes. These approaches sacrifice part of the global connectivity in exchange for acceptable computational cost, which is another trade-off between long-video consistency and computational cost.

Take a specific task as an example: generate a 6-second shot of “a pedestrian with a red umbrella crossing a waterlogged street.” If you simply feed that sentence to the model, the results are often hard to verify, because the word “crossing” does not specify any verifiable intermediate states. A more useful approach is to break the prompt down into an event sequence that can be verified.

First, lock down the subject: a pedestrian holding a red umbrella and wearing a dark coat; the scene is a wet street after rain. These are identity and appearance constraints that run through the entire video and must remain consistent frame by frame. Second, define temporal events by expanding the abstract “crossing” into a clear causal sequence: the pedestrian enters the frame from the left → one foot steps into a puddle → splashes water → continues walking to the right, with the camera slowly following. This event chain provides checkpoints that can be checked segment by segment, rather than a vague stylistic description.

If image-to-video generation is used, the first frame can lock in the initial identity and layout, giving subsequent generation a definite starting point; trajectory or depth information further constrains the direction of motion and camera movement, preventing the subject from drifting arbitrarily or the camera from shaking erratically. Generation is usually performed in two stages: first generate the complete, coherent motion at low resolution with a base model, then perform temporal and spatial super-resolution separately. This allows the motion skeleton to stabilize first, rather than sharpening each frame independently—independent sharpening destroys inter-frame continuity and introduces flicker.

During the verification stage, focus on the three most error-prone objects: the pedestrian, the umbrella, and the puddle. Track them frame by frame to check whether there is identity switching, whether an object reappearing after occlusion is still the same object, and whether the number of objects in the frame changes midway. Quantitatively, optical flow and flicker metrics can be used to detect jumps; qualitatively, a human must inspect several causal details frame by frame: whether the splash is really triggered by a foot stepping into the puddle, whether the contact between foot and ground is solid, and whether the camera movement is continuous.

Before publishing, there is also a high-risk check: if real people appear in the footage, their identity authorization must be confirmed; the source of any copyrighted material must be clear; and the finished video must carry a provenance marker. In summary, a time-expanded event graph constrains the video far better than piling up a large number of stylistic adjectives, because style words only affect appearance, while the event chain determines what actually happens on the timeline.

The entire generation pipeline can be summarized in one figure: Why is generating motion at low resolution first and then performing spatiotemporal super-resolution more stable than upscaling frame by frame? Figure 1 provides the answer and also shows the causal order among the stages.

The starting point of the figure is two conditional inputs: a text event graph that specifies “what state changes happen at what time”; and a first-frame condition that locks in the initial identity and layout. Together they enter a low-resolution video model; the output of this step is not final image quality but a temporal skeleton—a complete, cross-frame coherent motion sequence, just at low resolution and with few details.

Afterward there are two stages of super-resolution. Temporal super-resolution first fills in intermediate frames along the time axis to make motion smoother; spatial super-resolution then raises resolution and adds details frame by frame. The key is that these two steps are performed jointly on the already stable motion skeleton—frame interpolation and detail addition reference each other, rather than operating on each frame independently. Finally, object tracking and physical evaluation use the previously established identity constraints and causal events to inspect the final video.

The logic of the whole figure is “skeleton first, details later”: first solve the hardest temporal consistency in a low-resolution space, then use super-resolution to restore image quality. If you do the opposite and enlarge and sharpen each frame separately, each frame will drift a little toward its own optimum, and inter-frame continuity will be destroyed. What Figure 1 expresses is precisely that appearance, motion, and state are jointly constrained on a multi-scale time axis, rather than each operating independently at the pixel level.

Video generation is not a single task, but a family of tasks with different constraint strengths. The more known conditions there are, the fewer degrees of freedom the model needs to sample on its own, and the difficulty shifts accordingly. A common misconception is that providing the first frame to the model both improves identity consistency and may conversely restrict reasonable motion. Because the first frame locks in the initial appearance and layout, identity drift is suppressed, but the model also loses the freedom to sample motion from scratch—it must strike a balance between 'respecting the first frame' and 'producing reasonable motion', especially regarding motion magnitude and the content that reappears after an object is occluded.

Viewed by task type, the differences become clearer. Text-to-video generation has only high-level semantics as conditions; layout, identity, and motion all require the model to sample on its own, giving it the greatest degrees of freedom and making it the most prone to losing control. Image-to-video generation provides the initial appearance and layout, greatly reducing the burden of identity consistency, and the difficulty shifts to motion magnitude and content after occlusion. Trajectory or pose control provides a geometric path, imposing hard constraints on motion, but the new contradiction is the conflict between control and naturalness: strictly adhering to the path may make movement stiff. Video editing has the strongest conditions; the original video's motion and most pixels are known, requiring modification only of the target region and ensuring that modifications remain consistent across frames without affecting unrelated parts.

These differences mean that evaluation must align with task conditions. Comparing the stability of image-to-video generation directly with pure text-to-video generation is unfair—the former inherits a large amount of information from the first frame and is inherently more likely to maintain consistency. Reasonable comparisons should only occur between the same type of task and under equivalent conditions; otherwise, 'stronger conditions' can be misread as 'smarter model'.

Temporal consistency is not a property that a single metric can cover. A typical problem is: even when optical flow looks smooth, a cup in the frame can still suddenly turn into a phone out of nowhere. The reason is that optical flow only measures low-level consistency—whether pixels and features move smoothly with motion; it does not care what objects those pixels represent. An object changing identity while optical flow remains smooth is exactly the mechanism by which this discontinuity occurs.

Breaking consistency down, at least four layers can be distinguished. Low-level consistency looks at the smoothness of pixels and features with motion, answering the question “does the picture jump?”. Object consistency looks at whether identity, number, and texture are preserved, answering “is it still the same object?”. Scene consistency looks at whether the camera, lighting, occlusion, and spatial relationships among objects are self-consistent, answering “does this world still hold?”. Causal consistency looks at whether actions produce correct consequences, answering “does stepping into a puddle produce a splash?”. The reason a cup turning into a phone can fool optical flow is that it only breaks the object layer, while the low-level layer remains continuous.

Precisely because any single metric can only cover part of this, diagnosis requires combining multiple approaches: object tracking to confirm identity remains unbroken, re-identification to handle objects that reappear after being occluded or leaving the frame, depth to check spatial relationships, optical flow to measure low-level smoothness, event detection to verify causal chains, plus human review of temporal issues. Several types of stress tests are especially revealing: reappearance after occlusion, re-entry from outside the frame, and rapid rotation of objects. These are the moments when models are most likely to lose identity, swap objects, and disrupt spatial relationships, and they are high-value scenarios for verifying consistency.

Directly stitching short videos into a long video inevitably causes drift: each time you extend by 4 seconds, characters and props eventually slowly become distorted. The root cause is that the sliding-window mechanism only keeps the most recent frames as context, while character identity and overall world state need a summary to be passed along. This summary is lossy at each continuation—it cannot fully remember all previous details, and the error caused by the discarded information becomes the condition for the next generation segment. Errors accumulate segment by segment like compound interest, and after tens of seconds the character's facial features, clothing color, and number of props may look like a different person from the beginning.

The solution is not to infinitely extend the attention window, because that would again hit the wall of computational cost, but to switch to a layered planning-generation-editing workflow. The system first writes a scene table, shot table, and state table, specifying which characters are in each shot, what they wear, what they hold, and what space they stand in; then generates short shots segment by segment, and at each transition explicitly verifies whether the characters, clothing, props, and spatial layout match the state table. If they do not match, it corrects them before moving to the next segment. Composition happens after generation, rather than letting the model sample all at once.

Certain content also requires specialized conditioning signals to be done well: lip synchronization for dialogue, causality between actions and consequences, and the cinematic language itself (shot size, camera position, pacing). These are difficult to automatically emerge from a vague text prompt. Therefore, long-form video production is more like a production process that alternates among planning, generation, and editing, rather than one-shot sampling by a single model. Planning handles macro consistency, generation handles local motion, and editing intercepts errors at transitions.

Cascaded super-resolution has a capability boundary that is easy to overestimate. Suppose that in the base video a pedestrian's finger count has already drifted—from five fingers to six—the later-stage super-resolution model will not fix it back; instead, it will render the texture of the six fingers more and more clearly. The reason is that each stage of super-resolution only does “enhancement,” not “re-planning”: temporal super-resolution inserts intermediate frames between already generated keyframes, and spatial super-resolution adds texture to existing content, both of which are constrained by the trajectories set by the base stage. They inherit the already existing object identity, motion direction, and event order; if these are already wrong, later stages usually only preserve them, or even sharpen the errors more conspicuously.

To prevent super-resolution from introducing new artifacts, the conditions and noise at each stage must be shared across frames. If each frame is super-resolved independently, adjacent frames will each refine in slightly different directions, producing flicker. The correct approach is to have both frame interpolation and detail addition refer to the same cross-frame shared information, maintaining inter-frame continuity.

This naturally leads to a staged evaluation method: first check whether the base motion is correct, then check the consistency of the frame interpolation, and finally check whether the spatial details are clear. Each stage only verifies the responsibility of its own layer. Once a problem is found, do not expect later stages to fix it; instead, go back to the earliest stage that produced the fault to resolve it—if the base motion is wrong, fix it in the base model; if the frame interpolation jumps, fix the temporal super-resolution; only when the texture is blurry should spatial super-resolution be fixed. Attributing the fault to the correct layer is the key to repair efficiency.

A model can generate realistic splashes, yet the foot may not actually touch the water at all; this contradiction reveals the fundamental difference between video generators and the physical world. The generator learns statistical regularities in video data: it knows what splash appearance typically accompanies a frame of "foot stepping in water," so it can reproduce that appearance. But internally it has no hard physical constraints—no conservation of mass, no rigid-body rules, no contact detection, and no persistent identity constraints. It draws pixels that "look the part," not a world state driven by conservation laws. Correct appearance but wrong contact is exactly the typical failure of this "correlation learning."

Certain scenarios are especially fragile. Rare interactions (such as multiple objects colliding simultaneously), complex hands, mirror reflections, text in the frame, and precise counting are corners that are statistically hard to cover: in training data, samples of these situations are sparse or their patterns complex, so the model can only guess from weak correlations, and its error rate is far higher than for common actions.

Introducing world-model-style state representations or geometric conditions can improve this situation—for example, explicitly tracking objects' positions, velocities, and contact relationships so that generation is constrained by those states—but this still does not eliminate the need for verification. As long as the model remains fundamentally a statistical generator, there is a risk of using plausible appearance to mask incorrect physics. Therefore, in any context involving safety, scientific research, or use as training data, visual realism must not be treated as physical correctness; task simulators or real data must be used to verify that the motion and consequences actually hold.

The most insidious misconception in video evaluation is treating a single metric as a comprehensive conclusion. For example, a decrease in distribution metrics like FVD only indicates that the generated samples are overall closer to the distribution of real videos; it does not directly answer fine-grained questions such as "whether characters are more consistent in long takes." Distribution metrics measure differences at the sample-set level, not identity stability across frames within a single video.

Correct evaluation requires covering multiple perspectives simultaneously. Text-video similarity checks semantics, answering "whether the generated content matches the prompt"; frame-level image metrics assess the clarity of individual frames; optical flow or feature differences measure short-term stability; object tracking checks whether identity is maintained in long takes; event question answering verifies whether the temporal order of actions and consequences is correct. Each of these metrics covers one layer; none can replace the others.

But machine metrics cannot replace human judgment. Human blind testing needs to holistically evaluate naturalness, adherence to the prompt, whether motion is plausible, and ultimately whether the video is usable. Human preference can catch flaws beyond metric scores, such as a certain indescribable stiffness.

To make evaluation conclusions truly informative, you also need to slice the samples: group them by duration, motion speed, number of objects, camera cuts, on-screen text, and occlusion, and separately report the failure rate and worst time segments under each slice. Finally, there is another layer of risk from the evaluation tools themselves: the encoder used for scoring may also have its own preferences, for example favoring still frames or familiar content seen during training, thus giving inflated scores. Only by recognizing this can you avoid misinterpreting the encoder's tendencies as the actual quality of the video.

Adding watermarks to generated videos does not by itself solve the risk of deepfakes. Watermarks are one link in the verification chain, not the endpoint. What really needs to be controlled is the use itself: likenesses of real people, minors, impersonation in political or financial scenarios, unauthorized portraits, and copyrighted material. Even if these things carry watermarks, the harm does not disappear because of the watermark.

The value of provenance credentials, watermarks, and edit histories lies in supporting post-hoc verification—enabling people to trace who generated a video, when, and with what tool. But this verification chain is fragile: watermarks can be cropped out, diluted by re-encoding, or even erased by filming the screen again; detectors themselves also have false positives and false negatives, and they will experience distribution drift as generation technology evolves. A detector trained today may become ineffective tomorrow.

Therefore, practical protection requires a set of coordinated measures: restrict high-risk combinations of identity and action, for example, disallow generating certain sensitive behaviors of specific real people; record the authorized sources of all input materials to ensure that materials used in training and generation are legal; clearly label outputs so viewers know the content is synthetic; and provide impersonated individuals with appeal and takedown channels so victims can quickly stop the spread. It is also necessary to distinguish two things—the model's own capabilities and the release policies built around the model; they need to be evaluated separately. A highly capable model paired with strict release policies can be used safely, while a model with limited capabilities but missing policies can still cause harm. Safety is a policy issue, not just a technical issue.

High-resolution long videos have a typical failure mode during deployment: everything works fine at first, and the last few seconds suddenly crash and exhaust resources. The root cause is that GPU memory and computation grow rapidly with spatial resolution, frame count, and attention connections—the higher the resolution, the larger each frame; the more frames, the longer the tensor; attention connections scale quadratically with sequence length. When these factors stack up, resource demands may exceed hardware limits in the final stage, while users only see an unexplained failure.

Therefore, deployment must proactively set limits and make failures explainable. The system should set maximum frame count, resolution, sampling steps, and concurrency limits, keeping tasks within hardware capacity; at the same time, support cancellation at any time, and write checkpoints at each stage so that after a long task fails, it can resume from an intermediate stage rather than starting over. For long tasks, return clear progress and failure stage—users need to know whether it is stuck at base motion, frame interpolation, or super-resolution, not a vague "failure".

When generation must be split into segments, the segments should overlap with each other, and state should be passed from the previous segment to the next. Overlap provides room for continuity, and state passing ensures identity and layout continuity, thus avoiding seams and identity jumps across segments.

Reproducibility is the other half of deployment. Keeping only the final compressed video is insufficient; once auditing, repair, or re-export is needed, there is no starting point. The complete generation recipe must be saved: model version, VAE, prompt, control signals, random seed, and the versions of each stage of the cascaded super-resolution. Before release, it should be regenerated from these original conditions and verified, ensuring it does not rely solely on a final file that may have been damaged by compression. The complete recipe and the limited runtime together constitute a deployable, recoverable, reproducible video generation service.

By stringing together all the previous links, you can see a complete causal chain that leads directly from the problem to verifiable practice.

The starting point is to break down the prompt into subject, scene, event, and shot. A vague description is not enough to constrain the time axis; only by breaking it down into verifiable state changes does each subsequent step have something to check against. Then choose the task type based on the conditions at hand: if you have only text, take text-to-video; if you have a first frame, take image-to-video; if you have a geometric path, take trajectory control. The strength of the conditions determines the degrees of freedom the model must sample on its own, and also determines the baseline for subsequent verification.

The generation itself is first completed in latent space: use a compressed low-resolution tensor to generate a complete motion skeleton, let the most difficult temporal consistency stabilize first, and then let temporal modules maintain identity, occlusion, and state on top of the skeleton. After that, enter cascaded post-processing—temporal super-resolution fills in intermediate frames, spatial super-resolution adds details, and the two share cross-frame information to avoid the flicker caused by frame-by-frame independent processing. At the verification stage, track the objects in the picture, check whether the order of events and the physical consequences hold; at the same time, evaluate on multiple axes—frame quality, temporal consistency, semantics, and human preference—since any single metric covers only one of those layers. At final delivery, record provenance, restrict identity misuse, and perform versioned release to ensure reproducibility.

This chain also gives a method for verifying it. Between fixing and observing, each layer must simultaneously answer two questions: what is fixed, and what evidence is observed. The input layer fixes the same batch of samples, the same preprocessing pipeline, and permission boundaries, and observes input hashes, slice labels, and rejection reasons. The mechanism layer changes only one core variable while locking all other configurations, and observes key intermediate states and where the first deviation from expectation occurs. The output layer uses the same acceptance rules and resource budget, and observes differences in quality, cost, latency, and failure rate across layers. The falsification layer keeps a control group that does not enable the target mechanism, to see whether the benefit can be stably reproduced across different samples and random seeds. Only when a causal chain is self-consistent in the three directions of fixing, observing, and falsification is it not just a derivation in principle, but a practice that can be repeatedly tested.