Glitch Rider

a 4k PC intro released at Evoke 2019

[pouët]

This entry is a demonstration of how to fake the data moshing effect using fragment shaders in 4kb. It is the spiritual successor to Kill the Encoder, which was my last 4k intro at the time.

Data Moshing

Many video codecs use inter-frame compression. The idea is that a frame in a video typically looks similar to the frames immediately before and after it, so most of the time, you can get away with storing only what changed between frames rather than storing all the frames individually. Roughly speaking, videos compressed this way are encoded as a sequence of I-frames and P-frames, where An I-frame (intra-frame) specifies what an image looks like at the current point in time, and a P-frame (predicted frame) only encodes changes between frames, i.e. what transformations need to be applied to a different frame to obtain the current frame.

If an I-frame is corrupted or removed, the subsequent P-frames apply their transformations to the wrong base image. If this happens while the video cuts from one shot to another, you get the changes and camera movement from one shot but applied to another. This is what we call data moshing. There are plenty of tools that allow you to strategically remove I-frames from video files to create this effect yourself, but of course, I couldn't use those in a 4k intro.

How to fake it

The intro takes a very different approach. First, we do the usual camera ray intersection to find the surface point p. Next, we check if p was on-screen on the previous frame, first by checking if the point was occluded using another ray intersection, then by computing the pixel coordinate where p should be visible and checking if it's within the bounds of the frame. If we find p in the previous frame, we simply display its color, otherwise, we compute it from scratch.

This gives us a method to have textures "stick" to the geometry, similar to what TAA does. Note that we only ever think about where the camera was in the last frame, this method completely falls apart when the geometry moves, which is why the scene is completely static in the intro. In the shader, moving all the camera setup/movement into a separate camera function that takes the current time as a parameter makes this fairly easy to implement.

In its current state, if the camera jumps from one scene to another, it detects that none of the points it sees were on-screen in the previous frame and discards all of them. To get this data-moshing effect, we want to prevent it from detecting scene transitions like that. So, to "remove the I-frame", we base all our jumps between scenes of the time of the current frame inside the camera function. This way, we lie to the algorithm about where the camera was last frame, and that last frame sticks to the new geometry after the cut.

If you want to play around with that effect, I made Data Moshing Effect, a minimal Shadertoy that implements this algorithm. Or rather, the intro is a fork of this Shadertoy.

Some remarks

You might notice that the Shadertoy uses one more buffer than necessary. This is because every time the above algorithm is applied, it introduces a small amount of error when fetching the texture. So if you always fetch the previous frame, you get an extremely unstable feedback loop where the image quickly starts to blur or deform. To break this feedback loop, the shader only updates the "previous frame" texture every 8 frames.

Since posting the Shadertoy, I have received multiple requests to rewrite the algorithm to work on arbitrary videos. This is not feasible, as the algorithm relies on exact knowledge of the geometry in the scene and precise 3D coordinates for every pixel on the screen. If you want to data-mosh videos, messing with the video file directly is much easier.

Conclusion

This entry won first place in the 4k intro compo, though arguably Eisenerz by LJ & Virgill should have won ¹. When I first saw LJ's intro, I had doubts whether I should even submit mine. I'm glad the other Alcatraz members talked me into it.

Also, thank you to xTr1m and Gopher for helping me debug this mess.