The Making of Koi 3D

Koi 3D was creating using advanced techniques at the cutting edge of both art and science.

Each koi fish in Koi 3D is actually a digital model comprised of a set of points in 3D space that map the contours of the surface of the fish.

This digital model was carefully animated, frame by frame, to create numerous motion sequences that the fish could then choose to follow when swimming (for example, turning left, accelerating, swishing the tail).

Each fish in Koi 3D was carefully painted using 3D painting software that allowed the artist to render her vision onto a 3D canvas - almost like painting a sculpture in the real world, except that all the work is done inside a computer program.

Koi 3D contains a real-time simulation of water, complete with real world properties of water such as reflection and refraction. As the koi fish swim beneath this virtual liquid surface, they appear bent and distorted as waves move over them.

As they swim, each koi uses a little artificial intelligence "brain" to allow it to navigate the pond alongside all the other fish, avoiding collisions by predicting where the other fish are heading and steering clear of them accordingly.

Specific Techniques

The Water Surface Simulation

Before any rendering of the pond can occur, the water surface must be simulated. A variety of methods are known and discussed in the scientific and computing literature for doing this, but many of them are extremely computationally intensive and thus inappropriate for realtime-rendered applications.

The most accurate method of simulating water available relies on solutions to the 3D Navier-Stokes fluid equations. This method is most commonly used for non-realtime 3D graphics (for example, the CGI water found in many films) and requires a prohibitive amount of computational effort. Realtime applications, such as video games, usually involve some simplified form of solutions to the 2D versions of the Navier-Stokes differential equations.

For Koi 3D, we used a simple discrete "propagator" equation to model wave movement along discrete surface points of the water surface. Effectively, we represent the surface as a series of floating points connected by springs. This produces roughly similar results to a complete solution to the differential equations with less computational effort, and produces visually convincing water movement and response.

Reflection Effect

Modelling true reflection in realtime is very difficult. Reflection is a complicated light phenomenon in which light rays "bounce" off of multiple objects. To achieve this effect in realtime requires an alternative to a realistic physical model of reflection.

For our reflection effect, we rely on Cubic Environment Mapping, a feature built into most modern 3D graphics cards. Cubic Environment Mapping uses a virtual "camera" to take six images of the current scene from each of six different angles corresponding to the six faces of a cube (top, bottom and the four sides).

These six cube face images (the reflection "cube map") are loaded into your graphics card so they can be used to generate reflections from any possible angle. The reflection visible on a reflective surface (for example, the teapot below) can then be generated by sampling from the correct location on this virtual cube surface.

In Koi 3D, the reflection cube map is one of the few pieces of data that, for performance reasons, is not updated every frame. In fact, the reflection cube map doesn't change at all when the view camera is stationary, only when it is moving, and even then, it is only updated once every few frames due to amount of computational work required to render 6 separate images of the entire scene onto the water surface.

Refraction Effect

Refraction is a more complex phenomenon in real life. The actual amount of distortion, or bending of the light, created by a fluid surface depends on the index of refraction of the material through which the light is passing (in this case, water). The physical law that describes this distortion is called Snell's Law, and it relates incidence angles of light rays to refracted angles. Again, though, simulating the real physics of water is currently computationally prohibitive for a realtime application like Koi 3D.

As a result, we use a simplified technique called Projective Texture Mapping to create the appearance of refraction. This technique involves first rendering the portion of the image below the water surface (the pond bottom and fish in our case) onto a flat image in your computer's memory. This flat rendered image (or texture map) is then projected back up onto the water surface simulation. The water surface geometry is used to vary the region of the texture map that is sampled, creating the effect of refraction ripples around areas where the water surface is particularly distorted. The amount of distortion that results is similar to the index of refraction in a real liquid.

Fresnel Effect

To tie all these effects together, the various rendering techniques (reflection, refraction, and wave movement) need to be combined when drawing the finished product, a dynamic water surface that moves and shifts each frame.

The fresnel effect, or the amount of reflected light (or surface reflection) visible based on the viewing angle, is calculated each frame for each of a large number of sampling points on the water surface. This data is then used to blend the reflection and refraction images together to produce a seamless water surface, with bright, beautiful fish moving fluidly beneath it.