Backface Culling 101

This post is for artists, designers, and anyone else who has had backface culling shoddily explained to them. Perhaps I should have checked the Venn Diagram of AltDev readership before writing about a rendering technique specifically with non-programmers in mind. I know I've gotten embarrassingly confused by forgetting what's important as little as 2 months ago.

The "Simple" Answer is Misleading

Backface culling comes up all the time when people are first becoming acquanted with 3D game dev. All it takes is deleting half of a cylinder in Maya and dropping it inside of an easy to use engine like Unity. Then they're suddenly be trying to figure out what got screwed up that's causing the inside of it to be invisible. Inevitably the answer given is that those faces are backface culled to avoid rendering the inside of 3D models. That's true to some extent, but in my opinion very misleading.

The other part of the common answers is that backface culling works by only drawing the side with outward facing normals. This part, is not actually right and it leads to a lot of misunderstandings. I think this answer comes up because people like artists and designers are used to thinking of meshes in 3D space, rarely do people think about the process that turns it into a 2D image unless they get into rendering programming.

The Triangles are not Drawn twice

First off, understand that you have a big mess of triangles that you are using to represent your piece of art. One of the simplest representations of this is to have all the vertices in a big long list, and then to have a big long list of triangles defined from those points. Your computer is taking those triangles and transforming them into the 2D space that is displayed to you. That triangle is so simply defined to expedite rendering that it has no concept of "front" or "back" (well it does, but I'm getting to that). The vertices may have normal information, but that's used for lighting, not culling. Consider a camera pointed at a half-cylinder that renders properly in the first case, with the following badly drawn diagram taken from above. The green lines are normals:

Now consider the *exact same* list of triangles with the only change being to flip the normals at each vertex. If you try this on your own, it is probably best done by modifying your vertex list in the rendering code, for reasons that will be explained shortly.

With backface culling turned on, you will still see the cylinder in each image, except the lighting will be flipped in the second one. And if you turn off backface culling then the same images will be rendered, and it will be done so with the same number of triangles, because there are no backfacing triangles in the given example. Don't think that the GPU is rendering the backside and then going over it a second time when you have backface culling turned off for the front side, unless you actually did specify two triangles in that big list of triangles with vertices in the exact same locations.

Assuming you understand that what I described is not what typically happens, let me now explain the point that I'm making. When you flip the normals in a 3D modeling package it's doing more than just changing the vertex information, it's also changing that triangle list. This is because the front and back face of a triangle are determined through the novel concept of winding order, which is quite simply that the front of a triangle is the side of the face where vertices are viewed counterclockwise. Suppose you have a mesh with no normal data, just positions in 3D space for vertices. You can still use backface culling just fine as long as your triangle list is specified in the proper order, a point that I think is often missed by designers and artists when trying to understand backface culling.

If you want the proof in the pudding, here's the OpenGL call to specify culling:

glFrontFace(GLenum mode);

Where mode is either GL_CW or GL_CCW, standing for clockwise and counterclockwise, notice that this has nothing to do with vertex normals!

And here's a diagram to illustrate which side is the front for a triangle by default (you could assume that triangles are front clockwise as your standard if you really want to, but the default is counterclockwise). If the points defining your triangle are listed in an order that would cause the diagram on the left to be true after transformed into screen-space, then that triangle is front facing:

So what are we culling again?

We're culling triangles facing away from the camera, which are going to be obstructed by the forward facing polygons on a closed mesh. Those triangles that would be drawing the inside of that cylinder are just going to be covered up by the front-facing triangles that you would actually see. With the winding order in hand, the triangle's face normal can be determined and checked against the camera. I made this little diagram (insipred by the great explanation for backface culling in Real-Time Rendering). The triangles on the backside of the full cylinder are facing away from the camera, so they are culled (indicated by the dotted part).

So on a closed mesh like a full cylinder, you can avoid doing the work of rendering all of those triangles that you already know are going to be obscured. This is why disabling backface culling is typically not the correct answer when you have triangles culled that you didn't intend to, and it's also bad because that usually means that the backfacing triangles are being shaded incorrectly with backwards normals.

If you do intentionally cut that cylinder in half, the easy fix is to add front facing triangles along the inside of the mesh. I believe there is functionality these days (DX10 maybe?) to figure out which side of the triangle is being rasterized. Theoretically you could have the shader flip the normals based off of that information to still have correct lighting, but if you just read a post on the basics of backface culling, I bet that's not what you're looking to do.

Because 3D modeling programs are automatically adjusting the winding order for you based off of what direction your normals are, it's easy to mistakenly think of backface culled triangles as triangles with incorrect normal information, when really it's winding order that determines it. This is why sometimes you can really end up in weird situations while using a 3D modeling package. I know that as a Freshman in college, there were many models at game jams that had those stray triangles that artists just couldn't get to show up, and maybe if they did the lighting got all weird. Perhaps thinking about what's actually happening can help alleviate those pains.