Polygon Tessellation in OpenGL

Two sample shapes that are easily modeled "by hand" with triangle strips (above) and a wall shape that is not (below).

Motivation

OpenGL requires that all geometry to be rendered must consist of some combination of points, straight line segments, and triangles. To a certain extent, more complex shapes can be fairly easily constructed by clever use of the various triangle primitive types we have seen. (See the two sample shapes at the top on the right.)

If the holes and/or concavities are much more complex, however (e.g., the wall models at the bottom on the right), it becomes much more difficult – for all practical purposes, impossible – to create the model geometry without the help of some sort of sophisticated algorithm. This is where OpenGL's polygon tessellation functionality comes into play. You simply use the OpenGL glu tessellation sub-API to define all the contours (i.e., connected sets of edges that comprise the outer and inner boundaries of a desired object), and then OpenGL will use callback functions you have registered to tell you the sequence of triangle primitives that can be used to render the shape.

Details

To use the facility, the geometry (and, optionally, the additional per-vertex attributes) of the object's contours are specified once, typically during execution of a ModelView subclass constructor. After you have done so, the basic tessellation algorithm runs, and your code captures and stores in data structures specifications for the triangle primitives that are to be rendered. Then during display callbacks (specifically when your render method is called), you simply bind the vertex array as usual and use those data structures to issue a series of glDrawArrays calls.

Minor note: The most common spelling of "tessellation" is with two "l"s, however the OpenGL API uses one "l" in its data types and function prototypes.

Compiling and Linking

Part 1: Model Definition Time in Your Constructor

1. Create a tessellator object using gluNewTess():
   GLUtesselator* tObj = gluNewTess();
2. Define tessellation callbacks using a series of calls to gluTessCallback(tObj, cbType, cbFcn)
3. For each polygon to be tessellated and associated with this tessellated object (see example_Step3.c++):
   1. gluTessBeginPolygon(tObj, optionalPtr); // Details on "optionalPtr" later; for now, assume nullptr
   2. For each contour:
      1. gluTessBeginContour(tObj);
      2. gluTessVertex(tObj, xyz, pva); …; gluTessVertex(tObj, xyz, pva); // double xyz[3]; For now, assume pva == xyz as in "example_Step3"
      3. gluTessEndContour(tObj)
   3. gluTessEndPolygon(tObj)
   4. (The OpenGL tessellation algorithm begins when gluTessEndPolygon is called. Control does not return from gluTessEndPolygon until the tessellation algorithm has finished. As the algorithm runs, your callbacks will be invoked, providing you with the data you need to store in your data structures.)
4. Delete the tessellator object and any memory it has allocated using gluDeleteTess():
   gluDeleteTess(tObj);

1. The gluTessVertex call takes three parameters. The tessellation algorithm uses the coordinates passed in the second parameter, however the pointer that is returned to your GLU_TESS_VERTEX callback function is the third parameter. You saw in the "example_Step3.c++" code above that the same parameter was passed for both the second and the third parameters. This may seem odd, but it will make more sense when we look at how to keep general per-vertex attributes associated with vertices as they go through the tessellation process.
2. You must not overwrite any part of the data passed as the second or third parameters to gluTessVertex until the algorithm completes (i.e., until control returns from gluTessEndPolygon). Similarly, you must also ensure that the data don't get deleted, including by going out of scope while returning from a function call.
3. Vertices for each contour should be specified in step 3.b.ii above in an order such that the material of the face is "locally on the left" as you walk vertex to vertex. For example, the outermost contour would be specified in a counterclockwise order; an inner hole loop immediately inside this outermost contour would be specified in a clockwise order.
4. All of the geometry specified in step 3 for a polygon to be tessellated must be coplanar. It need not be in the xy-plane (or any plane of constant coordinate). However, the results are undefined if the vertices do not all lie in a common plane. (As we will see a bit later in the course, however, you can use the facility on parametric curved surfaces. We can define the contours in the parameter space of the surface – e.g., a "vertex" is (u_i, v_i, 0) – then as the vertices come back to your GLU_TESS_VERTEX callback function, you store the 3D points coresponding to the returned surface parameters into your VBO.)

Part 2: During Display Callbacks

This is the easy part. When the render method of your ModelView subclass is called, bind the appropriate vertex array and then simply loop through the "glDrawArrays buffer" you created:

for (int i=0 ; i<bufSize ; i++)
	glDrawArrays(buf[i].mode, buf[i].start, buf[i].num);

Additional Useful Capabilities

1. Preserving arbitrary per-vertex attributes
2. Additional callback function options

Caution – Caution – Caution

You may well find your way via google to what looks like a much simpler way to incorporate polygon tessellation into your code. There will be no need for buffering of anything, and all the routines you pass to gluTessCallback are OpenGL calls like glBegin, glVertex3dv, and glEnd. You would be right in thinking that this is a much simpler approach, however you would never get it to work because those calls are all deprecated and will not work with shader-based OpenGL programs. Trust me: the way I have described it here and in class is the only way it will work!

Finally...

This polygon tessellation facility is a holdover from the early days of OpenGL. It is still available, although it is not well advertised. There are many more options supported in this sub-API than what we have covered here. If you find an early version of the Addison-Wesley book "OpenGL Programming Guide" (a.k.a "The Red Book"), you will find that it has an entire chapter devoted to the facility. As of 2019 November 15, an online version of this chapter is available.