High-Level IVL Overview

IVL is different from other low-level APIs in that it provides more than just a mechanism for moving pixels from one location to another; it actually defines a pixel-processing pipeline that operates on pixels as they are transferred. The following figure contains a diagram of the IVL state machine. This diagram shows how pixel data moves around the system and operations that affect pixels during transfer.

In this diagram, the arrows represent the flow of data within the system. Words that begin with "gl" indicate IVL subroutines that provide state or input values for that particular operation. Boxes that are outlined with thin lines and have square corners are operations that are applied to pixel values. Boxes with bold outlines and rounded corners represent storage locations for pixel values.

Figure 2-5  Abstract Machine

On the left side of the above diagram you'll see the three IVL routines that provide pixel information to IVL: glConvolutionFilter2DEXT, glColorTableEXT, and glDrawPixels. All three of these routines pass in a pointer to pixel values stored in host memory.

Pixel values in host memory can be stored in a variety of formats. The glPixelStore routines provide IVL with the parameters necessary to extract the desired pixel values. Using these pixel unpacking values, each of the three routines obtains the indicated pixels from host memory and sends them to their destination.

In the case of glConvolutionFilter2DEXT, the pixel values are modified by the scale and bias values set by glConvolutionParameterEXT and then stored in convolution filter memory for later use. Similarly, glColorTableEXT causes pixel values to be extracted from host memory, modified by the scale and bias values set by glColorTableParameterEXT, and stored in color table memory for later use.

The ultimate destination for pixels specified by glDrawPixels is frame buffer memory. The pixel values follow a somewhat winding path to get to the frame buffer. The following figure is a more succinct diagram showing the steps that occur as part of the glDrawPixels routine. You should be able to trace a path through the state diagram in the previous section and see the same set of operations.

Figure 2-6  Pipeline Stages

The steps defined in the previous figure are described in the subsequent sections.

Unpack Pixels

For simple image transfers, the parameters for the glDrawPixels routine provides all the flexibility that is needed. The image is stored as a rectangle in host memory and you define the width, height, type, and format, along with a pointer to the start of the image data.

There are often times when additional flexibility is required. It is not uncommon for applications to maintain a large image (like 2K×2K) and transfer smaller portions of that image to the screen for display. The glPixelStore routine lets you transfer a subimage, and allows you to skip over padding at the end of each row of pixels.

A pixel rectangle is a two-dimensional array of pixels. Each pixel may contain one or four components depending on the format argument to glDrawPixels. Each pixel component has the machine data type specified by the type argument to glDrawPixels. The pixels in memory can be thought of as a rectangle that is arranged in a series of rows. If no scaling or rotation factors are in effect, the first pixel group in the first row will be displayed at the current raster position as the lower left corner of the image.

The GL_UNPACK_ROW_LENGTH attribute can override the number of pixels in each row. If GL_UNPACK_ROW_LENGTH is 0, the number of pixels in each row is assumed to equal the width parameter passed to glDrawPixels, otherwise the number of pixels in each row is GL_UNPACK_ROW_LENGTH. This attribute is typically used to extract a subimage with rows that are shorter than the real image stored in memory. If GL_UNPACK_ROW_LENGTH is less than the width you specify to glDrawPixels, you may see some "striping" effects. Keep in mind that this attribute indicates the number of pixels in each row, not the number of bytes. Note that IVL does not restrict the input arguments for pixel store operations to prevent these effects.

You can also skip a number of rows before reading the first row, and skip a number of pixels in that row (and each subsequent row) prior to the first pixel that is to be transferred to the display. GL_UNPACK_SKIP_ROWS defines the number of rows to skip, and GL_UNPACK_SKIP_PIXELS defines the number of pixels to skip in each row.

Figure 2-7  Skipping Rows and Pixels

The previous figure shows how these three pixel store values are used to extract a subimage from a larger image in host memory. The values indicated by width, height, and pixels are the values passed to the glDrawPixels routine.

To get to the first pixel that is to be transferred, the number of rows specified by GL_UNPACK_SKIP_ROWS is skipped. Each of these rows contains GL_UNPACK_ROW_LENGTH pixels. This positions the pointer at the beginning of the first row containing pixels to be transferred to the display.

Next, the number of pixels specified by GL_UNPACK_SKIP_PIXELS is skipped, positioning the pointer at the first pixel to be transferred. The next width pixels are transferred to the display. The pointer is then positioned at the start of the next row, and the process is repeated for each of the height rows that contain pixels to be transferred.

IVL also allows you to specify the data alignment per row via the GL_UNPACK_ALIGNMENT attribute. This attribute specifies whether each row begins at a memory address that is a multiple of 1, 2, 4, or 8 bytes. Set the alignment to 8 if each row begins at an address that is a multiple of 8, set it to 4 if each row begins at an address that is a multiple of 4, and so on.

The default value is 0 for the pixel store parameters that indicate row length, number of rows to skip, and number of pixels to skip in each row. The default value for alignment is 4.

Figure 2-8  Unpacking Pixels

There are three main steps that go into unpacking pixels, as shown in the above figure. First, the arguments to glDrawPixels together with the pixel store parameters are used to extract pixels from host memory as described above. The pixels that are extracted will have components that are either unsigned bytes or unsigned shorts, and each pixel will consist of either one or four components, depending on whether the format argument is GL_LUMINANCE or GL_RGBA.

In order to simplify subsequent operations on these pixel values, the next conceptual step is to convert all pixel components to floating-point values in the range [0,1]. This conversion is shown as the second box in the above figure. (Note that this is a conceptual step in order to make it easier to specify the semantics of subsequent imaging operations. Real implementations will be optimized to avoid this conversion step if it is unnecessary). After completing this step, all pixel components are the same type and can be processed similarly.

The final conceptual step in unpacking pixels is to convert luminance values to RGBA values by assigning the luminance value to each of the red, green, and blue pixel components and assigning a value of 1.0 to alpha. As the previous figure shows, the result of pixel unpacking is a stream of pixels that are uniform in type and format. This makes it easier to describe the operations that follow.

Convolution is a common image-processing operation used to filter an image. The filtering is accomplished by computing the sum of products between the source image and a smaller image called the convolution filter or convolution kernel. The convolution filter can be loaded with different values to achieve effects like sharpening, blurring, and edge detection.

If you want to perform a convolution operation as part of the pixel-processing path, the first thing you need to do is define the convolution filter. This can be done using the glConvolutionFilter2DEXT routine.

The pixels that make up the convolution filter pass through the "Unpack Pixels" stage of the pixel-processing path just as if glDrawPixels was called. However, instead of continuing on to the "Pixel Transfer" stage, the filter values are routed to the convolution filter memory for later use.

As the filter values are stored, they are modified by the current convolution filter scale and bias values. These attributes are set by glConvolutionParameterEXT.

These scale and bias values are provided in groups of four, one value for each of the red, green, blue, and alpha components. As each filter value is saved, its red component is multiplied by the red scale value and added to the red bias value, the green component is multiplied by the green scale value and added to the green bias value, etc. The resulting values are not clamped to the range [0,1] during this process. If internalFormat is set to GL_RGBA, each of the four resulting components is stored away in the convolution filter memory. If internalFormat is set to GL_LUMINANCE, only the resulting red values are stored away.

The convolution filter is a two-dimensional image indexed with coordinate (i, j) such that i increases from left to right, starting at zero, and j increases from bottom to top, also starting at zero. The convolution filter value at location (i, j) will be the Nth pixel, counting from zero, where N=i+j×filter_width.

Unless you really need to specify different convolution filter values for each of the red, green, blue, and alpha components, you should just specify an internal format of GL_LUMINANCE. This will result in a single component convolution filter that may provide slightly better performance than when using a four-component convolution filter.

The initial implementation of IVL currently supports convolution filters of size 3×3 only. The maximum supported values for the convolution filter width and height may be queried by calling glGetConvolutionParameterivEXT with the pname argument set to GL_MAX_CONVOLUTION_WIDTH_EXT or GL_MAX_CONVOLUTION_HEIGHT_EXT. It is a good idea to perform this query as part of your initialization routine so that you don't try to provide IVL with a convolution filter that is larger than it can handle.

The convolution operation is disabled by default and the default convolution filter is an empty filter. If you enable convolution without specifying a convolution filter, the input image will pass through the convolution stage unmodified. You can query the current convolution filter using glGetConvolutionFilterEXT.

When it is enabled, the convolution operation occurs as part of the pixel-transfer operation and affects pixel rectangles that are transferred with calls to glDrawPixels, glReadPixels, and glCopyPixels. To enable the convolution operation, call glEnable with the value GL_CONVOLUTION_2D_EXT. You can disable convolution by passing the same value to glDisable, and you can see whether convolution is currently enabled by calling glIsEnabled with this value. In the default state, the convolution operation is disabled.

The convolution operation is a sum of products of pixels in the source image and pixels in the convolution filter. At this stage in the pixel-processing pipeline, source image pixels are always RGBA (four-component) pixels. If the convolution filter has an internalFormat of GL_LUMINANCE, it will be applied equally to red, green, and blue components of the source image, and alpha values will pass through unmodified. If the convolution filter has an internalFormat of GL_RGBA, the red components of the convolution filter will be convolved with the red components in the source image, the green components of the convolution filter will be convolved with the green components of the source image, the blue components of the convolution filter will be convolved with the blue components of the source image, and the alpha components of the convolution filter will be convolved with the alpha components of the source image.

When the GL_CONVOLUTION_BORDER_MODE_EXT attribute is set to GL_REDUCE_EXT, the sum of products is computed in the following fashion:

In this equation:

When the convolution border mode is set to GL_REDUCE_EXT, the output image will have coordinates that range in i from 0 to the width of the source image minus the width of the convolution filter, and that range in j from 0 to the height of the source image minus the height of the convolution filter.

See the glConvolutionParameterEXT reference page for information on other supported border modes.

The next box shown in the Pixel Transfer figure is "Post-Convolution Scale and Bias." The filter values are not clamped when they are loaded. This means that you may specify values outside of the normal legal range (such as negative numbers). But this also means that you may cause an underflow or an overflow condition to occur. To alleviate this, IVL provides a way to specify scale and bias factors for each component that are applied once the convolution operation has been performed.

The post-convolution scale and bias factors can each be specified as a quadruple, with one value for each of the red, green, blue, and alpha components. These scale and bias values are only applied if convolution is enabled. After the convolution operation has been applied, all resulting red values are multiplied by the red post-convolution scale factor and added to the red post-convolution bias factor. Green, blue, and alpha components are treated similarly.

Once the scale and bias factors have been applied, resulting component values are clamped to the range [0, 1]. (Remember, at this stage of processing, all component values are treated as floating-point values with values normally in the range [0, 1].) The post-convolution scale and bias values are specified by calling the glPixelTransfer routine.

Another common imaging operation follows convolution in the pixel-processing pipeline. The image-transformation operation provides support for image scaling, rotation, and translation.

Like other operations in the imaging pipeline, the image transformation can be enabled by calling glEnable and disabled by calling glDisable using the value GL_IMAGE_TRANSFORM_2D_HP. By default, the image-transformation operation is disabled.

When enabled, the image-transformation operation uses the current set of image-transformation parameters to compute a new window coordinate for each incoming pixel. All of these parameters can be set with glImageTransformParameterHP. You can query any of the current image-transform parameters using the glGetImageTransformParameterHP routine.

Following the image-transformation stage of the pixel-processing path, it is possible to re-map all component values through the use of a look-up table. A color table uses each incoming pixel component value as an index into a table (that corresponds to that component). Thus, there are actually four look-up tables, one for each of the red, green, blue, and alpha channels. The value stored in the table at the indexed location is extracted and becomes the pixel component for subsequent processing.

The look-up table that immediately follows the image transformation stage of the pixel-processing pipeline is called the post-image transform color table. You can specify the contents of this table with glColorTableEXT.

The final step of the pixel transfer stage involves converting pixel values from their internal floating-point representation into values that map to the entire range supported by the destination color buffer. First, pixel components are clamped to the range [0, 1]. Next, if the destination buffer has m bits, each component, c, is converted to the fixed point value k where k={0, 1, ..., 2^m-1} and the fraction k/(2^m-1) is closer to c than any other value.

Another way of looking at this is with a concrete example. If your destination color buffer is 8 bits, then you have 2⁸ = 256 different values that are possible. The floating-point values in the range [0, 1] are mapped onto the 256 frame buffer values with 0.0 being mapped to frame buffer value 0 and 1.0 mapped to frame buffer value 255. The range [0, 1] is effectively divided into 255 bands, and any floating-point value in band k will get mapped to a frame buffer value of k.

The first fragment operation that is applied is called the pixel ownership test. The pixel location at which the fragment is to be written is checked to see whether it is part of the current drawable. It may be that the pixel location to be written is obscured by another window, or the pixel location might be ineligible since it would be clipped by a window clip list.

If the pixel location that is to be written is not owned by the current drawable, the window system determines what to do with the incoming fragment. Most of the time, the fragment will be discarded. However, this test is defined in such a way as to allow window-system-specific behavior for conditions such as overlapping windows.

IVL provides for a rectangular clipping region known as the scissor box. When the scissor test is enabled, the glDrawPixels routine can affect only the pixel values inside this rectangular region. When the scissor test is disabled, any of the pixels in the drawable can be modified. Scissor testing can be enabled or disabled by passing the value GL_SCISSOR_TEST to glEnable or glDisable. The glScissor routine may be used to set the current scissor box.

The scissor test is disabled by default. You can obtain the values for the current scissor box by calling glGetIntegerv with the pname parameter set to GL_SCISSOR_BOX. You can see whether the scissor test is enabled by calling glIsEnabled with the cap argument set to GL_SCISSOR_TEST.