Tom Malzbender

» Polynomial texture mapping (PTM)
» Fourier volume rendering
» 3D photography
» Interactive texture synthesis
» 3D ultrasound
» Microtelepresence
» Volume sampling
» Artery tracking
» Rosetta graphics tablet

Polynomial texture mapping (PTM)

Polynomial Texture Mapping (PTM) is a new method for increasing the photorealism of texture maps. Coefficients of a biquadratic polynomial are stored per texel, and used to reconstruct the surface color under varying lighting conditions. Like bump mapping, this allows the perception of surface deformations.

However, our method is image based, and photographs of a surface under varying lighting conditions can be used to construct these maps. Unlike bump maps these Polynomial Texture Maps (PTMs) also capture variations due to surface self-shadowing and interreflections, which enhance realism. Surface colors can be efficiently reconstructed from polynomial coefficients and light directions with minimal fixed-point hardware. We have also found PTMs useful for producing a number of other effects such as anisotropic and Fresnel shading models and variable depth of focus. Lastly, we have discovered several reflectance function transformations that act as contrast enhancement operators. We have found these particularly useful in the study of ancient archeological clay and stone writings. Download sample PTMs as well as code for viewing and creating PTMs.

We have found PTMs especially useful for the display and enhancement of ancient clay tablets as described in "Picturing the Past." In particular, extracting per-pixel surface normals can be done at high resolution. In fact no spatial kernal is used when extracting these normals, resulting in very little spatial blurring. Once extracted, these normals can be used to effectively re-light the surfaces. We have developed two methods, specular enhancement and diffuse gain, that are effective at bringing out surface detail not apparent to the human eye. A third method, Light Direction Extrapolation, is also useful for creating lighting conditions more oblique than physically possible.

Fourier volume rendering

Volume rendering is traditionally done in Euclidean space. Here we present a method that works from data represented in the frequency domain. It employs the Fourier projection slice theorem, used in various recontruction applications, most popular being Computed Tomography. Here we basically invert the process of CT.

A conventional uniformly sampled volume aquired with MRI, CT, PET, etc is transformed into the frequency domain as a preprocess. This representation can now be sampled along a two dimensional slice of the 3D fourier data. When inverse fourier transformed, this slice yields the spatial projection of the original dataset, in a direction perpendicular to the resampling slice in the frequency domain. The net result is a factor of 100 - 1000 in computational complexity over a spatial approach. This is due to the fact that traditional volume rendering requires sampling in 3 dimensions, whereas here only sample a two dimensional slab in frequency space.

3D photography

A central problem in computer vision is the estimation of 3D structure from multiple images of a scene. Although methods have been developed for specific classes of scenes, a general, robust solution to this problem has proved elusive. We are pursuing a new approach pioneered by Steve Sietz called voxel carving that uses an underlying volumetric representation. If the location and orientation of all cameras are know, one can project each voxel into all source images and determine the statistical color consistency between those footprints as evidence of the voxel containing a surface.

Our algorithms concentrate on 3 main contributions. Firstly, we can ensure information from all cameras available can be used. Secondly, our treatment of visibility can use layered depth images which provide computational advantages.

Interactive texture synthesis

Starting from a photograph of a stationary texture, we construct a statistical model automatically using color co-ocurrance measurements. This model can then be used to interactively synthesize new instatiations of the texture. For each pixel synthesized a non-causal neighborhood is analyzed and the pixel color chosen that minimized the difference in normalized co-occurrance statistics between the texture being synthesized and the original source texture. Fast incremental methods are used to allow this to happen at interactive rates. The result is a painting package that allows one to draw with textures that are context sensitive and blend in with the background texture. In the example shown the bee can be interactively erased by synthesizing pebble texture.

3D ultrasound

Two dimensional image slices acquired with ultrasound can be combined to reconstruct a three dimensional imaged volume. We use an Ascention magnetic position tracker to determine the spatial location and orientation of each slice acquired in real time. This data is resampled into a uniformly sampled array using filters that are fast and smooth. Particularly effective are the segmentation techniques provided by the ultrasound machine itself. Acoustic Quantification and similar methods can differentiate blood from tissue at frame rates. Voxels labeled with AQ are tesselated into a polygonal representation and rendered interactively using 3D graphics hardware. It takes a few seconds to patient, a few seconds to resample and then renderings are available in real time from arbitrary perspectives.

At right is a rendering of several veins in my liver, imaged with a HP Sonos 1500.

Volume sampling

A step in almost every volume rendering algorithm is the interpolation of volumetric sample values. A common example is during the ray casting of preclassified volume data. Here sample points are defined along the rays and color and opacity values are sampled in the volume, typically using trilinear interpolation.

The common method is to sample color and opacity values independently at each ray point. This method is flawed and leads to sampling artifacts. The correct approach is to interpolate the color values preweighted by their opacities (in a rendering of human vertebrae, above left). If this is not done colors assigned to empty regions of space (zero opacity) can wind up bleeding onto surfaces. The red color (above right) was assigned only for zero opacity voxels and should not appear in any final rendering. Note that it does appear when separately interpolating color and opacity.

Artery tracking

Atherosclerotic vascular disease is the most common cause of death and disability in the western world. Magnetic Resonance is on the verge of having the resolution required to image the coronary arteries and detect lesions and plaque buildup. Here we have developed a method that assists in the visualization of arterial datasets in general, regardless of imaging modality. Given a seed point on the artery, we can track the path of the artery in 3D and use this path to extract a ruled spline surface through the arterial medial axis. Intensity values from the raw imaging data can be mapped onto the resampling surface providing a visualization of the cross-section of the artery.

This cross section can be rotated interactively and is stereoscopically rendered and displayed from arbitrary viewpoints in conjunction with the original slice data. The challenging aspect of this method is the robust tracking of artery.

This is accomplished with a parametric finite impulse response filter. This center surround filter has a number of sample points arranged in two concentric rings. Under parametric control the filter can be arbitrarily oriented and sized and responds maximally when the interior sample points lie inside the artery and exterior sample points lie outside the artery. Since the cost of evaluating the filter is independent of size or orientation, the parameter space can be automatically searched in real time to find the path of the artery.

Rosetta graphics tablet

In 1984 Mike Berke and I wanted a graphics tablet for our home workstations, at the time these cost a few thousand dollars each. We came up with a way to keep the drive electronics cost low with a new method called Permuted Trace Ordering and began designing and building a graphics tablet around it. Eventually enough interest within the Personal Office Computer Division was yielded for HP to pick up the design. The tablet was manufactured in Singapore and sold as the HP 45911 for several product generations.

Permuted Trace Ordering uses a small number of drive lines (eight in each axis for Rosetta) used repeatedly in a spatial pattern that guarantees a unique signature of three traces that are closest to the stylus anywhere on the tablet.

This unique pattern of top three sorted stylus responses provides a coarse position estimate. Fine position can also be determined using these measures responses to a resolution of 1000 lines per inch. The code sequence was designed to be insensitive to a moderate degree of electrical noise. Additional details on the scheme are available in the HP Journal Paper (high resolution .pdf file, 12.74MB or low resolution .pdf file, 1.87MB). At the time, our HP 9836 workstation required several days of runtime to compute an acceptable code sequence.