PV: A tool for color palette visualization

A palette in PV consists of a 64x64x64 array of histogram buckets. Each bucket is addressed by a set of 18-bit rgb coordinates. To compute the palette of a 24-bit RGB color image, the pixels are scanned sequentially. For each pixel, an rgb bucket address is computed using the high-order 6-bits of its 8-bit components. The addressed bucket contains a counter which is incremented. At the end of the scan every bucket contains a count of all image pixels having its address. We refer to this count as bucket occupancy.

Description

PV accepts .tif format images, computes an image palette and stores it if necessary, and produces an interactive display of the palette and the image as shown in Figure 1.

Figure 1a is the "Picture" window which contains the image. Figure 1b is the "Palette" window which contains the palette. The palette is displayed in a colored wireframe cube showing the boundaries of the rgb space. In the following description we use boldface and <> brackets to denote interactive controls such as buttons and sliders. For example the <Reset> button.

Volume cursor

In the palette view there is a 3D white crosshair volume cursor which is initially set at the rgb bucket position corresponding to the RGB color of the upper left corner pixel of the image - in this case a dark gray background color. A typical RGB color value of the cursor position is also displayed numerically and as a color patch in the horizontal strip below the palette. In the case shown RGB = (160,175,167). There is also a 4th number - in this case 49 - listed in the strip. In general it has a value in the range [0:63] and is a monotonic function of the bucket occupancy which we call the occupancy-group. It is explained below.

The <CursorOff> button to the right of the horizontal cursor display strip, can be used to turn of the white crosshair display in case it is obscuring some feature(s) of interest in the palette.

Motion controls

The row of buttons below the RGB display strip are for controlling the view of the palette. From left to right in Figure 1b, their names and meanings are:

State toggles

Below the motion control buttons are a row of buttons which toggle the current state of the visualizer. Their names correspond to Boolean state variables. When the corresponding variable is set, the button's indicator is red. From left to right in Figure 1b, their meanings are:

Occupancy-group slider

Bucket occupancy counts depend upon the size of an image and can take on an extremely large range of values. Thus we use a 32-bit counter to hold them. Also the occupancy counts for a given image tend to be sparsely distributed over their range.

Below the state toggles lies the <Occupancy-group Slider>. It is used to set an occupancy-group threshold T (31 in this case). The volume renderer will treat as transparent all buckets having

T can take on values in the range [1:63].

Selection mechanism

The selection mechanism is designed to control which palette buckets are selected when the user points and clicks with the mouse over the palette or the image. It is modified by the <Fill> and <Fine> state toggles.

The <Fine> state toggle is meaningful only when <Fill> is off. In this case it modifies the range of colors selected for highlighting. The default (<Fine> off) is for pixels in the designated palette bucket and its 6 3D-face-neighbor buckets to be highlighted. The 3D-face-neighbor buckets are those immediately adjacent to a bucket in the R,G,B palette directions respectively. If <Fine> is on, then only pixels in the designated bucket are selected. In what follows we refer to the effect of <Fine> by the "(s)" in the phrase "selected palette bucket(s)".

In the <Fill> off (default) state, bucket selection works as follows:

1. If the user clicks the <Left> mouse button over a non-empty position in the palette, the white crosshair will move to that location, and all pixels in the image contained in the selected palette bucket(s) will be highlighted.
2. If the user clicks the <Left> mouse button over any position in the image, the white palette crosshair will move to the color at that position, however no pixels will be highlighted in the image.
3. If the user clicks the <Middle> mouse button over any position in the image, the white palette crosshair will move to the palette bucket containing the color at that image position, and all pixels in image belonging to selected palette bucket(s) will be highlighted. If the user drags the mouse over the image before releasing the button, the selection process will be repeated for all locations covered.

If the <Fill> state is true (toggle indicator is red), then the bucket designated in cases 1 and 3 above is used as a seed in a 3D fill algorithm. The effect is to expand the set of buckets selected to all those that are connected to the seed in the current palette view.

Successive selections using the mouse are cumulative. Thus more palette buckets and image pixels get selected at each relevant mouse click. The selection memory can be reset via the <UnSelect> button in lower right corner of the Picture window.

Highlighting mechanism and fade slider

The highlighting mechanism was designed to overcome the problem of how to highlight selected pixels in a color image with minimum color distortion to the pixels and their image context. The solution chosen was to fade all non-selected pixels into a neutral background. The amount of fading (in this case 50%) is interactively controlled via the <Fade Slider> at bottom of the Picture window. It ranges from 0 percent (no fading) to 100% whereby all unselected pixels are faded to the background color. The current background color (in this case black) is displayed in the square to the right of the <Fade Slider>. This too can be interactively modified by clicking on it as follows:

NB... The image is not faded in Figure 1a. In spite of the fade slider indicating 50 percent fade, the program initializes with no fade. However, as soon as a selection is made, the indicated (50 percent) fade will be executed.

Save button

The Picture window has a <Save> button which allows the user to save the image at any time in file <fname>.tif. A pop-up dialog allows the user to choose <fname>.

Transform_mode and controls

The Palette window has a button <XFM>. It enables transform_mode in which the user may specify a color transform in terms of hue, chromaticity, lightness, and contrast changes. The transform is applied to the selected pixels and the result is immediately displayed both in the Palette and Picture views. The XFM button is only functional if there are some colors selected; if not it just beeps and reports "RGB2XFM nothing selected (nothing done)" in the terminal window from which it was started. Figure 2 shows the result of setting the <Occupancy-group Slider> to the smallest value that leaves the reds as an isolated cluster in the Palette window (T = 10)

• selecting most of the flowers, by clicking on <Fill> and then clicking on the red blob in the Palette window with the <left> mouse button, and finally clicking on <Fill> to turn it off.
• setting the <Occupancy-group Slider> to a lower value (T = 4) to include some more reds in the Palette window, and then repeatedly clicking over the red cluster in different views to pick up more reds surrounding the original cluster, while avoiding the reds of the woman's lips
• clicking on <ShowSelect>, <CursorOff> and <XFM>. The Palette viewpoint snaps to the neutral axis above the white point to show the chromaticity projection, and the "Change Color" transform control window (Figure 2b) appears.
• changing the hue angle by a 50 degree rotation (see "Hue Increment" in the "Change Color" window of Figure 2b). The result appears in the Picture window 50 percent faded to highlight the selection (Figure 2c)
• clicking on the left end of the fade slider under the Picture window with <Button2> to undo the fade and see the result in the normal image context (Figure 2d). Figure 2e is a copy of Figure 1a for easy comparison with 2d.

Diagonal Von Kries type transform

The "Von-Kries" type scaling transform is a diagonal matrix with per-channel scale factors (Rs,Gs,Bs). This allows previewing addition or removal of color-casts in images.
The procedure is as follows:

• the user must make the usual selection of colors of interest (COI)
• the last point selected in the image of the COI with <middle> button click will be singled out as the "color to be changed" (CTBC).
• the user must then click on the <XFM> button, and use the transform controls to change the selected area into the color desired. The original RGBH value of the CTBC is stored in the "Change Color" transform control window as the "BaseColor" display. As color changes are made, the updated CTBC is stored in the "Palette" window RGBH display.
• clicking on the <VonKries> button causes (Rs,Gs,Bs) scaling to be applied to the whole image. The scales are ratios of the new/old values for each color channel of the CTBC.
• clicking again on the <VonKries> button will restore the old selection and allow for interactive modification of the scaling transform.

Figure 3a shows the effect of clicking on the <VonKries> button after all the operations above. The purple shade that the original flower sample was mapped into now gives an overall cast to the image. Note the color of the flowers is essentially the same in Figure 3 as in Figure 2d. However the viewer's perception of the color is affected by the color cast of the image, and thus the flowers in Figure 2d will appear redder to most viewers. Figure 3b is a copy of Figure 2d for comparison.

Early observations

Some early observations that we have made with PV are:

• Scaling an image changes its palette. Scaling up tends to give smoother palette shapes, while scaling down tends to make palette shapes more ragged and disconnected. This is consistent with the fact that scaling up via interpolation yields more pixels per bucket and creates intermediate colors at color boundaries - the net effect being to spread out and fill in the palette. Scaling down by interpolation also adds intermediate colors, but reduces the pixels per bucket to the point where bucket quantization effects interfere with palette occupancy-group thresholding, giving a ragged effect.
• Passive reflective surfaces have banana-shaped palettes:

• Palettes of images whose color has been reduced by quantizing, show up easily. So do those from images that have been contrast stretched.
• Many conjectures can be easily checked... e.g. that skin-tone diversity is 1-dimensional seems to be valid. That is, only skin lightness and not chroma varies across races.

History and acknowledgments

The present work is an outgrowth of our Pisano medical visualization project performed at the University of Pisa with Prof. Davide Caramella and Massimo Delsarto in the year 1992-1993. This was in collaboration with Prof. Francesco Beltrame and his students at the University of Genoa.

In Pisano the goal was to take advantage of the fact that MRI machines are capable of providing three physical measurements for each point in a diagnostic volume. We decided to treat the resulting three-parameter medical images - and stacks of images making up a volume - as RGB components of color images. This 'false color' technique had been widely used in remote sensing of the earth via satellites. We concluded that the method was capable of presenting the same information usually displayed as three-separate monochrome images, as a single color composite wherein the color contrast typically gave improved tissue discrimination.

The clinical verification of these early experiments remains to be done. Our collaborators at the University of Genoa are continuing the work with some background support from us.

Early experiments

The original Pisano 2D-data demo took advantage of the histogram statistics of three-parameter MRI brain scans to locate clusters of colors which in fact could be associated with the parametric characteristics of particular tissues - e.g. skin, bone, tumor, cerebro-spinal-fluid, gray-matter, white-matter, etc. By a combination of 3D rendering, and interactive thresholding of the histogram frequency (bucket-occupancy) counts, we were able to visualize the iso-occupancy surfaces of the histogram and thereby locate the cores of clusters representing the different tissues present.

The thresholding of histogram bucket occupancy was important to remove sparsely represented colors which were typically on the boundary between two different tissues types and thus - due to the finite resolution of the instrument - represented mixtures of two tissue types. In medical imaging this is referred to as the partial volume effect. The same effect appears as well in ordinary color photographs. These sparsely represented colors, when visualized, tended to obscure the cluster structure of the data.

Once an isolated histogram cluster was apparent, the image pixels belonging to it could be interactively highlighted by the user pointing and clicking with the mouse on the cluster. This caused a 3D fill operation on the histogram cluster and a subsequent scan of the image to find and highlight - via colored overlay - pixels contained in the filled set. The highlighting was done in the monochrome component images of our display. We purposely avoided the more difficult problem of how to effectively highlight colors in a color image. The user was also able to click on an arbitrary point in one of the monochrome component images and see the corresponding cluster highlighted in the histogram display.

Rendering methods

We initially used a monochrome surface-renderer - provided in our IAP visualization environment - to view the histogram iso-occupancy surfaces. This method lacked color and location queues to orient the user in color space, and also suffered due to the poor quality of the IAP surface-renderer output for such a low resolution complex dataset. We next briefly tried the IAP color-volume-renderer which was limited in that it would only handle palettes of up to 4096 colors. We mainly found it too slow for good interactive use and cumbersome as a rapid-prototyping vehicle.

The answer to our needs serendipitously appeared in the person of Milon Mackey of HP Labs who had been working with the Volpack fast volume-renderer from Stanford. Milon was able to take my histogram data, format it, and display it in 24-bit volume-rendered color. Moreover the renderer was fast enough to allow for the possibility of an interactive 3D orientation capability. The final bit of luck was that Milon also showed me vprdemo - an interactive X11/Motif-based program, provided with the the Volpack package. vprdemo came with source code in both C and the Tcl/Tk scripting language. We started with vprdemo, added color histogram input and many interactive tools over a period of about seven months, and arrived at PV.