Bioimage-oriented interfaces

As an example, consider MR (Magnetic Resonance) images. In normal clinical practice, MR images are generally used for qualitative examination. The aim of our study has been the quantitative spatial analysis of MR data using the three fundamental weighted images: PD (proton density), T1 (longitudinal relaxation time, spin-reticulum) and T2 (transverse relaxation time, spin-spin).

Starting from three 2D MR images, of 256x256 pixels (each pixel represented by 12 bits), at the same slice location in a given patient, a new single image representation of all three parameters has been generated by using the false-color technique on a HP 9000/730 workstation in a standard UNIX and X11 environment. A transformation linking together the MR parameters and the RGB (Red, Green, Blue) color components has been used. In particular, in our study, PD corresponds to Green, T1 to Blue, and T2 to Red, respectively.

The operator has several interactive controls for modifying the false-color compositing process. He/she interacts with the display shown in Figure 1.

The image in the center, is the composite result of the mapping of the three parameters by means of false-colors. It is displayed using a resolution of 24 bits per pixel via an HP CRX24 graphics board. This results in an image with high color definition which is desirable in order to better identify regions of interest for diagnostic purposes in radiology.

The operator may independently vary the mix of each of the three components in the false-color composite via three (R,G,B) mixing sliders. Each slider specifies a percentage of its associated component to be included in the composite. The operator can also do proportional "black-clipping" of the low intensity parts of the 3 components via the horizontal "Black Clip Percent" slider above the composite image. Furthermore, by using the pixel data in the PD, T1, T2 component images, a three-dimensional space containing the density distribution of these three parameters has been constructed.

This 3D histogram space is shown in Figure 1 (lower right side). It is constructed as follows: On the three 2D images (see Figure 1, left part), each coordinate pair (x,y) identifies a specific data triplet: PD, T1, T2. These data values, are the coordinate indices of a 3D histogram. The values of the histogram identify the number of pixels with the same PD, T1, T2. Memory requirements are critical in the realization of such a data structure. In fact, since each pixel is represented by 12 bits, in theory each coordinate of the histogram may range between 0 and . In a 3D space, the maximum dimension of the scatter-diagram will be pixels. Since each pixel has been defined as a "short int" (2 bytes), the image would occupy up to bytes. In order to reduce memory requirements, only six bits of each R, G, B channel have been considered. These are generated in 3-steps for each component: (a) the 12-bit data is (1D) histogrammed and mapped into an 8-bit range after clipping 1%off the high intensity (white) tail of the distribution, and (b) This 8-bit scaled component data may then be interactively attenuated by the mixing sliders and/or black-clipped as described above, and (c) the high order 6-bits of each processed component are extracted to yield an 18-bit address into a histogram array of 4-byte 'unsigned int' values. In this case, the scatter-diagram requires a memory space equal to .

The histogram (scatter-diagram) has been displayed in stereo: the stereo monitor allows the three-dimensional rendering of visual data through LCD (Liquid Crystal Display) shuttered glasses (Crystal Eyes by StereoGraphics, Inc.). Each eye shutter alternately opens and closes for half of a 72 Hz stereo cycle, while the graphics board synchronously displays a 144Hz sequence of left and right eye views. Using the mouse in this 3D space in a interactive way, it is possible to define regions of interest (ROI) in the tissue space by highlighting the anatomical zones corresponding to a given data cluster (i.e. color range). Also, a sort of inverse operation can be performed - i.e. to define some point of interest in one of the component parameter images, and see the data cluster containing its PD,T1,T2 values highlighted. This latter operation is more easily understandable to a clinician. That is, it can be explained as equivalent to asking the question: "Show me all pixels having similar tissue parameters as the one I am pointing at," the criterion for similarity being membership in a given data cluster.

We have what we think is a novel technique (patent applied for) for defining a data cluster. Namely we treat the histogram function as a data density function and apply a threshold T to it to define a set of histogram locations where H(PD,T1,T2) >= T. This set, in general will consist of several distinct connected components. These components are our operational definition of data clusters - i.e. all points in a given component are considered "similar". Moreover, the user of the system can interactively vary the threshold T, and view it's effect on the histogram clusters. Raising T will cause the clusters to shrink in a manner similar to peeling layers off an onion. Conversely, lowering T will cause the clusters to expand.

This kind of approach offers an easier interpretation of MR data and a clearer distinction between normal and pathologic tissues, allowing an immediate visual evaluation of the parameters of MR acquisition systems. The user can display in the same application the scatter-diagram, the component parameter images, and the false-colored composite tissue image. In such a way, it is possible to interact with all the available images, simultaneously reasoning about both qualitative and quantitative aspects.

Figure 2 shows a stereo pair which is an extension of this technique for a stack of brain slices making up a complete 3D tissue volume. It is from a 28 year old white male with three inoperable dural based tumors. 9 months prior to the MR scan a ventricular meningioma had been resected. The site of the resection can be seen as a dilation of the left lateral ventricle. Radiation therapy was completed three months before the MR scan, and was followed by chemotherapy. The growth of the frontal lesion suggests this is a meningeal sarcoma. The slice thickness of the MR scans is 5mm. The original data was 3 volumes of 27 slices 256x256x12 bits each. These were processed as follows:

• Each 12-bit component volume was histogrammed and scaled into 12-bits by
  • white_clip set by clipping .1% of the picture area off white histogram tail
  • black_clip set by clipping .1% of the picture area off black histogram tail
  • the reduced range [black-clip, white-clip] was linearly mapped into 12-bits.
• The resulting volume at 36-bits per voxel was reduced to a 12-bit per voxel volume and specific color palette by:
  • using the high-order 6-bits of each component value to form an 18-bit histogram bucket address in a 64x64x64 bucket color space.
  • choosing an initial palette consisting of the 4096 most popular buckets in the histogram.
  • the mean value of voxels in each bucket was used to represent each bucket.
  • voxels not lying in one of the chosen buckets were assigned to the nearest one (in color space) and the resulting bucket mean was updated accordingly.
• The resulting 12-bit paletted volume was then fed to the new ISG/IAP paletted color volume renderer. No z-interpolation was done. z-pixel dimension was set to be 3 times the x,y dimension.
• Palette colors were ordered on a HSV_12bit_order_code = vvvhhhssssss
  For reordered color index i, color[i] is composited using an opacity table interactively defined to be:

The result of this is that darker tissues have been made transparent, creating cavities in the stereo image which the user can peer into.

The intent is to add the 3D stereo histogram display to this application. A 24-bit (non-paletted) colored stereo pair view of the corresponding histogram for Figure 2 is shown in Figure 3.

This would allow simultaneous 3D visualization of false-colored tissue space and histogram space. Moreover, in contrast to Figure 2, the histogram cluster-selection technique can be used to rationally set the opacity of a cluster of interest to opaque (and everything else transparent), or alternatively transparent (with everything else opaque). This allows us to create a very powerful interactive virtual dissection tool.
Figure 4 shows a single stereo pair from a volume rendered cine-stereo sequence for flow enhanced heart imaging. This is a monochrome dataset acquired by a group at Picker, Inc. consisting of 10 volumes spaced in time over a complete heart cycle. Each volume consists of a sequence of 49 2D MR slices of a normal human thorax. For each given user azimuth angle (i.e. about the vertical axis) all 10 volumes were rendered into a cine loop of 10 stereo pairs. The resulting cine-stereo display was quite compelling and informative - several radiologists including at least one cardiac angiographer seemed interested enough to examine at it quite closely for about 10 minutes.