Google Image Visualization and Filtering

Introduction:

Welcome to my Google Image demo page! This demo is mainly to demonstrate how machine leaning, image analysis and visualization techniques can work together to enhance content-based image retrieval and junk image filtering.

Installation:

This system is built as a Java applet, so the only thing you need to have JRE (1.6+ preferred) installed on your machine before you can run this demo. If you do not have a JRE, you can download it for free here: http://java.sun.com/javase/downloads/index.jsp

Usage:

At your first time of visit, the browser will try to download the applet. After the download is completed, you need to approve the “Security Warning” for the demo to work, because the applet needs to access network to download images from Google.

If everything goes ok, you should be able to see the applet. And now you are ready to use its functions as described below:

(a) First, think of a keyword for image search, and type in the top left text box.

(b) Given the keyword, the system will download a stream of images from Google image search engine using its keyword-based search. By default, the system will download roughly 200 images from Google, but the number of images to be downloaded can be controlled through the 2^nd text box at the top of the applet. If everything goes fine, you should be able to see the layout of the search result.

The upper pane of the applet shows the similarity-based layout result, and the lower pane shows the list of images obtained from the original Google image search. An example is shown in Figure 1.

Notice that although downloading images is reasonably fast, computing the layout is time-consuming. Therefore you will probably experience slow response or may even run out of memory when the number of download images gets really large (400 for my machine).

(c) Since the system projects the obtained images based on their content similarities, so related images are clustered together while junk images are pushed out as outliers. Users can then easily identify and navigate to a sub-region of the display without caring about other unrelated images.

Figure 1: the layout of the search result using the keyword “moon”. Upper pane shows the similarity-based layout, while lower pane shows the original Google query result.

Figure 2: User indicates two relevant images by selecting them (highlighted in red boundary), the system then filter out irrelevant images based on this constraint and re-layout of the filtered search results.

(d) Users can then dynamically interact with the system with the following system functionalities:

a) Search Result Exploration: Zoom (scroll middle button), Pan (drag mouse while pressing left button), Rotate (drag mouse while pressing the SHIFT + left button) or change the size of icons (scroll mouse middle button while pressing the SHIFT key)

b) Image Acquisition: Obtain original images (double-click on a specific image and a new image window will popup) or link to the website where the image is originally found (click on the popup image window and it will lead you to the website).

Figure 3: Search using the keyword “apple”. The user selects two relevant images for QBE (highlighted in red), and the search result is shown in the lower image browse pane.

c) Relevant Image Selection: If certain images are relevant to user’s query concept, he can make selections by CTRL-CLICK on the images displayed on the upper visualization pane or on the lower image list pane. These two panes are synchronized, so selection from one pane will automatically light up the corresponding images from the other pane.

d) Image Filtering: First, you need to provide to the system a couple of positive examples through the selection functions mentioned above. After that, you can RIGHT-CLICK on anywhere inside the applet to fire a filtering operation. The system then takes your inputs as relevancy constraints and filter out irrelevant images (hopefully) before a new layout is computed. If the filtered result is not satisfactory, you can continue to select more relevant samples for filtering, or try to come up with new keywords to capture your targeting concept.

A filtering example is given in Figure 2, where most irrelevant images are filtered out by providing two relevant samples (highlighted in red boundary).

e) Query By Example: First, you need to provide to the system a couple of positive examples through the selection functions mentioned above. Then you can click on the “Query By Selected Examples” menu to fire a QBE search. The search result it listed at the lower pane. An example of using two “apple” images for QBE is shown in Figure 3.

Contact Information:

This work is accomplished while I was a PhD student at UNC Charlotte. If you have encountered any difficulties, feel free to contact me.

Thanks and enjoy playing the demo.

Yuli Gao