Photobook: image database browser --------------------------------- User's guide ------------------------------------------------------------------------------ 1 Introduction 2 Performing a query 3 Mouse controls 4 Viewing the result of a query 5 Entering a query filter 6 Option menus 7 Controls 8 Toggle windows 9 Customization ------------------------------------------------------------------------------ 1 Introduction Photobook is an interactive browser of image databases which uses machine vision to perform queries based on content. Traditional database browsers perform queries based only on textual annotations provided by humans. This program can perform queries based on annotations, but can also perform queries based on "content", as defined by which vision algorithm is used. For example, images may be queried based on low-level features such as coarseness, color histogram, directionality, or periodicity. Suitable vision algorithms may even allow queries based on higher-level features such as the presence of certain objects or the spatial relationships of those objects. Further sophistication may allow highly semantic queries such as "happy" scenes or "boring" scenes, according to the user's definition of these concepts. The philosophy behind Photobook is that low-level features can be combined in intelligent ways to produce an illusion of high-level features. Since the vision community seems to have achieved far more success in modeling low-level phenomena than high-level, this appears to be the most feasible approach at this time. Consequently, the current implementation supports a large number of low-level features for query. ------------------------------------------------------------------------------ 2 Performing a query The user is presented with a grid of still images (the "working set") and some control buttons. To compose a query, select some images ("query images") from the grid and/or enter an annotation filter ("query filter") in the text entry field below the grid. To perform a query, press the middle mouse button on an image (it doesn't matter what image), or press return in the query filter. The result of the query will replace the working set, and the user may again select images to perform another search. The result of a search is intended to mean: "these are the images with annotations satisfying the query filter, ordered by their content similarity with the query images". If no query images were specified, then the images appear in the order that they appear in the database. If no query filter was specified, then the result contains all images in the database. ------------------------------------------------------------------------------ 3 Mouse controls On an image displayed in the grid, the mouse buttons act as follows: L - select/deselect the image M - perform a search on the selected images using the current query filter R - pop up annotations of the image Elsewhere, the behavior is defined by the Motif widget set. ------------------------------------------------------------------------------ 4 Viewing the result of a query The number of images in the working set is displayed to the left of the grid. If the working set is too large to fit on the grid, it is "paged" into the grid. The currently displayed page can be changed with the arrow buttons to the right of the grid. The Photobook application window can be resized. If the grid doesn't have enough space, scroll bars will appear. If the grid has too much space, it will be padded with the background color. Pressing the "resize" button will change the number of rows and columns of the grid to best accomodate the amount of space provided by the appplication window. You can press buttons and interact with the images even as they are being loaded; in particular, you can advance the page before the current page has been loaded. Each image commonly has a black text area underneath it, which can display the image tag (roughly corresponding to its filename), the similarity "distance" of that image, and more. (The meaning of the distance varies from one search algorithm to another, but always decreases for images which are more similar to the query images.) Use the "Text..." toggle window to set what kind of text is displayed. ------------------------------------------------------------------------------ 5 Entering a query filter A query filter is a logical expression built from the following primitives: Literal image tags, e.g. boston.00010 - Matches the image with that tag. Specific annotations, e.g. place/boston - Matches those images which have a matching annotation category ("place") and category value ("boston"). The slash separates the category from the value. String annotations, e.g. "boston" (with quotes) - Matches those images which have a string annotation with a substring equal to the given string (e.g. images with a string annotation containing "boston" in it). No word parsing is performed, so "boston" matches with "in boston" as well as "bostonite". Primitives are delimited by whitespace, so multi-word primitives must be flanked with square brackets (e.g. place/[new york] or [place/new york]). Primitives are combined using the following prefix operators: And (e.g. (and place/boston scene/outdoor) ) - takes at least two operands. Matches with those images which match each of the operands, e.g. outdoor images in boston. Or (e.g. (or place/boston scene/outdoor) ) - takes at least two operands. Matches with those images which match any of the operands, e.g. outdoor images or images in boston. Not (e.g. (not place/boston) ) - takes one operand. Matches with those images which do not match the operand, e.g. images not in boston. Except (e.g. (except place/boston scene/outdoor) ) - takes two operands. Matches with those images which match the first operand but not the second, e.g. images in boston that are not outdoor. Equivalent to (and place/boston (not scene/outdoor)), but faster. Parentheses are used to enforce precedence, and are optional around primitives or the filter itself. Operators and annotations can also be entered via the "Symbols..." toggle window. ------------------------------------------------------------------------------ 6 Option menus The "database" option menu will change the image database used. The "display mode" option menu will select how each database entry is represented. Normally a database entry is represented by a single associated image, but a database can also define a set of alternative representations, such as the Fourier spectrum or NTSC gray component. Another common display mode is a bar graph of a set of image features, such as coarseness and contrast, or a color histogram. The database specification file determines what display modes are available for a database (see "annotate.txt"). The "search metric" option menu will select the algorithm used to determine content similarity. Since every algorithm requires pre-analysis of the images, each database defines which algorithms it supports. See the programmer's manual for more information on display modes and search metrics. ------------------------------------------------------------------------------ 7 Controls The "Initialize" button will perform a "null" query, i.e. with no query filter or query images. This has the effect of returning the working set to the start-up state. The "Shuffle" button will randomly reorder the working set. The "Load query" and "Save query" buttons will load/save the working set. The format is one tag per line. The filename is specified via a file selection dialog. The "Text..." dialog allows the user to specify what text appears under each image. It is composed of a list of toggle buttons and text entries. The order in which information is displayed under the images is determined by the order of these widgets. The "Show annotations" text entry can be used to specify a list of annotations, if you enter one annotation per line. The "Refresh cache" button will reload the images from disk, applying any new display mode parameters which may have been set (see "Configure display mode..." below). The "Jump to page" text box can be used to set the displayed page of the working set. The "Jump to item" text box will set the displayed page to include the named image, and will select that image. The "Quit" button will terminate the program. ------------------------------------------------------------------------------ 8 Toggle windows Toggle windows pop up when the toggle button is pushed in, and pop down when it is pushed out. The "Symbols..." toggle window is a convenient means of entering query filters. The left-hand menu lists the available symbolic categories, and the right-hand menu lists the possible values for the selected category. Selecting a value and pressing return on the keyboard (or double-clicking on the value) will insert the appropriate expression into the query filter. The "Label..." toggle window starts up the "Power-assisted Annotation" tool. The dialog box displays a list of possible labels which can be attached to images in the database (these labels are specified in the database specification file; see "annotate.txt"). Double-clicking on one of these labels causes it to be assigned to all selected images. Highlighting a label and pressing "Positive" has the same effect. Only one label can be assigned to an image; placing a different label on a labeled image will replace the old label. The system will propagate labels to other images based on the similarity trees named in the database specification (see "annotate.txt"). The "Negative" button will instruct the system that the selected images should _not_ have the highlighted label; this is a means of correction. The "Save" button will save the labeling state in an external file ("examples") which will be read in future labeling sessions. The "Revert" button will revert the labeling state to the point where it was last saved. The "Configure display mode..." and "Configure search metric..." toggle windows provide a control box whose contents vary depending on the current display mode or search metric. They allow interactive alteration of the display/search parameters. You must press enter in text entry boxes for the change to be registered. Changes generally take effect on the next cache update (the "refresh cache" button is useful for this) or the next search. See the programmer's manual for information on the parameters of display modes and search metrics. ------------------------------------------------------------------------------ 9 Customization The directory for the FRAMER annotation database used by Photobook is given by the environment variable FRAMER_SEARCH_PATH. The directory for the image and coefficient data is given by the environment variable PHOTOBOOK_DATA_DIR. Many appearance characteristics can be changed by modifying the file "Photobook", which contains X application resources. On an 8-bit display without a 24-bit visual, color images will be displayed in grayscale. ------------------------------------------------------------------------------ tpminka@media.mit.edu 12/22/94