A thesaurus is a book containing synonyms in a given language; it provides similarity links when trying to retrieve articles or stories about a particular topic. A ``visual thesaurus'' works with pictures, not words. It aids in recognizing visually similar events, ``visual synonyms,'' including both spatial and motion similarity. This paper describes a method for building such a tool, and recent research results in the MIT Media Lab which contribute toward this goal. The heart of the method is a learning system which gathers information by interacting with a user of a database. The learning system is also capable of incorporating audio and other perceptual information, ultimately constructing a representation of common {\it sense} knowledge.