Multimedia Retrieval Through Spatio-Temporal Activity Maps

Spatio-temporal activity maps serve as an efficient and intuitive graphical user interface for multimedia retrieval, particularly when the media streams are derived from multiple sensors observing a physical environment. This type of retrieval is important in many surveillance applications. This paper formulates the media retrieval problem in this context, and develops an architecture for interactive media retrieval by combining spatio- temporal "activity maps" with domain specific event information. Activity maps are derived from the trajectories of motion of objects in the environment. An example system is presented for map based indexing and retrieval of video in the sport of tennis. Using this example, it is demonstrated that the activity map based scheme significantly helps the user in a) discovering which portion of the data is interesting to them, and b) non-linearly retrieving the corresponding media streams.