4.889 Intelligent Interface Software Design Workshop

Project 4: An Intelligent Video Editor

Objective

The objective of this project is to gain some experience with intelligent video editing systems, interactive video presentation interfaces, rule-based systems, narrative structures and representation of video content. You will be making extensions to Suture-Self Video, a system developed by Abbe Don and Warren Sack.

Suture-Self Video

The system consists of two parts: the VideoWall, an interface to a knowledge base of video clips and Montage Rules , code for automatically putting together video sequences.

There are about 45 clips, digitized with Quicktime, that deal with the Iran-Contra hearings. The clips present a variety of points of view, including the Regan Administration, expert witnesses, and others. Each clip is annotated with information describing the speaker, the topic, the dialog, and other information. Rules can be written in a Prolog-like language to compose sequences of clips. You will write new rules and annotations for the existing material, and make extensions to the interface.

VideoWall

The VideoWall is a spreadsheet-like grid that contains rows and columns which present a collection of video clips, charactierized according to different relations. You may pick relations for the rows and columns from the VideoWall menu. The Create operation causes a new VideoWall to be computed with the currently selected row and column relations. The new VideoWall automatically sizes the cells according to the number of clips retrieved for that relation by the VideoWall rules.

Each clip may be played by clicking on it. Double-click starts the clip, single-click stops.

Knowledge bases

It is worth noting that VideoWall is unlike a spreadsheet or database in that the relations need not be indexed in advance explicitly in the data. The clips that are retrieved to fill each cell are compted by Prolog-like rules at the time the VideoWall is requested, and can be inferred from existing annotations. New relations can be added at any time by writing new rules.

Montage rule menu

The Montage Rule menu contains a list of rules for assembling clips into a sequence. You may pick an initial clip and a rule and the system can apply that rule to create a sequence starting with that clip.

The results of the rule application are displayed in the Montage Rule Bar, as a series of movies, each representing a single clip. A Quicktime viewer centered at the top allows the sequence to be played in its entirety.

Montage examples

The system stores examples of applications of the montage rules. Initially, there are no stored examples, but as you create examples, you can browse through the set of existing examples.

Understand how Suture-Self works

To load the system first type (in-package:cl-user)

Then, load the file load-suture-self.lisp

Ignore the warnings from the compiler.

When it's done loading, select Open from the Suture-Self-Video menu.

First, play with the interface to get a feel for how it works. Create some different VideoWalls by playing with the VideoWall menu. Choose some different relations for the rows and columns. For example, choose group-of-speaker from the Rows menu item and topic-of-dialog from the Columns menu item. Then, select the Create menu item.

Note how the layout changes according to what is retrieved. Play the clips in the cells. Can you understand the categorizations of the clips?

[You may have to know something about the Iran-Contra story to understand the clips out of context. If you are unfamiliar with the background of the Iran-Contra story, we can supply you with a short reading.]

Play with the Montage Rule menu. Shift-click on a clip from the VideoWall. Now, select a rule (e.g., point-counter-point) from the Montage Rule menu. For each rule, try it with a variety of initial clips. Can you understand how the sequences are being put together?

Annotating clips

Annotations for the clips are stored in the file Annotations.Lisp. The defclip macro creates an annotation for each clip. Look at some clips, then find their annotations to see what information the system stores about them.

Though the syntax is similar to CLOS, each of the "slots" expands into a Prolog assertion. The slot name is a Prolog predicate.

If you are not familiar with Prolog, or the idea of Logic Programming, you may want to read the chapter on Prolog in Introduction to Artificial Intelligence Programming by Peter Norvig. We can also suggest other references for Prolog.

Try some examples of retrieving information from the annotations.

From the Listener type:

(?- (speaker ?Clip-Name (Oliver North)))

retrieves the names of all clips in which North is speaking.

Type semi-colon repeatedly to see each of the names of the clips.

Type period when you are done.

Taxonomy

The rules that describe the hierarchy of concepts used in the annotations resides in the file Taxonomy.Lisp. This describes, for example, which people are members of groups that have a defined role in the affair, or share a common outlook. Relate the taxonomic rules to the information in the annotations of the clips.

Practice some retrievals that exercise the taxonomic rules.

(?- (and

(speaker ?Clip-Name ?Speaker)

(group-member ?Speaker

former-administration-insiders)))

finds all clips where former administrations insiders are speaking.

Type semi-colon repeatedly to see each of the names of the clips.

Type period when you are done.

VideoWall Rules

To see how the VideoWall works, we will trace a VideoWall rule.

Create the VideoWall that has group-of-speaker as the row relation, and topic-of-dialog as the column relation.

Trace the predicate

(trace topic-of-dialog/2)

[The /2 refers to the fact that this predicate takes 2 arguments, and must be mentioned when you turn on the trace].

Now do the query

(?-

(and (topic-of-dialog ?Clip Secrecy-Issues)

(group-of-speaker ?Clip Administration)))

to see how the VideoWall finds the clips to fill in the cell indexed by Secrecy-Issues and Administration.

Try tracing group-of-speaker/2 and issue the same query. Try it with other cells. The VideoWall rules are in the file vw-rules.lisp.

Montage Rules

The Montage Rules are stored in the file montage-rules.lisp. Look at the rule point-counter-point. Understand how it works, and trace an example as we did with VideoWall to watch how it works. Try running the montage rule point-counter-point by posing the following query in the Listener:

(?movie-

(point-counter-point sheehan1

?SequenceOfClips))

[The ?movie- form instructs the Prolog compiler to return a playable QuickTime movie. In contrast, the ?- form will simply return a list of variable bindings, rather than a QuickTime movie.]

Task: Extending Suture-Self

Write a new Montage Rule for Suture-Self. Trace an example as we did above. Make three or four example sequences with your new rule. Do the sequences produced fit together as well as you intended? Is the rule intelligible from watching the sequences?

Rules have the following form:

(<- head condition1 condition2 ... conditionN)

where head and each of the conditions is a list.

Rules can be evaluated into the Listener like any other piece of Lisp code. However, if you change the definition of a rule you must remember to evaluate the following before you evaluate the new definition of the rule:

(clear-predicate 'name-of-rule)

The function clear-predicate deletes the old version of the rule from the database.

Write some new annotations for the existing clips. You can do this by adding Prolog assertions of the following form to the database:

(<- (some-new-property some-clip-name

some-property))

Add a slot to the existing annotations that captures some other aspect of the material. To do this you will need to modify the defclip macro definition and the file annotations.lisp.

Write some new taxonomy rules, and perhaps a new montage rule, that take advantage of the new annotations and make some sequences with it.

Extend the interface

Critique the Suture-self interface itself. Are there other ways in which you could graphically present the video material or annotations? Other ways in which a user might want to interact with it? Design a significant extension to the Suture-Self interface and implement it. Most of the code for the current interface is in the file videowall.lisp.

For example, you could imagine a hierarchical tree representation rather than a grid representation for presenting the collection of clips. You could use the object-oriented tree grapher in the Lisp examples folder to display a tree of video sequences. Is there some way in which you could graphically represent the rules or annotations themselves?