Cati Vaucelle
bio
blog
portfolio
publications
♣ photos
SOFTWARE - by Cati Vaucelle
Picture This! allows movie making assembly at the extension of natural play. The software deals with the filtering algorithm of gestures I created as well as the gesture control of automatic video making assembly.
OVERALL DESIGN
FILTERING ALGORITHM
I designed two bag accessories for two toys. Data is transmitted in real time to a microcontroller. Each bag attached to the toy contains a microcontroller, a piezo vibration sensor, a printed circuit board, a funnel board and XBee wireless module per toy with a tailored video camera.
My software retrieves the output data of the microcontroller continuously so that each toy carrying an augmented bag can communicate with my program via specific gestures filtered by the software.
As input in Picture This I have the gestures of children playing with the dolls and their voice. A camera, micropone, piezo sensor mounted on a printed circuit board communicate to my software via a microcontroller.
My software conduct automatic assembly based on the gesture analysis of the children playing with toys and output different video controls: I’ve got playback preview, record, stop record along with the audio, so the kids can setup their final movie during the play. These mode are controlled by the children as they play.
I developed a filtering algorithm for gesture recognition through which angles of motions are detected and interpreted. The motions I chose to identify support natural character play movements, such as jumping and shaking, with the addition of video control functions to these character play movements.
For instance, if the child wants to playback the movie she just created, the two dolls have to be moved in synchrony, in essence, jumping horizontally together.
4 gesture detection modes: Preview, Record, Stop Record, Playback.
Example, Playback: The two dolls have to be shaken horizontally together. The software had concatenated the video segments captured. It now plays back the entire movie on display.
The sequences of video clips are automatically added to one another and the blurry frames from the gesture commands are removed. The final movie is played back on the display for the child to watch her final video composition.
The motions that are detected by the system are anthropomorphized. The dolls need to jump in synchrony at completion and shake for attention, as if the doll wants to say: “film me, film me!” To master the interaction with Picture This!, the child needs to alternate between projecting herself onto her toys and being the master-mind of the scene.
Software architecture in Picture This!
Published as a full paper in: Vaucelle, C., & Ishii, H. (2008). Picture This! Film assembly using toy gestures. Proc. Ubicomp ’08, pp. 350–360. ACM Press.
To detect gestures, I use a piezo film with accompanying electronics (see the hardware section of Picture This web site). The advantage is to be low cost, easy to use, offering a nice signal. However it is sensitive to temprerature variations.
Because the camera is on the doll, and because the child shakes to activate the preview or the recording, I could have just taken advantages of the camera as input to detext when it is moving.
I could have done simple optical flow calculation, detecting the vector of a moving object in a video sequence.
However there is a problem with the optical flow method: the child can easily occlude the camera by putting her fingers in front of it, or simply obstruct its line of sight without noticing it. Because the child shakes the doll that will be in the video, the other doll being the cameraman, the child does not have a direct feedback for motion detection occlusion. So I decided to go with a hardware method for prototyping the Picture This system.