::the work

We are witnessing an emerging act of storing everything happened to us in digital format, and sharing it online with friends or even strangers. The behavior of blogging in text, image, sound, and video formats, storing them on some central machines of the corporates, and making them taggable and searchable by the public, is what I am interested, and inspired me to produce this work, "your memory, connected."

We build an artbot that is able to take free form text input from users and generates a list of semantically related "concepts" that are pertain to the original text. This list of concepts are then used to retrieve related "photos" from flickr.com, the largest online photo sharing site in the english world. We see flickr as a site that collects a huge amount of personal memories in the form of photos contributed by the people around the world. "your memory, connected. I" is a system that collages the images retrieved from flickr into one single image that shows an aggregated view of the personal memories triggered by the input text. Imagine it as a system that allows you to shout to millions of people and to browse through the pictorial memories triggered by whay you said. Below is the collage generated by the system with the text input to be the word "belief:"

This work has been chosen and exhibited this year in SIGGRAPH Art Gallery 2006.

This work just got featured in the 34th issue of the Public Journal on Art|Culture|Ideas. The image chosen to be the cover image is generated by our system with text input to be the Sonnet 18 "Shall I compare thee" by William Shakespeare.

Invited by the College of Fine Arts, Tsinghua University, Beijing, China, we designed and exhibited an interactive installation version of the system. It consists of four PCs with eight LCD displays and 20 speakers. A viewer can wake up the system by typing text into the system. It will read the input and retrieve conceptually related images from flickr.com, display them on the LCDs, and read out (by voice syntheall the keywords that are tagged with retrieved images. We want to create an installation that acts as an agent to bring back personal memories online based on viewers' input.

Also exhibited is a three-monitor system that cycles through the images from flickr. The three monitors represent the year 2004, 2005, and 2006 respectively. Each row on the display is the collection of images that are taken at exactly the same particular "second" in time, while most likely are taken at different places. We would like to introduce to the viewers the nature of digital documentation: each of them has an presice time stamp. By putting them together we get a collective view of the world space of that particular second.

::technical details

Central to our system is the artbot, an engine that reads natural language (English), analyzes the sentences, and determines related concepts in terms of semantics and emotional attributes. The textual reasoning algorithm in this engine is based on two semantic networks, WordNet and ConceptNet. WordNet is an electronic lexicon developed by the Cognitive Science Laboratory at Princeton University. It is a fruitful resource of words’ definitions as well as their synonyms, hypernyms, hyponyms, etc. (E.g., Happiness is an emotional state). ConceptNet, on the other hand, provides common sense about everything happening in our lives, (e.g., People smile when they are happy). It was developed by Hugo Liu and his colleagues at the MIT Media Lab. Using ConceptNet, Liu also developed an affect sensing model that detects the emotional states within arbitrary text. We build the engine with these technologies. It is a general-purpose engine that is able to parse free text and output related concepts. Also, we will be able to determine how related two photos are, based on their descriptions and tagged keywords. This is how we determine which photos to download for text inputs.

::credits

collaboration with edward shen, pranav mistry, and di Z Ye.