Streaming Live Audio and Video from a Nokia N800 to Second Life

22 April 2007


[video of the final product (18.0 MB)]

Introduction

This document describes how to stream live audio and video from a Nokia N800 Internet Tablet into the Second Life virtual world. The N800 has a built-in microphone, web camera, and 802.11 wireless connectivity and comes with a video VOIP client preinstalled. The Second Life client can view any QuickTime compatible audio and video stream. Combining the two is fairly straightforward and just a matter of determining which piece connects to what; no code needs to be written or even compiled.

The basic idea is to use GStreamer on the N800 to encode live audio and video and transmit the streams via RTP to a QuickTime Streaming Server, which then reflects it as an RTSP stream to any interested clients, such as a Second Life client.

Requirements

Streaming audio and video from a Nokia N800 into Second Life requires the following:
  • Nokia N800 Internet Tablet
  • Second Life account that owns land
  • computer running QuickTime Streaming Server (or equivalent)

Preparing the Nokia N800

The procedure described in this document was tested with an N800 running Internet Tablet OS 2007 Edition version 3.2007.10-7. Instructions for downloading and installing the OS are available at the Maemo website. In addition to the OS, several other software packages will also need to be installed on the N800. To make this easier, the N800 should be put into red pill mode. Once in red pill mode, use the N800's Application Manager to refresh the package list and install the following packages:
  • dropbear (lightweight SSH2 server and client)
  • osso-xterm (X terminal)
  • All GStreamer packages (only a couple of these are actually used here, but all were installed and could be useful for modifying and extending this work):
    • gst-plugins-farsight
    • gstreamer-tools
    • gstreamer0.10-3gp-osso
    • gstreamer0.10-adpcm-osso
    • gstreamer0.10-ffmpeg-osso
    • gstreamer0.10-gnomevfs
    • gstreamer0.10-hantro
    • gstreamer0.10-id3lib-osso
    • gstreamer0.10-mpeg1demux-osso
    • gstreamer0.10-mpegstream-osso
    • gstreamer0.10-plugins-base
    • gstreamer0.10-plugins-base-extra
    • gstreamer0.10-plugins-good
    • gstreamer0.10-plugins-good-dbg
    • gstreamer0.10-plugins-good-extra
    • gstreamer0.10-plugins-good-extra-dbg
    • gstreamer0.10-tools
    • gstreamer0.10-wma

Preparing the Streaming Server

The procedure described in this document was tested with the Apple QuickTime Streaming Server (QTSS) running on a Mac OS X Server machine. Similar results are most likely possible with the open source Darwin Streaming Server (DSS). The only deviation from the standard QTSS set up was the placement of a Session Description Protocol (SDP) file in the /Library/QuickTimeStreaming/Movies/ directory of the server. The SDP file contains all the information needed by the client to properly find and understand the audio and video streams, such as type of encoding, size of each frame, and addresses and port numbers of the streams. This file could be crafted by hand (it is a plain text file), but it is probably easier to generate it with a program such as QuickTime Broadcaster or mp4live. QuickTime Broadcaster was used to generate the following SDP file (File->Export->SDP from within Broadcaster): 

v=0
o=- 37 614155991 IN IP4 127.0.0.0
s=QuickTime
t=0 0
a=range:npt=now-
m=audio 5432 RTP/AVP 0
c=IN IP4 XX.XX.X.XXX
b=AS:63
m=video 5434 RTP/AVP 96
c=IN IP4 XX.XX.X.XXX
a=rtpmap:96 H263-2000/90000
a=fmtp:96  
a=cliprect:0,0,144,176
a=framesize:96 176-144

This SDP file can be used once the XX.XX.X.XXX address listed in the file (in two places) is replaced with the address at which a streaming server is located. Changing any part of the streams coming from the N800 will most likely require changing the SDP file.

Preparing Second Life

Only the owner of a parcel of land in Second Life can determine if media can be streamed to the land and which URLs they are streamed from. Both the built-in (press F1 while using the Second Life client) and online help have a section on how to enable streaming media. In short, the World->About Land->Media menu offers options for audio and video streams. Only the video stream is needed, as it includes it audio with the video. The URL should be of the form rtsp://XX.XX.X.XXX/N800.sdp, where XX.XX.X.XXX is the same address as listed in the SDP file on the streaming server. The video stream will replace a chosen texture (image) on the parcel of land in question. In addition to configuring the land to stream media, it is also necessary to configure the client to stream media by going to the Audio/Video tab of Preferences (CTRL-P). The Second Life client can display streaming video only if QuickTime is installed on the same computer.

Putting It All Together

Once the N800, QTSS, Second Life land, and Second Life client have been configured as above, the only thing left to do is start the audio and video streaming from the N800 by running the following command from the terminal (either via SSH or xterm):

gst-launch-0.10 gconfv4l2src ! video/x-raw-yuv,width=176,height=144,framerate=\(fraction\)15/1 ! hantro4200enc stream-type=1 profile-and-level=1001 ! video/x-h263,framerate=\(fraction\)15/1 ! rtph263ppay mtu=1438 ! udpsink host=18.85.9.200 port=5434 dsppcmsrc ! queue ! audio/x-raw-int,channels=1,rate=8000 ! mulawenc ! rtppcmupay mtu=1438 ! udpsink host=18.85.9.200 port=5432

This command creates and starts playing a GStreamer pipeline that encodes the raw audio and video streams, stuffs them into RTP packets, and sends them to the streaming server. Once the GStreamer pipeline has been started, the N800's camera can be opened without the default camera application starting up. In any case, be sure the default camera application is not running before the GStreamer pipeline is started. From within Second Life, be sure to hit the play button at the bottom of the screen to start the stream playing in the parcel of land. The stream will only play if the texture it is meant to replace is currently being used somewhere in the parcel. A regular QuickTime client or any other media player capable of playing streaming media, such as mplayer, can be used to view the stream independently of Second Life, although with mplayer the -fps 15 option was needed to properly view the stream. The total time delay between real time and the audio and video streams depends on the network connections between all the points of interest, from the N800 to the streaming server to the Second Life client (the Second Life servers don't have anything to do with the streaming of the media; it all happens through the client). The shortest delay yet observed is approximately five seconds. With the screen turned off and the audio and video streaming, the battery lasts more than three hours.

Acknowledgements

Thanks to the folks on the FreeNode #maemo and #gstreamer channels for their generous advice and abundant patience, to Jun Ki Lee for providing access to his streaming server and advice about SDP files, and to Peter Boda at Nokia for getting me started with the N800 in the first place.