The IsisWorld Simulator for AI research

The IsisWorld simulator is available to researchers for building and evaluating problem solving and language learning systems with everyday problems that take place in a kitchen. Some documentation and arguments about theoretical underpinnings aimed toward the metareasoning community are available in this paper:

IsisWorld: An open source commonsense simulator for AI researchers. Dustin Smith and Bo Morgan. Submitted to AAAI–10 Workshop on Metacognition.
IsisWorld AAAI presentation about using simulators to ground abstract AI problems

Isis World Simulator v0.4

About IsisWorld

IsisWorld is a highly-customizable open source kitchen simulator written in the Python language. IsisWorld was created to as evaluation platform for evaluating and observing the behavior of commonsense reasoning systems. It has a number of unique features:

Source code and binaries are freely available for download and distribution
All source code is immediately available and binaries for most common platforms are available for download. The Panda3D libraries and the ODE framework are covered by the BSD 3-clause license. The rest of the project is covered by the GPLv3.
Task evaluation framework is built-in
It’s easy to write scenario templates and create custom evaluation scenarios. A scenario defines the default 3D environment and contains tasks with training settings and goal functions.
Environments are generated from a specification file according to commonsense defaults
Environment descriptions are procedurally generated: the sizes and orientation of the kitchen are determined each time the environment is generated as an effort to prevent AI systems from “overfitting” the task.
Client-server architecture
The agents in the simulator can be controlled by a client. Other than XML-RPC, no additional libraries are required.
Rigid-body physics simulation:
Objects’ position and orientation can be controlled by the Open Dynamics Engine: an object location is a function of its mass, and linear and angular forces (including gravity). Custom callbacks and triggers can be defined.
Default objects with commonsense behaviors
We are continuing to add new objects, actions and events to the environment.

Change History

What’s new in the Iris World simulator?

0.5b 2011–01–04
- specifying scale ranges for some of the common models sizes, the same way the size of the kitchen is chosen from a random range. in isis_agent:pick_object_up_with, special handling for objects on surfaces and objects in containers: Items being removed from containers first have their parent checked to see if is_open is True. If it is not, then the pickup action fails. Objects like the toaster, have is_open as a hidden property, that is turned to False iff the toaster is_on.
- properties of IsisObjects can be defined with IsisAttribute with consistency checks.
- scenario files allow defaults to be specified as keyword arguments. k = kitchen(width=10, height=7)
- tasks have goal states which check for whether the goal state is met
- migrated the kitchen.isis into an “scene initialization” section of the “scenario/” files.
- GUI for loading tasks, which is the default screen when the simulator loads.
- agent vision can be exported through XMLRPC or saved as a screenshot
- fixed broken ‘say’ command
- loads objects with default properties in objects.py
- object loader handles containment (“in”), surface contact (“on”) and part of relationships
- kinematic and dynamic physical modeling using the Open Dynamics Engine
- more generic Boxman models
0.3.1 2010–04–09 -fixed bug accepting text
0.3 2010–04–06
- Added body orientation (h,p,r) to proprioceptive position frame.
- Changed name from “Society of Mind Simulator” to Isis Simulator.
0.2 2010–03–17
- Initial public release.

Developing an agent to use the simulator

Perceptual and motor interfaces

By default, the simulator is paused, and no physical simulation can take place. The simulator can be unpaused and stepped, advanced incrementally, or run continuously. The character in the simulator, “Ralph”, can be controlled directly in the simulator using key commands or separately through a programmed agent (a client). The environment and agent are temporally decoupled. In this section, we review the agent-environment interface in terms of a perceptual frame and primitive actions.

Perceptual Frame

The agent can sense the environment, which returns a perceptual frame containing:

Object Information: All names of the items in the character’s field of vision; each item’s center’s relative x and y coordinates on the frustum: the 2D plane representing the projected 3D region of the character’s field of vision ¹; the item’s area of the entire field of vision; the item’s distance; and the item’s global orientation.
Proprioceptive Information: The character’s global position (x,y,z) and orientation (h,p,r) in the environment as well as the relative positions and orientations of the character’s head and limbs with respect to its body; and the names of the items within each of the character’s hands (if any).
Linguistic Information:: A buffer representing all of the text entered by the user since the last time the sense command was executed.

Primitive actions

The character has a set of primitive actions that can be taken:

turn_left-start and turn_left-stop: Starts and stops rotating the character’s body along the vertical Z axis: subtracts from the heading.
turn_right-start and turn_right-stop: Starts and stops rotating the character’s body along the vertical Z axis: adds to the heading.
move_forward-start and move_forward-stop: Starts and stops moving the character’s location in the environment forward (in the direction the body faces) a fixed amount, and changes the model to an animated walking character.
move_backward-start and move_backward-stop: Starts and stops moving the character’s location in the environment backward a fixed amount, and changes the model to an animated walking character.
look_left-start and look_left-stop: Starts and stops rotating the character’s neck, with respect to the body, along the horizontal X axis: adds to the neck’s pitch. (look- actions are bounded to prevent uncanny exorcist-style head rotations.)
look_right-start and look_right-stop: Starts and stops rotating the character’s neck, with respect to the body, along the horizontal X axis: subtracts from the neck’s pitch.
look_up-start and look_up-stop: Starts and stops rotating the character’s neck, with respect to the body, along the vertical Z axis: adds to the neck’s heading.
look_down-start and look_down-stop: Starts and stops rotating the character’s neck, with respect to the body, along the vertical Z axis: subtracts from the neck’s heading.
say(statement): Presents the a statement string above the character’s head, which gradually disappears over the course of two seconds as the simulator steps forward in time.
pick_up_with_right_hand(object): Attaches the object to the character’s right hand, so long as the object is graspable and the actor has a free right hand.
pick_up_with_left_hand(object): Attaches the object to the character’s left hand, so long as the object is graspable and the actor has a free left hand.
drop_from_right_hand: Releases any object from the character’s right hand.
drop_from_left_hand: Releases any object from the character’s left hand.

Some actions like turn_left are interval actions and have separate -start and -stop commands. Once started, they continue until they are stopped. Other actions are atomic and instantaneous, like drop_from_right_hand.

Building an agent

A simple implementation of an agent in Python:

import xmlrpclib as xml
# connect to environment via XML-RPC
e = xml.ServerProxy('http://localhost:8001')
# sense world
perceptions = e.do('sense')
# do something
e.do('say',{'statement':"Hello world!"})
# simulator is paused by default
# run for X=0.02 seconds
e.do('step_simulation',{'seconds':0.02})

Problems and solutions for Developers

Here is a list of problems

ImportError: No module named direct.directbase.DirectStart

This problem is probably that you are using a different version of Python than what Panda3D has configured itself for.

To determine which version of Python you are using, type:

python -V

Panda3D configures itself for Python 2.5. Somewhere in that directory is a “Panda3D.pth” file.

cat /path/to/old/python/2.5/site-packages/Panda3D.pth

Which should output “/installed/path/Panda3D/lib”, indicating the path to the required Panda3d libraries. But you now have to copy Panda3D.pth to the site-packages directory of default version of Python you are using. To see all paths to installed python (recognized by your $PATH variable), type:

which python

The first path output by this command, e.g., if it outputs

/another/path/2.6/bin/python

is the python executed by default when you type “python”. That’s the one who needs to have the Panda3D.pth file. Copy the file to the Python’s site-packages folder, e.g.,

sudo cp /path/to/old/python/2.5/site-packages/Panda3D.pth /another/path/2.6/lib/python2.6/site-packages/

Notice the “2.6/bin/python” from the output of which changed to “2.6/lib/python2.6/site-packages/”

Ubuntu install help

System: Ubuntu 9.04 or 9.10, 32 or 64 bit.

Problems:

Missing 3rd party dependencies.
Running the sample programs gives no errors but displays a grey window.

Solutions:

A list of dependencies to install:
- http://www.panda3d.org/phpbb2/viewtopic.php?p=52984&sid=52e0921545f72dcbf1fe9d4d0c51cdc7
Change /etc/Config.prc:
- http://www.panda3d.org/phpbb2/viewtopic.php?t=8071

Thanks to Sarina Canelake for this information.

Missing CG Framework

Download and install Nvidia’s Cg framework.

Running on Athena

To develop the simulator on Athena, you should compile Panda3D from source.

(0,0) denotes the center of the field of vision, where x and y are values in (–1,1) ↩

Setting up and Installing IsisWorld