by Push Singh, MIT Media Lab

April 1, 1998 (Revised December 7, 1999)

Introduction

The following is a list of many of the elements one needs to build a system capable of commonsense thinking.  No AI system contains all of them, though many contain more than a few of them.  I am trying to put these elements together into a system in order to control a simulated robot.

I have implemented many of them using standard AI building blocks (recognize, compare, retrieve, etc.)  I am working on a multiagent cognitive architecture to build them in, based on ideas from Marvin Minsky's Society of Mind theory, which is efficient, cooperative, and robust at responding to failures, but I will leave that discussion for another time.  Nevertheless, I see many of the following elements as easily programmed in any architecture.  This list of elements taken together implies a larger, more powerful architecture than the architecture that underlies it, so it is not clear how much it matters what the underlying architecture is.

One could probably go through the literature and find several hundred more elements.  This list came from working on the problem of building a robot capable of movement and manipulation, and learning what techniques could be applied to solving the many problems that came up, and figuring out how to organize them into a single system.

Despite the large size of this list, I think building this architecture is only one percent of the big problem of building a system capable of commonsense thinking.  There's a lot more you need to know, both in terms of how to think and what to think.  And in the end, everyone has their own "thinking style", which no single, simple architecture will be able to encompass.

Basic Vocabulary of Objects for Artificial Intelligence

This system is "object oriented" in the sense that there are a standard set of objects which the agents use and manipulate.  Here are some of the objects needed:
 

Mental Questions that Direct Common Sense Thinking

One way to control the direction of mental activity is by focusing on some set of questions.  Of course, these questions do not need to be expressed in english, and may be expressed as queries in various internal representations.  These questions initiate and prioritize inference processes that help answer them.  Here are some of the questions a robot should always be asking itself:

• What is going to happen next?
• What would explain this?
• What is the best thing for me to do now?
• What objects are around me?
• What is happening around me?
• Have I seen that object before?
• What can I learn from this event?
• What can I learn from this failure?
• How long will it take to perform this action?
• What sorts of things might go wrong while performing this action?
• What sorts of problems might occur, and how can I prepare for them?
• Am I spending too much time on this problem?

Mental Agents for Common Sense Thinking

Some of these agents are simple IF-THEN-DO rules.
Some are more complex IF-THEN-DECIDE/SEARCH rules.
Some are really complex IF-THEN-FIGURE-OUT rules!

Tracking, Recognizing and Describing

• given a partial state, try to find a frame that describes it (if you see a fridge and stove, think "kitchen")
• given a situation, find a more abstract representation of it
• given a description, find something additional to say about it
• given an object for which a set of categories match, look for a feature that discriminates between them
• binding problem: if an object might be the same as one seen before, impose temporal-behavioral constraints to verify

• given a goal that cannot seem to be achieved, give up on the goal
• given a goal that cannot seem to be achieved, censor the goal
• given a pair of goals, look for a causal chain that results in a conflict between them
• given a pair of goals, prioritize them with respect to higher priority supergoals
• given a goal, propose a set of subgoals to achieve it
• given a goal, find the difference between it and the situation in order to characterize the problem
• monitor for mania: believing you can achieve any goal
• monitor for scattered feeling: too many goals are active at once
• monitor for impatience: wanting too many things right away
• monitor for depression: excessively critical of your actions and goals
• checkpoint our mental state whenever something important happens

• given a problem, reset your mind state to an earlier state in which a similar problem was solved (k-lines)
• given a problem, retrieve from memory an abstract plan to solve it
• given a problem, try to chain a few methods to solve it
• given a problem, look for different ways to solve that problem
• given a plan, adapt it to the present situation
• given a plan, adapt it to the present goal context
• given a plan, check if you have tried it recently and it failed, and if so suppress it
• given a primary goal and secondary goals, retrieve a plan that achieves primary but does not undo secondary
• given a situation, look for methods you can execute
• given a goal, look for methods that have been known to achieve it
• given a method, look for ways to adapt it
• given a plan, criticize the plan by finding another reason why it will fail
• given a plan, justify the plan by finding another reason why it will work
• given a script, generate a version of that script that monitors important conditions
• given a plan to execute, assign monitors to its preconditions
• given a method under consideration, check if the method is undoable
• given a plan, estimate how long it will take
• given a plan, predict what resources it will take and what goals it will undo
• given a pair of plans that achieve the same goal, find some way to combine them into a better plan
• given a plan that is strongly criticized, censor it
• given a plan being constructed, add constraints one at a time
• given a plan being constructed, prefer plans that are undoable
• given that a plan cannot be adapted, raise an impasse
• given that no plan can be retrieved, raise an impasse
• given a set of subplans, combine them into a larger plan in which their mutual interactions are resolved
• given a problem, try to find a simple problem that resembles it and try to solve the simple problem

• given two plans, compare them to see which is better

• given a plan, look for a plan that is better than it in some regard, and try to improve it in that regard
• given a plan, improve its reliability and speed and minimize its use of resources
• given a plan, look for simple useful variations

• given a set of plans, run them at the same time and monitor for interactions

• given a plan, try the plan in simulation
• given a plan, consider the cost of failure of the plan
• given a plan, check if its preconditions are satisfied

• given an applied method, note any deviations from expected effects
• given a successful plan, try that plan in other situations and on other methods, to learn more about it
• given a successful plan, save the plan to memory
• given a successful plan, index the plan

• given a plan, execute it
• given an executing plan, monitor if the actions do as expected
• given an executing plan, check if its actions are taking too long
• given an executing plan, suppress it if it is predicted it will achieve a dangerous state

• given a failed plan, try the plan in simulation at finer temporal grain
• given a failed plan, try the plan in simulation while simulated more objects
• given a failed plan, try the plan in simulation while simulating causes
• given a failed plan, figure out which subaction didn't achieve its subgoal
• given a failed plan, find or generate an alternate plan
• given a failed plan, try another plan
• given a failed plan, mutate the plan slightly in a way that worked before, and try again
• given a failed plan that has been recently modified, regress to an earlier version of that plan
• given a failed method, check if its preconditions are still satisfied
• given a failed method, change a parameter of a method, like doing it harder or more smoothly
• given a failed method, do some experiments to test the method

• given a situation, predict the next state by memory (using a transframe)
• given a situation, predict the next state by dynamic simulation (like QSIM)
• given a simulation, control its level of detail (which and how many objects and properties are simulated)

• given an expectation failure, raise an impasse
• given an object that behaves unexpectedly, try to explain it by seeing it in terms of other frame

• given an object and a frame, look for a way to see an object in terms of the frame
• given an object, look for unusual features
• given a transframe, look for ways to see the current situation in terms of it
• given an observed event, try to break it down into achievable subevents
• given an object in the world, check if you saw it before
• given a situation, recognize the general context (inside, outside, etc.)

• given an event, look for its cause
• given a sequence of events, judge whether things are getting better or worse
• given an unusual conclusion, verify it by trying to prove it another way
• given a pair of situations, before and after, find a chain of events that links them
• given a failure, look for another frame in terms of which to describe the situation
• given an event, verify that it meets our expectations

• given an assertion, assert its opposite and monitor for a contradiction
• given a theory (a set of constraints), find an example from memory or in the environment
• given a theory, find a counterexample from memory or in the environment
• given a pair of theories, find analogies between them
• given a pair of theories and a goal, find analogies between them with respect to the goal
• given a property P, and P implies Q, then assume Q
• given a property Q, and P implies Q, then check if P is true
• given a pair of properties P and Q, if there is a constraint between those properties, verify the constraint
• if there is a constraint failure between properties, verify the properties involved
• given a failure of reasoning, describe the reasoning using your language machinery
• given a failure of reasoning, reformulate by modifying your theory

• given a failure, look for a failed assumption
• given a failure, look for a missing assumption
• given an explanation, look for a weakness
• given an X, criticize it

• given a successful plan, save the trace of how it was derived
• given a set of objects, look for a way to uniframe them
• given a set of events, build a theory that explains the behavior
• given an object, find a new frame that matches it
• given a new object, construct a frame for it
• given a new X, construct a frame for it
• given a set of descriptions, find a way to unify them as variations of a single description
• given an observed method, figure out how to copy it
• given a failure, construct a critic that complains when it sees the potential to fail that way again
• given a pair of correlated events, look for a causal dependency between them, and if found build transframe
• given a plan of action or reasoning, generalize it by variablizing its elements
• given a successful plan, take a snapshot of the mental state in which it was discovered (make a k-line)

• given no more resource, replenish that resource (if out of fuel, go get more)
• given no more resource, find another source of that resource (if fuel tank is empty, find another tank)
• given a resource, see how much of that resource is left (check gauge in fuel tank)
• given a function, find another way to achieve that function (you can also walk on your hands!)

• given an anti-goal, build pessimists by thinking of ways we might get to that anti-goal
• given a goal, build optimists by thinking of ways we might get to that goal
• given a plan, look for subplans that can be extracted and usable in their own right, and index them
• given a goal, make a list of the top 10 ways we might fail at that goal

• given a method, index it according to its effects
• given a set of operators, find a reformulation of those operators that "orthogonalizes" them
• given a pair of representations, look for an analogy
• given a transframe, look for scripts that match it
• given a pair of frames, find the differences between them
• given a set of plans that achieve the same goal, index them so we can quickly jump between them
• given a method, look for plans that can undo its effects

• given a method, check if it interferes with the goals of the other agents
• given a method, check if it might help other agents
• if an agent notices another agent doing something that might help it, help that other agent do it
• given a manager, look for ways in which its worker agents might interfere with one another
• given a pair of interacting agents, debug the interaction one at a time
• if a manager notices a set of agents working well together, consider them a team

• given a person, guess at their goals from their actions
• given a person model, simulate what they would do in some situation S
• given a person model, simulate what they would think in some situation S

• given a sentence, generate some possible syntactic parses of it
• given a sentence, generate some possible interpretations of it in other representations
• given a word, check if you know what it means
• given a sentence, record how the words in that sentence are used
• given a sentence, determine its type (examples: question, asserting a truth, rhetorical statement)
• given a desired mental state of a listener, generate a sentence that achieves that mental state (reduplication)

• given a visual scene, track the objects in it
• given one view of an object, retrieve other views of that object
• given a view of an object, compute its contour

• given a set of options, consider the consequences of choosing them

• given lack of progress or cycling, declare an impasse
• given a set of constraints between variables, look for a set of variable assignments that satisfy it
• given a causal dependency, monitor that if the independent variable changes, then the dependent one changes
• given bidirectional constraint between variables, monitor that if one changes so does the other one

Mental Representations for Common Sense Thinking

These agents operate on data structures that represent aspects of the world and of the mind of the robot itself.  Here are some examples of the kinds of concepts robots will need.  Each of these should be represented in multiple ways!

• time : before, after, duration, sensed-at-time
• temporal : repeating-event, sudden-event
• methods : expensive, cheap, one-time, undoable
• events : periodic, unspecified duration, momentary
• shape : round, square, changing, has-holes
• bodily : hand position, head position, orientation
• spatial : boundaries, paths, holes
• physical : solid, flexible, surfaces, connections
• perceptual : occlusion, clarity, depth
• geometric : above, near, inside, outside
• goals : closer-to-goal, interacting-goals
• resources : limbs-as-resource, time-as-resource
• balance : center-of-mass, leaning-to-the-left
• movement : moving-quickly, moving-sporadically
• energy : expensive, cheap
• difficulty : hard, easy