The object of this experiment is to make some observations on how people collaborate on a shared task, and how computers could provide active assistance to people working on a task, either alone or together.
The methodology is known a "Wizard of Oz" experiment. In this experiment, one person plays the role of the "user", who is working on a particular task. Another person plays the role of the computer, providing assistance to the user while doing the task. The object is to understand what kind of assistance the user requires, what sort of communication between the user and the computer is plausible during the task, and what knowledge and expertise it will take for the computer to provide the assistance. In the traditional "Wizard" experiments, the assistant is supposed to restrict him or herself to only providing the kind of assistance it would be plausible to have a computer do -- no fair relying on full natural language understanding or complete background knowledge. In our experiment, we will relax the role of the Wizard slightly, because we are interested in both the cases where a computer is providing assistance to a single person, and the case where two people are collaborating together with assistance from the computer.
The task we will consider is browsing the Web, a task presumably familiar to all of you. Each team will consist of three people: a Searcher, an Assistant, and an Observer. Try to form the team so that the person most expert in the Web has the role of Assistant, and the person least expert is the Searcher. The Observer is passive during the experiment and should only take notes as to what is happening. Each team will have one computer using Netscape on the Web.
Each team will be given the same list of questions to answer, in the style of the Internet Treasure Hunt. The Searcher must look for the answers to the questions on a Web site. Most of the time, the Searcher usually operates the Netscape interface looking for answers. The job of the Assistant is to provide assistance to the searcher. The Searcher may ask the Assistant questions, such as "What search engine should I use". The Assistant may offer advice, such as "Try Yahoo's subject catalog". The Searcher may ask the Assistant to take over control of the Netscape interface for a while, or the Assistant may ask for control.
The Searcher and the Assistant should "think out loud" while they are doing the task. Say why you chose to do what you did. e.g. the Assistant could say, "I see the Searcher is aimlessly wandering around, perhaps I should give him a hint now". The Observer should record all such observations. When the Assistant offers some assistance, the Searcher should evaluate it, e.g. "That was helpful". Everybody: GO SLOWLY, much slower than you normally would, to account for the time explaining and recording the actions takes.
The instructors will circulate around the room. Feel free to ask questions if you need to clarify the rules of the game.
· Any Web site is fair game, but no use of non-Web resources or other applications on the machine are allowed.
· Even if you know the answer off the top of your head, you should still search for an answer on the Web.
· You may give up on a question if you feel you have exhausted your personal resources for finding an answer. You should declare when you are giving up.
· Don't be afraid to ask for help if you need it. Don't be afraid to decline help when you don't want it.
· You may recommend specific sites, or general strategies, such as "Try a search engine first".
· You can't tell the Searcher the answer directly, they have to find it on their own.
· If you can lead the Searcher to discover the answer and learn a new way of finding things, that is better than just giving them the place to go to get the solution.
· You may use your own personal knowledge and experience of the Web, but try to avoid relying on general background knowledge [e.g. you have studied the subject and already know the answer. How would you have looked for it if you didn't know the answer?].
· Don't be afraid to offer advice when you feel it will help.
· Be responsive to feedback you get from the Searcher.
· Record each action of the Searcher and each action of the Assistant on paper. Was a link clicked, a URL typed, a bookmark made?
· If they say why they are doing an action, record that too. If they don't say why, ask "Why did you do that?". Otherwise, don't interfere with the interaction of the Searcher and the Assistant.
· Record any of your own observations, such as disagreement or tension between the Searcher and the Assistant.
· Ask the Searcher or Assistant to slow down or wait if you can't keep up recording.
Good luck, everyone.
Write up your observations, individually. You may discuss your observations with the other team members.Each team will report to the class next week. Please write things up in a format so that we can post the results on the Web [HTML or plain text].
How helpful was it to have the Assistant?
Were there times when the Assistant was more distracting than helpful?
Was there any kind of assistance you might have liked from the Assistant that you didn't get?
What could the Assistant have done if he/she knew more about your personal likes, dislikes, experience, etc.?
If the Assistant were a computer rather than a person, what could it have done or what would be impossible?
If the Assistant were another person [but not in the same room], how could a computer have facilitated interaction between you?
How hard was it to provide assistance?
Could the Searcher have provided any kind of input or advice that would have helped you give assistance?
Could you have done a better job giving assistance if you had had more time to work on the question by yourself?
Would you have an easier time giving assistance to the same person next time as a result of your experience?
Did you learn anything from assisting this person that you could apply if you were assisting someone else?
How much of your assistance could have been provided by a computer agent [assuming current technology]?
If the Searcher were another person [but not in the same room], how could a computer have facilitated interaction between you?
Did the Assistant provide effective assistance for the Searcher?
Were their interactions smooth and communication between them good?
Were there times when one of them misunderstood the other?
What could either the Searcher or the Assistant have done to facilitate working together?