Selected Papers Indexed by Theme
Theory of language/semantic grounding: T2, 47, 48, 65, 72
Robots and language: T2, 36, 41, 52, 53
Games and language: 44, 45, 61, 68, 69, 71
Child language acquisition: T2, 28, 60
Automatic language acquisition: T2, 26, 27, 28, 36
Modeling human-human situated language use: 42, 56, 65, 66
Multimedia data mining and visualization: T1, 5, 7, 46, 62, 67, 70
Assistive technology: 1, 12, 24, 37, 38
Papers
72. Deb Roy. (2008, in press). A Mechanistic Model of Three Facets of Meaning. Chapter to appear in Symbols, Embodiment, and Meaning, de Vega, Glenberg, and Graesser, eds. pdf
71. Jeff Orkin and Deb Roy. (2008, in press). The Restaurant Game: Learning Social Behavior and Language from Thousands of Players Online. Journal of Game Development. pdf (3.9MB)
70. Rony Kubat, Philip DeCamp, Brandon Roy, and Deb Roy. (2007). TotalRecall: Visualization and Semi-Automatic Annotation of Very Large Audio-Visual Corpora. Ninth International Conference on Multimodal Interfaces (ICMI 2007). pdf (491K)
69. Michael Fleischman, Brandon Roy and Deb Roy. (2007) Unsupervised Content-Based Indexing of Sports Video Retrieval. 9th ACM Workshop on Multimedia Information Retrieval (MIR). Augsburg, Germany. pdf (264K)
68. Michael Fleischman, Brandon Roy and Deb Roy. (2007) Temporal Feature Induction for Baseball Highlight Classification. ACM Multimedia Conference. Augsburg, Germany. pdf (317K)
67. Michael Fleischman and Deb Roy. (2007). Situated Models of Meaning for Sports Video Retrieval. HLT/ACL 2007, Rochester, NY. pdf (293K)
66. Michael Levit and Deb Roy. (2007). Interpretation of Spatial Language in a Map Navigation Task. IEEE Transactions on Systems, Man, and Cybernetics, Part B, 37(3), 667-679. pdf (386K)
65. Peter Gorniak and Deb Roy. (2007). Situated Language Understanding as Filtering Perceived Affordances. Cognitive Science, 31(2), 197-231. pdf (1.7MB)
64. Stefanie Tellex and Deb Roy. (2007). Grounding Language in Spatial Routines. AAAI Spring Symposia on Control Mechanisms for Spatial Knowledge Processing in Cognitive / Intelligent Systems. pdf (116K)
63. Michael Fleischman and Deb Roy. (2007). Representing Intentions in a Cognitive Model of Language Acquisition: Effects of Phrase Structure on Situated Verb Learning. AAAI Spring Symposia on Intentions in Intelligent Systems.
62. Michael Fleischman, Philip DeCamp, and Deb Roy. (2006). Mining Temporal Patterns of Movement for Video Content Classification. Proceedings of the 8th ACM SIGMM International Workshop on Multimedia Information Retrieval. pdf (323K)
61. Peter Gorniak and Deb Roy. (2006). Perceived Affordances as a Substrate for Linguistic Concepts. Twenty-eighth Annual Meeting of the Cognitive Science Society. 6 pages. (pdf)
60. Deb Roy, Rupal Patel, Philip DeCamp, Rony Kubat, Michael Fleischman, Brandon Roy, Nikolaos Mavridis, Stefanie Tellex, Alexia Salata, Jethran Guiness, Michael Levit, Peter Gorniak. (2006). The Human Speechome Project. Twenty-eighth Annual Meeting of the Cognitive Science Society. 6 pages. (pdf)
59. Peter Gorniak, Jeff Orkin, and Deb Roy. (2006). Speech, Space and Purpose: Situated Language Understanding in Computer Games. Twenty-eighth Annual Meeting of the Cognitive Science Society Workshop on Computer Games. pdf (313K)
58. Nikolaos Mavridis and Deb Roy. (2006). Grounded Situation Models for Robots: Where Words and Percepts Meet. Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 6 pages. (pdf)
57. Dong Zhang, Daniel Gatica-Perez, Deb Roy, Samy Bengio. (2006). Modeling Interactions from Email Communication. IEEE International Conference on Multimedia & Expo (ICME). 4 pages. (pdf)
56. Stefanie Tellex and Deb Roy. (2006). Spatial Routines for a Simulated Speech-Controlled Vehicle. Proc. Int. Conf. of Human-Robot Interaction (HRI2006), 8 pages. (pdf)
55. Kai-yuh Hsiao, Peter Gorniak, and Deb Roy. (2005). NetP: A Network API for Building Heterogeneous Modular Intelligent Systems. AAAI 2005 Workshop in Modular Construction of Human-Like Intelligence. (pdf)
54. Nick Mavridis and Deb Roy. (2005). Grounded Situation Models for Robots: Bridging language, Perception, and Action. AAAI Workshop on Modular Construction of Human-Like Intelligence. (pdf)
53. Kai-yuh Hsiao and Deb Roy. (2005). A Habit System for an Interactive Robot. Proceedings of the AAAI Fall Symposium 2005: From Reactive to Anticipatory Cognitive Embodied Systems. (pdf)
52. Deb Roy and Niloy Mukherjee (2005). Towards Situated Speech Understanding: Visual Context Priming of Language Models. Computer Speech and Language, 19(2), pages 227-248. (pdf)
51. Dong Zhang, Daniel Gatica-Perez, Samy Bengio, and Deb Roy. (2005). Learning Influence among Interacting Markov Chains. Neural Information Processing Systems (NIPS). (pdf)
50. Deb Roy. (2005). Grounding Words in Perception and Action: Computational Insights. Trends in Cognitive Sciences, 9(8): 389-96. (pdf)
49. Deb Roy and Ehud Reiter. (2005). Connecting Language to the World. Artificial Intelligence, 167(1-2): 1-12. (pdf)
48. Deb Roy. (2005). Semiotic Schemas: A Framework for Grounding Language in the Action and Perception. Artificial Intelligence, 167(1-2): 170-205. (pdf)
47. Michael Fleischman and Deb Roy. (2005). Why are verbs harder to learner than nouns? Initial insights from a computational model of situated word learning. Twenty-Seventh Annual Meeting of the Cognitive Science Society, 6 pages. (pdf)
46. Philip DeCamp, Amber Frid-Jimenez, Jethran Guiness and Deb Roy (2005). Gist Icons: Seeing Meaning in Large Bodies of Literature. IEEE Information Visualization 2005 Conference, 2 pages. (pdf)
45. Peter Gorniak and Deb Roy (2005). Probabilistic Grounding of Situated Speech using Plan Recognition and Reference Resolution. Seventh International Conference on Multimodal Interfaces (ICMI 2005), 6 pages. Best Paper Award. (pdf)
44. Michael Fleischman and Deb Roy (2005). Intentional Context in Situated Language Learning. Proc. Ninth Conference on Computational Natural Language Learning, 8 pages. (pdf)
43. Peter Gorniak and Deb Roy (2005). Speaking with your Sidekick: Understanding Situated Speech in Computer Role Playing Games. Proc. Artificial Intelligence and Interactive Digital Entertainment, 2005, 6 pages. (pdf)
42. Peter Gorniak and Deb Roy. (2004). Grounded Semantic Composition for Visual Scenes. Journal of Artificial Intelligence Research, Volume 21, pages 429-470. (pdf)
41. Deb Roy, Kai-Yuh Hsiao, and Nikolaos Mavridis. (2004). Mental Imagery for a Conversational Robot. IEEE Transactions on Systems, Man, and Cybernetics, Part B, Volume 34 , Issue 3, pages 1374-1383. (pdf)
40. Joshua Juster and Deb Roy. (2004). Elvis: Situated Speech and Gesture Understanding for a Robotic Chandilier. Proc. Int. Conf. Multimodal Interfaces. (pdf)
39. Deb Roy. (2004). 10x: Human-machine Symbiosis. BT Technology Journal, Vol 22, No. 4. (pdf)
38. Rupal Patel, Sam Pilato, and Deb Roy. (2004). Beyond Linear Syntax: An Image-Oriented Communication Aid. Journal of Assistive Technology Outcomes and Benefits, 1(1), 57-66. (pdf)
37. Deb Roy, Yair Ghitza, Jeff Bartelma, and Charlie Kehoe. (2004). Visual Memory Augmentation: Using Eye Gaze as an Attention Filter. Proc. IEEE International Symposium on Wearable Computers, 4 pages. (pdf)
36. Deb Roy. (2003). Grounded Spoken Language Acquisition: Experiments in Word Learning. IEEE Transactions on Multimedia, 5(2): 197-209. (pdf)
35. Kai-yuh Hsiao, Nikolaos Mavridis, Deb Roy. (2003). Coupling Perception and Simulation: Steps Towards Conversational Robotics. Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems. (pdf)
34. Peter Gorniak and Deb Roy. (2003). Augmenting Interfaces with Adaptive Speech Commands. International Conference on Multimodal Interfaces, Vancouver. (pdf)
33. Peter Gorniak and Deb Roy. (2003). A Visually Grounded Natural Language Interface for Reference to Spatial Scenes. International Conference on Multimodal Interfaces, Vancouver. (pdf)
32. Niloy Mukherjee and Deb Roy. (2003). A Visual Context-Aware Multimodal System for Spoken Language Processing. Proc. Eurospeech, 4 pages. (pdf)
31. Brian Whitman, Deb Roy, Barry Vercoe. (2003). Learning Word Meanings and Descriptive Parameter Spaces from Music. Proceedings of the HLT-NAACL03 workshop on Learning Word Meaning from Non-Linguistic Data. (pdf)
30. Deb Roy, Kai-Yuh Hsiao and Nikolaos Mavridis. (2003). Conversational Robots: Building Blocks for Grounding Word Meaning. Proc. NAACL Workshop on Word Meaning.
29. Peter Gorniak and Deb Roy. (2003). Understanding Complex Visually Referring Utterances. NAACL Workshop on Word Meaning.
28. Deb Roy and Alex Pentland. (2002). Learning Words from Sights and Sounds: A Computational Model. Cognitive Science, 26(1), 113-146. (pdf)
27. Deb Roy. (2002) Learning Words and Syntax for a Visual Description Task. Computer Speech and Language, 16(3). (pdf)
26. Deb Roy, Peter Gorniak, Niloy Mukherjee, and Josh Juster. (2002). A Trainable Spoken Language Understanding System for Visual Object Selection. Proceedings of the International Conference of Spoken Language Processing. (pdf)
25. Deb Roy. (2002). A Trainable Visually-Grounded Spoken Language Generation System. Proceedings of the International Conference of Spoken Language Processing.
24. Ewa Dominowska, Deb Roy and Rupal Patel. (2002). An Adaptive Context-Sensitive Communication Aid. Proceedings of the CSUN International Conference on Technology and Persons with Disabilities. (pdf)
23. Deb Roy. (2000/2001). Learning Visually Grounded Words and Syntax of Natural Spoken Language. Evolution of Communication. 4(1). (pdf)
22. Deb Roy. (2001). Situation-Aware Spoken Language Processing. Royal Institute of Acoustics Workshop on Innovation in Speech Processing, Stratford-upon-Avon, England. (pdf)
21. Fred Cummins and Deb Roy. (2001). Using Synchronous Speech to Minimize Variability. Royal Institute of Acoustics Workshop on Innovation in Speech Processing, Stratford-upon-Avon, England. (pdf)
20. Deb Roy. (2000). Grounded Speech Communication. Proceedings of the International Conference on Spoken Language Processing. (pdf)
19. Deb Roy. (2000). Learning from multimodal observations. IEEE Int. Conf. Multimedia and Expo (ICME), New York, NY, (Invited paper). (pdf)
18. Deb Roy. (2000). Integration of Speech and Vision using Mutual Information. Int. Conf. Acoustics, Speech and Signal Processing. (pdf)
17. Roy, D. (2000). A computational model of word learning from multimodal sensory input. International conference of Cognitive Modeling, Groningen, Netherlands. (pdf)
16. Rupal Patel and Deb Roy. (1999). Adaptive spoken communication aids. Proceedings of the American Speech and Hearing Association annual Conference, San Francisco, CA.
15. Deb Roy, Bernt Schiele, and Alex Pentland. (1999). Learning Audio-Visual Associations using Mutual Information. International Conference on Computer Vision, Workshop on Integrating Speech and Image Understanding. Corfu, Greece. (pdf)
14. Deb Roy and Alex Pentland. (1998). Learning Words from Audio-Visual Input, Int. Conf. Spoken Language Processing, Sydney, Australia. Volume 4, p. 1279. (pdf)
13. Deb Roy and Alex Pentland. (1998). Word Learning in a Multimodal Environment. ICASSP, Seattle, WA. (pdf)
12. Deb Roy and Alex Pentland. (1998). A Phoneme Probability Display for Individuals with Hearing Disabilities. Assets '98. (pdf)
11. Rupal Patel and Deb Roy. (1998). Teachable Interfaces for Individuals with Dysarthric Speech and Severe Physical Impairments. AAAI workshop on Integrating Artificial Intelligence and Assistive Technology, Madison, WI. (pdf)
10. Deb Roy and Alex Pentland. (1998). Learning Audio-Visually Grounded Words from Natural Input. AAAI workshop on The Grounding of Word Meaning: Data and Models, Madison, WI. (pdf)
9. Deb Roy, Michal Hlavac, Marina Umaschi, Tony Jebara, Justine Cassell and Alex Pentland. (1997). Toco the Toucan: A Synthetic Character Guided by Perception, Emotion, and Story. Visual Proceedings of Siggraph, Page 66.
8. Deb Roy and Carl Malamud. (1997). Integration of a Large Text and Audio Corpus Using Speaker Identification. Proceedings of the AAAI Spring Symposium on the Intelligent Integration and Use of Text, Image, Video and Audio Corpora, Palo Alto.
7. Deb Roy and Carl Malamud. (1997). Speaker identification based text to audio alignment for an audio retrieval system. Proceedings of the International Conference of Acoustics, Speech and Signal Processing, Munich, Vol. 2, pp. 1099-1103. (pdf)
6. Deb Roy. (1997). Speaker indexing using neural network clustering of vowel spectra. International Journal of Speech Technology, 1(2):143-149. (pdf)
5. Deb Roy and Chris Schmandt. (1996). NewsComm: A Hand-Held Interface for Interactive Access to Structured Audio. ACM Conference on Computer Human Interaction, Vancouver. (pdf)
4. Chris Schmandt and Deb Roy. (1996). Using acoustic structure in a hand-held audio playback device. IBM Systems Journal, Vol. 35, Nos. 3&4. (pdf)
3. Deb Roy and Alex Pentland. (1996). Automatic Spoken Affect Analysis and Classification. International Conference on Automatic Face and Gesture Recognition, Killington, VT. (pdf)
2. B.N. Chodirker, D. Roy, C.R. Greenberg, M. Cheang, J.A. Evans, and M.H. Reed. (1991). Computer assisted analysis of hand radiographs in infantile hypophosphatasia carriers. Pediatric Radiology, 21:216-219. (pdf)
1. Joseph Pear, Witold Kinsner and Deb Roy. (1987). Vocal shaping of retarded and autistic individuals using speech synthesis and speech Recognition. IEEE Ninth Annual Conference of the Engineering in Medicine and Biology Society, pages 1787-1788. (pdf)
Theses
T2. Deb Roy. (1999). Learning from Sights and Sounds: A Computational Model. Ph.D. in Media Arts and Sciences, MIT. (pdf)
T1. Deb Roy. (1995). NewsComm: A Hand-Held Device for Interactive Access to Structured Audio. M.Sc. in Media Arts and Sciences, MIT. (pdf)
Please report to me any broken links.