(01) 2019 - Facebook Research PhD Fellowship 2019.
(12) 2018 - VizNet paper got accepted to CHI 2019.
(11) 2018 - selected as a Distinguished Program Committee member of the 27th International Joint Conference on Artificial Intelligence (IJCAI).
(09) 2018 - received the MIT Arts Scholars honor.
(08) 2018 - received the MIT-SenseTime Alliance on Artificial Intelligence Grant ($100,000), MIT Quest for Intelligence.
(05) 2018 - received the Graduate Teaching Award, presented annually to one MIT professor or teaching assistant from each school, for excellence in teaching a graduate level course.
(04) 2018 - invited talk, "Human-AI Collaboration for Sustainable Market Design", at the Ethics of AI event by the Media Lab and Berkman Klein Center, Harvard University.
(01) 2018 - represented Daemo at a European Dialog on the Platform Economy event by the European Trade Union Institute and partners, Brussels.
(11) 2017 - AI-algorithmic ethics paper got accepted at the AAAI-2018, New Orleans, USA.
(11) 2017 - TEDx talk "the Future of Markets in the Era of AI", Boston.
(10) 2017 - invited to the Industry Panel at the AAAI HCOMP conference, Canada― decided to step down to increase the panel's gender diversity.
(09) 2017 - invited at the Hasso Plattner Institute and Stanford Design Thinking Research Program, Potsdam Germany.
(01) 2017 - guided inventors across India to solve pressing societal challenges in agriculture and healthcare using AI [MIT, Innovating for Billions].
(06) 2016 - EteRNA featured in Werner Herzog's film ``Lo And Behold: Reveries of the Connected World’’ (Trailer).
(09) 2015 - Daemo, crowdsourcing marketplace completed its first project with Microsoft Research.
(02) 2014 - EteRNA paper got accepted to PNAS.
(01) 2011 - co-launched EteRNA, crowd-computing game that helps design RNA molecules and invent medicine.
Neil S. Gaikwad
MIT Media Lab
Massachusetts Institute of Technology
Neil Gaikwad is a third-year doctoral researcher at the MIT Media Lab, specializing in human-centered machine learning and socio-economic development. He develops human-AI collaboration algorithms and tools to study, model, and (re-)design socio-technical systems, analyzing large-scale datasets emerging from social processes, Earth remote sensing satellites, and socio-economic interactions. Neil holds a master’s degree from the school of computer science at Carnegie Mellon University where he worked at the Robotics Institute. Currently, he is a member of the Space Enabled Research Group, where he works with his advisor Danielle Wood.
Neil's research focuses at the intersection of interactive machine learning, community-centered design, and economics (game theory, collective choice, etc.) to provide real-world solutions to challenging wicked problems in crisis informatics, market design under weak institutions, just public policy design, and urban environmental planning. This work is informed by experiences in academia, quantitative finance research on Wall Street, and farming communities in the Western Ghats of India (Sahyadri, सह्याद्री) . A major focus of his current work is to decode the impact of the Earth’s physical processes on human societies and institutions. Drawing upon this understanding, he is designing and engineering more efficient and equitable precision agriculture markets to help small-scale farmers mitigate the impact of meteorological disasters and institutional failures. His research is part of the MIT Quest for Intelligence, an institute-wide initiative that aims to unlock the nature of intelligence and harness it to make a better world.
Neil’s research has been published in premier artificial intelligence and human-computer interaction conferences (AAAI, ACM UIST, ACM CSCW, ACM CHI) and a scientific journal (PNAS), and featured in the New York Times, Bloomberg, WIRED, and the Wall Street Journal. Some of the examples of his socio-technical system research include Daemo, a self-governed crowdsourcing marketplace and EteRNA, a crowd-computing system that harnesses a massive scale human-machine collaboration to help solve computational problems that neither humans nor machines can solve alone. Neil was one of the principle creators and founding members of Daemo and EteRNA. He led Daemo's technical architecture and invented Boomerang, an incentive-compatible reputation system. His foundational contributions draw on fundamentals from game theory (incentive design for pro-social behavior), structured finance (guilds organization as tranches), and human-centered design (tasks composition). This research has been deployed and is used by people across the world. Daemo has been used to create the SQuAD, Stanford's reading comprehension dataset for Natural Language Processing, and EteRNA has reached over 100,000 citizen scientists across the world.
One of the major missions of Neil's life is to democratize STEAM opportunities to empower people across the globe. He was one of the core members of the Stanford Crowd Research Initiative and part of the MIT's Innovating for Billions in Emerging Worlds Leadership Council. His honors include the Facebook PhD Fellowship award, an MIT Arts Scholar, an IJCAI Distinguished Program Committee Member, the ACM UIST Honorable Mention, the CHI Special Recognition for Outstanding Reviews, and the MIT Graduate Teaching award, presented annually to one MIT professor or teaching assistant from each school, for excellence in teaching a graduate level course. Neil is a TEDx speaker and his talk presents the vision for just markets in the era of artificial intelligence.
Recent awards and honors
Facebook Research PhD Fellowship, 2019
Microsoft Research PhD Fellowship Finalist, 2019
CHI Special Recognition for Outstanding Reviews
Distinguished PC, IJCAI
The MIT Arts Scholars, MIT
Graduate Teaching Award, MIT
Adobe Research PhD Fellowship Finalist
Best Paper Honorable Mention, ACM UIST
VizNet: Towards A Large-Scale Visualization Learning and Benchmarking Repository
CHI 2019: The ACM CHI Conference on Human Factors in Computing Systems(Acceptance rate of 23.8%)
Researchers currently rely on ad hoc datasets to train automated visualization tools and evaluate the effectiveness of visualization designs. These exemplars often lack the characteristics of real-world datasets, and their one-off nature makes it difficult to compare different techniques. In this paper, we present VizNet: a large-scale corpus of over 31 million datasets compiled from open data repositories and online visualization galleries. On average, these datasets comprise 17 records over 3 dimensions and across the corpus, we find 51% of the dimensions record categorical data, 44% quantitative, and only 5% temporal. VizNet provides the necessary common baseline for comparing visualization design techniques, and developing benchmark models and algorithms for automating visual analysis. To demonstrate VizNet's utility as a platform for conducting online crowdsourced experiments at scale, we replicate a prior study assessing the influence of user task and data distribution on visual encoding effectiveness, and extend it by considering an additional task: outlier detection. To contend with running such studies at scale, we demonstrate how a metric of perceptual effectiveness can be learned from experimental results, and show its predictive power across test datasets.
AAAI 2018: Association for the Advancement of Artificial Intelligence. (Acceptance rate of 24.6%)
We present a general approach to automating ethical decisions, drawing on machine learning and computational social choice. In a nutshell, we propose to learn a model of societal preferences, and, when faced with a specific ethical dilemma at runtime, efficiently aggregate those preferences to identify a desirable choice. We provide a concrete algorithm that instantiates our approach; some of its crucial steps are informed by a new theory of swap-dominance efficient voting rules. Finally, we implement and evaluate a system for ethical decision making in the autonomous vehicle domain, using preference data collected from 1.3 million people through the Moral Machine website.
PRESS: Social Media Has Failed Its Self-Driving Test (Bloomberg) | Researchers go after the biggest problem with self-driving cars (Axios)
Belongie, Serge., Goel, Sharad., Davis, James., and Bernstein, Michael.
UIST 2017: ACM Symposium on User Interface Software and Technology, (Acceptance rate of 22.5%)
Research experiences today are limited to a privileged few at select universities. Providing open access to research experiences would enable global upward mobility and increased diversity in the scientific workforce. But, how do we coordinate a crowd of diverse volunteers on open-ended research? How could a PI have enough visibility into each person's contributions to recommend them for further study? We present Crowd Research, a crowdsourcing technique that coordinates open-ended research through an iterative cycle of open contribution, synchronous collaboration, and peer assessment. To aid upward mobility and recognize contributions in publications, we introduce a decentralized credit system: participants allocate credits to each other, which a graph centrality algorithm translates into a collectively-created author order. Over 1,500 people from 62 countries have participated, 74% from institutions with low access to research. Over two years and three projects, this crowd has produced articles at top-tier Computer Science venues, and participants have gone on to leading graduate programs.
PRESS: A Stanford-led Platform for Crowdsourced Research Gives Experience to Global Participants (Stanford News)
PDF Web [ACM UIST Best Paper Honorable Mention, top 2.5% of the technical paper submissions]
CSCW 2017: The ACM Conference on Computer-Supported Cooperative Work and Social Computing (Acceptance rate of 25%)
Crowd workers are distributed and decentralized. While decentralization is designed to utilize independent judgment to promote high-quality results, it paradoxically undercuts behaviors and institutions that are critical to high-quality work. Reputation is one central example: crowdsourcing systems depend on reputation scores from decentralized workers and requesters, but these scores are notoriously inflated and uninformative. In this paper, we draw inspiration from historical worker guilds (e.g., in the silk trade) to design and implement crowd guilds: centralized groups of crowd workers who collectively certify each other’s quality through double-blind peer assessment. A two week field experiment compared crowd guilds to a traditional decentralized crowd work model. Crowd guilds produced reputation signals more strongly correlated with ground-truth worker quality than signals available on current platforms, and more accurate than in the traditional model.
UIST 2016: The ACM Symposium on User Interface Software and Technology, (Acceptance rate of 20.6%)
Paid crowdsourcing platforms suffer from low-quality work and unfair rejections, but paradoxically, most workers and requesters have high reputation scores. These inflated scores, which make high-quality work and workers difficult to find, stem from social pressure to avoid giving negative feedback. We introduce Boomerang, a reputation system for crowdsourcing that elicits more accurate feedback by rebounding the consequences of feedback directly back onto the person who gave it. With Boomerang, requesters find that their highly-rated workers gain earliest access to their future tasks, and workers find tasks from their highly-rated requesters at the top of their task feed. Field experiments verify that Boomerang causes both workers and requesters to provide feedback that is more closely aligned with their private opinions. Inspired by a game-theoretic notion of incentive-compatibility, Boomerang opens opportunities for interaction design to incentivize honest reporting over strategic dishonesty.
* In the readings of Human Computation classes at KAIST, Cornell University, University of Washington
RNA Design Rules From a Massive Open Laboratory
Gaikwad, S., Yoon, Sungroh., Treuille, Adrien., Das, Rhiju., and EteRNA Participants
PNAS 2014: Proceedings of the National Academy of Sciences of the United States of America
Self-assembling RNA molecules present compelling substrates for the rational interrogation and control of living systems. However, imperfect in silico models—even at the secondary structure level—hinder the design of new RNAs that function properly when synthesized. Here, we present a unique and potentially general approach to such empirical problems: the Massive Open Laboratory. The EteRNA project connects 37,000 enthusiasts to RNA design puzzles through an online interface. Uniquely, EteRNA participants not only manipulate simulated molecules but also control a remote experimental pipeline for high-throughput RNA synthesis and structure mapping. We show herein that the EteRNA community leveraged dozens of cycles of continuous wet laboratory feedback to learn strategies for solving in vitro RNA design problems on which automated methods fail. The top strategies—including several previously unrecognized negative design rules-were distilled by machine learning into an algorithm, EteRNABot. Over a rigorous 1-y testing phase, both the EteRNA community and EteRNABot significantly outperformed prior algorithms in a dozen RNA secondary structure design tests, including the creation of dendrimer-like structures and scaffolds for small molecule sensors. These results show that an online community can carry out large-scale experiments, hypothesis generation, and algorithm design to create practical advances in empirical science.
PRESS: Videogamers are recruited to fight Tuberculosis and other ills (Wall Street Journal) | RNA game lets players help find a biological prize (New York Times) | How turning science into a game rouses more public interest (WIRED) | Game lets citizen scientists participate in creating large-scale library of synthetic RNA designs (Scientific American) | Rebooting science outreach (ASBMB Today) | Why video games are key to modern science (CNN) | Online game helps predict how RNA folds (New Scientist)
Crowd Research: Open and Scalable University Laboratories
Belongie, Serge., Goel, Sharad., Davis, James., and Bernstein, Michael.
Design Thinking Research: Looking Further: Design Thinking Beyond Solution-Fixation
Springer Nature 2019
Seafloor to Satellites: Crowdcomputing and Data Across Scales
All Hands on Deck 2018: National Ocean Exploration Forum, NOAA Office of Ocean Exploration and Research and MIT Media Lab
Deep ocean ecosystems are an important component of our planet, but they remain scientifically under explored. Traditionally, research on the ocean ecosystems has been decentralized and scattered, involving small teams of experts, on single ships, traveling to distant locations, and mapping oceans as much as possible within a limited time. This has restricted rapid advancement and sharing of scientific knowledge of the ocean with broader communities. While the advent of deep learning and an unprecedented growth of the seafloor and satellite datasets provide an exciting opportunity to decode patterns in deep ocean ecosystems, these datasets are unlabeled, decentralized, and disorganized. In addition, existing deep learning algorithms often fail to capture and incorporate human insights and local knowledge in scientific explorations. This heightens the need for designing a centralized citizen science platform for collectively studying and exploring the ocean. Toward this goal, we present foundational and methodological building blocks required to design the platform. This work makes several contributions to the open ocean exploration community. First, we illustrate the science of crowd powered computational ecosystem. Second, we explore incentive mechanisms to accomplish high cognitive overload tasks and augment human-AI performance at scale. Third, we discuss ethics and best practices of designing and empowering a sustainable community of citizen scientists. We provide an important opportunity to harness crowdcomputing and AI for deep ocean exploration.
IC2S2 2017: International Conference on Computational Social Science
Historically, scientific experiments have been conducted at a small scale either with artificial environments or with the expertise of limited number of scientists. While social science literature investigates very deep questions to understand human behavior, many experiments are usually limited by the number of participants and duration of a study. On the contrary, computer science literature exploits advanced computational techniques to crunch voluminous datasets, but research designs are generally not experimental, which limits the opportunity to generate causal inferences. In this tutorial we demonstrate how crowdcomputing can enable computational social scientists to engage with millions of users on the Internet and study human behavior at scale for a longer time. We showcase pitfalls and lessons learned from various crowdcomputing and citizen science projects. Furthermore, we provide insights about how to build a sustainable citizen science community to scale science beyond the traditional laboratories. We envisage this tutorial will help computational social scientists effectively use crowdcomputing to investigate deep research questions and longitudinally validate their hypotheses in large scale experiments.
Decoding the Evolution of UN’s Sustainable Development Goals using Bayesian Inference
Santa Fe Institute, Complex Systems Summer School Proceedings 2018
The lack of progress on sustainable development has received considerable critical attention from the United Nations. As a result, the UN has created the Sustainable Development Goals (SDGs) to provide a blueprint for global development. In order to evaluate the success on SDGs, the UN’s Inter-Agency and Expert Group (IAEG-SDGs) introduced the global indicator framework with consensus on 232 parameters. However, these parameters are broad and tend to miss socio-cultural context along with local constraints across the world. Therefore, the generalisability of the global indicator framework is problematic. A number of studies have examined interlinks and interactions between SDGs and respective indicators (e.g. Blanc, 2014, Nilsson et al., 2016, Weitz et al., 2018, Nilsson et al., 2018) through subjective measures, but to date none has systematically studied creation of SDGs beyond government agreements on indicators. In this paper, I provide a conceptual theoretical framework based on Latent Dirichlet Allocation (Blei et al., 2003), a Bayesian inference model, to study the interlinks and evolution of SDGs in the context of local constraints. I envisage that this research will provide an important opportunity to engineer quantitative measures for evaluating the success and refinements of sustainable development programs.
AAAI HCOMP 2017: AAAI Conference on Human Computation
Low-quality results have been a long-standing problem on microtask crowdsourcing platforms, driving away requesters and justifying low wages for workers. To date, workers have been blamed for low-quality results: they are said to make as little effort as possible, do not pay attention to detail, and lack expertise. In this paper, we hypothesize that requesters may also be responsible for low-quality work: they launch unclear task designs that confuse even earnest workers, under-specify edge cases, and neglect to include examples. We introduce prototype tasks, a crowdsourcing strategy requiring all new task designs to launch a small number of sample tasks. Workers attempt these tasks and leave feedback, enabling the requester to iterate on the design before publishing it. We report a field experiment in which tasks that underwent prototype task iteration produced higher-quality work results than the original task designs. With this research, we suggest that a simple and rapid iteration cycle can improve crowd work, and we provide empirical evidence that requester “quality” directly impacts result quality.
CSCW 2017: The ACM Conference on Computer-Supported Cooperative Work
The success of crowdsourcing markets is dependent on a strong foundation of trust between workers and requesters. In current marketplaces, workers and requesters are often unable to trust each other’s quality, and their mental models of tasks are misaligned due to ambiguous instructions or confusing edge cases. This breakdown of trust typically arises from (1) flawed reputation systems which do not accurately reflect worker and requester quality, and from (2) poorly designed tasks. In this demo, we present how Boomerang and Prototype Tasks, the fundamental building blocks of the Daemo crowdsourcing marketplace, help restore trust between workers and requesters. Daemo’s Boomerang reputation system incentivizes alignment between opinion and ratings by determining the likelihood that workers and requesters will work together in the future based on how they rate each other. Daemo’s Prototype tasks require that new tasks go through a feedback iteration phase with a small number of workers so that requesters can revise their instructions and task designs before launch.
CI 2017: Collective Intelligence Conference
A key principle of Daemo is to provide crowd workers and requesters with a means of governing the development and evolution of the platform into the future. To facilitate this, we introduced the Daemo constitution, outlining the goals of the platform, relationship between members of the community and its standards, methods for seeking ideas, amending the constitution and resolving conflicts. The Daemo constitution has been a collaborative effort by the Stanford Crowd Research Collective. However, we have also endeavored to solicit feedback from the MTurk worker and requester communities via TurkerNation, Reddit and other channels.
PDF AVAILABLE UPON REQUEST
UIST 2015: The ACM Symposium on User Interface Software and Technology
Crowdsourcing marketplaces provide opportunities for autonomous and collaborative professional work as well as social engagement. However, in these marketplaces, workers feel disrespected due to unreasonable rejections and low payments, whereas requesters do not trust the results they receive. The lack of trust and uneven distribution of power among workers and requesters have raised serious concerns about sustainability of these marketplaces. To address the challenges of trust and power, this paper introduces Daemo, a self-governed crowdsourcing marketplace. We propose a prototype task to improve the work quality and open-governance model to achieve equitable representation. We envisage Daemo will enable workers to build sustainable careers and provide requesters with timely, quality labor for their businesses.
PRESS: The future of work the peoples uber (Pacific Standard)
PDF Poster System
Simoiu, Camelia., Veit, Andreas., Wilber, Michael., Zhou, Sharon., Belongie, Serge., Goel, Sharad., Davis, James., Bernstein, Michael.
HCI+Design Open House 2016: Stanford University
Scientific research is becoming increasingly collaborative, yet primarily limited to professional researchers in labs and universities. Can we scale traditional research approach and invite anyone from around the world to participate, and crowdsource large-scale, open-ended research problems? We're scaling the research process to solve open-ended novel problems, by providing access and connecting hundreds of people with top researchers.
Click to see my detailed analysis of PageRank & Credit Distribution
BioX 2011: Interdisciplinary Initiatives Symposium Poster Session, Stanford University
Mentored as a teaching assistant
Conference on Fairness, Accountability, and Transparency (FAT), 2018
ACM CHI Conference on Human Factors in Computing Systems, 2017
Organizational Behavior and Human Decision Processes, 2013
Students Supervised & Mentored
MIT Undergraduate Research Opportunity Program (UROP)
Kealani Finegan (MAS, Wellesley College), Project: Precision Agriculture Markets, Fall 2018
Sourav Das (MIT EECS), Project: Computational Sustainability, Spring 2018
Alice Jin (MIT EECS), Project: Superforecasters IARPA, Co-supervised, Fall 2017
Some of the students with positions/scholarships they acquired after the projects:
Aditi Mithal (Awarded Google Venkat Panchapakesan Memorial Scholarship, CS grad program at the University of California, Los Angeles)
Aditi Nath (CS grad program at Arizona State University)
Ankita Sastry (Info Security grad program Carnegie Mellon)
Karan Rajpal (CS grad program at Cornell University)
Nalin Chhibber (Mathematics grad program at the University of Waterloo)
Prastut Kumar (The Google Summer of Code Berkman Klein Center Harvard University)
Prithvi Raj (Awarded Dr. Shanker Dayal Sharma Medal at the IIT Kanpur)
Rahul Sheth (Undergraduate program at the University of California, Los Angeles)
Radhika Bhanu K (CS grad program at Cornell University)
William Dai (Undergraduate program at the University of California, Berkeley)
Teaching and research mentoring have been an integral part of Neil’s quest: to help grow and inspire a generation of creative thinkers, scientists, and inventors who will collectively engage in resolving some of the biggest societal challenges of our time. He gets a great satisfaction when his students succeed, and their accomplishments give him an immense sense of gratification that cannot be measured. Similar to Dr. A.P.J. Abdul Kalam, he would like to be remembered as good teacher and research mentor. You can listen to his interview at MIT's WMBR 88.1 FM Post-It Wall.
At MIT Neil has spearheaded new initiatives that provided the institute wide platform for graduate and undergraduate communities to come together, share their research, and find mentors. He leads the MIT SP Graduate Student Dinner Seminar Series initiative, and he has partnered with MIT's UROP and Office of Graduate Education (OGE) to host research talks and mentorship workshops for MIT students and post-doctoral associates.
Outreach & Service
AI and HCI Conferences
Reviewer, The ACM CHI Conference on Human Factors in Computing Systems, CHI-2019
Program committee, 27th International Joint Conference on Artificial Intelligence Conference on Artificial Intelligence IJCAI-ECAI 2018
Reviewer, 21st ACM CSCW 2018
Reviewer, 26th Int. World Wide Web Conference WWW 2017
Reviewer, iConference, University of Illinois Urbana-Champaign, 2010
Service at MIT
Co-chair, MIT SP Committee on Scholarly Interactions (CoSI), a graduate organization dedicated to bringing top scholars to the MIT campus and the Sidney Pacific graduate residence
Lead and co-founder, MIT SP Graduate Student Dinner Seminar Series, (visit)
Leadership Council, MIT's Innovating for Billions in Emerging Worlds Media Lab Design Rep for the, MIT GradRat Ring
Service at Carnegie Mellon Robotics Institute
Volunteer Robotics Teacher, taught s kids in the Big Brothers Big Sisters program to build robots. This initiative was a part of 100 Robots for 100 Kids, a grassroots project by graduate students at the CMU Robotics Institute.
Co-organizer, EteRNA sessions for Leap@CMU, summer enrichment program for high school students.
Neil is a practicing photographer and an MIT arts scholar. His artwork inspires his scientific endeavors. His exhibition Beyond the Boundaries captures the complexities of our planet, including 3,000 years old glacier landscapes, the Western Ghats (Sahyadri) of India, cultures, and pressing societal challenges such as climate change. Neil's photography work has been published in National Geographic. For more information, please visit `Explore the Planet Earth’ on Facebook and Instagram.
The Game of Cricket
Neil’s research draws inspiration from sports and team dynamics. He was a former cricket player— opening batter and leg spinner. In the US leagues he kept wickets (similar to being a catcher in Baseball) for Jermaine Lawson, former international cricketer and Harshal Patel, Royal Challengers Bangalore's fast-bowler in the IPL.
Some of the archived man of the match scorecards: [146 not out]
Sports Quote: "and so you touch this limit, something happens and you suddenly can go a little bit further. With your mind power, your determination, your instinct, and the experience as well, you can fly very high" -- Ayrton Senna, one of the greatest Formula One drivers of all time.
Networks in the Massive Open Online Research
Where The Mind is Without Fear
Where knowledge is free
Where the world has not been broken up into fragments
By narrow domestic walls
Where words come out from the depth of truth
Where tireless striving stretches its arms towards perfection
Where the clear stream of reason has not lost its way
Into the dreary desert sand of dead habit
Where the mind is led forward by thee
Into ever-widening thought and action
Into that heaven of freedom, my Father, let my country awake.
Nobel Laureate Rabindranath Tagore
The Impossible Dream (The Quest)
To fight the unbeatable foe
To bear with unbearable sorrow
To run where the brave dare not go
To right the unrightable wrong | To love pure and chaste from afar
To try when your arms are too weary | To reach the unreachable star
This is my quest | To follow that star
No matter how hopeless | No matter how far
To fight for the right | Without question or pause
To be willing to march into Hell | For a heavenly cause
And I know if I'll only be true | To this glorious quest
That my heart will lie peaceful and calm | When I'm laid to my rest
And the world will be better for this
That one man, scorned and covered with scars
Still strove with his last ounce of courage
To reach the unreachable star!
Joe Darion (From The Man of La Mancha)