Students will work on quarter-long research project that proposes a contribution to the fields of collective intelligence, human computation and crowdsourcing, or uses crowdsourcing in a creative way to make a contribution to another research domain. Successful projects raise interesting and open research question(s) and create a prototype system or conduct a preliminary study that demonstrates a potential contribution to the academic literature. This project will be completed in groups of 2-3 students per team.
To propose a project that will make a unique contribution to the research literature
To understand how iterative design methods can inspire and guide research contributions
To connect project work with insights from the readings and discussions
To engage peers as participants and to recruit online participants beyond the classroom
We will have interim project deadlines throughout the quarter:
Weeks 1-3: At the end of each class, we will collectively generate research ideas. Students to form teams.
Weeks 4-6: Teams pitch 1 or 2 project ideas in class for critical feedback.
Week 7-8: Teams make a prototype or collect pilot data (and give updates during class)
Week 9-10: Final demos and presentations in class
Week 11: Submit a 4-6 page short paper on your research proposal (no class this week)
Everyone has ideas! Let's create a collective repository of possible research projects (link will be shared on first day of class). Feel free to list anything related to the course topic. For each idea, include a brief descriptive title, a paragraph that describes the potential contribution, and your name so that peers may reach out to you about collaborations through our Slack channel. We encourage everyone to contribute ideas asynchronously outside of class. Start communicating with each other about potential collaborations for the final project.
This project will be completed in groups of two to three students. No single-member teams. When discussing a potential partnership, you should strive to create a team with diverse backgrounds and skills (e.g., programming, design, writing, study design) but overlapping interests, availability, and motivation levels. It's important to communicate and follow through on commitments. Keep a clear, consistent line of communication open with your teammate.
You'll likely iterate on your ideas throughout the quarter -- try to avoid fixating on one idea early on. Collaboration with people not enrolled in the class is allowed as long as their contribution is clearly identified.
Add your ideas to the Ideation tab in the Course Dashboard. When you form a team, add your team to the ProjectTeams tab.
During class in Week 4, 5 or Week 6, teams will present a short “pitch” that describes 1 or 2 of your favorite ideas. Your talk and discussion should be relatively short (no more than 10 minutes per team) and should cover these points:
What's the idea? Create a quick mockup of your idea to make it concrete!
What makes the idea valuable or important? How could this idea impact others?
How does the research idea expand on other projects or knowledge already published? (include citations!)
What are your hypotheses?
How might one evaluate the idea? What are a few things might you measure?
What system or study will you develop to test the idea?
What are the riskiest risks? Point out the riskiest component of your project (e.g., cannot get the hardware you need, the algorithm may take too long, the difference between conditions may not be measurable, etc).
After your pitch, the instructor and peers will provide comments, ideas, and feedback. If your team presents more than one idea, we can all vote on which of your ideas has the most promise. Make sure to get the instructor's approval before moving on to the prototyping/piloting stage.
When your "Pitch" slides are ready, add a link to the ProjectTeams tab (by W4 or W5 or W6 class).
After getting feedback and approval on your project pitch, spend several weeks creating an initial prototype of your proposed system or creating a draft of the materials you will need to conduct a study. Your team's prototype should be a concrete representation of your project that people can actually try out. It could be a working prototype of some aspect of an envisioned system or it could study materials for an intervention you want to test empirically. Whether it's a system or intervention, every team should develop an initial study protocol (e.g., desired participants, procedures, measures, etc.).
Iterate on your prototype system or study. Then try to collect data from actual participants. Then gather preliminary data on how people interact with the prototype/intervention (ie., behaviors) as well as how they feel about it (ie., impressions). Teams should also create a short survey to collect comments and feedback directly after people try it.
You may recruit people by any means. This includes posting a link to your social media or to online forums related to your topic, or by paying people from a crowdsourcing marketplace. Try to collect data from at least 10 participants and conduct a preliminary analysis.
Each team will present during week 9 or 10 (no paper discussions those weeks) for up to 25 minutes (~15 min for talk/demo and 5 min for Q&A/discussion). The goal should be to discuss all aspects of your research process and results, as if you were giving a conference research talk or a departmental research talk. Cover these points:
Motivation. What's the context? Why is the idea important? How could it impact people?
Background. How does the idea build on prior work (include citations)? How will it expand knowledge?
Questions and hypotheses. What's your core set of research questions? What are your hypotheses?
Prototype. Show what you made. A video or live demo would be rad. This can be a system design or a study intervention.
Method. Give details on your evaluation. This includes who and how you recruit participants, what you measured, and the procedures. Also, what changes would you make to the protocol for a more extensive study.
Early results. Summarize what you learned so far from your data collection. Include charts/graphs/quotes to give an overview of the main insights. Speculate on whether the study has legs worthy of collecting a full data set and writing a full paper submission.
When your "Final" slides are ready, add a link to the ProjectTeams tab (by W9/W10 class). Please be sure to include all teammates' names on the first and last slide.
For the final team report, your goal is to write a 4-6 page paper that would serve as a good first draft of an actual submission to CHI, HCOMP, CI or any academic conference. Please use the ACM paper format (in Overleaf or Word). All of the text and figures/tables should be included within the 6-page limit, while any number of references/citations can go beyond the six-page limit. Appendices are acceptable and optional (they don't count towards the page limit). Include details a keen reader might want to see (e.g., survey questions, interview questions, ChatGPT prompt engineering, etc.).
The research paper should include the motivation for the research, background literature, method, results, discussion, and conclusion. The goal is to draft a paper worthy of publication in a top-tier research venue.
Some advise:
Find a paper that you can use as a template / model. This paper does not need to be on the same topic, but should map onto your work in terms of type of contribution (e.g. a tool paper vs. a theory paper). This will give you guidance as to how to structure your paper.
The title and abstract are the most important parts of a paper, and should clearly convey what you did. Motivate your specific problem and focus on what you did. After reading the abstract, the reader should understand your contribution. Do not just say what you did generally, but what you specifically learned.
Have separate headers in your paper labeled 'method', 'results', and 'discussion'. (This is standard APA heading structure.) Make sure to discuss how your analyses ties to your initial hypotheses.
Use visual representations to show UI screens and data. Graphs should aggregate data across participants (include standard error bars). Figures should be captioned with what you believe the reader should infer from the figure. Again, gather than a generic description of the data, give readers a hint about what they should interpret from the graph.
When your Final Report is ready, add a link to the ProjectTeams tab (by Friday of W11).
In general, when students do research as part of a class project, IRB approval is not required. You can pilot your study and collect data; it counts as your educational experience. However, this means that the data you collect may not be published in an academic research venue. This is not something you can change later. If you anticipate that you might want to publish your data later, we encourage you to apply for an exempt IRB protocol BEFORE you start collecting data. If you would like to apply for IRB approval do the following:
complete the CITI training course on Social and Behavioral Research (Basic/Refresher);
write drafts of a consent form and a research protocol (SBER) based on the templates on this page;
create drafts of all research instruments (e.g., surveys, interview guides, etc.) to include in the IRB submission;
ask Prof. Dow to create a seed protocol. He will add your team as study personnel on a new protocol. Your team can finish the process and submit.
Even for class projects that are not subject to IRB jurisdiction, you still have a responsibility to protect the welfare of those participating in your class project. That means, creating a consent form that spells out what you are asking from each participant in terms of information, time, and effort. Specify what data will be collected from each participant and how you will protect this data.
The expectation is that every student will present one paper per week, participate actively in discussions around all papers, and make significant contributions to a team research project as described above. If you are fully engaged, you can expect to get an A. Prof. Dow will not provide interim grades on any deliverables, but will happily provide feedback throughout the quarter. Prof. Dow will also let you know if he thinks your research paper has the potential to make a contribution to the research community. After the class ends, you will have a good start on a full paper submission with detailed feedback and you will have collected at least some of your data. It's your team's choice about whether to carry forward to submit your work to an academic conference.