Project
Students will work on quarter-long research project that proposes a contribution to the fields of collective intelligence, human computation and crowdsourcing, or uses crowdsourcing in a creative way to make a contribution to another research domain. Successful projects raise interesting and open research question(s) and create a prototype system or conduct a preliminary study that demonstrates a potential contribution to the academic literature. This project will be completed solo or in groups of two.
Project goals:
To propose a project that will make a unique research contribution to the literature
To understand how iterative design methods can inspire and guide research contributions
To engage the class as participants and to recruit online participants beyond the classroom
To connect project work with insights from the readings and discussions
Project timeline:
We will have interim project deadlines throughout the quarter:
Weeks 1-3: At the end of each class, we will collectively generate research ideas. Students to form teams.
Weeks 4-5: Teams pitch 1 to 3 project ideas in class for critical feedback.
Week 6-8: Teams create a prototype or prepare materials for a preliminary study.
Week 8: Show your prototype or study materials. Be ready to let others try it out during class.
Week 9: Iterate on your prototype. Collect data and conduct analyses.
Week 10: Deliver a final presentation .
Week 11: Submit a 4-6 page short paper on your research proposal.
Project details:
Weeks 1-3: Ideas and teams
Everyone has ideas! Let's create a collective repository of possible research projects (link will be shared on first day of class). Feel free to list anything related to the course topic. For each idea, include a brief descriptive title, a paragraph that describes the potential contribution, and your name so that peers may reach out to you about collaborations through our Slack channel. We encourage everyone to contribute ideas asynchronously outside of class. Start communicating with each other about potential collaborations for the final project.
This project will be completed in groups of two. If we have an odd number of students, there can be one team of three. No teams with a single member. When discussing a potential partnership, you should strive to create a team with diverse backgrounds and skills (e.g., programming, design, writing, study design) but overlapping interests, availability, and motivation levels. It's important to communicate and follow through on commitments. Keep a clear, consistent line of communication open with your teammate.
You'll likely iterate on your ideas throughout the quarter -- try to avoid fixating on one idea early on. Collaboration with people not enrolled in the class is allowed as long as their contribution is clearly identified. A higher level of quality will be expected with additional collaborators.
Add your ideas to the Ideation tab in the Course Dashboard. When you form a team, add your team to the ProjectTeams tab.
Weeks 4-5: Pitches
During class in Week 4 or Week 5, teams will present a short “pitch” that describes 1-3 of your favorite ideas. Your talk and discussion should be relatively short (no more than 10 minutes per team) and should cover these points:
What's the idea? Create a quick mockup of your idea to make it concrete!
What makes the idea valuable or important? How could this idea impact others?
How does the research idea expand on other projects or knowledge already published? (include citations!)
What are your hypotheses?
How might one evaluate the idea? What 2-3 things might you measure?
What system or study will you develop to test the idea?
What are the riskiest risks? Point out the riskiest component of your project (e.g., cannot get the hardware you need, the algorithm may take too long, the difference between conditions may not be measurable, etc).
After your pitch, the instructor and peers will provide comments, ideas, and feedback. If your team presents more than one idea, we can all vote on which of your 2-3 ideas has the most promise. Make sure to get the instructor's approval before moving on to the prototyping stage.
When your "Pitch" slides are ready, add a link to the ProjectTeams tab (by W4 or W5 class).
Weeks 6-8: Prototypes
After getting feedback and approval on your project pitch, spend weeks 6-8 creating an initial prototype of your proposed system or creating a draft of the materials you will need to conduct a study. Your team's prototype should be a concrete representation of your project that people can actually try out. It could be a working prototype of some aspect of an envisioned system or it could study materials for an intervention you want to test empirically. Whether it's a system or intervention, every team should develop an initial study protocol (e.g., desired participants, procedures, measures, etc.).
During class in week 8, each team will demonstrate their "prototype" by either demoing the prototype, or even better, by conducting a short trial session with your peers as pilot participants. Use this opportunity to gather preliminary data on how people interact with the prototype (ie., behaviors) as well as how they feel about it (ie., impressions). Teams should also create a short survey to collect comments and feedback directly after people try it. We will end the session with a discussion about how to improve the prototype and study protocol.
Build on your earlier slide deck. Include details about the prototype and instructions for how to interact with the prototype. Additional written instructions will help insure that everything is consistent across participants. The slides should include links to the system or study prototype, as well as to any forms you create to collect feedback.
When your "Prototype" slides are ready, add a link to the ProjectTeams tab (by W8 class).
Weeks 9-11: Data Collection, Presentation, and Report
Data collection. Iterate on your prototype system or study. Then try to collect data from actual participants. You may recruit people by any means. This includes using UCSD's SONA system, posting a link to your social media or to online forums related to your topic, or by paying people from a crowdsourcing marketplace. From Week 8 to Week 10, collect data from at least 20 participants and conduct some preliminary analysis.
Presentation. Each team will present during week 10 (no paper discussions that week) for up to 15 minutes (~10 min for talk and 5 min for Q&A). Please be sure to include all teammates' names on the first and last slide. The goal should be to discuss all aspects of your research process and results, as if you were giving a conference research talk or a departmental research talk. Cover these points:
Motivation. What's the context? Why is the idea important? How could it impact people?
Background. How does the idea build on prior work (include citations)? How will it expand knowledge?
Questions and hypotheses. What's your core set of research questions? What are your hypotheses?
Prototype. Show what you made. A video or live demo would be rad. This can be a system design or a study intervention.
Method. Give details on your evaluation. This includes who and how you recruit participants, what you measured, and the procedures. Also, what changes would you make to the protocol for a more extensive study.
Early results. Summarize what you learned so far from your data collection. Include charts/graphs/quotes to give an overview of the main insights. Speculate on whether the study has legs worthy of collecting a full data set and writing a full paper submission.
When your "Final" slides are ready, add a link to the ProjectTeams tab (by W10 class).
Report. For the final team report, your goal is to write a 4-6 page paper that would serve as a good first draft of an actual submission to CHI, HCOMP, CI or any academic conference. Please use the ACM SIGCHI paper format. All of the text and figures/tables should be included within the 6-page limit, while any number of references/citations can go beyond the six-page limit. Appendices are acceptable and optional (they don't count towards the page limit). Include details an interested reader might want to see (e.g., survey questions, interview questions, ChatGPT prompt engineering, etc.).
The research paper should include the motivation for the research, background literature, method, results, discussion, and conclusion. The goal is to draft a paper worthy of publication in a top-tier research venue.
Some advise:
Find a paper that you can use as a template. This paper does not need to be on the same topic, but should map onto your work in terms of type of contribution (e.g. a tool paper vs. a theory paper). This will give you guidance as to how to structure your paper.
The title and abstract are the most important parts of a paper, and should clearly convey what you did. Motivate your specific problem and focus on what you did. After reading the abstract, the reader should understand your contribution. Do not just say what you did generally, but what you specifically learned.
Have separate headers in your paper labeled 'method', 'results', and 'discussion'. (This is standard APA heading structure.) Make sure to discuss how your analyses ties to your initial hypotheses.
Use visual representations to show UI screens and data. Graphs should aggregate data across participants (include standard error bars). Figures should be captioned with what you believe the reader should infer from the figure. Again, gather than a generic description of the data, give readers a hint about what they should interpret from the graph.
When your Final Report is ready, add a link to the ProjectTeams tab (by Friday of W11).
IRB Review:
In general, when students do research as part of a class project, IRB approval is not required. You can pilot your study and collect data; it counts as your educational experience. However, this means that the data you collect may not be published in an academic research venue. This is not something you can change later. If you anticipate that you might want to publish your data later, we encourage you to apply for an exempt IRB protocol BEFORE you start collecting data. If you would like to apply for IRB approval do the following:
complete the CITI training course on Social and Behavioral Research (Basic/Refresher);
write drafts of a consent form and a research protocol (SBER) based on the templates on this page;
create drafts of all research instruments (e.g., surveys, interview guides, etc.) to include in the IRB submission;
ask Prof. Dow to create a seed protocol. He will add your team as study personnel on a new protocol. Your team can finish the process and submit.
Even for class projects that are not subject to IRB jurisdiction, you still have a responsibility to protect the welfare of those participating in your class project. That means, creating a consent form that spells out what you are asking from each participant in terms of information, time, and effort. Specify what data will be collected from each participant and how you will protect this data.
Expectations and Grading:
The expectation is that every student will present one paper per week, participate actively in discussions around all papers, and make significant contributions to a team research project as described above. If you are fully engaged, you can expect to get an A. Prof. Dow will not provide interim grades on any deliverables, but will happily provide feedback throughout the quarter. Prof. Dow will also let you know if he thinks your research paper has the potential to make a contribution to the research community. After the class ends, you will have a good start on a full paper submission with detailed feedback and you will have collected at least some of your data. It's your team's choice about whether to carry forward to submit your work to an academic conference.