This schedule is subject to change.
Week Topics and activities
Readings and homework due
Sept 28
Topic: Introduction to Crowdsourcing and Human Computation (all lecture PDFs will be posted here after class)

Activities: enrollment, icebreakers, syllabus, discussion forum sign-up

Special guest:  Haakon Faste

Readings due today (discussion sign-up, papers):
  • Quinn and Bederson (2011) Human Computation: A Survey and Taxonomy of a Growing Field
  • Kittur et al. (2013) The Future of Crowd Work
  • Faste et al. (2013) Brainstorm, Chainstorm, Cheatstorm, Tweetstorm
Oct 5
Topic:  Platforms of & for People

Watch: Luis von Ahn TED talk
Watch: Turking for a Living
Check out: Guidelines for Academic Researchers Using Amazon Mechanical Turk
Check out: TurkerNation, Turkopticon

  • von Ahn and Dabbish (2004) Labeling Images with a Computer Game
  • Peer et al. (2015) Beyond the Turk: An Empirical Comparison of Alternative Platforms for Crowdsourcing Online Behavioral Research
  • Martin et al. (2014) Being a Turker
  • Salehi et al. (2015) We Are Dynamo: Overcoming Stalling and Friction in Collective Action for Crowd Workers
  • Zyskowski et al. (2015) Accessible Crowdwork? Understanding the Value in and Challenge of Microtask Employment for People with Disabilities

Homework due: Be a Worker

Oct 12
Topic:  Managing micro-work

Special guest:  Jaime Teevan

  • Cheng, Teevan, et al (2015) Break It Down: A Comparison of Macro- and Microtasks
  • Little et al. (2009) TurKit: Human Computation Algorithms on Mechanical Turk
  • Kulkarni et al. (2012) Collaboratively Crowdsourcing Workflows with Turkomatic
  • Dow et al. (2012) Shepherding the Crowd Yields Better Work
  • Rzeszotarski and Kittur (2012) CrowdScape: Interactively Visualizing User Behavior and Output
  • Dai et al. (2010) Decision-Theoretic Control of Crowd-Sourced Workflows
Homework due:  Be a Boss
Oct 19
Topic:  X-sourcing: Community-sourcing, Learner-sourcing, Citizen-sourcing...

Try: DuoLingo
Try: ProjectCobi
Try: CycleAtlanta
Try: TiramisuTransit

Special guest:  Chris Le Dantec

  • Le Dantec, et al (2015) Planning with Crowdsourced Data: Rhetoric and Representation in Transportation Planning
  • Kim, et al. (2014) Crowdsourcing Step-by-Step Information Extraction to Enhance Existing How-to Videos
  • Dontcheva et al. (2014) Combining Crowdsourcing and Learning to Improve Engagement and Performance
  • André (2013) Community Clustering: Leveraging an Academic Crowd to Form Coherent Conference Sessions
  • Heimerl (2012) Communitysourcing: Engaging Local Crowds to Perform Expert Work Via Physical Kiosks
  • Zimmerman et al. (2011)  Field Trial of Tiramisu: Crowd-sourcing Bus Arrival Times to Spur Co-design
Homework due:  Project ideation

5 Oct 26
Topic:  Crowds and Machines

Special guest:  Julian McAuley

  • He and McAuley (2016) VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback
  • Snow et al. (2008) Cheap and Fast — But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks
  • Deng et al. (2009)  ImageNet: A Large-Scale Hierarchical Image Database
  • Gingold et al. (2012) Micro perceptual human computation for visual tasks
  • Lasecki et al. (2012) Real-Time Captioning by Groups of Non-Experts
  • Zitnick and Parikh (2013) Bringing Semantics Into Focus Using Visual Abstraction

Homework due:  Project pitches
6 Nov 2
Topic:  Collaborative Innovation

Play:  The Johnny Cash Project
Watch: Karl Ulrich on Innovation Tournaments

Special guest:  Mickey McManus

  • MacDonald and McManus (2016) Building High-Performance Teams for Collaborative Innovation (MAYA vision paper)
  • Kornish and Ulrich (2011) Opportunity Spaces in Innovation: Empirical Analysis of Large Samples of Ideas
  • Yu and Nickerson (2011) Cooks or cobblers? Crowd Creativity through Combination
  • Boudreau and Lakhani (2015) Open disclosure of innovations, incentives and follow-on reuse: Theory on processes of cumulative innovation and a field experiment in computational biology
  • Chan et al. (2016) Improving Crowd Innovation with Expert Facilitation
  • Siangliulue et al. (2016) IdeaHound: Improving Large-scale Collaborative Ideation with Crowd-powered Real-time Semantic Modeling
Homework due:  Project prototypes
7 Nov 9
Topic:  Crowd-powered applications

Special guest:  Greg Horowitt

  • Bernstein et al. (2010) Soylent: a Word Processor with a Crowd Inside
  • Bigham et al. (2010) VizWiz: Nearly Real-time Answers to Visual Questions
  • Horowitz and Kamvar (2010) The Anatomy of a Large-Scale Social Search Engine
  • Noronha et al. (2011) Platemate: Crowdsourcing Nutritional Analysis from Food Photographs
  • Lasecki et al (2014) Glance: Rapidly Coding Behavioral Video with the Crowd
  • Retelny et al. (2014) Expert Crowdsourcing with Flash Teams
Homework due:  Project revised prototypes
8 Nov 16
Topic:  Collective intelligence and sensemaking

Special guest:  Anita Woolley

  • Engel, Woolley, et al. (2014) Reading the Mind in the Eyes or Reading between the Lines? Theory of Mind Predicts Collective Intelligence Equally Well Online and Face-To-Face
  • Andre et al. (2014) Crowd synthesis: extracting categories and clusters from complex data
  • Chilton et al. (2014) Frenzy: collaborative data organization for creating conference sessions 
  • Kittur et al. (2014) Standing on the schemas of giants: socially augmented information foraging
  • Willett et al. (2013) Identifying Redundancy and Exposing Provenance in Crowdsourced Data Analysis

Homework due:  Project study details
9 Nov 23 (thanksgiving week)
Topic:  Incentives and Microvolunteering

Special guest:  Kate Starbird

    • Starbird and Palen (2011) Voluntweeters: Self-Organizing by Digital Volunteers in Times of Crisis
    • Brady et al. (2015) Gauging Receptiveness to Social Microvolunteering
    • Mason and Watts (2009) Financial Incentives and the Performance of Crowds
    • Mitra et al. (2015) Comparing Person- and Process-centric Strategies for Obtaining Quality Data on Amazon Mechanical Turk
    • Thebault-Spieker et al. (2015) Avoiding the South Side and the Suburbs: The Geography of Mobile Crowdsourcing Markets

    Homework due:  Project data analysis
    10 Nov 30
    Topic:  Games

    Special guest:  Adrien Treuille

    Watch: Adrien Treuille on Science games

    • Cooper, Khatib, Treuille, et al. (2010) Predicting protein structures with a multiplayer online game
    • Lee, Treuille, et al., (2014) RNA Design Rules from a Massive Open Laboratory
    • Krause and Kizilcec (2015) To Play or not to Play: Interactions between Response Quality and Task Complexity in Games and Paid Crowdsourcing
    • Tuite et al. (2011) PhotoCity: Training Experts at Large-scale Image Acquisition through a Competitive Game

    Homework due:  Project drafts for presentation and paper

    11 Dec 7 (finals week)