Homework 1: Be a Worker

For this assignment, you will become a crowd worker on a web site like Mechanical Turk and see how much you can earn in one hour. You will then reflect on this experience via a set of short-answer questions.

The goal of the assignment is to gain experience as a crowdworker and to see how much you can earn in one hour. These instructions below focus on Amazon Mechanical Turk. If you do not have a social security number or a credit card, you might not be able to use it. In that case, pick another crowdsourcing site (see from the Alternative Crowdsourcing Sites listed below).

You will turn in your assignment using this form.

Amazon Mechanical Turk

Sign up

First, head over to Amazon Mechanical Turk (MTurk) and sign up for an account. At this point, you’re there to “make money by working on HITs,” so click that side of the page. Later in the course you’ll be a requester, but one of the best ways to become good at crowdsourcing is to experience what it’s like as a worker.

The navigation on MTurk is notoriously confusing. One way to force new user registration is to click the “Sign In” link at the top of the page. During the signup procedure, you may have to associate your account with a payment number (SSN or the like). This is sometimes only necessary when you want to actually withdraw your funds, but you might find that it asks you at the beginning. The general rule is that Mechanical Turk will not report these earnings until you’ve made at least $600 (a limit set by the IRS). If you have concerns about this, please let the instructor know. One approach is to use a friend’s account to complete the work.

Finding HITs (Human-Intelligence tasks)

Once you’re registered, the next step is to go out and find work to do. The jobs (HITs) available to you will be displayed if you click the “HITs” tab on the Mechanical Turk page.

The screen shot shows there are 232,666 HITs available. You can sort the available jobs on various dimensions: the price they pay, how recently they were posted, time allotted, etc.

Not all of the jobs will be available to you. Some require you to have a specific “qualification,” which is assigned by a requester. Sometimes you can take a test to gain a qualification, other times the qualification is given to you because you participated in an earlier HIT of the requester, e.g. you participated in one survey, which qualified you to participate in the next one. You may also not qualify for some jobs because you haven’t yet completed enough HITs for other people, the acceptance rate of your HITs is too low, or you’re not connecting from the right region.

Regardless, if you click the “HITs Available To You” only those HITs that you can actually work on will be displayed in the list.

Accepting and Completing HITs

When you find a HIT that looks interesting in the big list, click on it to read more about what you would do if you accepted the work. If it looks reasonable, then click the “Accept HIT” button. Only after doing that, you can start working on the HIT.

HITs vary a lot in what they ask you do, although common task types involve writing, visual classification and labeling, and audio transcription. This makes sense because these are tasks that are still tricky to get computers to do reliably.

When you’re done with the work as described by the requester, submit the HIT. It will then be reviewed by the requester and either accepted or rejected. If it is accepted, then you will get the money promised; if it’s rejected, then you won’t get the money. Your ratio of accepted HITs stays with you as your “reputation.” Requesters are not required to review your HIT quickly, although workers obviously prefer those who review their HITs quickly. Of course, that information isn’t made readily available to workers, but we’ll get into that more later.

Your Assignment:

As soon as you find the first HIT you want to perform, keep track of the time and see how much you can earn in one hour. This should allow you to calculate a rough hourly wage. Check out Jeff Bigham's wonderful post on his experience being a worker for a half day.

Read the questions below and think about them as you work. Provide short answers to each in a Google Document that you'll then share with the instructor using this form.

1. What kinds of tasks did you preform as part of this assignment?
2. What was your HIT acceptance rate? 0-100%
3. What hourly wage did you earn? What was your hourly wage if you include the time you spent searching for tasks as well?
4. How much money or how many tasks did you earn overall? (include a screenshot of your payments page showing the final amount)
5. List one or more problems that you ran into while earning your hour of wages.
6. What effect(s) do you think these problems had on the work you submitted?
7. What effect(s) do you think these problems have on the “average” Mechanical Turk worker?
8. What ideas do you have on how to improve Mechanical Turk based on your experience?

Alternative Crowdsourcing Sites

Many crowdsourcing platforms exist other than Mechanical Turk (thanks to Jeff Bigham, and by proxy, Chris Callison Burch, for a list). I specifically picked platforms where people perform micro-work for pay:

CrowdFlower, Freelancer, leadGenius, Microworkers, ODesk, TaskRabbit, topcoder.

If you cannot preform the assignment above on Amazon Mechanical Turk, you can instead pick one of these platforms, preferably one you have not used before, and do an hour of work there. Your assignment will be to answer the following questions and to provide short answers to each in a Google Document that you'll then share with the instructor using this form.

1. What platform and what kinds of tasks did you preform as part of this assignment?
2. What hourly wage did you earn? Or what did you get as a reward (if not money)?
3. List one or more problems that you ran into while working on tasks for an hour.
4. What effect(s) do you think these problems had on the work you submitted?
5. What effect(s) do you think these problems have on the “average” worker on this platform?
6. What ideas do you have on how to improve the chosen platform based on your experience?

For reference, here's the much longer list of "crowd" platforms that we will not be using for this assignment:

538, 99Designs, airbnb, ALEC, ALICE, Amazon reviews/product recommendations, benefunder, Book Country, Buy Amazon Reviews, change.org, CloudFactory, coursera, Crowdee (German), CrowdMed, EatWith, eBay buyer/seller ratings, edX, Field Agent, Flattr, Foursquare, Freebase, Iceland’s Crowdsourced Constitution, Idibon, IndieGoGo, Innocentive, Instacart, iStockPhoto, Intrade, Iowa Election Markets, Kaggle, Kickstarter, Kiva, Lending Club, Lyft, Microsoft Prediction Lab, MTurk List, Netflix recommendations, OpenStreetMap, Parchment PatientsLikeMe, Prosper, PublicStuff, Quirky, Quora, Rotten Tomatoes, Samasource, Silk Road, Stackexchange, Stackoverflow , Threadless, Thumbtack, TurkOpticon, Uber, Udacity, Ushahidi, Waze, We the People, Wikipedia, XPRIZE, Yelp.