Assignments
For Wed Sep 23: Learning about the Amazon Mechanical Turk
- Read these two blog entries:
- Hamm (blog) Can You Actually Earn Reasonable Money from Mechanical Turk? (Also read through visitor comments at the end.)
- Ipeirotis (blog) Mechanical Turk: The Demographics (also see links at bottom if interested)
- Contest:
- Sign up for a worker account at the Amazon Mechanical Turk if you don't already have one.
- Spend exactly one hour trying to make as much money as possible.
- We'll see who earns the most.
- Take rough notes on what you did.
- Email Adam (af@cs.princeton.edu)
- Three points for discussion from the blogs and your experience.
- Your notes from Mechanical Turk
For Wed Sep 30:
- Reading about web-based studies:
- Gosling, Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires
- Kraut, Psychological research online: Report of Board of Scientific Affairs' advisory group on the conduct of research on the internet
- Nosek, Banaji, and Greenwald, E-research: Ethics, security, design and control in psychological research on the internet
- Spend 10 minutes perusing this web site looking at a few online studies.
- Bring a for a 4-minute discussion a proposed experiment that you think would be interesting and that would be feasible to run on the Mechanical Turk.
For Mon Oct 5:
- Do the NIH Training course prior to discussion with IRB
For Wed Oct 7:
- Be prepared to discuss each proposed experiment for about 10 minutes.
For Mon Oct 12:
- Read this paper:
- Kittur et al, Crowdsourcing user studies with Mechanical Turk
- Read the first two sections and skim remainder in this paper:
For Wed Oct 13:
- Further discussion of experimental designs.
For Mon Oct 19:
- von Ahn and Dabbish, Designing games with a purpose
- Hacker and von Ahn, Matchin: eliciting user preferences with an online game
Syllabus
- Part 1: Readings (Sep-Oct)
- Part 2: Project Planning (Oct-Nov)
- Part 3: Pilot Testing (Nov)
- Part 4: Experiments (Dec)
- Part 5: Analysis (Dec-Jan)
- Part 6: Reports (Jan)
This schedule is preliminary and will be filled in as the seminar crystallizes:
| Week 1 | Sep 21,23 | Part 1: Introduction, case studies |
| Week 2 | Sep 28,30 | Part 1: Case studies (Note religious holiday 9/28 - no class) |
| Week 3 | Oct 5,7 | Part 1: Guidelines for human subjects experiments / IRB |
| Week 4 | Oct 12,14 | Part 1: Games with a purpose |
| Week 5 | Oct 19,21 | Part 1: Experimental planning |
| Week 6 | Oct 26,28 | Part 2: Project proposals and critique |
| --- | Nov 2,4 | FALL BREAK |
| Week 7 | Nov 9,11 | Pilot testing... |
| Week 8 | Nov 16,18 | We'll see... |
| Week 9 | Nov 23,25 | (Note T-day 11/25) |
| Week 10 | Nov 30,Dec 2 | |
| Week 11 | Dec 7,9 | |
| Week 12 | Dec 14,16 | |
| --- | January XX | Final presentations |
Sample Readings (in no logical order yet)
- Mason and Watts, Financial incentives and the "performance of crowds"
- Winteram, Guide to Experiments on Amazon’s Mechanical Turk
- Grossklags, Experimental Economics and Experimental Computer Science: A Survey
- Kittur et al, Crowdsourcing user studies with Mechanical Turk
- Skitka and Sargis, The internet as psychological laboratory
- Watts, A twenty-first century science
- Gosling, Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires
- Kraut, Psychological research online: Report of Board of Scientific Affairs' advisory group on the conduct of research on the internet
- Nosek, Banaji, and Greenwald, E-research: Ethics, security, design and control in psychological research on the internet
- von Ahn and Dabbish, designing games with a purpose
- Hacker and von Ahn, Matchin: eliciting user preferences with an online game
- Cole et al, How Well Do Line Drawings Depict Shape?
- http://blog.doloreslabs.com/2008/09/amt-fast-cheap-good-machine-learning/
- Note: list is in progress...