Incites gained from the last class discussion:
1. Multiple different wiki pages can be used - search for premade quizzes on subjects on teaching resource websites.
2. Mechanical Turk can impose restrictions on who is allowed to complete your request, and you can pay them based on their quiz responses to give them incentive to actually answer correctly and put some thought into it rather than just skipping through.
3. Measuring the timing accurately is difficult but important to both judge if the experimenters are actually performing the task and to compare against treatments.
Unfortunately, I have been sick for the past week+, so not much progress has been made. I would like to outline here my plans of the next week or so.
Changes I need to implement:
1. Modify the php to accurately keep track of sessions to record the correct time of completion for each step of the process.
2. Add on 2-3 more wikipedia pages of different subjects to add to the single page I have now. Possibly write a script to automate the conversion of wikipage into a quiz page. Match this with quizzes found for known subjects online so I don't make up the quiz questions myself.
3. Add another treatment with a pre-quiz given instead of or in addition to the quiz in the middle of the article.
4. Possibly add in a javascript timer to the pages that visually imposes some time limit for reading and completing quizzes, and use the time allotted for reading and for quiz completion as a variable in the trials.
In general if you incentives users to do something, you will see significantly different results (some times as high as 10x, but usually around 1.5-2x) than if you did not incentivize users. I am not sure if you can do this with Turk, but you can add some game mechanics something like, "answer 10 questions with high quality and get bonus points".
ReplyDeleteFor the content, I would make sure you have enough different types of wikipedia pages because the users may be biased towards different types of content. i.e. maybe a wikipedia page on a scientific explanation may not be very interesting to a turker and thus they take the task with a grain of salt and try to blow through the quiz.
A pre-quiz treatment is a great idea! I would like to see the results of that!