Here is a list of things that must be done, in the order I intend to complete it and date by which they must be done:
Migrate code to my own server to avoid using stanford web space (fix timing issue) - 5/14
Find 2 more combinations of quizzes and subjects for a Wikipedia page to my testing trials - 5/16
Convert those quiz/subject pages into web pages in my work flow to test users with - 5/16
Add in a new page with a pre-quiz for a new treatment of trials for each subject - 5/17
Start running a larger experiment with 20-30 users per treatment on AMT - 5/18
Process the results for the trials and present to the class - 5/20
Depending on if there is a large variation in time taken to complete the reading and quizzes in each treatment, I may add in another set of treatment pages using visibly prominent javascript timers with a set small amount of time on either the reading portion or quiz portion or both. This would be purely aesthetic, but I would be testing if the artificial deadline effects performance on time and correctness. Time to add the timer - 5/24
Run another set of AMT trials with the timer present - 5/25
Process the results of the new set of trials and test the effect of the visible timer - 5/27
I'm excited to see how the AMT results turn out, hopefully I'll get to comment on that further tomorrow in class. I think we learned last week that AMT testing is a tricky art, especially when you're expecting to have people display a non-trivial level of mental engagement (as opposed to something simple like image tagging). You talked a lot last week about not accidentally incentivizing people to google the answers to questions by giving them a bonus but it just occurred to me that one method to prevent this would be to write questions that can't be googled easily. This is kind of tough since Wikipedia itself has a high google ranking and writing a question that can be gleaned from a Wikipedia article kind seems counter to that goal; however if you phrase the question without using keywords from the article, then maybe you can get away with it.
ReplyDeleteAnother strategy you might try, assuming you get this opportunity, is to conduct your analysis on participants who did many tasks for you (not sure whether or not you're allowing this). When I ran AMT experiments back in the day, the people who did many tasks were generally people who liked doing them, and overall did a more faithful and better job. The people who do multiple quizzes are likely those who like taking the quizzes and will take them genuinely. However depending on your experimental design this may not be feasible.