Experimental Data

This week I first collected 2 extra wikipedia article subjects and created questions for them. I found that finding pre-made questions online on teacher resource cites did not provide enough overlap for subjects that had concise wikipedia article pages, so I made the questions myself again.

Three articles used in trials:

Gravity Well
Tardigrade (Water Bear)
Tercio

Some articles had certain sections removed to be roughly the same length, about 3-4 small sections.

I conducted 4 trials this week on Amazon's Mechanical Turk with each combination of the 2 variables:

1. quizzes within or not within
2. random assignment of topic or choice of topic

The results I got actually were the opposite of the pilot experiment, which is discouraging, but I hope there should still be something useful from the data I collected once I analyze it more. The graph below shows the percentage of total correct questions for each of the treatments.

  The figure above shows the distribution of the average time spent reading the article (and answering questions within the article for those treatments), and average time spent on the final quiz, and then average total time spent. It is interesting that the within quiz treatments show a higher total time, even though they have worse performance on the final quiz. Also, the quizzes within trials have more of their time spent reading the article then doing the final quiz, which may provide some insight into why they scored worse than trials with no quizzes within. 

List of things to do

Here is a list of things that must be done, in the order I intend to complete it and date by which they must be done:

Migrate code to my own server to avoid using stanford web space (fix timing issue) - 5/14
Find 2 more combinations of quizzes and subjects for a Wikipedia page to my testing trials - 5/16
Convert those quiz/subject pages into web pages in my work flow to test users with - 5/16
Add in a new page with a pre-quiz for a new treatment of trials for each subject - 5/17
Start running a larger experiment with 20-30 users per treatment on AMT - 5/18
Process the results for the trials and present to the class - 5/20
Depending on if there is a large variation in time taken to complete the reading and quizzes in each treatment, I may add in another set of treatment pages using visibly prominent javascript timers with a set small amount of time on either the reading portion or quiz portion or both. This would be purely aesthetic, but I would be testing if the artificial deadline effects performance on time and correctness. Time to add the timer - 5/24
Run another set of AMT trials with the timer present - 5/25
Process the results of the new set of trials and test the effect of the visible timer - 5/27

Update

Incites gained from the last class discussion:
1. Multiple different wiki pages can be used - search for premade quizzes on subjects on teaching resource websites.
2. Mechanical Turk can impose restrictions on who is allowed to complete your request, and you can pay them based on their quiz responses to give them incentive to actually answer correctly and put some thought into it rather than just skipping through.
3. Measuring the timing accurately is difficult but important to both judge if the experimenters are actually performing the task and to compare against treatments.

Unfortunately, I have been sick for the past week+, so not much progress has been made. I would like to outline here my plans of the next week or so.

Changes I need to implement:
1. Modify the php to accurately keep track of sessions to record the correct time of completion for each step of the process.
2.  Add on 2-3 more wikipedia pages of different subjects to add to the single page I have now. Possibly write a script to automate the conversion of wikipage into a quiz page. Match this with quizzes found for known subjects online so I don't make up the quiz questions myself.
3. Add another treatment with a pre-quiz given instead of or in addition to the quiz in the middle of the article.
4. Possibly add in a javascript timer to the pages that visually imposes some time limit for reading and completing quizzes, and use the time allotted for reading and for quiz completion as a variable in the trials.