A couple of months ago, the folks at codecademy were nice enough to respond to a complimentary e-mail I’d sent them by writing something nice back about me and publishing it on their codecademy.com/stories page.

cadecademic

It was great to get this sort of coverage on a site I think is fantastic (despite the picture of me with a rather supercilious looking one-year old on my back). The only problem was that I didn’t have a working example of code available for them to link to. The JavaScript experiments I had previously coded had run their course, garnering approximately 200 participants each, and had been taken offline, leaving the lab experiment page with nothing for people to try their hand at.

That changed today.  I have a very simple new experiment for people to try their hand at, which can be accessed via the experiments (online) link on the right, or directly, here: http://www.st-andrews.ac.uk/~oclab/memorywords/.

I’m afraid it doesn’t quite do justice to the “neuroscientist discovers javascript” headline on the codecademy stories page, but it’s something.

The lab’s first Javascript experiment has been online for about 3 weeks now, and has amassed close to 200 participants. It’s been a great experience discovering that the benefits of online testing (60+ participants a week, many of them run while I’m asleep!) easily outweight the costs (the time expended learning Javascript and coding all the fiddly bits, particularly the informed consent procedures and performance-appropriate feedback).

On top of the study completion data that’s obvious from the 7 KB csv file that each happily-debriefed participant leaves behind, the Google Analytics code embedded in each page of the experiment provides further opportunity to explore participation data.

Attrition

As the experiment structure is entirely linear, it’s possible to track the loss of participants from each page to the next.

Study Attrition

The major point of attrition is between the Participant Information Page and the Consent Form – not surprising given quite how text-heavy the first page was, and how ‘scary’ headings like “Are there any potential risks to taking part?” make the study sound. The content of that first page is entirely driven by the Informed Consent requirements of the University of St Andrews, but the huge attrition rate here has prompted a bit of a redesign in the next follow-up study.

Browser

New Visits by Browser

Other information useful for the design of future studies has been the browser data. As might be expected, Firefox and its relatives are the dominant browsers, with Chrome a distant second and Internet Explorer lagging far behind. Implementing fancy HTML5 code that won’t work in Firefox is therefore a bad idea. On top of that, despite how tablet- and phone-friendly the experiment was, very few people used this sort of device to complete the study – it’s probably a waste of time optimising the site specifically for devices like iPads.

Study Completions by Browser
Study Completions by Browser

Curiously enough, when the data for study completions are explored by browser, the three major platforms start to level up. Chrome, Firefox and IE all yield similar completion statistics, suggesting that IE browsers are far more likely to follow through and complete the study once they visit the site. I’m speculating here, but I suspect that this has something to do with a) this being a memory study and b) IE being used by an older demographic of internet user who may be interested in how they perform. Of the three major browsers, Firefox users have the worst completion rate.

Location

Another consideration with word-based experiments is the location of participants. This could impact on the choice of words used in future studies (American or UK spellings) and could be considered important by those who are keen to exclude those who don’t speak English as their first language. Finer grained information about participants’ first languages is something we got from participant self-reports in the demographic questionnaire, but the table of new visits and study completions is still rather interesting.

New Visits and Study Completions by Country

Once again, there are few surprises here, with the US dominating the new visits list, though one new visit from a UK- or India-based browser is more likely to lead to a study completion. A solid argument for using North American spellings and words could also be made from these data.

Source of Traffic

The most important thing to do to make potential participants aware of an online psychology study is to advertise it. But where?

Study Completions by Source

While getting the study listed on stumbleupon was a real coup, it didn’t lead to very many study completions (a measly 2.5%). That’s not surprising – the study doesn’t capture the attention from page 1 and doesn’t have much in the way of internet meme-factor. That is, of course, something that we should be rectifying in future studies if we want them to go viral, but it’s tough to do within the rigid constraints of the informed consent pages that must precede the study itself.

The most fruitful source of participants was the psych.hanover.edu Psychological Research on the Net page. It was much more successful at attracting visits and study completions than facebook, the best of the social networks, and the other online experiment listing sites on which we advertised the study (onlineresearch.co.uk and http://www.socialpsychology.org/expts.htm). What’s more, there has been a sustained stream of visitors from the psych.hanover.edu page that hasn’t tailed off as the study has been displaced from the top of the Recently Added Studies list.

These statistics, surprised me more than any other.  I assumed that social networking, not a dedicated experiment listing page, would be how people would find the study. But in retrospect, it all makes sense. There is clearly a large number of people out there who want to do online psychology studies, and what better way to find them than to use a directory that lists hundreds of them.  If there’s one place you should advertise your online studies, it’s psych.hanover.edu.

Points and badges

For the past week or so, I have been working my way through Codecademy’s JavaScript tutorials. I can’t recommend them highly enough.

As things stand, I have a full house of 480 points and 35 badges and, as the Codecademy creators would undoubtedly hope, I am rather satisfied with the JavaScript proficiency I have attained. ‘Attained’ is probably the wrong word to use though. Being a self-taught Matlab hacker, I have found most of my coding know-how has translated fairly well into Javascript. A few concepts (recursion in particular) have presented me with some difficulty, but the overall experience has been more like learning a new coding dialect  than a new language altogether. I haven’t attained a proficiency, so much as uncovered a hidden one.

Which brings me to why I sought out Codecademy in the first place (thanks to @m_wall for the twitter-solicited tip-off) – I am preparing to teach Psychology undergrads how to code. From 2012/2013 onwards, my academic life is going to be a little more ‘balanced’. As well as the research, admin and small-group teaching I currently enjoy, I’m also going to be doing some large-group teaching. Although I have plenty to say to undergraduates on cognitive neuroscience and cognitive psychology, I think giving them some coding skills will actually be much more useful to most. As my experience with Codecademy has recently reinforced to me, coding basics are the fundamental building-blocks of programming in any language. They will hold you in good stead whatever dialect you end up speaking to your computer in. What’s more, they will hold you in good stead whatever you end up doing, as long as it involves a computer: coding is the most versatile of transferable skills to be imparting to psychology graduates who (rightly) believe they are leaving university with the most versatile of degrees.

With all this in mind, one of Codecademy’s limitations is the difficulty with which its students can translate their new-found JavaScript skills into useful ‘stuff’ implemented outside the Codecademy editor. As Audrey Watters points out, there is barely any acknowledgement within the Codecademy tutorials that the goal of all of these points and badges is to encourage you to write interactive web contact in an IDE. Indeed, last night when I thought about how I would use JavaScript to administer online  memory experiments, I had to do a lot more reading. This could all be about to change though. If the latest Code Year class on HTML is anything to go by, the folks at Codecademy are mindful of this limitation, and are attempting to remedy it.

It’s just a shame that the html integration has come so late in the Code Year (yes, I say this with full awareness that we’re only on week 13).  If the HTML-Javascript confluence had come a little further upstream, I think there probably would have been a fledgling memory experiment linked to from this blogpost!