This is a guest post from Radka Jersakova (@RadkaJersakova), who did her undergraduate degree in my lab and is now working on her PhD at Leeds University and the Université de Bourgogne in Dijon. Radka has embraced online experimentation and has run many hundreds of participants through an impressive number of experiments coded in Javascript.

Akira

Onscreen Experiments

Recently, Crump, McDonnell and Gureckis (2013) replicated the results of a number of classic cognitive behavioral tasks, such as the Stroop task, using experiments conducted online. They demonstrated that, despite what some people fear, online testing can be as reliable as lab-based testing. Additionally, online testing can be extremely fast and efficient in a way that lab-based testing cannot. I have now completed my 7th online experiment as well as having helped others in creating and advertising theirs.  This post is a review of things I have learned in the process. It summarises what I did not know but now wish I had when I was planning my first study and answers some questions I got asked by others along the way.

 

CREATING ONLINE EXPERIMENTS

In terms of conducting online experiments, the best method remains programming as it is by far the most flexible approach. As someone who has learned programming on my own from free online courses, I can confirm that this is not as difficult as some people think it to be and it really is quite fun (for some tips on where to get started this TED blog post is quite useful.). At the same time, many people do not know how to code and do not have the time to learn. The good news is that for many experiments, the current survey software available online remains flexible enough to create large number of experiments although the potential complexity is naturally limited. My favorite is Qualtrics as even the free version allows a fair amount of functionality and number of trials.

 

FINDING PARTICIPANTS

A major advantage of the Internet is that one can reach many different communities. With online testing, one can reach participants who are simply interested in psychology experiments and volunteering in a way that is preferable to testing psychology undergraduates who are coerced into participating for course credit. Once you have an experiment to advertise, the challenge is to find the easiest route by which to reach these people.

There are many websites that focus directly on advertising online experiments. The one I have found the most useful is the Psychological Research on the Net website administered by John H. Krantz. Alternatively, the In-Mind magazine has a page where they post online experiments, which they also share on their Facebook and Twitter account.  Other websites that host links to online studies are the Social Psychology Network  and Online Psychology Research.

The most powerful way for a single individual to reach participants is, quite unsurprisingly, social media. Once a few people start sharing the link, the interest can spread very quickly. The simplest thing to do is to post your study on your Facebook page or Twitter account. Something I haven’t tried yet but that might be worth exploring is finding pages on Facebook or hashtags on Twitter that might relate to the topic of the experiment or psychology in general and post the link to the experiment there. One of the biggest successes for me though, remains reddit. Reddit has a very strong community and people spend time their because they are actively searching for new information and interesting projects. There are a number of subreddits that are specific to psychology so yet again, visited by people interested in these particular topics. To give a few examples: psychology; cognitive science; psych science; music and cognition; mathematical psychology and the list goes on! There is even a subreddit specific to finding participants to complete surveys and experiments simply called Sample Size.

The last resource I have tried a number of times is using more general advertising sites such as craigslist. There is always a ‘volunteers’ section, which is visited by people looking to volunteer for a project of some sort. In that sense it can be a good place to reach participants and the sample will be fairly diverse. This for me has never been as successful as using social media but a few times it has worked fairly well.

 

USEFUL CHECKPOINTS

The most commonly heard argument against online testing is the lack of control. Really what this means is that data collected online might include more noise, making it easier to miss existing effects, than traditional lab-based experiments. As already mentioned, Crump et al. (2013) replicated a number of classic tasks online suggesting that this might not be as big a worry as it at first seems to be. The range of tasks they have chosen demonstrates nicely that the same results can be obtained in the lab as well as on the Internet. Nevertheless, there are a number of ways one can track participants’ behavior to determine whether sufficient attention was given to the experiment. The simplest way is to measure the time participants took to complete the study. If you are using existing survey software, this information is usually automatically provided. If you are programming the study yourself, requesting a timestamp for when the study begins and for when it ends is an easy way to track the same kind of information. If participants are abnormally slow (or fast) in completing a task, then one might have sufficient reasons to exclude the data.

One of the biggest problems I have encountered is a participant completing one part of the task (e.g. a recognition test) but not completing as faithfully another part of the same experiment (e.g. free report descriptions of particular memory experiences from her daily life). While due to ethics we were not allowed to force participants to respond to any question, I have found that simply asking if they are sure they want to proceed, in case that they haven’t filled out all the questions on a page, increased report rates dramatically. As such it can be useful to provide such pointers along the way to make sure participants answer all questions without forcing them to do so.

Crump et al. (2013) also point out from their experiences of online testing that it can be useful to include some questions about the study instructions.  One could simply ask participants to describe briefly what it is that they are expected to do in the experiment. This way one has data against which to check whether participants understood the instructions and completed the task as anticipated. It will probably also help to ensure that participants pay close attention to the instructions. This is particularly useful if the task is fairly complex.

 

DEALING WITH DROP OUTS

A big disadvantage of online testing can be dropout rates. This isn’t something I have tested in any formal way but it does seem that there is at least some relationship between the length of the study and dropout rates. This means that online testing is definitely most suitable to studies, which are up to 15 or 20 minutes in length to complete and this might be something to consider. It is also certain that tasks, which are more engaging, will have lower dropout rates. A good incentive I have found is to give participants at the end of an experiment a breakdown of their performance. I have had many participants confirm that they really enjoyed the feedback on how they performed on the memory task. Such feedback is a simple but efficient way to increase participation and decrease dropout rates.

The second worry is participants’ dropping out in the middle of an experiment and then restarting it. It is not something that would be common but it could happen. One way to deal with this is to ask participants to provide at the beginning of the study some code that should be unique to each participant, anonymous and yet always constant. An example is asking participants to create a code consisting of their day and month of birth and ending with their mother’s maiden initials. This is hardly a novel idea, I have participated in experiments, which asked for such information to create participant IDs that allowed to link responses across a number of experimental sessions. The idea is to find some combination of numbers and letters that should never (or rarely) be the same for two participants but that remains the same for any one participant, whenever he is asked. Once in the data-analysis stage, one can simply exclude files that contain repetitions of the same code.

Once the study is up and running, other than finding suitable places to advertise it at, one can leave it and focus on other things until the data has been collected. It is possible to reach large samples quickly and these samples are often more diverse than your classic psychology undergraduate population. There is a certain degree of luck involved but I have in the past managed to collect data for well over 100 participants in a single day. That is not to say that all studies are suitable to online testing but it is definitely a resource well worth exploring.

I often bang on about how useful twitter is for crowd-sourcing a research community. Today I was reminded of quite brilliant the people on twitter can be at helping to overcome an ‘I don’t know where to start’-type information problem.

I’m currently helping to design an fMRI study which could benefit considerably from the application of multivoxel pattern analysis (MVPA). Having no practical experience with MVPA means I’m trying to figure out what I need to do to make the MVPA bit of the study a success. After a few hours of searching, I have come across and read a number of broad theoretical methods papers, but nothing that gives me the confidence that anything I come up with will be viableOf course, there’s no right way of designing a study, but there are a tonne of wrong ways, and I definitely want to avoid those.

So, I turned to twitter:

Relays and Retweets from @hugospiers, @zarinahagnew and @neuroconscience led to the following tweets coming my way (stripped of @s for ease of reading… kind of).

Sure, I could have come up with as many articles to read by typing “MVPA” into Google Scholar (as I have done in the past), but the best thing about my twitter-sourced reading list is that I’m confident it’s pitched at the right level.

I’m humbled by how generous people are with their time, and glad so many friendly academics are on twitter. I hope collegiality and friendliness like this encourages many more to join our ranks.

On Monday I gave a talk on how internet tools can be used to make the job of being an academic a little easier.  I had given a very short version of the talk to faculty in the department over a year ago, but this time I was given an hour in a forum for  early career researchers, PhD students and postdocs.  The subject of twitter, covered early on in the talk, aroused a lot of interest, probably because I got very animated about its benefits for those in the early stages of their careers.

To provide a little context for my enthusiasm, it probably helps to know a few things about me, about my situation, and about my recent experiences.

  1. I am an introvert.  Despite my best (and occasionally successful) efforts to project a different image, I do not find talking to people I don’t know very enjoyable.
  2. I am an early career cognitive neuroscientist keen to build my own research programme and develop links with other researchers.
  3. Last month I attended the Society for Neuroscience conference, at which I attended the best conference social I have ever attended.

Given the received wisdom that people in my position ought to be networking, I often drag myself kicking and screaming to conference socials. The result tends to be a lot of standing around on my own drinking beer, which gives me something to do, but which I could do much more comfortably with one or two people I know well.  The major problem at these events is not my nature, or my status as an early career researcher, but the fact that the people I have imagined myself talking to usually don’t know who I am.  Conversation is therefore awkward, one-sided and introductory.  Once the niceties have dried up, and the level of accumulated conversational silence edges into awkward territory I invariably finish my drink and bugger off to get another one, ending the misery for all involved.  This is probably a universal experience for those starting out in academia, though thankfully it is happening less and less to me as I build something of network of real friends who attend the same conferences as me.  But as a PhD student and postdoc, the experience was excruciating.

I had a totally different experience when I attended the SfN Banter tweetup*.  The event, organised by @doc_becca and @neuropolarbear, was a social for neuroscientists who use twitter and changed my view of conference socials.  They do not have to be endured, even by those doing PhDs and postdocs. They can be enjoyed.

I was excited about going and that excitement didn’t leave me feeling shortchanged by the time I left.  I spoke (actually spoke!) to everyone I wanted to speak to.  Moreover, I had good conversations with people to whom I was speaking for the first time. The reason is fairly obvious – twitter allowed us to build on a body of shared (or at least assumed) knowledge. I follow people, they follow me,  I reply to or retweet their tweets, they do the same – and this is all before we’ve introduced ourselves. When I finally meet someone with whom I have such a history of communication, introducing myself is the least awkward thing I can do. The barriers to conversation are removed**.

Sure, this pattern followed for most interactions at the tweetup because we were all there to do exactly that.  Would the experience be the same at the ‘fMRI social’? No.  But, I don’t think that matters.  If I could have had one of those conference social experience during my time as a PhD student, it would have given me an idea of what I might have to look forward to from conferences if I stuck at it.  Light at the end of the tunnel, a pot of gold at the end of the rainbow, a variable-ratio schedule-determined stimulation of the limbic system following an umpteenth lever press.

It will take a while (there’s no point joining in September 2013 and expecting great things at the SfN tweetup in San Diego), and it’s probably not the primary reason to join twitter (see  Dorothy Bishop’s blog and Tom Hartley’s blog for far more comprehensive discussions  of how and why you should join), but it’s another reason, and it’s one that could make you feel good about your role in academia.  It’s worth a shot.

 

* tw(itter) (m)eetup, see?

** What you do afterwards is up to you.  I still had some awkward interactions, but I think that’s probably down to me (see context point 1).

Academics are a diverse bunch. Those in my department of just under 40 lecturers and teaching fellows span an estimated 40-year age-range, at least 10 different nationalities and the full spectrum of technological competence. Some were introduced to e-mail in their teens, others in their 40s – all of us use it as the primary mode of communicating with students. A student making e-mail contact with an academic therefore needs to make a few allowances for the recipient.

Below are a few thing to watch out for when e-mailing academics for the first time.

1) Use your university-provided e-mail account.
It’s your ‘work’ e-mail, so set it up correctly (make sure you change the account setting to display your sender name etc.) and use it for work-related correspondence. I don’t know what to think when I get an e-mail from fluffyspuggle@freemail.com.

2) Use an appropriate greeting.
“Dear <title> <surname>,”  will never let you down. Yes, there are some who believe that it’s too formal for the medium, but you’re e-mailing someone about whom you know very little.  They might like being addressed this formally, they might not care, but they certainly won’t think any worse of you doing it. In my view, the less formal “Hello <title> <surname>” is equally appropriate, though straying into “Hi” or “Hey” gets risky – I don’t mind “Hi” but I really hate receiving e-mails that open with “Hey” from someone I don’t know. The total absence of a greeting offends me.

3) Don’t get your academic’s gender wrong.
I have a name that ends in the letter A.  As a result of a Western European naming heuristic, people often assume that I am female. This assumption is fine from telemarketers and those who want to send me trial subscriptions to Red magazine, but not students.  If you’re not a professional cold-caller, getting your e-mail recipient’s gender wrong just suggests that you’re lazy. If you’ve haven’t heard the name before, Google it.

4) Don’t get your academic’s title wrong.
Is the recipient’s gender even relevant to your e-mail? If you’re e-mailing an academic with “Dear Mr/Mrs/Ms”,  you’re already taking an etiquette risk. Some academics get very tetchy about their title (Dr, Professor). If in doubt, just Google it. The search term <first name> <surname> <institution> usually does the trick.  If they have PhD or Dr on their page, use “Dr”.  If they have Prof or Professor on their page, use “Prof”.  If none of these apply, use a gender-appropriate title.

5) Don’t make inappropriate gender assumptions about your academic.
Women get PhDs.  Women also get made Professor. I don’t know anyone who would argue with these facts, but coffee-room conversation suggests that female academics have to deal with a greater number of inappropriately-written e-mails than their male colleagues.

6) Use appropriate language to communicate.
Communicating over e-mail isn’t like communicating over twitter or facebook (at least your academic probably doesn’t think so). Yes, keep it concise, but not so concise that you have to omit vowels.  Smileys in your first e-mail may make your recipient >:  And be polite… please?

7) Sign off.
Something more formal than “Cheers” usually does the trick. “Regards,” “Best,” and “Thanks,” are all fine by me, though I would err on the side of “Regards,” if you started with “Dear”.  And do sign off with your name (first or first and surname) and no kisses. Just friends, ok?

These recommendations become a little looser as you build a history of communication with your academic.  In second e-mails you can probably ditch the <title> <surname> business in favour of the academic’s sign-off name or something appropriate to the tone of their response (or slightly more formal).  For instance, if your academic signs off with  “Best, Akira” or “Thanks, A” feel free to follow-up with “Dear Akira,” or “Hello Akira,”.  If your academic responds with “Hey,” they have absolutely no reason to get offended by a similar reply from you.

Just make sure you get that first e-mail right. It does matter.  You’re going to be relying on tutors, project supervisors and lecturers to give you guidance and opportunities during and after university. Making a bad first-impression isn’t insurmountable, but it’s definitely something you can do without.

Having deleted my facebook account nearly two years ago, as I activated a Google+ account this week I was wary of repeating previous mistakes.  Back in 2009 I had decided that I wasn’t getting as much out of facebook as I was putting into it. Specifically, I was ashamed at the amount of my time it consumed, I was worried, not so much about my privacy, as the disregard of my right to it (even if I chose not to take it up), and I was anxious about expressing myself too freely lest I cause offence to my friends.  Google+ has lessened my anxiety with its subdivision of friends into circles, though, of course, its potential to cause me shame and worry over my time and privacy are just as real as they ever were with facebook.

With all this in mind, my early experiences of Google+ have been very positive. As I was hoping to, I have re-connected with some lovely friends who had remained on facebook and never ventured onto twitter. Perhaps most encouragingly though, it looks like Google+ might be able to reach beyond the social, and enrich my professional life too. The following exchange, which I started to try and learn more about the use of Amazon’s Mechanical Turk in psychological research, is the sort of thing that’s making me very excited about this possibility.

Full Google Plus conversation

The discussion went in to far more detail than I could have hoped, and for those who are interested, a text-searchable pdf of the exchange complete with clickable links is available here.  (Incidentally, the reason I have had to go to so much trouble to provide a link to the thread with jpgs and pdfs, as opposed to the sort of easy html permalinking offered by twitter, is down to Google’s as-yet imperfect post-hoc sharing system. Once I decided that the thread deserved a wider audience, the options available to me were a) to re-share my original post, without the comments, to anyone on the web, or b) to provide a permalink to the whole thread that was only accessible to those with whom I had originally shared my first post. An option to change the sharing permissions for the entire thread, with the permission of all contributors of course, would be highly appreciated!)

As to why the question about Mechanical Turk generated so much useful information, there are three reasons I can think of.  The first is a simple affordance of the length of posts and comments.  Unlike twitter, detail can be provided when detail is required.  Whilst I have read the thoughts of writers praising the cognitive workout required to condense their tweets to be both eloquent and informative, it is limited medium that doesn’t lend itself to information-rich content or detailed evaluation.  Google+provides a clean, long-format forum in which ideas can be effectively transferred.

The second reason lies in the flexibility of the medium to provide relevant information to those who care.  Circles can be used to selectively share updates with certain groups.  This means that scientific updates can be restricted to my ‘Science’ circle, posts on running can be restricted to my ‘Runners’ circle,  and users may be feeling the effects of a more targeted dose of updates and information.  Comments aren’t driven by a desire to appear funny to a large number of people who probably share your boredom at the fact that, as it’s Sunday, Akira has once again completed a 6.3 mile run in a smidgen over 50 minutes – you’d probably only reach 5 people who would actually be rather preoccupied with trying to work out why Akira hasn’t managed to improve on his 6-mile time despite having done the same run every week for about 6 months.  Depending on your willingness to invest time in the categorisation of contacts, you can be taken as seriously as you want.

Finally, and maybe most importantly, Google+ is current rife with early-adopters. These are technologically ‘switched-on’ folk, who are willing to take a punt on a new medium, testing its capabilities and its uses as they go. To illustrate, Tom Hartley, Tal Yarkoni and Yana Weinstein all maintain current blogs/websites of their own and all contributors to the above thread are active twitter users (and well worth following).  Asking a question about how to conduct science using a nascent technology via a nascent communication technology stood every chance of being successful given the overlap in the Venn diagram of technology users.  Add to that the diminished risk of being called out as a ‘geek’, we’re all geeks here even before uber-geeks are further isolated within the ‘Geek’ circle, and we have the optimum conditions in which to find out about Amazon’s Mechanical Turk.

This isn’t to say that Google+ won’t be successful for non-technological academic discussion, or for technological discussion even after the the laggards arrive.  But I think that success depends on the parameters for its use in academia being established now.  If academics recognise that Google+ can be used to exchange work-related ideas early on in its life-cycle, then it has a much better chance of taking off and even being further developed with this use in mind. It already seems to me a far more attractive site for academics than academia.edu which has comprehensively failed to do anything other than act as a repository for electronic papers and CVs.

So, I’m quietly optimistic… until the next big thing comes along and I jump ship, desperately trying to keep up with all the other early-adopters.

[youtube http://www.youtube.com/watch?v=suJgV9HhJp8]