Monday, 13 July 2009

Bait for the MedWeb2Skeptic

Gah, @amcunningham beat me to a proper look at this paper on web2 use in medical education. To be fair, I was on night shift at the time, so wasn't really in the right frame of mind to write anything longer than 140chars. Still, feeling quite chuffed that I got in there early with the critique, even if it was a little... concise.

Anyway, there isn't a massive amount to add to Anne-Marie's skewering of this survey-based paper on use of Web2 tools in medical/nursing education - she rightly critiques the low response rate, confusion & conflation of web2 / social media tools, and the authors' rather bold conclusions (subsequently echoed around the twittersphere).

The authors do acknowledge one of the paper's weaknesses when they state:

"...given the small sample size, it is difficult to predict whether the use of Web 2.0 tools portends a growing trend in education or merely represents a passing fad"

But although they note the small sample size, they still accept their findings as significant, albeit perhaps transient. To be honest, in this paper, the future of web2 use in medical education is not 'difficult to predict', it's completely outwith any of the conclusions that could possibly be drawn from the data.

But just a few more points...

1. A survey of web2 usage by medical/nursing institutions by a fairly open-access survey, with a very poor response rate means that any conclusions must be interpreted with a degree of caution. But it's not just the low response rate that sounds a note of caution. One also has to question why those particular people bothered to respond (selection bias). It's easy to hypothesize that survey recipients who'd never heard of Moodle etc would just delete the email, while those who were evangelical about using wikis and youtube would reply in their droves. So the sample biases itself.

2. I think there's two other ways to do this kind of research - either spend some time identifying IT/education leads at medical schools and send them a better-designed survey asking questions about overall web2 tool use in medical school, or survey a large number of medical students from several medical schools with a very short survey to ask what tools they actually use on a regular basis.

3. As Anne-Marie mentioned, the qualitative data isn't mentioned. My guess is that there wasn't very much of it. The question is too broad and vague "please briefly describe how these tools are used in your institution". This makes it difficult to answer (therefore most respondents probably don't bother) and unlikely to identify any common themes, as the responses given are likely to be highly heterogenous. If you've ever tried to get useful qualitative responses from questionnaires, you learn this lesson pretty quickly. I did, and I was doing an MSF in my spare time.

So, having kicked the corpse a bit, what's the real issue here? I think it's this - apart from generating headlines, what use is this kind of research anyway? So 45% of medical/nursing schools use web2 tools. Big woop. Who uses them? What for? How? How often? And most importantly, why? If a web2 tool can deliver a better educational outcome (or an equivalent one more cheaply / easily / quickly) than a conventional teaching method, that's a good thing. But just using web2 education tools isn't important - it's what you do with them that counts.

Ref: Lemley T, & Burnham, J (2009). Web 2.0 tools in medical and nursing school curricula Journal of the Medical Library Association : JMLA, 97 (1), 50-52


  1. Hello Colin

    Great post! Mainly I was annoyed at how people could share and comment on this paper when it had so many obvious holes in it.

    I think that we need to start sharing how we are using web 2.0 tools and why? What is working and what is not working? But I am going to be controversial and suggest that we don't need to wait to do a survey etc, we can start blogging about this now. Everyone who is using these tools in education should be sharing their experiences as part of a process of peer review and dissemination which leads to scholarship in education.
    When we start to understand more about the use of web 2.0 tools in medical education- or any kind of education- then we can formulate appropriate research questions.
    By the way, I don't think medical educators should only be blogging about web 2.0 tools. Everything should be out in the open.

    Look forward to more discussion at #asme09!

    Anne Marie

  2. Clearly there's innovation going on and it makes sense to share experiences and develop best practice. But the field definitely needs robust research as well. It's difficult enough to show that educational interventions work even with well-designed education research. Anecdotal evidence isn't enough, although as you say we're probably still at the stage where using a 'virtual faculty' to develop ideas is a crucial step to lead on to formal research.