The curious case of the game show neuroscientists, or how NOT to research an online community

I’m a fond member of the blogging/social networking site, Livejournal.   Over the last few days, I’ve seen the most incredible shitstorm unfold, over the cack-handed efforts of two rogue academics to research what they were pleased to call ‘the cognitive neuroscience of fanfiction’.

Background

First, a bit of background: Livejournal (one of the original social networks) is a vast and varied set of subcultures, and interconnected blogs, dominated by film, TV, book and gaming fans.    It is more counterculture than culture, really: it tends to be left-wing, creative and anarchic.

One of the many subcultures in the mix is fanfiction writing:  stories that people write using characters from books, film, music and TV.  Fanfic writing is female-dominated, and some of it (but by no means all) is very explicit.   There is fanfic for everything, from Jane Austen through Doctor Who (rewriting the works of Russell T. Davies) to The Mighty Boosh.

Fanfic writers have an odd hobby, but they are a pleasant and literate bunch who are much studied by academics.   In fact, academics (like Henry Jenkins) completely adore this stuff  – it pulls feminism, transgression, social networking and copyright laws all into one place. What’s not to like.

The questionnaire is launched

Anyway, a few days ago a friend forwarded me a link to an online questionnaire that she found intriguing.  It was about fanfiction, it seemed a bit amateur, and what did I think of it?   The link was banner-style, and it looked a lot like the Cosmo-style pop quizzes that are memed all over the place on social networks.   There was a reassuring link to a FAQ page giving the names of the researchers, Ogi Ogas and Sai Gaddam, and their academic affiliations at Boston University (the BU links no longer exist).  This page also gave a long explanation of their interests in cognitive neuroscience, and what this had to do with fanfiction…

“We’re deeply interested in broad-based behavioral data that involves romantic or erotic cognition and evinces a clear distinction between men and women. Fan fiction matches this criteria perfectly.”

…Uhuh.

The researchers had apparently also consulted a couple of well-known bloggers in the areas, and got their guidance and feedback and endorsement.    Apart from the fact that the academics weren’t making the changes suggested, it all seemed fine.

The online questionnaire itself (captured here in two parts on an LJ Dr Who site – you may see an generic age warning for content on some LJ pages) was a rather different story. I took a look.  There were  70 questions in all (one per page), starting with some brusque questions about one’s gender, age and ethnicity.   It even asked for your SAT scores.  The questionnaire proceeded to a number of fantastically detailed and rather odd questions dealing with fanfiction reading habits; and then it got heavily intimate, asking (amongst other things), exactly what kinds of sexy stories the respondent read and (deep intake of breath) whether they ever had rape fantasies.

The questionnaire… does not go down well.

The questionnaire was barely up before LJers started complaining about the content.  LJ people love to complain at the best of times, and there was a lot of ground to cover here:

a)      Terrible questionnaire design

b)      Inaccurate, amateurish and homophobic wording

c)      Prurient lines of questioning

d)      No attempt to screen out under-18s

e)      Lack of the usual information on privacy, anonymity and confidentiality

f)       And (my favourite) frequent criticisms of the methodology.  How in the name of heaven the researchers were going to draw any valid conclusions whatsoever about subcortical processing, given their data collection methods?

What the researchers hadn’t bargained for was the thoughtfulness of the response.  Livejournal people are a fairly literate bunch.   Stuff like feminist analysis of television casting decisions is a walk in the park for many of them.   At least some of the people who came across the questionnaire were social researchers, lecturers, feminist academics, and indeed neuroscientists.  They didn’t like what they saw.

Ogi attempts to engage with respondents

The lead researcher opened a journal (now showing a single entry, an apology) for the purposes of answering questions about the research; and in the space of about two days, that journal moved from polite, rather subservient requests for clarification, to a full-on flamewar, as the lead researcher put up his questions for comment. As he engaged, he revealed more and more of his (very strange) thinking (he’s deleted his comments on this thread, but you can work some of them out), and his subjects began to research him in earnest.

Google is your friend (and Wikipedia, and Youtube)

Turns out, Ogi Ogas had forgotten to mention a few things:

  1. He wasn’t actually affiliated with Boston University any more
  2. While they were indeed neuroscientists, their Ph.Ds were on visual processing and artificial intelligence
  3. The lead researcher’s Ph.D was funded by the US Department of Homeland Security
  4. The lead author gained earlier infamy as a successful contestant on the American version of ‘Who Wants To Be a Millionaire’

And, last but not least, there was another teeny fact missing:

The authors had just signed a substantial book deal with Penguin for a popular science book entitled: ‘Rule 34: What Netporn teaches us about the brain.’

(As one commenter put it: ‘What? You think we can’t Google?’)

(NB – the literary agency has changed the book title now, to ‘Rule 34′)

So they asked about these Netporn theories, and then the shit really hit the fan.  It’s hard to follow the logic, but his theory (screencapped here)  drew on data-mining of adult sites aimed at men, and posited that explicit fanfiction for women could be equated with male interest in male-to-female transsexuals  (?!) and that both of these things could be used to model subcortical processing (whatever that is) in male and female  brains.  Or something.

Somewhere around there, people stopped arguing with him and started taking direct action.  The academics started complaining to Boston University, the creatives started creating cat macros, the neuroscientists started writing long introductions to neuroscience and the specialists in gender identity just started screaming.  There were a few more updates, and then Ogi locked his journal.  He issued a few wandering emails, and removed most of his journal (and indeed many of the comments that he’d left elsewhere).  Naturally, the LJers (being used to the ways of flamewars) took screenshots of the more alarming content well in advance.

Aftermath

From beginning to end, Ogi Ogas maintained that he wasn’t doing social research, he was just collecting data.

The day after the shitstorm, someone reported their conversation with the University of Boston’s rearch ethics board: he wasn’t formally affiliated, and he didn’t have ethics board clearance.  His university pages have now disappeared, the questionnaire is down, and at time of writing, he seems to be deleting all his comments elsewhere.

On the face of it, this is simply an extreme example of shoddy and unethical  research which will reflect badly on anyone who tries to do research online, especially within a community or subculture.   Anyone who approaches that particular community in the future is going to encounter deep suspicion.

It goes further, though.  One of the very odd features of the whole story is that Ogi Ogas and his colleague took a lot of care to approach prominent people. He got a great deal of help from some of them (he also got a magnificent brush-off from one, but that’s another story*).   All of those people are writing to explain that he seemed genuine, and they trusted him.   They offered the same critique of the questions that anyone would.  He seemed to listen, but went ahead with his own version.  This is either arrogance or sociopathy.

One of the people he approached has written to apologise for being taken in, and to reprint some of their correspondence.  She warns him that his attempts to research this particular community are probably dead in the water.   In his reply to her, he’s chirpy.

‘Eventually we’re going to go through this all over again with the far right. It will be interesting to see who throws the meaner punch.’

And I’m left thinking: is this the ultimate troll?

The book is due out in 2010.

*My favourite part of these people’s very lengthy smackdown is the grand postmodern refusal:

‘And so we decline to be interviewed by you; we decline to be the objects of your fascination; we decline to be naturalized; we decline to allow our political project to be cited in support of the very discourses we are trying to question.’

ETA: When respondents bite back

I actually hesitated in writing this up, because I was worried that mainstream researchers will see this as a distant kerfuffle in an unlikely subculture.   But I agree strongly with the writer at the Rough Theory blog (see below), who suggests that Ogas may fail to take valid community criticisms seriously, because he has so thoroughly Othered them as respondents.  In other words, ‘they’re so weird, we don’t have to be careful with them.’

The second general learning point for anyone thinking of attempting a controversial online questionnaire, is how quickly things go viral.  Ogas was terribly happy about the response rate (reliability and validity were not a concern); that same speed of process led, very rapidly, to critique, opprobrium, and direct action.   Before you engage?  Do us all a favour and go on that Methodology course.

Some other quick links:

Rough Theory’s roundup

Unfunny Business’s summary of the whole mess

Feminist SF

Jonquil’s thoughts on respondents who bite back

28 Responses

  1. […] post has been picked up at Josh Jasper's blog at Publisher's Weekly, as well as at Alison Macleod's the human element. Macleod's blog has a very clear overview of how the whole thing unfolded, as well, for folks new […]

  2. Excellent summary!

    There is an extra forward slash in your link to the “magnificent brush-off” (which is very worth reading!)

    Presumably this post by eruthros

  3. This post has been included in a linkspam roundup.

  4. […] The curious case of the game show neuroscientists, or how NOT to research an online community « The… What the researchers hadn’t bargained for was the thoughtfulness of the response. Livejournal people are a fairly literate bunch. Stuff like feminist analysis of television casting decisions is a walk in the park for many of them. At least some of the people who came across the questionnaire were social researchers, lecturers, feminist academics, and indeed neuroscientists. They didn’t like what they saw. (tags: surveyfail surveydesign fandom) […]

  5. Very intriguing.I am not ultra familiar with fan fiction (except for the Austenian variety) but this was a great introduction to what promises to be an enduring issue. Thanks for sharing!

  6. Thanks for sharing!

    I am not a very big fan of Livejournal. Yes, you can find something interesting there, but mostly young people who try to prove one another who is “cooler”. The biggest amount of people there joined the communities which do not discuss anything more or less interesting, just sex, machos, hot chicks or something like this. Sometimes it is really funny to read their posts.
    Fortunately, not everything is so bad. You can find some interesting and useful information there which is possible to discuss without any personal “hittings”

  7. “But I agree strongly with the writer at the Rough Theory blog (see below), who suggests that Ogas may fail to take valid community criticisms seriously, because he has so thoroughly Othered them as respondents. In other words, ‘they’re so weird, we don’t have to be careful with them.”

    Maybe, but I think he might just be a bit thick. It’s pretty common, even in people with PhDs. Not sure there’s any deeper lesson here.

    • Heh. It might indeed be that too, although he seems to be a very special kind of thick, I feel.

  8. @givenchance

    It’s interesting that this comment illustrates some of the same assumptions involved in SurveyFail.

    No doubt givenchance knows live-journal communities where the demographic is young and interested in ‘hot chicks’.

    The livejournal communities I frequent skew heavily towards my demographic and interests, which are folks aged mid-thirties through late fifties in age, with graduate degrees.

    I estimate that close to one-quarter of the people I follow on livejournal work in academia (University-level).

    There is probably no overlap between the livejournal givenchance knows and the one I know.

    That simply proves that one needs to know one’s audience, which the SurveyFail ‘researchers’ did not.

    LURK MOAR.

    • What you said. Anyone (and this clearly includes the researchers, who seem unfazed by all the well-expressed doubts regarding their sexist, determinist agenda) who starts out saying “______________ is all just a bunch of …” might want to think about their assumptions, and about how much they actually know about what they’re stereotyping, before they finish that sentence.

  9. I apologise, I hit send too soon.

    At any rate, LURK MOAR is what the researchers needed to do. This is internet slang for knowing the audience and social conventions of an online group before one jumps in.

    Allison, thank you for your summary of this conflict. It is very a very helpful distillation of a sprawling internet meltdown. I can’t imagine how much work went into putting this summation together.

    • You’re welcome! Lurking moar is always a good strategy

      I’m slightly sad to realise that my own corner of LJ is free from the machos and hot chicks. It’s an excellent point: in large social networks, we create our own cosy patch around us, hardly realising that the landscape may look quite different to someone else.

  10. I have come across many LJ pages that were fan fic pages, but never had the time or inclination to read any one page in its entirety, I guess its because the internet induces short attention span.

  11. […] There’s not much need to do the same for Ogas and Gaddam here, as great critiques already exist. N Pepperdell at Rough Theory first called out to Neuroanthropology through a post covering the survey controversy, Wearing the Juice: A Case in Research Implosion. Sabrina discussed the present state of our knowledge and statistical differences versus real differences in I Need To Walk Away From This Trainwreck. Neededalj took on the neuroscience side in Why Ogi Ogas and Sai Gaddam Are Phrenologists. Those looking for a general account of the failings, look no further than the wiki SurveyFail. Finally, Alison Macleod provided an excellent overview in The curious case of the game show neuroscientists, or how NOT to research an online community. […]

  12. I know this was a pretty poorly executed survey, but this still seems like a pretty huge overreaction from an online group known for overreactions. The assumption of malice built into a lot of these comments seems uncalled for.

    • The thing is, you have to understand the history of fandom to know why we seem to overreact to things. Fandom is not entirely, but quite substantially, made up of people who do not fit into societal norms, who are ostracised and sidelined in real life for various reasons – be it sexuality, gender identity, political beliefs, or just plain being “weird”. Added to that, as has been noted fandom has an awful lot of very intelligent people in it who spend a lot of time analysing the subtext and messages in source texts, who have degrees in a wide range of subjects and who are well aware of the nasty side of humanity and how subtle hints tie in to more dangerous actions (for example, Othering people leading to considering those different from you as deserving of hatred, or connections between making jokes about rape or gay people and the rate of violence towards women and minority sexualities).

      To many, it may look as though we’re overreacting, because we are looking at the ideas behind what is being said in a way that most people don’t have to. And yes, we are sensitive to being sidelined and studied and treated as different from “normal” people, because it’s been happening to us all our lives. Fandom is our safe zone. It’s where there are other people like us, who we can discuss thorny issues with and know that someone will understand what we’re trying to say. We understand each other.

      After a while, acceptance and resignation to the idea that we’re seen as strange and weird and odd becomes pride in that. We’re taking back our strangeness and making it ours, and now we don’t want these mainstream “scientists” coming in to study us and deliver promises of putting us in books so “normal” people can read about us and gawk like they do over books and tv shows about the supernatural or unsolved crimes or cultures vastly different from our own. People have tried to study us in the past, and even if they have the best of intentions their conclusions have been inaccurate and their presentations of us have been patronising or sensationalised.

      We don’t want to be studied. We just want to be left alone, or at the very least treated with some respect.

    • The strong reactions aren’t coming from fandom alone – from those of us looking on from academia, whether from within or outside fandom, this violates very deeply held ethical principles about professional conduct and strongly held ideals about what it means to do science.

      The poor execution isn’t the only or the primary objection made by academics (although that poor execution implies a disregard for basic professional standards). Instead, the decision to represent themselves as academics with a university affiliation – which implies to the public that certain ethical procedures and standards of professional conduct can be assumed – while not actually adhering to any of that, in incendendiary, even without prior exposure to fandom.

      • Yes, exactly. Pretty much everything about this “study” is troublesome, from the misrepresentation, the execution that is worse than I’ve seen from intermediate school students (11-12 here) including changing questions while the survey was live!, the fact that they had just signed a book deal and it was evident that they had already reached their conclusions, and just wanted some evidence they could quote. Even the advertising banner looked like a banner for a silly meme – that’s what I assumed it was until I saw the first beginnings of people pointing out problems. The slogan was something like “how sexy is your fandom’s brain?”

        I’d almost be interested in reading the book if they ever manage to publish it, just for the trainwreck factor – but not if I have to pay for it.

  13. Funny to see that some theories could be real fiction-stories ;)

  14. Some contact Penguin and let them know that just signed a contract with a researchers who did not properly and get permissions for their “research, and this will consequently open them up to several hundred lawsuits. It goes without saying that if this is done, and this information is provided, the “researchers” will be dropped by Penguin like hot potatos – and with good reason. It’s bad enough that the “research” itself is crap – but to profit off the online community like this, lying the entire way – it’s not only unethical, it’s illegal.

  15. The first warning should have been the researchers’ inability to use the plurals “data” and “criteria” properly.

    We’re deeply interested in broad-based behavioral data that involves romantic or erotic cognition and evinces a clear distinction between men and women. Fan fiction matches this criteria perfectly.

    I’m surprised that they had actual, if unrelated, academic credentials.

  16. […] academics becoming aware of the issue as it was […]

  17. Hey very nice blog!! This was what I needed to know. Keep it coming.

  18. […] there are screencaps around and there are certainly a number of posts that eloquently and summarize what happened in extensive detail (I’ve tried to link to some of the best throughout this post). I, personally, recall taking […]

  19. […] The curious case of the game show neuroscientists, or how NOT to research an online communityWhat the researchers hadn’t bargained for was the thoughtfulness of the response. Livejournal people are a fairly literate bunch. Stuff like feminist analysis of television casting decisions is a walk in the park for many of them. At least some of the people who came across the questionnaire were social researchers, lecturers, feminist academics, and indeed neuroscientists. They didn’t like what they saw. fandom fandom:meta fandom:surveyfail […]

Comments are closed.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: