A while ago I promised that I’d speak a little more about Max Wilson‘s RepliCHI SIG at CHI’12 — alongside the discussions I had about interactions between research and practice, this SIG was one of my CHI highlights (CHIlights? No? Just me? Oh, ok then).

Anyways: there are lots of reasons to care about replication in the HCI community. As Max pointed out in his slides, reasons we should care include:

• replication is a cornerstone of good science
• we’re a global community, yet report on local studies — do results hold in different contexts?
• technology moves fast — again, (how) do older findings still apply?
• how do we deal with proprietary data / software / methods?
• the community prioritises novel work, ground-breaking work — not replications
• it can cause conflict — if Alice replicates Bob’s Awesome Experiment and finds it doesn’t work, that can be difficult for all parties

Max Wilson on replication

It was a neat SIG: after a brief intro from Max we broke into separate groups and bounced ideas around, before feeding back at the end. It was great to see some people heavily involved in the organisation of the CHI community present — always a good sign that discussions are being taken seriously. And they need to be taken seriously: there’s very little replication done in the HCI community.

I’ve always been interested in how we go about research (hence past posts on methodology and mixed methods, and introspection about the make-up of the Web Science community). For some time I’ve been mulling over the issue of replication of industrial research — specifically, research from Big Players, research that involves Big Data.

Say you’re Google, or Facebook. You can do a wealth of fascinating work using the mountains of data from your millions of users. More power to you! But, oh wait. You’re an industry player — you have a different approach to intellectual property, and want to keep your cards close to your chest. Furthermore, there’s a history of privacy issues in sharing data (I’m sure I wrote about this last year) — if you’re going to release your data, you need to work really hard to ensure it’s safe to share. What’s the motivation? Why bother?

Say you’re anyone else in the same community. Well, here’s an unhealthy situation! The Big Industrial Playas are publishing some interesting work, but there’s absolutely no way to a) verify or b) build on that. This is bad, both as a scientific practice and for building on existing work.

What now?

Conversations about this are increasing — for instance, last month the NY Times ran a piece on this. The proposed approach of insisting that publications are accompanied by accessible data appeals to me, but I’ve no doubt it raises all sorts of qualms for industrial researchers. I’d like to catch up with some of ‘em.

Indeed, I suspect this issue shall be discussed a week from now at WebSci’12 in the panel on Challenges & Opportunities for Industry Research. Although I’ll be at WebSci’12, I have to leave early — I’m really gutted, as I’ll miss both this panal and danah boyd’s keynote. Naturally, I’ll be prodding attendees to find out what goes on at the panel.