Monday, November 30, 2009

Good blog-post about the problem of large-scale software simulations in science.

I'm obviously a fan of science-through-simulation so I think that the problems raised don't kill it. Or rather, I think that these issues are not a sign that something has "gone wrong" with science. They are an inevitable part of the maturing of simulation as a tool.

But clearly there's a unarguable need for the code to be available for reviewers. And ideally, to a wider community (hey! OPTIMAES) In fact, this is what I called "the dialogue of models", where competing models are presented and criticised as a way of refining everyone's understanding of the issues.

But what else can be done? Because, frankly, even when the code is out there, the number of people with sufficient understanding and time to analyse it, is going to be vanishingly small. And code is big and complex and time consuming. So the chance of it being "properly" peer-reviewed is low.

Higher-level languages make it easier to express more, more concisely. But they require abstractions are commensurately hard to unpack.

Another option is to use common toolkits like Repast where the task of debugging and verifying the underlying infrastructure is shared among many peers. Similarly, the data-sets need to be accepted and shared within the peer community. (There seems to be a list of climate data repositories here.)

But what else could be done?


zby said...

Publishing the code has the additional beneficial effect that people would just be ashamed to brush dirt under the carpet. This effect does not require anyone really doing the analysis - it is enough that the publisher knows that there are people capable of doing that.

John Powers said...

I see all sorts of advantages to the proposal of Open Source Science Eric Raymond is in favor of. But it does cut across the grain of establishment science. Nonetheless modeling is so important and the stakes are so high, for example this piece in "Nature" about evolution, that the scientific community may well lead the charge in this direction.