RIP John McCarthy

Another great loss. The inventor of Lisp has died.

Interestingly, this, turns out to have been my most popular post evah (I think) ... 276 hits within the last few days.

Is it all those Haskell programmers?

Update : seems to be due to Reddit.

This is a pretty good explanation of monads. Let's see if StackOverflow can help me out with Y-combinators

Put together : programming languages for chip-design + chemistry labs on chips, how soon before you can start programming chemistry directly?

Live Together, Die Alone

When I first heard about Joe Holmburg's illness I immediately started trying to find practical responses.

Suddenly my online world looked spectacularly inadequate to handle this. I went looking for peer-to-peer computing projects that looked likely to do some good. And eventually, I found the World Community Grid and particularly the Help Conquer Cancer project.

Well, maybe that's worth a shot. I've donated a few hours of computer time to it since then - but far too little. Even though I plan to keep up with it.

I wondered what all this social networking could do. There is The Joe Holmberg Appreciation Society Which is a nice idea and I hope was instrumental in organizing something.

But, really, are we trying hard enough to look after ourselves?

Are we putting the undoubted co-ordinating power of the web to work fighting disease? Are we using the new sense of "autonomy" that we get as netocratic "dividuals" to take responsibility for our health?

There are, of course, some companies promising to help us. But can we trust them? Is the market the right way to organize this? Is the government? Is a blogosphere?

Those questions haunt me. But let me self-indulgently digress for a second.

A few months or so ago I went with some friends to a kind of party that is called a "sarau" here in Brazil. A party based on the guests reading poetry, singing or performing some other kind of entertainment for each other. I go, and, in a sense admire this. But, late-20th century anglo-saxon that I am, I can't exactly enjoy it. I don't feel comfortable. I don't sing.

Not because my voice is particularly terrible. At least, no worse than some of my friends who sing away quite lustily. My usual excuse : that I can never remember the lyrics. But at this party, we had a laptop hooked up to the internet, and any lyric you could imagine was easily available.

But I still couldn't sing. And I had a strange insight into why not.

None of the songs I like are social.

The songwriters I like, that I call my "favourites" : Momus, Current 93 etc. are obscurantist, sly, cynical, an elitist in-joke. Not the kind of thing that can be innocently shared, or entered into as a group. And I realize this is my experience of music in general. I grew up in the UK in the 80s. I listened to John Peel on the radio in the evenings. I collected exotic recordings which I listened to alone in my bedroom, and occassionally swapped with like-minded afficionados through the medium of "mix-tapes". As an adult I programmed computers to produce music to amuse myself, caring little about any other audience. I shared Indie Rock Pete's plaintive motto : "Nothing is any good if other people like it.

What does music have to do with other people?

But how can that experience be compared to, or even considered as part of the same universe as the sarau where everyone sings their party piece?

Or carnival? Where everyone goes out to sing together in the street.

Or the incredibly rich popular tradition in Brazil which has dozens of styles of music, from different regions, played during specific festivals throughout the year, localized but where the whole city takes part?

Recently I wrote about Jacques Attali's "Noise" and in particular, I wrote approvingly about the mode of "composing". My mood was celebratory, interpreting "composing" as an era of small-pieces of autonomous self-expression loosely joined by "sampling" or reference or quotation or linking. Music made to the ideals of an isolated English teenager in the 80s with a hoard of cassettes, an 8-bit sampler and "tracker" software on his home computer. I celebrated because I saw in "composing" the exit from an era of mass-production and the capitalist market, a music that could herald the wider break-down of such oppressive social and economic institutions.

In this I was explicitly influenced by netocracy theory and so, I now realize, indirectly by Deleuze and Guattari who see the world as composed of "desiring machines" and we, people, as nexuses in impersonal flows of desire. In this tradition, unleashing the flows of desire and breaking free from constraining institutions is utopian. (Or at least, the only way to lead an "anti-fascist" life.) Bard and Soderqvist cast the elite netocratic class as enthusiastic Deleuzians, and flows of desire are reinterpreted as manoeuvrable network connections. But their dividual is also fragmented and mercurial.

But there's another way to read this. That the lonely teenagers listening to their personal hoard of recordings in their bedrooms herald a quasi-autistic world; a netocracy of the worst sort. A netocracy where we cut ourselves off entirely from each other, shielding ourselves with a technology that only lets filtered, software mediated, connections through. Where Facebook et al have succeeded in enclosing and privatising our friendships and routinely rent them back to us (rent in the sense of making us look at adverts while we engage in them). The isolation we mistakenly believe is autonomy and self-management and escape from overbearing institutions (as we carry our protective musical shells around in our MP3 players) is really the absence of connection, the flight from responsibility and the destruction of the shared tastes and traditions that hold communities together.

In such a world there's no place for the sarau, for a year clocked by samba at carnival and forro at the juninho party.

Put that way, "composing" sounds rather horrible. Opportunistic appropriation is a very meagre way of interacting compared to being a member of a band. Neighbourhood as "pilfery". Squabbling sea-birds always robbing pebbles from each other's nests.

Of course, the other world, the world of shared festival is actually the primitive mode that Attali would call "sacrificing" ... where music exists to mask the violence of everyday life. And, true enough, for all the apparent community you might read into carnival, Brazil is riven by class differences and prejudices to a degree that the 20th century English kids would find hard to imagine (except through reading 18th century literature). Violence in the "communities" is high - between rival gangs and between gangs and police it is astronomical. Nowhere in Brazil escapes the shadow of violence. Nor the fear which keeps the middle-classes in their fortified houses and condominiums. John Robb's post-nation-state world of global guerrillas and "armed suburbs" sems closer to reality in Brazil than his US and certainly more than my UK. (Or is that so? You don't feel that a gang of teenagers in Brazil is more dangerous than in the UK. In Brazil, violence is forgotten or ignored easily. In the UK, moody kids glower menacingly from under their hoodies (allegedly))

So which kind of music prevails? The nation state, "project man", (the mode (roughly) that Attali calls "representing" in that music is intended to represent the harmony of society) is in decline. Are we reverting to medieval "sacrificing" or going forward to "composing". And is composing a true, post-modern, post-capitalist utopia where our economic activity is (roughly again) "look around at cool stuff and spontaneously re-appropriate it to make something of your own that's cooler" (the peer-production economy of free-software, blogging, scratch video and sampling, Facebook, and free-improvisation) Or is that a happy facade, the reality for a few elite netocrats while the rest descend into the drudgery and banality of the consumtariat?

One can, perhaps, gain a new insight here : In netocracy, the netocrats "steal" the social links from the consumtariat.

While the netocrats are busily managing their portfolio of links, the technology that trickles down from them pollutes the social space. Slide and RockYou are destroying my friendships! People who I was rather glad to meet up with and find out about every couple of years are now nothing but virtual conduits for an endless stream of meaningless polls about 10 things I have no interest in and would never do but somehow have to put in order of which I'd be more likely to, just to find that I'm only 73% similar to my erstwhile friend.

The flip-side of the coin of "everyone can have their own little media empire", is that "everyone is as ignorable as a media celebrity we don't care about" On the internet it doesn't matter that X was a guy who I quite liked to go to the pub with a few years ago. Now he's just another blog that isn't interesting enough to read.

The end-game ... we consumtariat stick our white earbuds into our ears and filter out the community. We are "composing" with instruments handed down to us by the elite. (As Attali warned) And so we are isolated.

When people die, it's often written that their loved ones and family were by their side. I realized, long ago, that I don't want to die surrounded by friends and family. What on earth would you talk about?

Are you expected to make light chat about what they plan to get up to once you (and the boring bureaucracy of the funeral) are out of the way? Are they just gonna sit round looking gloomy?

No, long ago, I figured that I'd like to die alone, but listening to music. Only music is soothing and mentally engaging enough (without being taxing) to cushion the sheer terror of waiting for it to happen. I could almost be relaxed waiting for death ... if I have control over the sound-track.

And yet, now I worry, is this not, perhaps the final end-game logic of isolation? This is the state to which netocracy reduces all of us. By making social connections a means to an end rather than an end in themselves, a tool rather than the condition within which we live, we have stripped them of all real significance or value. They have become dead. We have become dead. Out capacity to participate has been eroded even as we gained a capacity to curate.

Or is this getting overly morbid and pessimistic?

An old joke seems apposite at this point : Someone dug up Mozart's grave, and found the corpse busily erasing the marks from pieces of scored manuscript paper. "What are you doing?" they asked. "Why?", said Mozart, "I'm decomposing". :-)

And so, rambling on, more new years resolutions.

6) I'm not planning on dying yet. And nor should you. But what should you do instead?

My first enthusiastic announcement of OPTIMAES was once billed as "Phil vs. Capitalism : Round 1" - nothing wrong with a bit of ambition. Now I think hackers (of both the technical and social kinds) should be looking into how to use their art to tackle the big issues. One thing I want to think about this year is what can a society moving into a netocratic / spime era (ie. with ubiquitous connectivity and computing power) do to confront death : to take on the big scary diseases : cancers, AIDs, malaria, Parkinson's, Alzheimer's, heart disease etc. We really *aren't* trying hard enough, are we? Compared to all the other fripperies we waste our time and money on.

7) I'm reading Deleuze and Guatari's Anti-Oedipus at the moment. I will finish it.

8) I'm back to writing simulations, currently of competitive network formation (more on this soon). In Erlang! And Erlang, it turns out, is very very cool. It's a little bit ugly to my eyes; I wish it could have been a bit cleaner (in the Python / Haskell direction). But nevertheless, it is very concise. I've managed to write my whole agent framework in about 150 lines (compared to about a 800 to do something similar in OPTIMAES in Python; although that was, admittedly, when I was still writing Python as if it was Java). My evolution into a functional programmer is continuing apace.

9) Perhaps most importantly - I'm looking for participatory, social, music in my life. I was wrong to think this was about "Brazilian" music. What's really important is the other understanding of composing that Attali gives (it was me who goes astray into digital sampling and name-dropping) - the "hippie" spontaneous communitarian, improvisatory, free-jazz, jamming, lick-swapping "composing". The sarau and singalong around the camp-fire. This is, after all, the way to stay in touch. To stay alive. However embarrassing it sometimes seems. ;-)

Canned lambda-calculus

I promised to explain that little bit of python black-magic I posted a couple of days ago. It may take a couple of posts ...

But to start with, here's some code I just found, that I wrote about 18 months ago, to create a "pipeline" to do various transformations to lines of text. Basically I want to transform wiki markup into html, but I wanted a flexible mechanism which allowed me to add or remove various transformations at runtime so I can rejig the same transformation for slightly different targets. A good example of this occurs in the original SdiDesk, where the production of HTML to be viewed in SdiDesk is *almost* the same as the production of HTML for export to a static site, except for the link tags, who's href property for internal use needs to be "About:Blank" whereas the external site needs the actual URL of the page.

Here's my original code (with a couple of example transformations) ...

class PipeNode :
    """Base class for pipeline nodes. 
       They must implement 
       process() and getName()"""
       
    def process(self, item) :
        return item

    def getName(self) : return "PipeNode"
    
class Pipeline :

    def __init__(self) :
        self.pipe = []
        self.index = 0
        self.item = None

    def addNode(self, n) :
        self.pipe.append(n)

    def load(self, item) :
        self.index = 0
        self.item = item

    def next(self) :
        self.item = self.pipe[self.index].process(self.item)
        self.index = self.index + 1

    def getCurrentItem(self) :
        return self.item

    def getCurrentIndex(self) :
        return self.index

    def getCurrentNode(self) :
        return self.pipe[self.index]

    def eop(self) :
        """End of pipe"""
        return self.index == len(self.pipe)

    def run(self) :
        while (not self.eop() ) :
            self.next()

class BlankLines (PipeNode) :

    def getName(self) : return "blankline-to-para"
    
    def process(self, item) :
        if item == '' :
            return "<p/>"
        else :
            return item

class Bullet (PipeNode) :

    def process(self, item) :
        if item[0] == '*' :
            return '<li>' + item[1:] + '</li>'
        else :
            return item
        
class LinewiseStringPipeline (Pipeline) :

    """
    Based on the generic pipeline, this cuts a long string into lines
    (separated by '\n') and processes them one by one
    """
    
    def __init__(self, sep='\n') :
        Pipeline.__init__(self)
        self.sep = sep

    def runAll(self, s) :
        if len(self.pipe) == 0 :
            return s
        build = ''
        lines = s.split(self.sep)
        for x in lines :
            self.load(x)
            self.run()
            build = build + self.getCurrentItem() + self.sep
        build = build[0:-1]
        return build

# test it
pl = LinewiseStringPipeline(r'\n')
pl.addNode( BlankLines() )
pl.addNode( Bullet() )   

page = """here

is

* some
* data"""

print pl.runAll(page)

You'll see it's all good object oriented stuff. I define a base-class generic "PipeNode" which doesn't really do anything, but acts as a definition of the interface of a section of the pipe. Its important method is process, which takes an input and delivers some kind of output.

Then there's the pipe object itself. A straight-forward list of nodes which knows how to run ie. pull the items through each of the processing nodes. It keeps track of which section of the pipe the item is in using index.

Then there's the LinewiseStringPipeline which knows how to cut a large marked up page into single lines (wiki markup is applied on a line-by-line basis), run them separately through the pipe, and then concatenates them back together at the end.

Finally there are a couple of example PipeNode filters ... one which turns blank lines into blank paragraphs, and * into list-items.

I hope it's comprehensible. It's a little over-engineered with extra methods that are not strictly necessary for the operation. (Such as admin functions I guessed would be useful.)

And it is extremely verbose.

Now compare the same thing in my new, functional style which dispenses with objects and classes but seriously uses generators and closures.

def makeSection(f) :
    def gen(source) :
        for x in source :
            yield f(x)
    return gen

  
def makeFilter(test) :
    def gen(source) :
        for x in source :
            if test(x) : yield x
    return gen

  
def pipeline(generators) :
    def p(source) :
        old = source
        for g in generators :
            new = g(old)
            old = new
        return old
    return p

  
def blankLines(item) :
    if item == '' :
        return "<p/>"
    else :
        return item

  
def bullet(item) :
    if item[0] == '*' :
            return '<li>' + item[1:].strip() + '</li>'
    else :
        return item

        
def linewisePipe(page, pipe) :
     return '\n'.join(pipe(page.split('\n')))
     

# test it
pipe = pipeline([ makeSection(blankLines) , 
                  makeSection(bullet) ])
page = """here

is

* some
* data"""

print linewisePipe(page,pipe)        

So, how does it work? Instead of handling each stage of the pipe with an object with only one significant method : process, it now handles each with a single function. Except that function is, in fact, a generator (ie. it's a function who's execution frame doesn't disappear when it yields it's return value, but continues in memory, and gets resumes where it left off).

The reason for this, is that these generators take a in iterator for their argument. And only pull and process the next item from the iterator before yielding it. The use of a generator is important because once it's up and running it's the generator itself which is remembering which item is currently being processed.

Of course, writing generators is a mildly more complicated than writing functions, so I've written some "make" functions to turn ordinary functions into them. makeSection(f) takes an ordinary x -> y function and returns the generator as a closure; note how the actual processing function is bound to f at the call of makeSection.

The result of calling makeSection, then, is a new generator which a) sucks something from an iterator (called "source"), b) calls f on it, c) yields it (ie. returns it, but hangs around with the next item from the iterator in "x", which it will yield next time it's called)

To simply use one of these sections on its own you could do something like this :

s = makeSection(lambda x: x * 2)
for x in s( [1,2,3,4,5] ) :
   print x

Closure s takes an iterator as an argument (in this case, the list [1,2,3,4,5] ). It's result is also an iterator. (Which is what Generators look like from outside)

In fact the "for" loop keyword is implicitly calling next() on it every time to get a new value for x. When it does so, it pulls the next value out of s. Now, inside s, we have the body of "gen" as defined in makeSection. This is applying the function f (which doubles its argument x) on each value that comes through.

So sctions are always generator functions that are started with an iterator as input and at every step of the iteration return the next item from their source iterator transformed. The transformation itself depends on the argument given to the call to makeSection.

Filters are the same, except in this case, they only pass on items that meet the "test" criteria.

The next part of the program is to chain a number of these generators together into a pipeline. In fact, "pipeline" is another function which returns a closure. It takes a list of sections and filters and it's this list which is stored in the closure. (Be careful here, the reference to the list not the actual list content - so there's a danger of changing them after the execution of pipeline)

What comes out of pipeline is a new generator. One which takes an initial iterator as source, and on each next, sucks the next item from that source all the way through all the stages of the pipeline and spits it out.

So what are the advantages of this over my original object oriented version? It's certainly far "cleverer". To me, today, it looks more elegant. And it's much easier to start using. My specific transformation functions don't need to know anything about pipelines - they don't need to be defined as methods on new classes that are explicitly sub-classed from PipeNode. Instead each function can be written in ignorance of it's role, and then turned into a section or filter when needed. (Often just before assembling the pipe.)

Pipes are easy to assemble using the pipeline function, and I believe are very intuitive to use.

Of course, the object version could be tweaked to look more like the functional version to the outside user. The Pipeline object could be an iterator, making it possible to use it in a for loop. We could perhaps make it take a list of sections in it's constructor rather than force the programmer to write multiple explicit addNodes.

Nevertheless, I'm smitten with the new style. I'm replacing the old code with the new today.