Geek Questions for 2014 : Programming on Tablets?

My latest geek question for 2014 (over on Smart Disorganized). How the hell do I program on my tablet?

GeekWeaver Lives!

A story over on SmartDisorganized about the return of GeekWeaver and why my software is like Cthulhu.

Teach Programming In Schools

A petition to the UK government.

RIP Dennis Ritchie

Forget Steve Jobs, the guy who invented C has died!

I'm a big fan of Alan Kay (inventor of Object Oriented programming, and general interesting thinker) so I bought the Points of View book.

As usual, this is a connection competition. Suggest interesting parallels, contrasts, connections between the different recent acquisitions shown here.

An example of magical thinking / user-interface that I didn't notice before :

We are about to study the idea of a computational process. Computational processes are abstract beings that inhabit computers. As they evolve, processes manipulate other abstract things called data. The evolution of a process is directed by a pattern of rules called a program. People create programs to direct processes. In effect, we conjure the spirits of the computer with our spells.

Been quiet again, hasn't it?

I'm busy.

Hopefully over in a few weeks though ...

meanwhile, some random news / thoughts :

1) I have a bike! And biking in London is wonderful. Especially if most of your commute is along the canal tow-path. :-)

2) ThoughtStorms is blowing itself out!

Yes! Really!

I used to love that wiki. Now, for the umpteenth time in the last couple of years, it's frozen up on me because of file-system restrictions on the hosting service. Spammers forced me to close it to other contributors. I rarely update it, and then only as a LinkBin. It needs a bloody good refactoring and clean out of old, redundant, information.

And suddenly, I'm sick of it. I need something new ...

What that is, I don't know. My rewrite of SdiDesk in Python is ... well, 5 years late. I don't like all these new fangled wikis with their wysiwyg editors and massive plugin configurations and a markup-languages that I don't know. (Hrrrumph!)

I don't think it's blogging, although, as you can see, I do more blogging that wikiing now.

In fact, I'm starting to suspect that BillSeitz was right and I was clever but wrong in thinking that blogs and wikis couldn't really be integrated due to conceptual rather than technological incompatibilities. I'd love to be able to pool the best of ThoughtStorms writing with Composing / SmartDisorganized / PlatformWars etc.

I'd love refactoring tools to help sift and mine and reorganize my writing from one place to another.

I'd love the power of git or mercurial distribute source-control behind my writing on different devices.

Yeah, but do I love it enough to sit down and write it?

3) Watch this awesome talk by Tim O'Reilly. Good inspiration for anyone thinking what to do next.

Good Zed Shaw talk (in tweets)

(Covers everything from python to music to banks and economic crash.)

Tomorrow is Ada Lovelace Day.

Nine year old releases painting program for iPhone.

With jQuery and the Firebug plugin for Firefox - wow! - browser-based development is way better than it used to be 10 years ago :-)

Interestingly, this, turns out to have been my most popular post evah (I think) ... 276 hits within the last few days.

Is it all those Haskell programmers?

Update : seems to be due to Reddit.

This is a pretty good explanation of monads. Let's see if StackOverflow can help me out with Y-combinators

I'm debating Simon Wardley (read comments) over what generalist computer programmers with their understanding of abstraction can bring to specialist science.

Meanwhile he finds a great video of embedding electronics in paper.

Why Geeks are doomed in a Suit world. :-(

Joel Spolsky said something interesting in his critique of Java-oriented Computer Science eduction.

Pointers and recursion require a certain ability to reason, to think in abstractions, and, most importantly, to view a problem at several levels of abstraction simultaneously. And thus, the ability to understand pointers and recursion is directly correlated with the ability to be a great programmer

This is so true. Great geeks understand the importance of the mental ducking and diving and wheeling and loop-the-looping up and down the layers of abstraction. The Geek's job is to see at all levels so that the requirements of one level can be solved by implementations further down.

But now consider the perenial Suit complaint of the Geek : that he starts getting bogged down in unnecessary technical details when he should just be giving a high-level progress report or talking to the customer about her problems.

He doesn't "understand business" the Suit thinks.

How tragically, irritatingly wrong. The Geek's job is to make the abstraction levels fit together. Of course the only way to achieve this is with an understanding of both the higher and the lower levels : simultaneously. And the easy shifting from one to another.

But Suits love to keep the levels separate. Their whole reason for existence, their positions depend on the notion that the levels are distinct. The rigid company hierarchy is the reification of non-traversable levels of abstraction.

It is obvious to them that "business" can be understood through the abstraction of accounting. And that the senior managers make visionary plans which require highly abstract inferences about strategic relations but don't require understanding the gritty details of technology.

To the Suit, Geek thinking, that swoops between the layers of abstraction, that claims the right to think of the problem from any perspective, is anathema.

In light of my current dalliance with Erlang and that recent post on setting up pipelines in a functional stylee (in Python), this comparison of process-based concurrency in Erlang and Stackless Python is very cool.

Talented Friend Watch #9 : Oli Sharpe reinventing programming as we know it with Semantic Programming


(I'm going to follow and write more about this over on Smart Disorganized, which is increasingly my "powerful programming language" blog. That's partly because I'm seeing a lot of overlap between the values of powerful programming languages and tools for Smart Disorganized Individuals in general. Both need to support powerful abstractions, (and the creation of new ones), while remaining very fluid and expressive and easy to use.

Semantic Programming promises to be powerful ... I'm pushing Oli for more details, but I hope it will be easy too.

Hmmm .... this does mean though that both sides of the conversation have to retain state.

This looks like a must-read for me : Monads in Python

... and another Monad Tutorial I should read so I can understand it.

I promised to explain that little bit of python black-magic I posted a couple of days ago. It may take a couple of posts ...

But to start with, here's some code I just found, that I wrote about 18 months ago, to create a "pipeline" to do various transformations to lines of text. Basically I want to transform wiki markup into html, but I wanted a flexible mechanism which allowed me to add or remove various transformations at runtime so I can rejig the same transformation for slightly different targets. A good example of this occurs in the original SdiDesk, where the production of HTML to be viewed in SdiDesk is *almost* the same as the production of HTML for export to a static site, except for the link tags, who's href property for internal use needs to be "About:Blank" whereas the external site needs the actual URL of the page.

Here's my original code (with a couple of example transformations) ...

class PipeNode :
    """Base class for pipeline nodes. 
       They must implement 
       process() and getName()"""
       
    def process(self, item) :
        return item

    def getName(self) : return "PipeNode"
    
class Pipeline :

    def __init__(self) :
        self.pipe = []
        self.index = 0
        self.item = None

    def addNode(self, n) :
        self.pipe.append(n)

    def load(self, item) :
        self.index = 0
        self.item = item

    def next(self) :
        self.item = self.pipe[self.index].process(self.item)
        self.index = self.index + 1

    def getCurrentItem(self) :
        return self.item

    def getCurrentIndex(self) :
        return self.index

    def getCurrentNode(self) :
        return self.pipe[self.index]

    def eop(self) :
        """End of pipe"""
        return self.index == len(self.pipe)

    def run(self) :
        while (not self.eop() ) :
            self.next()

class BlankLines (PipeNode) :

    def getName(self) : return "blankline-to-para"
    
    def process(self, item) :
        if item == '' :
            return "<p/>"
        else :
            return item

class Bullet (PipeNode) :

    def process(self, item) :
        if item[0] == '*' :
            return '<li>' + item[1:] + '</li>'
        else :
            return item
        
class LinewiseStringPipeline (Pipeline) :

    """
    Based on the generic pipeline, this cuts a long string into lines
    (separated by '\n') and processes them one by one
    """
    
    def __init__(self, sep='\n') :
        Pipeline.__init__(self)
        self.sep = sep

    def runAll(self, s) :
        if len(self.pipe) == 0 :
            return s
        build = ''
        lines = s.split(self.sep)
        for x in lines :
            self.load(x)
            self.run()
            build = build + self.getCurrentItem() + self.sep
        build = build[0:-1]
        return build

# test it
pl = LinewiseStringPipeline(r'\n')
pl.addNode( BlankLines() )
pl.addNode( Bullet() )   

page = """here

is

* some
* data"""

print pl.runAll(page)

You'll see it's all good object oriented stuff. I define a base-class generic "PipeNode" which doesn't really do anything, but acts as a definition of the interface of a section of the pipe. Its important method is process, which takes an input and delivers some kind of output.

Then there's the pipe object itself. A straight-forward list of nodes which knows how to run ie. pull the items through each of the processing nodes. It keeps track of which section of the pipe the item is in using index.

Then there's the LinewiseStringPipeline which knows how to cut a large marked up page into single lines (wiki markup is applied on a line-by-line basis), run them separately through the pipe, and then concatenates them back together at the end.

Finally there are a couple of example PipeNode filters ... one which turns blank lines into blank paragraphs, and * into list-items.

I hope it's comprehensible. It's a little over-engineered with extra methods that are not strictly necessary for the operation. (Such as admin functions I guessed would be useful.)

And it is extremely verbose.

Now compare the same thing in my new, functional style which dispenses with objects and classes but seriously uses generators and closures.

def makeSection(f) :
    def gen(source) :
        for x in source :
            yield f(x)
    return gen

  
def makeFilter(test) :
    def gen(source) :
        for x in source :
            if test(x) : yield x
    return gen

  
def pipeline(generators) :
    def p(source) :
        old = source
        for g in generators :
            new = g(old)
            old = new
        return old
    return p

  
def blankLines(item) :
    if item == '' :
        return "<p/>"
    else :
        return item

  
def bullet(item) :
    if item[0] == '*' :
            return '<li>' + item[1:].strip() + '</li>'
    else :
        return item

        
def linewisePipe(page, pipe) :
     return '\n'.join(pipe(page.split('\n')))
     

# test it
pipe = pipeline([ makeSection(blankLines) , 
                  makeSection(bullet) ])
page = """here

is

* some
* data"""

print linewisePipe(page,pipe)        

So, how does it work? Instead of handling each stage of the pipe with an object with only one significant method : process, it now handles each with a single function. Except that function is, in fact, a generator (ie. it's a function who's execution frame doesn't disappear when it yields it's return value, but continues in memory, and gets resumes where it left off).

The reason for this, is that these generators take a in iterator for their argument. And only pull and process the next item from the iterator before yielding it. The use of a generator is important because once it's up and running it's the generator itself which is remembering which item is currently being processed.

Of course, writing generators is a mildly more complicated than writing functions, so I've written some "make" functions to turn ordinary functions into them. makeSection(f) takes an ordinary x -> y function and returns the generator as a closure; note how the actual processing function is bound to f at the call of makeSection.

The result of calling makeSection, then, is a new generator which a) sucks something from an iterator (called "source"), b) calls f on it, c) yields it (ie. returns it, but hangs around with the next item from the iterator in "x", which it will yield next time it's called)

To simply use one of these sections on its own you could do something like this :

s = makeSection(lambda x: x * 2)
for x in s( [1,2,3,4,5] ) :
   print x

Closure s takes an iterator as an argument (in this case, the list [1,2,3,4,5] ). It's result is also an iterator. (Which is what Generators look like from outside)

In fact the "for" loop keyword is implicitly calling next() on it every time to get a new value for x. When it does so, it pulls the next value out of s. Now, inside s, we have the body of "gen" as defined in makeSection. This is applying the function f (which doubles its argument x) on each value that comes through.

So sctions are always generator functions that are started with an iterator as input and at every step of the iteration return the next item from their source iterator transformed. The transformation itself depends on the argument given to the call to makeSection.

Filters are the same, except in this case, they only pass on items that meet the "test" criteria.

The next part of the program is to chain a number of these generators together into a pipeline. In fact, "pipeline" is another function which returns a closure. It takes a list of sections and filters and it's this list which is stored in the closure. (Be careful here, the reference to the list not the actual list content - so there's a danger of changing them after the execution of pipeline)

What comes out of pipeline is a new generator. One which takes an initial iterator as source, and on each next, sucks the next item from that source all the way through all the stages of the pipeline and spits it out.

So what are the advantages of this over my original object oriented version? It's certainly far "cleverer". To me, today, it looks more elegant. And it's much easier to start using. My specific transformation functions don't need to know anything about pipelines - they don't need to be defined as methods on new classes that are explicitly sub-classed from PipeNode. Instead each function can be written in ignorance of it's role, and then turned into a section or filter when needed. (Often just before assembling the pipe.)

Pipes are easy to assemble using the pipeline function, and I believe are very intuitive to use.

Of course, the object version could be tweaked to look more like the functional version to the outside user. The Pipeline object could be an iterator, making it possible to use it in a for loop. We could perhaps make it take a list of sections in it's constructor rather than force the programmer to write multiple explicit addNodes.

Nevertheless, I'm smitten with the new style. I'm replacing the old code with the new today.