Custom Search

Wednesday, November 23, 2011

Why the Mac UI sucks

The problem

One of my ex-bosses - a very sharp man whom I respect immensely - says that one of the problems with X is that there are too many choices for window managers. This causes people to waste time trying different window managers and playing with configurations rather than actually doing work.

The second part of that statement is certainly true - people do spend an inordinate amount of time essentially playing with window managers. I've certainly spent more than my share of time with them, starting with extremely configurable managers like gwm and sawfish, and then exporting a window managers internal objects via CORBA to make them available to other scripting languages.

I eventually realized that most of my work was trying to get the window manager to make efficient use of the available space with minimal effort on my part, so switched to working on managers that already did that, like ratpoison, and eventually writing an extension for plwm to provide a UI similar to ratpoison. Lately, I've started using dynamic window managers, like wmii and xmonad, which do all that for me, and also provide for windows that don't play well with their paradigm.

The reason people spend time configuring window managers - and even writing new ones - is because they are important! The window manager is something you have to deal with every time you use a computer, on an ongoing basis.

I even invented a benchmark for window manager quality based on this: the curses benchmark, which is how many times a day you curse because your window manager misbehaved. A good window manager will have a curses value well below one a day. The Mac doesn't makes that cut. About the only good thing that can be said about the Mac UI is that it's better than MS Windows.

Going back to the original statement, I agree with the first part. X has too many window managers. I suspect that's because the alternatives have too few. So people who can't stand the paradigm they provide wind up writing their own window managers to try and get a paradigm that they can stand.

Tiling managers

One of the things I observed - about myself and others - is that we tend to lay out collections of windows in a non-overlapping fashion using all available space. This is called a tiling. Much of my work with scripting managers was trying to get tilings built automatically. There are window managers that do this for you. They were popular in early windowing systems, but were quickly replaced by the overlapping model most often seen today. They didn't completely die out - at least not on X.

The tiling paradigm has three advantages over the overlapping windows paradigm. First is that they will use all the space available to them automatically, with no effort on your part. Second, since there is usually a visual order to the windows, there's an obvious paradigm for navigating between windows with the keyboard. For that reason, they appeal to people who don't like using pointing devices. Personally, I think window management is to important to be left to devices with the low bandwidth of pointing devices. Third, they actually manage your windows, rather than providing you with tools and making you do it yourself.

As I mentioned, lately I've started using dynamic managers. These will change the tiling automatically as windows are added and removed. In particular, I've recently switched to xmonad, which seems to fit my work flow really well. It has the notion of a master window, along with some secondary windows. One half the screen (normally left-right, but up-down is available for small screens) is reserved for the master window or windows (since you can ask for more than one). The other half is filled with secondary windows. When you open a window with the master window having the focus, the new window will replace the master, and the master will be added to the secondary windows. Opening a window with a secondary active causes the new window to be added to the secondary windows. No overlapping, no minization, no actions required by the user.

A task comparison

So lets compare xmonad to an overlapping window manager doing a typical work flow, like reading mail. For most messages - at least for me - the dynamic manager won't make a difference. I'll just delete and/or save most messages as I read them, and move on to the next. I may click a link or two in one, which will open in the browser that I've got in the secondary windows area.

But now I want to answer one. So I click reply, and a reply window opens up. While the pointy-clicky editor built into the mail reader is OK for short things, for a long message I'll want to use a real editor, so I choose mail readers that have an option to edit a draft message in an external editor. Choosing that option will open a third window for the editor.

With an overlapping window manager, I've now got two windows stacked on top of the mail reader, probably with no real relation to it. If I want to refer back to it - say to check that I didn't delete to much text in the editor, or look at the text of a different message I want to refer to while working on the draft, I usually have to bring one or more to the front and re-arrange things to get them where I can see one of the two older windows. This extra activity generally results in a curse or two, and happens more than once a day.

With xmonad, no such problem occurs. As each new window opened, the old one was moved to the secondary area, with no areas obscured. So they aren't in the way, but are accessable. It's probably also resized, so I may need to scroll a bit. Should I want better access to it, I can either swap it with a master window, or add it to the master area (where it would presumably have more space).

Bottom line

This actually matches my normal work mode almost exactly. I normally work with a single window - or maybe two - that I'm working with, and a few (between two and four) others that I refer to every once and a while. xmonad optimizes that behavior. I don't spend time worrying about window sizes and placement. I occaisionally swap a pair of windows, or maybe switch one to full screen mode (yes, xmonad supports that), but normally things are where I need them, with no time or effort on my part.

Isn't that kind of efficiency worth a little time spent configuring things?

Friday, October 28, 2011

Leveraging Haskell's derived classes.

One of the mantras of dynamic languages is "Minimize boilerplate". By boilerplate they mean repeated code sequences that just change a few names. A typical example is the setter/getter methods of Java objects - at least most of  them. Python, for example, deals with those by providing a decorater that turns an attribute get or set into a method invocation. This eliminates the primary motivation for setter/getter boilerplate, since you can turn attribute accesses into methods without changing the external API.

I recently wrote enough boilerplate code for a Haskell application that I went searching for a way to eliminate it. Sure enough, I found one. It uses a technique that doesn't seem to be discussed on the web, even though it's so powerful that I eliminated all of the boilerplate that had driven me to this search.

A little Haskell data

This section explains some of the syntax of Haskell. If you're already familiar with that, you should probably skip down to the next section.

Haskell uses about the simplest construct possible for creating the equivalent of a record or structure (or maybe class) in other lanugages:

    data Name = Constructor [ Type ... ]

This creates the new data type Name with a constructor function named Constructor. Following that is a list of the type of each element that is going to be in Name.

A union type isn't much different, and explains why you need both a Constructor and Name in the data declaration:
    data Name = Constructor1 [ Type1 ... ] | Constructor2 [Type2 ... ] [| .... ]

Which says that an object of type Name will be either a Constructor1 object or a Constructor2 object or a Constructor from that last ..., each of which can be built by the appropriate constructor name.

The last bit of syntax is an optional
deriving clause at the end of a data statement. It lets you specify a list of typeclasses - from a relatively small set - and the compiler will automatically provide the boilerplate to make your new type implement the interfaces for those typeclasses. Besides the limit on which typeclasses this works for, all the components of your type have to implement the interfaces you want to use.

The problem data type

One of the Project Euler problems involves evaluating poker hands. So, given the appropriate definitions of Rank (the value of a card - Ace, Two, etc) and Card ( a Rank and a Suit), a data type that represents poker hands would be: 

    data Hand = HighCard [Card]
              | PairOf Rank [Card]
              | TwoPair Rank Rank [Card]
              | ThreeOfAKind Rank [Card]
              | Straight [Card]
              | Flush [Card]
              | FullHouse Rank Rank [Card]
              | FourOfAKind Rank [Card]
              | StraightFlush [Card]
                deriving (Show, Eq, Ord)

I'll get straight to the punchline: the above is sufficient for evaluating poker hands. The Ord typeclass provides the ordering functions <>, and similar as well as min and max. The automatically derived code will work such that hand1 > hand2 if and only if hand1 is a better poker hand than hand2. Provided, of course, that the code that creates the hands is correct.

The derived operators

The operators you get from Ord have straightforward behavior (though I couldn't seem to find an explanation of it outside the language specification). A list - which in haskell holds homogeneous types, but can be of different lengths - is compared elementwise from front to back. The first non-equal pair determines the result. If one list ends before that and before the other then it has a lesser value. If they end at the same time without any differences, they're equal. This is a pretty standard behavior for list comparison, implemented by most modern languages. Tuples (not used above) are a fixed-size list of heterogeneous types. The elements are also compared in order. If two tuples are of different lengths, comparing them is a compile time type error. Ditto if a pair of elements that need to be compared aren't comparable. Again, the derived behavior is pretty standard behavior for modern languages.

A record data type is, to a large degree, isomorphic to a tuple of the same list of types. The derived behavior follows that - the elements in the record are compared in the order they appear in the constructor. This is rather unusual. At least, I'm not aware of other languages that do this automatically, or advertise it if they do. Many modern ones don't really have a record type, but instead have a class type that mixes in other behaviors as well.

Finally, the union type. The comparison is simple: those that appear first in the data statement have lower values. The closest thing to this I've seen elsewhere are enum types. Oddly enough, if none of your constructors have an argument - meaning they're just a bare token - you have what amounts to an enum type, and that same ordering rule applies.

In action

So, we now know how the derived ordering behaves, and have a relatively complex data type to see how it works.

Given two hands to compare, the first check will be whether or not they use the same constructor. If so, those will be compared as such. If not, then the first one in the list is the lesser value. Checking the declaration, that means  a StraightFlush beats everything, FourOfAKind everything but a flush, etc. Exactly what we want.

If the two hands are the same type, then the elements of the hand are compared. That's why there's a Rank or two for some hands - you compare the cards that give the hand it's type before comparing high cards in the rest of the hand. So if those Ranks are correct - and in the right order for TwoPair and FullHouse - then the derived comparison will work correctly. As an aside, for some games you only need the first rank for a FullHouse and don't need the [Cards] for it, ThreeOfAKind and FourOfAKind, because two hands where those match are impossible. But providing them makes the code more symmetric and saves code in the long run.

Finally, if any Ranks present are the same, the [Card]s are compared. It's sufficient to make sure the [Card]s are sorted. While the cards in the Ranks aren't part of that comparison in poker, they will either wind up in the same place in the lists and be equal or the comparison will be done before their places are checked.


So, having written the data statement properly, and making sure that all Hands are properly formed, the derived code will do the right thing. All of the essentially boilerplate code to compare two Hands has been eliminated.  I can make sure that all Hands are properly formed by exporting a Hand-making function that does that, and not exporting any of the Hand constructors. This means the only way client code can get a Hand to compare is to use my Hand-making function, and anything else compared to a Hand will cause a compile time type error.

Wednesday, October 19, 2011

"More Power" is not always a good thing

In Succinctness is Power, Paul Graham ponders why Python — or any language — would choose readability over power. He isn't sure what is meant by regularity, and discusses several definitions of readability, but never gets to what I think is the core of the what Paul Prescod meant by his statement.

Readability isn't simply about the line or character counts that Paul talks about. It's about how quickly you can pick up code that someone else wrote, or that you wrote six months ago, understand it, and fix a bug or add a feature. Paul's discussion of programming features focuses on how easy it is to write code to do new things. If you're developing new code, that's what you want. On the other hand, if you're maintaining code, the hard part is usually figuring out what the original coder was up to, not making the required changes. In this latter case, readability is more important than power. In extreme cases, it's easier to rewrite the code from scratch than to figure out what it's doing. In fact, this is one of the rules Kernighan and Plauger's Elements of programming style: Don't fix bad code, rewrite it.

Readability also helps when the goal is to explain new things to other people. If you're planning on publishing the code to demonstrate or document a new algorithm of some kind, then readers need to be able to pick it up and understand it. A powerful feature that saves you 60 minutes of development time but costs thousands of readers a minute or two each to understand is a net loss.

Regularity is part of this. If a language is regular, then you can expect that if some language feature works for some type in the language, it'll work for all similar types; that all statements will be terminated in a similar manner; that function arguments will be evaluated in the same order everywhere; that — well, the list is potentially infinite. A language being regular improves readability by making it reasonable to infer the behavior of things you aren't familiar with based on the behavior of things you are familiar with, which saves you time when reading a program.

Paul argues that a language can't be too powerful, because the author can always write things out in the less powerful idiom if they don't like the more powerful one. The reader doesn't have that option. If the author is allowed to choose the more powerful option and does, then the reader has to read — and understand — the more powerful version.

Some people claim that a language is powerful if it has lots of features, so that whenever you need to do something, you have lots of ways to choose from for achieving the task at hand. This is in line with succinctness being power, in that if you get to choose between lots of ways to do something, you can chose the most succinct way. This form of power also leads to a less readable language. Some very popular languages have so many features that few, if any, people actually program in the entire langauge. Almost everybody programs in a private subset of that language — their dialect. So when you sit down to read a piece of code, you have to translate from the authors dialect to yours — which may require consulting the manual if it's sufficiently different from yours. In the worse case, you'll be reading code that was written by one person and then maintained by others, so you may well be switching between multiple dialects in reading a single piece of code — meaning that two apparently different bits of code may well provide identical functionality, but the dialect is different.


Let's look at some language features to see how power and readability trade off. We're going to look at one area of language design: passing arguments to a function. Functions are one of the most powerful features of modern languages. They allow code to be reused in multiple places. The arguments to the function allow the same code to be used with different data. Functions also improve readability. Once you read and understand a function, you don't have to do it again. If the author failed to use functions, but simply repeated the code each time, you'd have to read the code to verify that it's actually the same in each case. So functions are pretty clearly good for everyone involved. In particular, we're going to look at one task: figuring out how a variable in our source code gets the wrong value.

Let's start with the least powerful version of arguments, and one found in many modern languages. The language simply evaluates each argument, and passes the resulting value to the function. The function can't change the value of any variables named in the arguments. This is a very readable construct, as if you're trying to figure out what happened to a variable, you can ignore the function calls, because you know they didn't change the variable.

Now let's add a feature: let's let a function be able to change the value of a variable passed to it. This provides a more succinct way of using a function that changes the value of a variable. You can simply write:

def inc m
    m = m + 1

x = 3
inc x
inc x
inc x
print x

And it would print 6. If inc can't change the valaue of it's argument, then the equivalent code would have to be written something like so:

def inc m
   return m + 1

x = 3
x = inc x
x = inc x
x = inc x
print x

Each of the lines that comprise the function has twice as many symbols. More complex examples, involving changing more than one variable and possibly having a return value from the function as well, are much worse. So this new feature make the language more succinct, and hence more powerful.

From the readability standpoint, this is a disaster. Now you have to read every function that the variable you're interested in is passed to, because they may well change it. The amount of code you've got to examine has gone up by an indeterminate amount.

So if your language prefers readability to power, you'd want to give some serious thought to whether or not you want to add this feature to your language. If you do so, you will probably eventually realize that the problem is that there's nothing at the point the function is called to note that the variable could be changed. If you do things that way, then you only need to read functions that are passed variables flagged as changeable at the call point. The language has the same number of features as before, is only slightly less succinct, requiring one more symbol than the original version.

Now that we can change a variable passed to a function, let's look at the case where the variable of interest is passed to multiple function that are allowed to change it, and the values returned by those functions are passed to a function. That isn't inherently harder to read, until you ask the question What order are the arguments evaluated? If the language specifies an evaluation order, then you can read the functions that might change your variable in that order, and ignore the function those functions return values are passed to, unless it has your variable as an argument. If the language doesn't specify the order of evaluation, then you have to consider each possible calling order. But let's make our language really powerful! Let's let the called function decide not only what order the arguments are evaluated in, but allow it to evaluate them multiple times, including not at all if it decides that's appropriate. Lisp macros have this ability, and it's part of what makes Lisp programmers hate programming in languages without them. The invocation syntax is the same as functions, so for the purpose of reading code that uses them, they are the same as functions.

From a readability standpoint, this is a disaster of the same order as letting functions change the values of variables passed to them. It's not quite as bad, because it only affects functions which are called as part of evaluating arguments to other functions, which may or may not be all that common. But every time you examine such a case, you have to read through the outer function to figure out whether or not the function call you're actually interested in is evaluated. Worse yet, the answer may be maybe, as it will depend on the value of some variable in this new function. So now you're starting over with a different function and variable.


While these last examples are exaggerated, and real-world LISP programs are generally better written than that, the point is not to say that these features are bad, but to explain how more powerfull is not the same thing as more readable, and provide some motivation for preferring the latter to the former. I don't expect to change anyones mind about what programming features — and hence languages — they prefer. I hope I've helped someone understand why other people might prefer readability — and a language that values it.

Thursday, October 6, 2011


I was listening to a podcast review of the game Dark Souls, and the reviewer gave it a near-perfect score of 9.5 out of 10. He believes that it will be a contender for one or more Game of the Year awards. What was interesting was that he didn't think the game was fun. Instead, it was frustratingrewarding and satisfying. The problems the game presented were frustrating, but so well done that when he finally solved them, he felt satisfaction at a job well done and properly rewarded for his efforts.

What's funny is that I knew exactly what he was talking about. I stumbled on Project Euler a week or so ago, and have been working the problems there. Yeah, they're frustrating. But the ones that are the most frustrating are also the most satisfying when solved. That's only part of the reward of solving them, as you usually learn something interesting along the way (but don't tell the kids!). Anyone who took any of Dr. Andree's computing classes at the University of Oklahoma will recognize the feel of the problems there - he would have loved this site. The real puzzle is how I managed to fail to find it before now.

Until hearing that podcast, I never would have thought of Project Euler as a game. But it has badges, awards and trophies like many modern games. It also provides the same sort of satisfaction as a game expected to contend for a game of the year, so I'd have to say it is a game.

If you're interested in logic and math puzzles, especially those which a computer can help with, it's definitely worth a look. I feel that the proper tools are a programming language that supports infinite lists (aka streams) and has a REPL to use as a calculator. I'm using Haskell, switching to Python if I want a mutable array.

Wednesday, September 21, 2011

The myth of "costs nothing"

A common argument to hear when a new feature is proposed for some program or programming language is "If you don't use it, it costs nothing." This argument is generally false, and I'm getting tired of hearing it. So I'm going to explain why it's false here where I can easily refer to it.

The first problem is that nothing is actually almost nothing. For the cost to be nothing, then the code paths would have to be identical to the code paths if the feature wasn't  in the program. If that's the case, then how does the feature get used at all? At the very least, there needs to be a test somewhere that decides whether to use the feature. That code almost certainly adds to the size of the application - even if the feature is only in a shared library, you have to load it to use the feature. Data files and the like may also get bigger.

The difference between nothing and almost nothing is more than a quantity, it's a quality. In particular, you can add arbitrarily many nothings together, and you still have nothing. But if you add enough almost nothings together, you get software bloat. This is why they call it feature creep: each feature adds almost nothing, so the bloat creeps into your application over time.

The second problem is that you are seldom the only user who gets a say in whether a feature is used. Especially in our networked era, where data is routinely shared between users of a program, and collaboration is easy. The problem is that when you're using data from some other user, they get a chance to use a feature even if you've decided not to. So you may wind up paying for that feature, or at least paying the cost of removing the use of it.  Word and document processors share documents as part of a collaboration, and often have a shared template library. Accounting software can download transactions from financial institutions. And so on.

If you're writing a program, the problem is worse, as you may wind up having to read and comprehend the language features you have decided not to use. You almost certainly use the standard libraries that come with the language, and quite often use third party libraries as well. If someone writing one of those libraries decides to use a feature, you're going to pay the cost for it. Either that, or you're going to pay the cost of forking the library to confirm to your coding standard, and of maintaining that fork. More on this in the future.

Bottom line: the problem with "if you don't use it, it costs nothing" is that you includes people that may disagree with a personal or corporate decision, and nothing is only almost nothing, which is actually something.

Saturday, July 30, 2011

Eddie - shell scripting with Haskell

The last few years have seen an increase in interest in functional programming languages, as they are better suited to dealing with the concurrency issues found on modern multi-core desktops. Functional programs have functions that are functions in the mathematical sense: the result depends only on the input parameters, with global state not playing a role. No global state means no global state to change means not having to worry about whether or not you are running the code concurrently, which is certainly easier than having to worry about such things.

I've written about my own exploration of Clojure for just this reason. One of the recurring requests in the Clojure community was wanting to use it for the kinds of things that scripting languages - most notably Perl and Python - were used for. This was sufficiently common that someone even posted a bounty for a tool that would facilitate such usage. This was interesting enough that I designed the tool, though I never got around to writing it.

Function shell scripting isn't quite as strange as it seems. Shell filters are generally functional in nature - the output depends strictly on the input and the parameters of the function. So building filters with a functional programming language should be a natural thing. This turns out to be true.

When I eventually decided that the Java infrastructure made Clojure - lovely as the language might be - unsuitable for my needs, I decided to investigate Haskell next. While I don't see requests for shell scripting tools for Haskell, it became apparent that my design for a Clojure shell tool would work even better in Haskell, as Haskell has a much cleaner syntax for combining functions in various ways.

The result is eddie - a tool for using Haskell functions as shell filters. Eddie itself  is fairly simply - it's mostly infrastructure to compile the function argument, and arrange for the file or files to be fed to the resulting function, possibly one line at a time.

Eddie is pretty simple to use: eddie function files will run the Haskell function function over the contents of files concatenated together, assuming that it has a type of String -> String. The simplest example is to reverse a file, which is simply eddie reverse file. The project site for eddie includes a page describing the various ways you can reverse a file (with eddie commands for each of them) as well as eddie commands to simulate a number of Unix commands, so I won't go into more detail here.

If you're a Haskell programmer, you should have cabal installed, and can use that to fetch eddie from Hackage and install it. I'm not otherwise distributing eddie, except as source tarballs from the project page. The next paragraph explains why.

I think the examples show that eddie is a really flexible tool for doing odd things from the shell. The downside is that it has to drag most of the Haskell compiler along with it. The standard Haskell compiler - ghc - is a 40 megabyte file on my Unix box. Eddie weighs in at around 28 megabytes on the same box. As a result, the first time you run eddie, it winds up loading most of that into the cache, which causes a noticable delay. Further runs are then reasonably quick. Not great, but not horrible. This Haskell compiler implements strings as linked lists of characters, which slows it down.  There are libraries to avoid this, but given the size handicap it's already suffering from, fixing this just doesn't seem like a priority.

The bottom line is that I think that eddie shows that the idea of using a functional language for oddball scripting like this works well, but the implementation has sufficient problems to make using it a bit impractical. What I think such a tool needs is a much simpler language. Eddie does not need all of Haskell's type machinery, just strings, characters, integers and possibly floats. Add in the libraries for dealing with those, and a large selection of higher order functions, and that should be nearly as powerful as having all of Haskell available for the purpose of writing shell filters.

Sunday, June 5, 2011

Dynamic types in a statically typed language

One of the more interesting surprises in Haskell was the regular expression library - or rather, libraries, as the same API can be used with a number of different regular expression engines. Before looking at that, let's look at a couple of more conventional languages.

Python has a typical OO regex API: the module includes a couple of routines for creating regex objects or running common sequences of actions using them (compile, find, match), the regex objects have a couple of methods for matching (find, match, findall, finditer) one or more times or running common sequences of actions for this regexp, most of which return match objects that have methods to query the result of the match (group, groups, pos, span, etc.).

Perl, on the other hand, has a single regex type that is built with the / or m operators and then matched against a string.  The result of the match is either a list of groups from the regex (in a list context) or a boolean indicating whether or not it matched (in a scalar context), with variables set to the groups as a side effect. Various regex features can be used with loops to iterate through multiple matches, etc.

Other functional languages have regex objects and collections of functions for manipulating them, similar to the methods of the Python compiled regex objects.

You would expect that Haskell - being neither OO nor dynamically typed - would follow those other functional languages. Except that Haskell's type system is very flexible. In particular, it uses type variables to stand in for actual types (what some languages call generics) - even in compiled code. It also has a typeclass system that can be used to constrain type variables to types which implement a specific set of operations. The type returned by a function can be a typeclass, and the compiler will check all the known instances of the typeclass against the type the calling context expects, then choose an instance of the typeclass that matches the calling context.

The Haskell regex library includes a typecless - in Text.Base.Regex.RegexLike - and instances - in Text.Regex.Base.Context - that includes instances for Bool, groups from a regex, lists of matches (as returned by Python's findall), lists of match positions, and more. There's essentially one operator =~ (=~~ is for use in monads) that matches a string representation of a regex against a string and return the typeclass instance for that context.

Instead of having a scalar context, the Haskell regex operator has three of them - Bool contexts get a did/didn't match return, Int contexts get a count of matches in the string, and String contexts get the string for the first match. There are also tuple contexts that can return things like start & length information for the match, before/match/after strings, before/match/after/groups, and so on. Contexts of lists of scalars or tuples should return a list of those values for all the matches found.

So the answer then is that Haskell's regex operators are dynamically typed, like Perl's - only with a much finer set of contexts, since they are more completely typed. Further, whereas Perl's dynamic regex operators are built into the language and not something you could write in user code, Haskell's dynamic regex operators are written in user code, and the technics can be used anywhere it might be appropriate. As a final bonus, Haskell's operators are statically typed, so if the type you ask for isn't one of the types that the regex operator returns, you will get a type error at compile time.

Monday, May 9, 2011

Mobile development keeps getting bettter

I just found a nifty web site/application for those who do mobile development - or just think about coding problems while running around town.

The folks at have a pretty spiffy programmers tool - at least for small programs. It will compile (if needed) and execute a source text - possibly with standard input. They even have an API so "native" applications running on various devices can use this service - and the applications seem to exist. I've used the android one, which is ugly but functional.

There are limits on memory, time and processes, and you can't do any network programming. Except for that last bit, nothing that's really a problem for the trivial types of things this interface is suited for.

What it is good for - at least if you don't have the language on your mobile device, which lack some manufacturers seem to see as a good thing - is a quick check on an idea that you would otherwise have to write a note about, and would probably bug you (or at least they bug me) until you got someplace to check them out.

Now that fossil scm has been ported to Android, and a language interface that stores program files as text on the sdcard, you can even put a real development model in place.

Wednesday, April 27, 2011

Why Haskell?

This should not be considered an expert overview of the language. It's a deep language, and I'm still learning it. This is a discussion of what I've seen so far to explain why I chose it.

As I mentioned, I'm not learning Haskell because I expect it to be a marketable skill; I'm learning Haskell because I expect it to improve my programming skills in general. That should happen because it's radically different from other languages I know, so it's idioms should be new to me, and will thus provide more options when working in other languages. At least one person regrets this - because they wind up thinking about how much simpler things would be in Haskell.

As a final bonus, there appears to be more higher mathematics in use when writing Haskell than in most other languages. So I'm hoping to get a little more use from my mathematics degree than I've gotten so far.

Things I look for in a language

Being well-designed

I've heard it claimed that Haskell is in the LISP family. Looking at a typical program, you'd have to wonder about that - it looks more like Perl, with obscure operators scattered throughout the text. Once I figured out what's really going on, I could see the logic of the claim. It's at least as valid as the claim that Scheme is a descendant of Algol. Outside of the type system (more on that later), there's a bit more syntax than LISP, but not a lot. Most of a programs code consists of function calls and binding the value of those calls to names - just like functional code in LISP. The most prominent difference is that, where LISP has one way to express a function call - (function arg arg arg) - Haskell has two: function arg arg arg (without LISPs parenthesis) and arg `function` arg for functions of two arguments. The tricky part is that a valid symbol quoted as shown is an operator, and an operator enclosed in parenthesis is a function name. So 2 + 3 = (+) 2 3 = 5.  Basically, it's a syntax tweak that lets the programmer decide if something is better used as a function or an operator both at definition time - by choosing a name that's an operator or a symbol - and at use time, by choosing to change the function name from one to the other.

So it's not as chaotic as it seems. And the bit about math plays into it - the design of the various functions and the function application system all seem to have a bit of mathematical logic in them.

High-level data types

Haskell has a nice selection of high-level data types. Unlike most languages I've dealt with - Clojure being the most notable exception - they're all immutable. Since evaluation is lazy (another thing that makes it different from other languages), lists - or streams, if you prefer - feature prominently. Another thing making it a LISP language. But it also has tuples, arrays, and hash-maps - though the latter aren't quite up to the performance levels I'm used to. Something about writing immutable hash maps with both a fast lookup and insert (which creates a new map) being difficult.

Boilerplate minimization

This is part of the static vs. dynamic type checking debate. One argument goes that static type checking catches errors at compile time rather than test time, so shortens development time. The counter-argument is that typing in type information takes time and adds more places for errors, so lengthens development time.

Haskell lands squarely on both sides. The compiler does static type checking at compile time. However, it also infers the types of most things, so you almost never have to provide type information. From what I can tell, type information is provided more often specifically to nail down a complex functions type so the compiler will complain if you get it wrong than because the compiler can't infer a type.

Further, Haskell one ups most dynamic type checking languages when it comes to reducing boilerplate. If not having to provide type information for parameters cuts down development time by removing places you can make mistakes, then not having to provide names for the parameters should be even more effective. There's a Haskell style that encourages doing just that. Instead of defining a function with parameters that returns  the result of the calculations with those parameters, you define a function without parameters that returns a function that does the calculations when passed those parameters.

Support for concurrent programming

Seems to be there as well, in a set of features similar to Clojure. Both languages use immutable data structures and functional techniques to make the bulk of a program concurrency-safe by default. Clojure has some container types that will raise exceptions if used in ways that aren't concurrency-safe. Haskell has a type system that captures the notion of not concurrency-safe, so you get compile-time errors if you try such things accidentally.


Yup, the Haskell implementations I've looked at all have a REPL.

Plays well with others

At least one implementation can compile to native code on a number of systems as well. There seem to be wrappers for most of my favorite C/C++ libraries, so the external function support should be quite good.

A note about homoiconicity

Haskell isn't homoiconic. That means it can't have real (i.e. LISP) macros. Which means LISP is more powerful. Right?

Maybe. LISP macros are generally used for one of three things:
  1. Controlling evaluation to create new constructs. That's built into Haskell.
  2. Creating domain specific languages. The ability to make operators functions and vice versa pretty much covers this.
  3. Automated code creation. Um...
Ok, Haskell doesn't have that one. And that is one of the more powerful uses of macros. There is an extension to Haskell (Template Haskell) that covers this. It's not portable. It's poorly documented. It's works an ASTs instead of source code. But it is there.

I'll be blogging about my experiments with Haskell as well, and possibly revisiting some of the things I did in Clojure. Stay tuned...

Tuesday, April 26, 2011

Personal choices in marketability

Having figured out what skills are going to be in demand, the next step is to filter in your personal attributes: Which things you enjoy doing - I assume you're a programmer because you enjoy it, we certainly don't do it for the fame or money. Which things you are good at - in particular, what innate abilities might be required for some choice, and how well you do at those.

My tastes and skills

Probably the single most important factor in what I choose to do is that I can't do design work involving graphics. I've studied the classics in the field, and can critique a design or UI, but my own work is ... well, the further I get from "bland" and "boring", the worse it gets. This blog is about as far as I dare go.

That ties into the things I enjoy doing. While I enjoyed working with web 1.0 technologies, the later versions bother me. JavaScript seems to combine the worst features of Perl and Java. Most of the web frameworks seem to be designed to let designers inject code into a design - which means they play to my weaknesses. I prefer toolkits that are designed for programmers - SeaSide and QtWui come to mind. But those aren't very popular.

If that hasn't make it obvious, I'll state it outright: I'm a picky SOB.  I don't want to work with things I don't enjoy: Windows, Java, C++, Perl. JavaScript I already mentioned not liking, which drags me even further away from doing client side work. I managed to survive the last 35 years with DOS and Windows work being at most minimal parts of my work; I suspect I can survive what's left of my career without having to deal with web client software, and only dealing with server software that I like.

As for what I like: languages that are obviously designed to meet some goal, as opposed to being a mere accumulation of features.  Anything that minimizes the boilerplate in the programming process is good in a language, but not an IDE. I want high-level data types, because those are sufficient for many tasks by themselves. Well done, they can be used for lots of things that aren't necessarily obvious. Finally, I consider a REPL to be one of the best tools for learning a language a beginner could ask for.

The languages I've chosen in the past are Scheme, Python, and Eiffel - and they are all fun. Python is fairly marketable. I like C - at least, the early versions, when it was designed to displace assembler as a systems programming language, as opposed to what it grew into as it was co-opted for ever more things, and that's also marketable, though I'd rather be writing Python.

Finally, there's what I'm looking for in working with concurrent programs. Basically, I want tools that I think of as relating to concurrency issues the way garbage collection relates to memory management. They should hide the details from me and at the same time prevent obvious mistakes. Rather than alloc (lock) and free (unlock) calls, I want high level protection mechanisms that can't deadlock or leave dangling locks. Preferably, a more advanced tool than CSP or variants. And of course, if I use something that needs to have access and/or modification protected without proper protection, I'd like a warning about it - at either run time or compile time.

So what am I studying now?

Well, this blog has chronicled my experiments with clojure. I chose that because it runs on the JVM, which has the potential of letting me write mobile applications, and meets my concurrency requirements. It also exposed me to the Java environment, which is nothing like the Unix environment I'm used to. The major difference is that - well, it's targeted at enterprise applications. Which is also the problem - the amount of boilerplate required to do simple things isn't noticeably less than to do those enterprise applications. I got over it, and created WAR files for a simple app, and got an education.

One of the problems with Scheme (and LISPs in general) is that they don't play well with others. Clojure solved that by integrating tightly with Java. But this makes it inappropriate for use in the Unix world. Writing a quick little hack to automate some task in a JVM-based language doesn't work very well - because the overhead of starting the JVM keeps it from being a quick little hack. People are "solving" this by making the JVM a server process you connect to to handle running your quick little hack. Basically, making that quick little hack part of an enterprise-class system.

With that in mind, I'm now studying Haskell. This is more a "learn it to improve your programming" than something that's directly marketable. I'll discuss why in the next entry.

Saturday, April 23, 2011

Keeping yourself marketable

A recent question on LinkedIn asked about the skill set to be working on if you're hoping to make yourself more marketable. This is something every professional programmer should be thinking about as long as they want to keep doing technical work. I've been thinking about this for a bit, and it ties into what I wanted to write about next in any case.

Before considering what skills you want to be working on, you have to start with what skills are going to be in demand - which means thinking about users are going to want from their computers in the future.

Enter the mobile internet

Unless you've been hiding under a rock for the last few years, you've run into some form of mobile device with internet access - smartphones, netbooks and tablets of various kinds. In particular, Apple (and I'm only an Apple fanboy when compared to Windows fanboys) changed the world with the iPhone. Full screen touchpad devices - with or without keyboards - are everywhere.  It changed the pocket computer (with or without a network connection) from something a small percentage of geeks carried to something almost a quarter of the people in the country have. And they're just getting started, with tablet prices dropping drastically since Apple introduced the iPad, and wireless broadband prices falling.

I've believed since the 90s that the mobile internet was going to be the real growth area for mobile devices. You might never be able to carry enough storage in your pocket to keep a database of, for example, every movie currently playing in the US, but even in the late 90s you could access that information from a cell phone or internet-connected personal digital assistant.

Lately, I've seen examples of what I think people are going want from applications in this environment. Users will want their data to be available no matter what device they're using.  As they move from adding notes on a tablet or laptop in a meeting to major editing on their desktop to a quick fact check on a cell phone while out at lunch, there's no reason to actually have to worry about making sure that everything is in sync - it should just happen. Save will no longer mean put this in local storage, but also imply and copy it to the server, and Open will mean check the server for updates. Variants include classic client/server relationships, where the user edits directly on the server, or saves and opens only go to the server - the user won't really care. In this environment, not only are the devices mobile, but the data is as well.

Technologies for mobile data

The development model for such applications is still being worked on. Obviously, they'll need a network (invariable web) back end no matter what platforms are targeted. The front end can run in a browser - which has the advantage of being portable to many platforms out of the box. With HTML5 and proper CSS, it will even work reasonably well on everything from small mobile devices to the desktop. However, In the brief history of such devices, users have shown a strong preference for native applications over browser-based applications, if for no other reason than mobile devices can't always reach the network. Mobile platforms - unless they're running a lightweight version of a desktop OS - tend to have restrictions on the development environment that vary from platform to platform. Finally, the back end may well run on a cloud server such as Amazon's Elastic Cloud or Google's App Engine, which may have it's own set of restrictions.

The desktop client can be written in whatever technologies are most suitable for the desktop, which will be discussed in the next section.

Mobile applications depend on the platform. The only one I follow closely is Android; I'll expect corrections from readers for the rest. 
For iOS, it's pretty much restricted to Objective C. Symbian is C++. Blackberry is Java. All three of these allow some selection of web tools like JavaScript, Flash and AIR - though which varies from platform to platform. Android is primarily Java-based, though it's open nature means you can develop for the embedded Linux system using anything in the Gnu Compiler Collection. Google supports a variety of scripting languages via the Android Scripting Environment. Finally, that the Android market is open means a number of different programming are available for Android. Many are ports of Java languages.

While EC is unrestricted, AE is restricted to Python and Java. Given Java's popularity in the enterprise computing world, it's unlikely that any cloud server won't allow things running on the JVM.

Exit the clock race

Another relatively recent change is the ending of the chip manufacturers wars to turn out the chip with the highest clock rate. Basically, it got to the point where it was more cost effective to wring more processing out of chips by putting more processors on them than by cranking up the speed. So my old 3.8GHz dual-core Pentium 4 is the fastest (and hottest) popular cpu sold in the US, but slower than many modern 2.xGHz systems. And these days, multi-core chips are the norm for desktops, laptops and tablets, and expected in the generation of smart phones and similar sized devices just being released.

This changes the development world since using the extra speed in these processors means running your code in parallel - which means dealing with concurrency issues. As a result, a number of different tools for dealing with concurrency at a higher levels than locking code segments have emerged. Eventually, these tools will show up in mainstream languages, but there's no harm in getting a head start on them if you can do so.

Summing up

The most valuable people will be able to work on multiple front end platforms - and on the back end as well. Since an applications first implementation is liable to be a pure web application, this makes the web tools - JavaScript and libraries, advanced HTML, and CSS - the tools to study. There isn't much that can be done about that.

Mobile platforms are liable to be the second front end. Those web technologies and Java are the only things that show up on multiple platforms (unless you're writing to the embedded OS under the application layer).

The back end could be any number of things. Ruby on Rails and Python with a number of web frameworks (Django, Pylons, Zope) are popular. Both SQL and the new noSQL databases are frequently used in such applications, and hence worth keeping in mind.  Java is also popular, at least at the enterprise level, where Java's web server technologies are excellent.

So web technologies and Java would seem to be the thing to study. However - Java does not necessarily mean the programming language here, as there are a number of languages that run on the JVM but aren't Java. Most notably, Jython and JRuby are Python and Ruby running the JVM, so those might well be a good combination with a web framework for the language. The other thing to note is that some of these languages - like Clojure and Scala - have interesting concurrency features.

Those who've been following this blog will have read about my experiments with Clojure. Next I'll discuss my preferences, and how that factors into what I choose to do next.

Thursday, April 7, 2011

Almost like a real web app

I took a long weekend over the first, and did the next round of changes to my X10 controller web app. The changes revolve around loading the config information from an SQL database rather than using lists wired into the code, and arranging things so that changed data could be reloaded without restarting the app. A database is normally a critical part of a web app, so this almost makes this a real web app.


The config.clj file has pretty much been rewritten from scratch. It's picked up code from controllers.clj that built the maps used for rendering, etc. The rest of the controllers code has moved into core.clj.

First, the ns changes to pick up clojure.contrib.sql. I chose that SQL module because this is a simple SQL application, so there's no reason to go to the work of finding a more sophisticated solution:
(ns x10.config
  [:use [clojure.set :only (difference)]
         :only (with-connection with-query-results transaction)]]
  [:require x10.core]
  [:import [com.micheldalal.x10 CM17A]])
This also picks up the set code for later use, the controller types now in x10.core, and the X10 controller and Java IO classes.

The globals have changed to references - so that it can reload the data later - and the one that isn't mutable has changed it's name to use + instead of * for ears (CL uses *'s for globals, Clojure for mutable's):
;; The *devices* ref holds the devices map for rendering and finding
;; controllers.
(def *devices* (ref {}))

;; The *ports* ref holds the map from port names (aka /dev/*) to open
;; port objects
(def *ports* (ref {}))

;;; Map from controller type names to code.
;; Note that the keys here must match the types in the controllers
;; table in the database. There should be one entry for each unique
;; controller module. It maps to a function to produce a new
;; controller given a port and delay.
(def +controller-type-map+
  {"CM17A" (fn [port delay] (x10.core.CM17A-controller. (new CM17A) port delay))})
All the actual database interaction happens in load-database, which recreates - with some tweaks - the lists that were wired into the original code from the database:
;; Load the device database into a set of lists.
;; Since the results are lazy, we have to force them with doall before
;; getting the next set of results.
(defn load-database [db-file]
  (with-connection  {:classname "org.sqlite.JDBC" :subprotocol "sqlite"
                     :subname db-file}
     [(with-query-results controllers ["select * from controllers"]
        (doall controllers))
      (with-query-results names ["select * from names order by name"]
        (doall names))
      (with-query-results groups ["select * from groups order by name"]
        (doall groups))
      (with-query-results ports ["select distinct port from controllers"]
        (doall ports))
      (with-query-results codes ["select * from code"] (doall codes))])))
Mostly, this just loads tables into lists of maps, and returns a vector of those. The  things to note are that it collects the port names from the controllers table, and that the four queries are wrapped in a transaction to insure consistent values in the tables (assuming anything updating the database does the same). And, as noted in the comments, the lists are lazy, so they have to be consumed before running another query, which the doall takes care of that.

The last step is to translate those lists of dictionaries into the map that is going to be stored in *devices*. That is straightforward, if a bit long. The let in load-devices takes care of it, step at a time:
;; Load the device database in db-file into the *devices* map.
(defn load-devices [db-file]
  (let [[controllers names groups ports codes]
         (load-database db-file)
         (fn [f list] (apply sorted-map (mapcat f list)))
         (make-map (fn [{:keys [name module port delay]}]
                       [name (agent ((+controller-type-map+ module)
                                      port delay
                                     :error-mode :continue)])
         (make-map (fn [{:keys [name controller code unit]}]
                       [name (x10.core.Module.
                               (controller-map controller)
                               name code unit (atom nil))])
         (make-map (fn [[name group]]
                       [name (x10.core.Group. 
                               name (map name-map group))])
                   (map (fn [[key vals]] [key (map :device vals)])
                            (group-by :name groups)))
         (conj (make-map (fn [{:keys [name command]}]
                             [name (x10.core.Command.
                                     name command)])
                {"Reload" (x10.core.Command.
                            "Reload" (str "(x10.config/load-devices \""
                                          db-file "\")"))})]
     (map #(.close %) 
           (let [[ports-map closing] (make-ports-map (map :port ports)
                                                     (ensure *ports*))]
             (doall (map #(x10.core/set-port @% ports-map)
                         (vals controller-map)))
             (ref-set *ports* ports-map)
             (ref-set *devices* {"Devices" name-map
                                 "Groups" group-map
                                 "Code" code-map})
It loads the lists with load-database, and then uses make-map to transform each dictionary into the thing actually used in the code. The controllers list gets turned into controller objects, and that is then used to attach controllers to the devices created from the names list, which are then used to create groups of devices that can be controlled at once. The groups list needs to be tweaked, as it's maps of name/device pairs, and is turned into a list of name/list of devices before being passed to make-map. Finally, the code table is used to create code entries - a new feature - with a Reload entry that runs the load-devices function again to reload the database. After all that is done, the function starts a Clojure transaction with dosync, and in the body of that calls make-ports-map to create the new set of ports being used and a list of those to be closed, adds the ports to the devices in controller-map, and changes the two ref objects. The transaction returns the list of ports to be closed, which has .close map'ed over it to actually close them.

Note that everything done inside the dosync can safely be run more than once. It may attach a port to a controller more than once, but the second one is a nop.

Nothing this does depends on the old value of *devices*, so changes to it won't change what happens here. The set of ports used doesn't depend on that old value, so this always attaches the same set of ports to controllers. The set of ports closed depends on the old value of *ports*, since a port is only closed if it was in the old value. If another thread changes the value of *ports* to no longer include some port, then that thread will be responsible for closing that port.

These changes had remarkably little impact on the rest of the application. Things that used to reference *devices* needed to use @*devices*. The page rendering code picked up the new page of options automatically. The other changes to handle this were mostly plumbing changes.

The most significant ones change to the Controller protocol and type: instead of having open/close, it has set-port (mandatory) and get-port (a debugging aid):
;;; Bottom level abstraction - an X10 controller object.
;; Controller "devices" set other devices to a state (currently true
;; for on and false for off). They actually go out and *do*
;; things. Must be eating their Powdermilk Biscuits.
(defprotocol Controller
  "X10 Controller Modules"
  (set-device [this code unit state]
    "Set the device at address code/unit to state")
  (set-port [this portmap] "Set the com port to use")
  (get-port [this] "Return the port this controller is attached to."))

;;; To add a new controller type
;; Write the appropriate defrecored that instantiates the Controller
;; protocol, then add an entry to the config.+controller-type-map+ to
;; map the name used in the config database to the record.
(deftype CM17A-controller [^CM17A controller port delay]
  (set-device [this code unit new-state]
    (.setState controller (first code) unit new-state)
    (Thread/sleep delay)
    this)       ; we send set-device, so return ourselves
  (set-port [this portmap]
    (.setSerialPort controller (portmap port)))
  (get-port [this]
    (.getSerialPort controller)))
set-port is a bit odd, in that it takes the map from port names to open ports to look up the open port object to use. Other than that, this is straightforward.

There's also the new Command record, which uses read-string to translate the string from the database into Clojure code that it then eval's when turned on:
;;; A command is clojure code that we run it's turned on.
;; This also keeps a state to make the renderers happy.
;; We use eval here, instead of at load time, so any symbols get resolved here
;; instead of in config. Not sure why....
(defrecord Command [name value]
  (set-state [this new-state]
    (when new-state
      (eval (read-string value))
      (str "Evaluated " name "."))))
The war.clj file was changed to include the new init code, as well as some shutdown code:
(ns x10.war
 [x10.web :only (handler)]
        [x10.config :only (load-devices close-ports)]
        [ring.middleware.stacktrace :only (wrap-stacktrace)]
        [ring.middleware.file :only (wrap-file)]
        [ring.util.servlet :only (defservice)]]
  (:gen-class :extends javax.servlet.http.HttpServlet
              :exposes-methods {init initSuper}))
(defn -init
  ([this config]
     (. this initSuper config)
     (let [db (.getInitParameter this "device-db")]
       (load-devices db)
       (.log this (str "Setting up *devices* from " db))))
  ([this]))     ; because the super config will eventually try and call this.

(defn -destroy [this]

(def app (wrap-stacktrace #'handler))
(defservice app)
The -init function takes some explaining. The war framework will try and invoke -init both with and without an argument, from two different ancestors. If you provide one, Clojure will fail to find the other when that invocation happens. Further, the version with an argument needs to invoke the superclass method of the same name, so the :exposes-method keyword in the ns macro is used to make that available with the name initSuper.

Finally, the web.xml file has additions to run the init code at startup and provide the database file name as an argument:
  <!-- Servlet class taken from first :aot namespace -->
  <!-- Servlet is mapped to / by default  -->
Future work

To turn this into a real web app, it needs an interface to allow editing the various controllers, groups, etc. in a web interface. Since these change rarely - a few times a year is typical - and I'm happy with an SQL prompt, I'm not likely to do that. Adding a DSL for macros and variables is probably next.

As usual, the code can be downloaded from the google code repository.