Custom Search

Tuesday, November 23, 2010

Evaluating Clojure

The point of the X10 Controller web site project was to put together a simple but not trivial project in Clojure to evaluate the language. I think it served that purpose well.

Clojure is a modern LISP. It reuses the best parts of LISP - simple syntax, functional programming, and a powerful macro facility - and adds modern data structures, high level concurrency tools and concepts like abstract data types. And then it all runs on top of the JVM, making it relatively easy to generate the various Java blobs that deploy as applications, to web servers, or out on the cloud. The parts mostly fit together well, but it's still a work in progress. For instance, the protocol, records and types facilities are new in 1.2, and flagged as "experimental".

Overall, I like the language. It still retains enough of LISPs flavor that my time spent with similar languages makes me comfortable there. Better still, the modern data structures are integrated into the language better than I remember them being in older LISPs, with tools for reaching through multiple levels of structure much like one would use the c[ad]*r functions on lists. The end result is that I can create short, conscise, but readable code for navigating relatively complex structures. The simple
(get-in *devices* [display what]) which finds a device or returns nil if it doesn't exist would take several lines in most language, as you'd either have to check the first index to see if it exists, or deal with the exception thrown if it doesn't.

While I wasn't really dealing with concurrency issues here, the concurrency features - which are part of what drew me to the language in the first place - included a facility that exactly met my needs, even though they aren't what it was intended for. That kind of serendipity is a good sign, and I'm looking forward to exploring those features in depth later.

The Java integration, on the other hand, is a two headed beast. Some consider it an abandonment of what makes LISP LISP. On the other hand, it's also part of what attracted me to Clojure. Classic LISP systems did not integrate well into the environment - they wanted to be the environment. There have been a number of attempts to deal with this issue since I last worked with LISP, but none that really captured my attention.

Clojure solved that problem, mostly very neatly. I had little trouble using the available Java libraries for dealing with either serial ports - which would have required wrapping non-Java code at some point in any case - or the MC17A library. Creating a deployable executable was a bit more work, but more on that later. Finally, another bit of serendipity: the protocol and record structures - designed to make it easy to generate objects that interface with Java code - also turned out to be exactly right for both X10 controllers and X10 devices - at least as I needed them. They felt much lighter than the Python versions of these things I had been contemplating, though in all fairness they are also less capable. On the other hand, that they are designed for Java interop instead of being LISP functions means I had to wrap them in functions to be able to use them the way I wanted.

Now for the bad side. In spite of running on Unix systems, the Java environment does not share the philosophy of Unix systems. In particular, I'm using to simple things being relatively simple. I.e. - I can build an executable and deploy it without needing some kind of file describing all the bits and a tool that uses that to pull them together, though that facility is there if I really want it. In the Java world - and hence in the clojure world as well - I don't seem to have a choice. The Java environment, in particular, seems to follow the language in being very verbose for these things, taking a lot of what's essentially boilerplate and repeated information to get things done. Where the tools have been reimplemented in Clojure, things aren't so bad. But in this area, as elsewhere, things are still a work in progress.

The environment makes Clojure a poor choice for projects much smaller than the X10 controller - the capabilities scattered across four files and another four support files would probably have wound up in three files with no support files in Python. On the other hand, the expressiveness and power of the language makes it quite pleasant to use in projects large enough that the infrastructure is a convenience instead of a pain, or where deploying a Java blob of some kind is part of the goal.

I look forward to getting to know Clojure better in the future.

Tuesday, November 2, 2010

An X10 Controller in Clojure

Having finally dealt with the disasters, personal matters, and found the pieces I needed, I've finished the project I started to study clojure. Or at least enough of it to be useful.

The goal is to have a web page to let me control the various X10 PLC modules scattered around the house: mostly it's lights, but the thermostat, a grill, a printer and the garage door opener are all controllable this way. This is an ideal type of project for learning about a language and it's environment. It's about as simple as a project can get without being trivial. A small collection of data, a little bit of state information, and a web page to display that information - or parts of it - along with controls to change that state - or parts of it. It can reasonably be done as one page, but has natural representations on multiple pages as well.

For those not familiar with them, X10 devices are used to control power to arbitrary bits of electronics. They and the controllers the user uses plug into the wall, and communicate with each other over the power line (hence Power Line Control). Later tools include RF receivers as dual-purpose control & power units, that RF remotes transmit signals to which they echo over the power line, plus being able to control one plug themselves. The modules have addresses consisting of a house code - or code - between A and P and a unit number between 1 and 16.

The computer accesses all of these through a dongle hanging off the serial line. Rather, it can transmit RF signals to the receivers. There are quite a few nice packages for doing all kinds of fancy things with these devices, both open-source and commercial, for pretty much any platform, that include the functionality I wanted. They are all overkill for what I want, and none have - or rather, had when I started down this path - web pages formatted for use on a smart phone.

In theory, access to serial line devices from Java is via Suns comm module. In practice, Sun only releases jars for Solaris and Windows. The solution is to use the rxtx module, an LGPL'ed rewrite of the comm module. That was available from the package system for my target platform - all well and good.

The X10 serial line protocol is - well, ugly doesn't really do it justice. Fortunately, Michel Dalal has already written a CM17A driver that worked with my hardware and the Sun comm module. While the comm module can supposedly be configured to use the rxtx module on non-Solaris Unix platforms, that never worked right. Possibly a version problem, or an issue with the build for my platform. Changing the type of the serial comm object CM17A used and recompiling it solved that - I could use the rxtx objects directly, and pass those to CM17A.

X10 Controllers protocols



The bottom level abstraction is a wrapper around the CM17A class. The file src/x10/controllers.clj includes it. At this point, controllers have a simple protocol: open them to use, close them because I'm done, and set an x10 device's state:

(defprotocol Controller
"X10 Controller Modules"
(open [this] "Whatever is needed to open the controller and ports")
(close [this] "Close & free the port for possible reuse.")
(set-device [this code unit state]
"Set the device at address code/unit to state"))

Since I only have one device, I only have one device type, hence only one implementation of this protocol, the CM17A-controller:

(deftype CM17A-controller [^CM17A controller ^String port delay]
Controller
(open [this]
(when-not (.getSerialPort controller)
(.setSerialPort controller
(.open (CommPortIdentifier/getPortIdentifier port)
"Mike's X10 Controller" 2000))
(Thread/sleep (Integer/parseInt delay)))
this)
(close [this] (.close (.getSerialPort controller)) this)
(set-device [this code unit new-state]
(.setState controller (first code) (Integer/parseInt unit) new-state)
(Thread/sleep (Integer/parseInt delay))
this))

I'm eventually going to build a map from human-legible device names (like "Office Light") to these controllers. The tricky part is that I can't open the serial port for the controller when I build the map, as that will cause building the war for deployment to open the port, and fail (because the port is in use by the running application) or hang (not sure about that one - seems to be an issue with the -war plugin for lein). So open is a distinct action, and it checks to make sure the device isn't already open before it does anything so I can safely invoke it on every request.

To go from the config information - which will eventually be in a database of some sort - to actual code, I use a map from the name of the controller type to a bit of code that creates both a java CM17A object a clojure CM17A-controller object:

(def *controller-type-map*
{"CM17A" (fn [port delay] (CM17A-controller. (new CM17A) port delay))})

This is wrapped up as a function, as I want to create a new controller object for each port one is attached to. To actually create it, I map it over the *controllers* list from the config information:

(def *controller-map*
(walk (fn [[name module port delay]]
{name (agent ((*controller-type-map* module) port delay))})
#(apply merge %)
*controllers*))

Since each x10 controller is a bit of hardware - something you can kick - that is usually only serially reusable, I wrap the actual instances in a clojure agent. This will serialize access to the controller. The agent interface expects the processes that it applies to it's data to create new data, which they then return. Since I'm using mutable Java objects, I don't create new data - but I still need to return the old one, which is why each method in the type ends with this.

In the end, *controller-map* and the open, close and set-device methods are the only things a client of this module needs.

X10 Devices protocol



Since I wanted to control things other than single devices - groups of devices to start with, and eventually some kind of macro facility, and possibly variables - this seemed like another place to use the Clojure 1.2 protocol/record system. Ok, maybe it's overkill - we really only have one method for the protocol: set-state, which sets the state of the device, returning a string describing what it did. The multimethod support might work as well, but using a protocol eliminates the need to write dispatching code. The protocol is:

(defprotocol X10
"Protocol to set the state of x10 objects."
(set-state [this newstate] "Set object state. Returns a string to show user."))

As noted, it set-state returns a string to tell the user what it did (or, given the nature of X10 PLC kit, what it tried to do). The actual implementation for an X10 module is straightforward:

(defrecord Module [controller name code unit state]
X10
(set-state [this new-state]
(do
(send controller open)
(reset! state new-state)
(send controller set-device code unit new-state)
(str "Turned " name (if new-state " on." " off.")))))

The fields of the module record are the name to display for this module, the controller it's attached to, the code and unit of it's address, the state - an atom holding the state we last set it to - and a delay that is a the number of milliseconds needed to for this controller to change a devices state. This is where the open method gets used to insure that the controller is open and ready to accept commands.

I also wanted to make sure that things other than hard devices would work. The simplest of those is a Group, which is just a collection of devices that toggle on and off with a single command. The implementation for those is trivial:

(defrecord Group [name devices state]
X10
(set-state [this new-state] (join " " (for [device devices]
(set-state device new-state)))))

Finally, I need a map from device and group names to actual devices and groups that the presentation layer can use. That is the *devices* map:

(def *devices*
(let [name-map (walk (fn [[name controller code unit]]
{name (Module. (*controller-map* controller) name
code unit (atom nil))})
#(apply merge %)
*names*)]
{"Devices" name-map
"Groups" (walk (fn [[name group]]
{name (Group. name (map name-map group) (atom nil))})
#(apply merge %)
*groups*)}))

This builds the device map first, so that it can be used to translate the device names in the group configuration into references to the actual devices.

The web page



So now I need to connect that to a web server. The standard for doing that with Clojure is the Ring module. This calls a handler with a single argument, a Clojure map containging the request information, and expects a map containing the result - including a page to be displayed - to be returned. My handler is defined like so:

(defn my-handler [req]
(let [{:strs [command display what] :or {display "Devices"}} (:query-params req)
result (if-let [device (get-in *devices* [display what])]
(do-command command device)
(if what (str "Unknown device " what " in " display ".")))]
{:status 200
:headers {"Content-type" "text/html" "Cache-Control" "no-cache"}
:body (write-page *devices* display (:uri req) result)}))

(def handler (wrap-params my-handler))

The first part is the handler proper. The second part wraps it with a Ring tool to parse the query parameters. The handler proper destructures those query parameters to get a command, the device type (as display) and what device in particular to use. If that display and device type is found in *devices*, the do-command function will take care of running the command on the device. Otherwise,
the result of the command is going to be an unknown device message.

do-command is a trivial function to look up the command in a case statement and execute the code associated with it, or return an unknown command message:

(defn do-command [command device]
(case command
"on" (set-state device true)
"off" (set-state device false)
(str "Unknown command " command ".")))

That leaves the actual body of the web page, which is generated by the write-page function, using the bitchen html template engine that I wrote to start this project:

(defn write-page [devices display uri response]
(html (head (title "X10 Controller")
(meta_ {:name "viewport" :content "width=160"}))
(body
(h1 "X10 Controller")
(p (join " * " (cons
(a {:href (format "%shelp.html" uri)} "Help")
(for [new-display (keys devices)]
(if (= display new-display)
(span {:style "font-weight: bold"} display)
(a {:href (format "%s?display=%s" uri new-display)}
new-display))))))
(apply table
(for [device (map second (devices display))]
(let [fmt (format "%s?command=%%s&display=%s&what=%s"
uri display (:name device))]
(tr (td (a {:href (format fmt "on")} "on"))
(td {:align "center" :style (case @(:state device)
true "color:green"
false "color:red"
"color:inherit")}
(:name device))
(td (a {:href (format fmt "off")} "off"))))))
(p response))))

This is fairly straightforward: Other than the headers, a paragraph that lists the top-level maps in *devices* - "Devices" and "Groups" - along with a link to a "Help" page. This is followed by a table with one line for each module in the chosen map, with "on" and "off" buttons, and the module name. The module text's color is chosen depending on the state, either green (on), red (off) or inherited if it hasn't been set. Then the last paragraph, with the results of running the command just requested.

Yup, this is a plain, web-1.0ish interface. Then again, that was the goal. It's plain because my designs get uglier the further I get from plain; I'd have to get help to make it pretty. It could be extended to make the "on" and "off" buttons use AJAX to run the command, change the color of the module name and update the result paragraph, but for an application that's never going to have more than a half-dozen users all on the local LAN, that's overkill.

That completes the project as originally conceived. The code is now available in the bitbucket repo, along with an update to clojure.st to support clojure 1.2.

Next, the post-mortem.