Custom Search

Thursday, April 7, 2011

Almost like a real web app

I took a long weekend over the first, and did the next round of changes to my X10 controller web app. The changes revolve around loading the config information from an SQL database rather than using lists wired into the code, and arranging things so that changed data could be reloaded without restarting the app. A database is normally a critical part of a web app, so this almost makes this a real web app.

Config

The config.clj file has pretty much been rewritten from scratch. It's picked up code from controllers.clj that built the maps used for rendering, etc. The rest of the controllers code has moved into core.clj.

First, the ns changes to pick up clojure.contrib.sql. I chose that SQL module because this is a simple SQL application, so there's no reason to go to the work of finding a more sophisticated solution:
(ns x10.config
  [:use [clojure.set :only (difference)]
        [clojure.contrib.sql
         :only (with-connection with-query-results transaction)]]
  [:require x10.core]
  [:import gnu.io.CommPortIdentifier [com.micheldalal.x10 CM17A]])
This also picks up the set code for later use, the controller types now in x10.core, and the X10 controller and Java IO classes.

The globals have changed to references - so that it can reload the data later - and the one that isn't mutable has changed it's name to use + instead of * for ears (CL uses *'s for globals, Clojure for mutable's):
;; The *devices* ref holds the devices map for rendering and finding
;; controllers.
(def *devices* (ref {}))

;; The *ports* ref holds the map from port names (aka /dev/*) to open
;; port objects
(def *ports* (ref {}))

;;; Map from controller type names to code.
;; Note that the keys here must match the types in the controllers
;; table in the database. There should be one entry for each unique
;; controller module. It maps to a function to produce a new
;; controller given a port and delay.
(def +controller-type-map+
  {"CM17A" (fn [port delay] (x10.core.CM17A-controller. (new CM17A) port delay))})
All the actual database interaction happens in load-database, which recreates - with some tweaks - the lists that were wired into the original code from the database:
;; Load the device database into a set of lists.
;; Since the results are lazy, we have to force them with doall before
;; getting the next set of results.
(defn load-database [db-file]
  (with-connection  {:classname "org.sqlite.JDBC" :subprotocol "sqlite"
                     :subname db-file}
    (transaction
     [(with-query-results controllers ["select * from controllers"]
        (doall controllers))
      (with-query-results names ["select * from names order by name"]
        (doall names))
      (with-query-results groups ["select * from groups order by name"]
        (doall groups))
      (with-query-results ports ["select distinct port from controllers"]
        (doall ports))
      (with-query-results codes ["select * from code"] (doall codes))])))
Mostly, this just loads tables into lists of maps, and returns a vector of those. The  things to note are that it collects the port names from the controllers table, and that the four queries are wrapped in a transaction to insure consistent values in the tables (assuming anything updating the database does the same). And, as noted in the comments, the lists are lazy, so they have to be consumed before running another query, which the doall takes care of that.

The last step is to translate those lists of dictionaries into the map that is going to be stored in *devices*. That is straightforward, if a bit long. The let in load-devices takes care of it, step at a time:
;; Load the device database in db-file into the *devices* map.
(defn load-devices [db-file]
  (let [[controllers names groups ports codes]
         (load-database db-file)
        make-map
         (fn [f list] (apply sorted-map (mapcat f list)))
        controller-map
         (make-map (fn [{:keys [name module port delay]}]
                       [name (agent ((+controller-type-map+ module)
                                      port delay
                                     :error-mode :continue)])
                   controllers)
        name-map
         (make-map (fn [{:keys [name controller code unit]}]
                       [name (x10.core.Module.
                               (controller-map controller)
                               name code unit (atom nil))])
                    names)
        group-map
         (make-map (fn [[name group]]
                       [name (x10.core.Group. 
                               name (map name-map group))])
                   (map (fn [[key vals]] [key (map :device vals)])
                            (group-by :name groups)))
        code-map
         (conj (make-map (fn [{:keys [name command]}]
                             [name (x10.core.Command.
                                     name command)])
                             codes)
                {"Reload" (x10.core.Command.
                            "Reload" (str "(x10.config/load-devices \""
                                          db-file "\")"))})]
    (doall
     (map #(.close %) 
          (dosync
           (let [[ports-map closing] (make-ports-map (map :port ports)
                                                     (ensure *ports*))]
             (doall (map #(x10.core/set-port @% ports-map)
                         (vals controller-map)))
             (ref-set *ports* ports-map)
             (ref-set *devices* {"Devices" name-map
                                 "Groups" group-map
                                 "Code" code-map})
             closing)))))) 
It loads the lists with load-database, and then uses make-map to transform each dictionary into the thing actually used in the code. The controllers list gets turned into controller objects, and that is then used to attach controllers to the devices created from the names list, which are then used to create groups of devices that can be controlled at once. The groups list needs to be tweaked, as it's maps of name/device pairs, and is turned into a list of name/list of devices before being passed to make-map. Finally, the code table is used to create code entries - a new feature - with a Reload entry that runs the load-devices function again to reload the database. After all that is done, the function starts a Clojure transaction with dosync, and in the body of that calls make-ports-map to create the new set of ports being used and a list of those to be closed, adds the ports to the devices in controller-map, and changes the two ref objects. The transaction returns the list of ports to be closed, which has .close map'ed over it to actually close them.


Note that everything done inside the dosync can safely be run more than once. It may attach a port to a controller more than once, but the second one is a nop.


Nothing this does depends on the old value of *devices*, so changes to it won't change what happens here. The set of ports used doesn't depend on that old value, so this always attaches the same set of ports to controllers. The set of ports closed depends on the old value of *ports*, since a port is only closed if it was in the old value. If another thread changes the value of *ports* to no longer include some port, then that thread will be responsible for closing that port.


Plumbing
These changes had remarkably little impact on the rest of the application. Things that used to reference *devices* needed to use @*devices*. The page rendering code picked up the new page of options automatically. The other changes to handle this were mostly plumbing changes.


The most significant ones change to the Controller protocol and type: instead of having open/close, it has set-port (mandatory) and get-port (a debugging aid):
;;; Bottom level abstraction - an X10 controller object.
;; Controller "devices" set other devices to a state (currently true
;; for on and false for off). They actually go out and *do*
;; things. Must be eating their Powdermilk Biscuits.
(defprotocol Controller
  "X10 Controller Modules"
  (set-device [this code unit state]
    "Set the device at address code/unit to state")
  (set-port [this portmap] "Set the com port to use")
  (get-port [this] "Return the port this controller is attached to."))

;;; To add a new controller type
;; Write the appropriate defrecored that instantiates the Controller
;; protocol, then add an entry to the config.+controller-type-map+ to
;; map the name used in the config database to the record.
(deftype CM17A-controller [^CM17A controller port delay]
  Controller
  (set-device [this code unit new-state]
    (.setState controller (first code) unit new-state)
    (Thread/sleep delay)
    this)       ; we send set-device, so return ourselves
  (set-port [this portmap]
    (.setSerialPort controller (portmap port)))
  (get-port [this]
    (.getSerialPort controller)))
set-port is a bit odd, in that it takes the map from port names to open ports to look up the open port object to use. Other than that, this is straightforward.

There's also the new Command record, which uses read-string to translate the string from the database into Clojure code that it then eval's when turned on:
;;; A command is clojure code that we run it's turned on.
;; This also keeps a state to make the renderers happy.
;; We use eval here, instead of at load time, so any symbols get resolved here
;; instead of in config. Not sure why....
(defrecord Command [name value]
  X10
  (set-state [this new-state]
    (when new-state
      (eval (read-string value))
      (str "Evaluated " name "."))))
The war.clj file was changed to include the new init code, as well as some shutdown code:
(ns x10.war
  [:use
 [x10.web :only (handler)]
        [x10.config :only (load-devices close-ports)]
        [ring.middleware.stacktrace :only (wrap-stacktrace)]
        [ring.middleware.file :only (wrap-file)]
        [ring.util.servlet :only (defservice)]]
  (:gen-class :extends javax.servlet.http.HttpServlet
              :exposes-methods {init initSuper}))
  
(defn -init
  ([this config]
     (. this initSuper config)
     (let [db (.getInitParameter this "device-db")]
       (load-devices db)
       (.log this (str "Setting up *devices* from " db))))
  ([this]))     ; because the super config will eventually try and call this.

(defn -destroy [this]
  (close-ports)
  (shutdown-agents))

(def app (wrap-stacktrace #'handler))
(defservice app)
The -init function takes some explaining. The war framework will try and invoke -init both with and without an argument, from two different ancestors. If you provide one, Clojure will fail to find the other when that invocation happens. Further, the version with an argument needs to invoke the superclass method of the same name, so the :exposes-method keyword in the ns macro is used to make that available with the name initSuper.

Finally, the web.xml file has additions to run the init code at startup and provide the database file name as an argument:
<web-app>
  <!-- Servlet class taken from first :aot namespace -->
  <servlet>
     <servlet-name>x10</servlet-name>
     <servlet-class>x10.war</servlet-class>
     <load-on-startup>1</load-on-startup>
     <init-param>
       <param-name>device-db</param-name>
       <param-value>/usr/local/etc/x10.db</param-value>
     </init-param>
  </servlet>
  <servlet>
    <servlet-name>default</servlet-name>
    <servlet-class>org.mortbay.jetty.servlet.DefaultServlet</servlet-class>
  </servlet>
  <!-- Servlet is mapped to / by default  -->
  <servlet-mapping>
    <servlet-name>default</servlet-name>
    <url-pattern>/help.html</url-pattern>
  </servlet-mapping>
  <servlet-mapping>
     <servlet-name>x10</servlet-name>
     <url-pattern>/*</url-pattern>
  </servlet-mapping>
</web-app>
Future work

To turn this into a real web app, it needs an interface to allow editing the various controllers, groups, etc. in a web interface. Since these change rarely - a few times a year is typical - and I'm happy with an SQL prompt, I'm not likely to do that. Adding a DSL for macros and variables is probably next.

As usual, the code can be downloaded from the google code repository.