To address this situation in some of the projects we are working on, we've been looking at algorithms that operate on streams of data.  One such algorithm for estimating GLMs is stochastic gradient descent.  SGD allows one to implement "online" statistical learning algorithms in the sense that as new data points arrive, the model parameters are updated in real-time.  This has some tremendous advantages - it is highly scalable (memory and computation time) as it operates on one data point at a time, it is real-time and the current optimal model is always available, etc.  In order to evaluate the effectiveness of this optimization technique, I put together a demonstration of using SGD to learn an ordinary least squares regression model of a single variable.

The demonstration shows the global likelihood surface of the regression model.  The blue point is the maximum likelihood estimate - equivalent to the least squares solution.  The red point is the current values of the SGD algorithm.  If you click through to the interactive demonstration, you can observe how the model parameters update as each data point is passed through the SGD algorithm.  As more data points are fed to the SGD algorithm, the SGD parameter estimation converges on the maximum likelihood estimation.  It is important to note that while we are showing how the algorithm converges in the global likelihood surface, it is in fact not using the surface at all - rather it is operating on the local gradient attributable to each new data point.  This is what makes the algorithm scalable and on-line.  

The image on the right shows the SGD regression line and how it approaches the MLE regression line.  

There are some potential drawbacks to the SGD algorithm.  For one, it may not converge on the optimal model parameters.  Second, it is hard to imagine how feature selection could be performed online without a significant ramp up in computational time and memory usage.  Third, complex logic needs to be incorporated into the algorithm to decay the impact that old data has on the real-time model and to accentuate the significance of the most recent data.  Nevertheless, the overall technique provides significant boosts to efficiency which may outweigh the issues.

The graphics object that represents a recursive partitioning on a dataset is a dendrogram.  Mathematica provides a Dendrogram function that will visualize nested clusters, but it does not provide a way to label the branches on the Dendrogram.  The original dendrogram looks like the following.

In Mathematica, this object can also be manipulated in its original form.  The original form is an expression.

Graphics[List[List[RGBColor[0, 1, 0], List[]], 
  List[List[], 
   List[List[
     Line[List[List[1, 0], List[1, 2], List[2, 2], List[2, 0]]], 
     Line[List[List[3, 0], List[3, 2], List[4, 2], List[4, 0]]], 
     Line[List[List[1.5`, 2], List[1.5`, 4], List[3.5`, 4], 
       List[3.5`, 2]]]]]], 
  List[Text[Text[Style["B", RGBColor[0, 0, 1]]], 
    Offset[List[0, -4], List[1, 0]], List[0, 1]], 
   Text[Text[Style["A", RGBColor[1, 0, 0]]], 
    Offset[List[0, -4], List[2, 0]], List[0, 1]], 
   Text[Text[Style["A", RGBColor[1, 0, 0]]], 
    Offset[List[0, -4], List[3, 0]], List[0, 1]], 
   Text[Text[Style["B", RGBColor[0, 0, 1]]], 
    Offset[List[0, -4], List[4, 0]], List[0, 1]]]], 
 List[Rule[PlotRange, All], 
  Rule[AspectRatio, Power[GoldenRatio, -1]]]]

In the dendrogram above, each path represents a region in space that would be classified with the label at the leaf node.  Each branch node represents a rule that decides which branch of the tree should be followed to classify new data points.  (See the demonstration at the end of the post for details).  In order to get the appropriate text onto the branch nodes, I had to destructure the graphics object and construct a parallel graphics object with the textual elements of the tree.  Mathematica has Position function that takes any Mathematica expression and destructures it using a rich pattern matching library.  Getting the positions of the branch nodes is done as follows:

Extract[FullForm[d], 
   Position[FullForm[d], Line[__]]] /. {Line[{_, h_, i_, _}] -> {h, i}}
{{{1, 2}, {2, 2}}, {{3, 2}, {4, 2}}, {{1.5, 4}, {3.5, 4}}}

In the expression above, the call to Position passes in the Line[___] pattern to match at any nested level the Line objects in the graphics object.  Then, the positions are extracted from the full form of the dendrogram graphic object and the center two vertices of the line are pulled out as well.  These center two vertices refer to the horizontal line of each branch node.  We can use these vertices' positions to construct the textual objects appropriately.  The following tree is the final result.

I have tested the algorithm on 20 Newsgroups data set.  I started with only two newsgroups to see how well the algorithm performed.  The following chart shows the two sets of documents projected into two dimensions of the concept space.

The blue points represent documents from the sci.space newsgroup and the red points  from the rec.sports.baseball newsgroup.  One can see that the algorithm has effectively separated these two groups in the concept space.  There is some overlap in the center as well as some outliers.  As a result of the overlap, there was some mis-classification.  However, the actual clustering implemented so far is not very sophisticated.  It simply selects the most similar centroid based on cosine similarity.  A more effective clustering implementation would involve agglomerative clustering or some form of k-means clustering.

Another nice effect of SVD is the ability to extract the concept vectors.  These serve to characterize the clusters.  One can use these concept vectors to induce labels or to profile clusters.  Some of the concept vectors for the above example are:

• us mission abort firm pegasus data pacastro system communic m contract ventur servic probe commerci  market space satellit launch
• homer win astro saturday eighth friday sunday hit doublehead klein cub second third home game run score inning doubl

These are just two of the concept vectors.  There are k concept vectors where k is the specified reduced rank supplied to the LSA algorithm.  The next step is to map the cluster centroids to the concept vectors.

Currently, the LSA algorithm uses Parallel Colt's SVD so the matrix algebra is done in-memory.  This means that it will only work for small numbers (300-500) of documents.  The next step is to investigate moving to Apache Mahout's distributed matrix library.

Consider the case of searching for the nearest road from a point on a map.  A natural way of performing this search is to expand a bounding box around the point and use the && operator to select the roads that intersect that bounding box.  Then, the distance to each of those roads in the returned subset is computed and the minimum distance is returned.  Observe the following scenario:

The tiny square in the upper right (just below and to the right of the rectangle) is the bbox of the point from which we wish to find the nearest road.  The yellow rectangle is the bbox of the nearest road.  The large transparent blue rectangle is a bbox of the next nearest road.  So the only overlapping bbox for the point is the large rectangle.  Thus, the && operator does not find the nearest road and our calculation is wrong.

The solution I have come up with so far is to use the bbox operator, compute the nearest distance, and use a new bbox expanded around the point using the just computed distance.  This operation will find any overlapping bbox within that range and will come up with the correct nearest road.  I don't like this solution as it requires two && searches and multiple distance computations - not very optimal.

To implement the GLM in Clojure/Incanter, we first need to implement the IRLS algorithm.  If we assume that we know the link function (and its inverse, derivative, and the weight function), then IRLS is implemented as follows:

(defn irls [y X B invlink dlink weight eps]
  (let [
    _irls (fn [Bnext] 
          (let [
        eta (mmult X Bnext)
        mu (invlink eta)
        z (plus eta (mult (minus y mu) (dlink mu)))
        W (diag (weight mu))]
        (mmult 
         (solve (mmult (trans X) W X)) 
         (trans X) W z)))
    ]
    (last 
     (last 
      (take-while 
       (fn [x] (> (euclidean-distance 
           (first x) 
           (last x)) eps)) 
       (partition 2 1 (iterate _irls B)))))))

In the above code, we define the update step as an internal function of the updated coefficients variable.  Then, we iterate over an infinite sequence of updates until the condition that the euclidean distance between successive iterations is less than epsilon.

Next, we need to define the link functions and other associated functions of each member of the exponential family of distributions.  I have shown Gaussian and Binomial distributions below:

(defstruct family :link :invlink :dlink :weight)
(def families
     {
     :gaussian (struct-map family
           :link (fn [x] x)
           :invlink (fn [x] x)
           :dlink (fn [x] 1)
           :weight (fn [mu] (repeat (length mu) 1)))
     :binomial (struct-map family
           :link (fn [x] (log (div x (minus 1 x))))
           :invlink (fn [x] (div (exp x) (plus 1 (exp x))))
           :dlink (fn [x] [x] (div 1 (mult x (minus 1 x))))
           :weight (fn [mu] (to-vect (mult mu (minus 1 mu)))))
      })

I have used the struct-map technique from Clojure which gives me a sort of family type.  Additional families would be specified here.  Now, similar to R, we can pass the family type to a general GLM function and have one estimation technique (the IRLS defined above) for all families.  The GLM function is shown:

(defn glm 
  ([y X & opts]
      (let [opts (when opts (apply assoc {} opts))
       family (or (families (:family opts)) 
              (:gaussian families))
       intercept (or (:intercept opts) true)
       eps (or (:eps opts) 0.01)
       bstart (:bstart opts)]
       (irls y 
         X 
         bstart (:invlink family) 
         (:dlink family) 
         (:weight family) 
         eps))))

The GLM function simply delegates to the IRLS function with the distribution specific link, inverse link, etc functions.

To test the GLM, I used the example from the Incanter linear-model documentation:

user=> (use '(incanter core stats datasets charts))
nil
user=> (def iris 
  (to-matrix (get-dataset :iris) :dummies true))
#'user/iris
user=> (def y (sel iris :cols 0))
#'user/y
user=> (def x (sel iris :cols (range 1 6)))
#'user/x
user=> (def iris-lm (linear-model y x))
#'user/iris-lm
user=> (:coefs iris-lm)
(2.171266292153149 0.4958889383890437 0.8292439122349187 -0.31515517332664444 -1.0234978144907245 -0.7235619577805039)

Now, does the GLM with the Gaussian family give the same coefficients?  First, we add an intercept column.

user=> (def x (bind-columns (repeat 150 1) x))
#'user/x
user=> (glm y x 
     :bstart (matrix [1 1 1 1 1 1]) 
     :family :gaussian)
[ 2.1713
 0.4959
 0.8292
-0.3152
-1.0235
-0.7236]

Finally, to test the binomial family, I used the "infert" dataset from R:

user=> (def sp (matrix [2 0 0 0 1 1 0 0 1 0 0 1 0 0 1 0 1 1 1 1 1 0 2 1 1 2 2 2 2 0 1 0 0 2 0 2 1 2 0 1 2 0 0 1 0 0 2 0 0 2 2 2 1 1 2 2 0 2 1 2 2 1 1 2 0 1 1 2 2 0 0 1 1 2 2 1 1 0 1 1 0 1 1 0 0 0 1 0 1 0 0 1 1 0 1 0 1 0 0 2 0 1 0 0 0 0 0 1 0 0 0 2 1 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 1 0 0 1 1 1 0 0 0 1 0 0 0 0 0 0 0 1 0 1 0 1 0 0 0 1 2 0 0 0 0 0 0 1 1 0 0 0 2 0 2 0 1 0 1 1 1 0 2 0 0 2 0 1 0 0 0 0 1 2 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1 1 1 0 0 0 0 0 0 0 0 2 0 0 0 2 0 0 0 0 2 1 0 1 1 1 0 0 1 1]))
#'user/sp
user=> (def in (matrix [1 1 2 2 1 2 0 0 0 0 1 2 1 2 1 2 2 0 2 0 0 2 0 0 1 0 0 0 1 2 0 1 1 0 1 0 0 1 0 0 0 0 0 1 0 1 0 2 2 0 1 1 1 0 0 0 1 0 0 0 0 0 0 0 1 0 0 1 0 1 1 0 0 0 0 0 0 1 0 0 2 0 0 2 0 2 0 2 1 0 2 0 0 0 1 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 1 0 0 0 0 2 0 1 1 0 0 0 1 0 1 2 1 1 2 1 1 1 1 1 1 2 1 1 2 1 0 0 0 0 0 2 1 0 1 0 0 0 0 2 0 0 0 0 0 0 2 0 2 0 0 0 0 0 1 0 0 0 0 0 1 0 1 1 1 1 0 0 2 0 0 0 0 0 2 1 0 0 0 2 0 0 0 2 0 0 0 0 0 0 0 0 1 2 1 1 2 2 2 0 1 0 2 1 0 1 1 1 0 1 0 1 0 2 0 1 0 1 0 0 1 1 0 0 0 0 2 0 0]))
#'user/in
user=> (def case (matrix [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]))
#'user/case
user=> (def X (bind-columns (repeat (length sp) 1) sp in))
#'user/X
user=> (glm case X 
  :bstart (matrix [0 1 1]) 
  :family :binomial 
  :eps 0.001)
[-1.7078
 1.1972
 0.4182]

To continue learning Clojure and Incanter, I implemented the algorithm as follows:

(defn mc-pi-calc [n] (/ (count (filter #(<= %1 0.5)
 (map #(euclidean-distance (vec %1) [0.5 0.5]) 
   (partition 2 (sample-uniform (* 2 n)))))) 
     (* n 0.5 0.5)))

user=> (mc-pi-calc 10000)
3.1432

The function takes the number of samples as input and uses the sample-uniform function to generate n points in the unit square.  Then, it counts the number of points that fall within a circle inscribed in the unit square using the euclidean distance function and divides this count by the total number of samples to get the area of the circle to area of the square ratio.  From this, the value of PI is easily calculated.

;; define the sample points
(def data (trans (map vec (partition 2 
    (sample-uniform (* 2 10000))))))

;; plot the sample points
(def p (scatter-plot (first data) (second data)))

;; overlay the points in the circle
(def data2 (trans 
   (filter #(>= 0.5 
      (euclidean-distance %1 [0.5 0.5])) 
   (trans data))))
(add-points p (first data2) (rest data2))

;; view the resulting plot
(view p)

(def data (pmap (fn [nsamples] 
    (take 100 (repeatedly (fn [] 
       (take nsamples 
           (repeatedly #(mc-pi-calc 100))))))) 
    (range 10 100 1)))

(pmap (fn [exp] (let 
   [d (map #(/ (sum %1) (count %1)) exp)] 
   (/ (count (filter #(<= (abs (- %1 3.14159)) 0.01) d))
      (double (count d))))) 
   data)

I wanted to investigate Incanter, but first I needed to wrap my head around Clojure.  Folding and nesting are common procedures in functional programming languages and in numerical methods.  You can find an excellent discussion of folding here.  Many algorithms follow the iterate and accumulate procedure that naturally maps to the folding and nesting paradigms.  Newton's Method for polynomial root finding follows this paradigm as do many optimization algorithms that converge on an extrema.  As an avid Mathematica enthusiast, I often used NestList to implement and visualize the steps of nesting algorithms, but I haven't found the equivalent built into the libraries of any of the other three languages.  So, to dive in to Clojure and Incanter, I decided to implement the root finding in Clojure, Scala, and R for comparison.  First, we need a NestList equivalent.  Then, we can use the NestList function to implement the root finding algorithm which we'll test out on the trivial polynomial x^2-5 which obviously has its root at ~±2.23606797749979.

The root finding algorithm using NestList in Mathematica can be viewed here.

In Clojure, the implementation and usage is as follows:

(defn nestlist [fn iv n] (take n (iterate fn iv)))
(defn findroot [f df iv n] 
     (nestlist #(- %1 (/ (f %1) (df %1))) (double iv) n))

user=> (findroot #(- (* %1 %1) 5) #(* 2 %1) 1.0 20)
(1.0 3.0 2.3333333333333335 2.238095238095238 2.2360688956433634 2.236067977499978 2.23606797749979 2.23606797749979 2.23606797749979 2.23606797749979)

The iterate function is exactly what I was looking for.  The lazy evaluation of the sequence makes it easy to work with and the definition of the nestlist function is just syntactic sugar on the iterate function.

In Scala, the implementation of nestlist uses the foldLeft function and accumulates its results 

def nestlist(f:(Double)=>Double, iv: Double, n: Int): List[Double]={
   (0 until n).foldLeft(List(f(iv)))
        ((xs,i) => xs ++ List(f(xs.head)))
}
def findroot(f:(Double)=>Double,
     df:(Double)=>Double,iv:Double,n:Int):List[Double]={
   nestlist((x)=>x-f(x)/df(x),iv,n)
}

 The accumulating list is a bit awkward.  I'm sure there are cleaner methods for implementing nestlist in Scala.

 Usage of the method:

findroot((x)=>x*x-5,(x)=>2*x,1,10)
res1: List[Double] = List(3.0, 2.3333333333333335, 2.238095238095238, 2.2360688956433634, 2.236067977499978, 2.23606797749979, 2.23606797749979, 2.23606797749979, 2.23606797749979, 2.23606797749979, 2.23606797749979)

In R, I had to resort to a very non-FP for loop. I looked at the apply family of functions and replicate but couldn't come up with a good algorithm quickly so here is the result.

nestlist=function(f, iv, n) { 
   acc=as.vector(iv)
   for(e in 1:n) acc=append(acc, f(acc[e]))
   acc
}
findroot=function(f, df, iv, n) 
   nestlist(function(x) x-f(x)/df(x), iv, n)

And its usage:

> findroot(function(x) x^2-5, function(x) 2*x, 1, 10)
 [1] 1.000000 3.000000 2.333333 2.238095 2.236069 2.236068 2.236068 2.236068
 [9] 2.236068 2.236068 2.236068

Next steps are to start playing around with Incanter and implement some statistical procedures that utilize the root finding algorithm.

There are situations in which one would like to know all the nearest neighbors between one feature and another feature.  This could be useful in diverse areas such as real estate planning (what is the best place to develop given proximity requirements to schools and grocery stores?) to disaster management (what is the most accessible and safest point between these hardest hit areas?).  The nearest neighbor query now has to proceed sequentially along all geometries of one feature computing nearest neighbors to all geometries along another feature.  This becomes computationally intractable as the size of the feature tables grow.

An easy way to improve this type of query is to exploit the spatial proximity of adjacent features in a table.  Using a smart sequential scan, the algorithm "remembers" the nearest neighbor distance from the last feature and uses that as the initial box size for the next feature.  The Python language extension provides a static dictionary that a stored procedure has access to to facilitate this kind of operation.  This may be quite obvious to everyone and of course it is plainly listed in the Postgres documentation, but somehow I still managed to overlook it for the longest time while trying to solve this problem.  The nearest neighbor distance can be stored and retrieved as in the following code block:

create or replace function nn(geom geometry, 
   featuretable text, 
   geomcol text, 
   initdist double precision, n int) 
   returns double precision
AS $$
    curdist = initdist
    key = featuretable
    if SD.has_key(key):
            curdist = int(SD[key])
    else:
            SD[key] = initdist

    // perform nearest neighbor query using curdist

$$ LANGUAGE plpythonu;

Keep in mind that this is not very threadsafe.  Two simultaneous nearest neighbor queries on the same table will interfere with each others' stored distance.  I'd imagine that one could use some data about the query plan to store the distance uniquely, but that is beyond my Postgres skills for the moment.

The following map shows a section of Philadelphia with zip codes labelled.  The median age is shown color coded where lighter green indicates a younger median age and blue means an older median age.  I wanted to determine if median age is correlated with homicides.  If it turns out that median age is correlated, then law enforcement could use this information to update deployment allocations when a new census comes out.  Homicides are marked using a star symbol and are shown for June 2009 to December 2009.  

When I included median age in my model, it came out as a significant predictor.  I generated a prediction for the following week using the model that includes median age.

A visual inspection verifies that incidents cluster on lighter green tracts (lower median age) and the prediction falls along the same lines as median age is considered a significant predictor variable.  This analysis is a bit quick and dirty since I spent so much time transforming the census data.  I will post a more rigorous analysis of median age and other census variables as time allows.

First, I retrieved the incident data from the resource described in an earlier post.  After some awk and sed wrangling, I got the data into a format where it could be imported into a PostgreSQL table with the following structure:

philly=# \d incidents 
        Table "public.incidents"
  Column   |           Type           | Modifiers 
-----------+--------------------------+-----------
 id        | bigint                   | 
 date      | timestamp with time zone | 
 geom      | geometry                 | 
 zip       | integer                  | 
Indexes:
    "inc_gist_idx" gist (geom)

Notice that there is a geometry column which specified the geocoded location of the homicide.  The data came down projected using SRID 26918 - UTM Zone 18.  I had to reproject the zip code geometries as they came as unprojected lat/lon.  The zip code table (which I retrieved from the source listed in an earlier post) had the following structure:

philly=# \d philly
             Table "public.philly"
   Column   |         Type          | Modifiers 
------------+-----------------------+-----------
 gid        | integer               | 
 area       | numeric               | 
 perimeter  | numeric               | 
 zt42_d00_  | bigint                | 
 zt42_d00_i | bigint                | 
 zcta       | character varying(5)  | 
 name       | character varying(90) | 
 lsad       | character varying(2)  | 
 lsad_trans | character varying(50) | 
 the_geom   | geometry              | 
Indexes:
    "philly_gist_idx" gist (the_geom)

Now, the zip code column of the incidents table is empty.  I used the following select statement to populate the zip code column with the proper zip code which it falls in:

update incidents set zip=
    (select cast(name as integer) from philly 
     where contains(transform(the_geom,26918),
           geom));

The statement is selecting the zip code name from the zip code table where the incident point falls within the zip code polygon.