Swamp Fox Analyst

Children of Prolog

2010-06-13T15:05:00.000-07:00

Recently, I have become interested in functional programming, largely because of the remarkable success of Erlang in producing robust web applications. Erlang seems to have 'solved' concurrent programming, which is one of the growing challenges in programming - especially as we move into a post Web 2.0 world. Erlang was initially implemented in Prolog.

Today, I listened to Joseph M Hellerstein's Keynote address at ACM PODS 2010, The Declarative Imperative: Experiences and Conjectures in Distributed Logic. The slides are also available. He uses Daedalus, a derivative of Datalog. Datalog is, syntactically, a subset of Prolog. Hellerstein also addresses the concurrency conundrum, a.k.a., the Programming Crisis by retelling children's classics

“The sky is falling! The sky is falling! Computers won’t get any faster unless programmers learn to write parallel code!” squawked Chicken Licken.

Henny Penny clucked in agreement: “Worse, there is Cloud Computing on the horizon, and it requires programmers to write parallel AND distributed code!”

This is more fun than your average academic talk. Of course, he then goes on to provide his solution to the programming crisis. He shows how Datalog has been used to solve a series of practical problems in computer science.

The solutions Hellerstein expounds rely upon declarative programming in Datalog variants. He describes a series of practical code for distributed crawlers, networking protocols, query optimizers, distributed file systems and a map-reduce scheduler. What is as impressive as the breadth of topics is his claim that his declarative solutions have code bases that are orders of magnitude smaller that existing, imperative solutions.

His code samples describe solving problems with a time dependence that evolves in steps, like a turn-based game. This was a little disconcerting to a physicist, since we like to think about time as continuous, still it was interesting to find a computer scientist shift from 'practical' solutions to a rather abstract discussion on the meaning of time and causality. At this point, I am starting to think about the InfoQ talk by Rich Hicky on Persistent Data Structures and Managed References.

From a Prolog beginning, Ericsson developed Erlang. Erlang is having a renaissance that is leading to high concurrency solutions for web programming. Next, we have Hellerstein using Datalog to address the programming crisis. Where are the imperative, OO solutions? Declarative logic & functional programming - are these the tools that will lead out out the the programming crisis?

Trends in programming and Amdal's Law

2010-06-05T12:00:00.000-07:00

in my previous blog, I explained trends I saw that will affect programmers. The main takeaway is that we are moving into a world of concurrent programming. If we do concurrent programming with memory shared by several threads of execution, life is going to be miserable for programmers. Fortunately, message passing via actors is a viable path to concurrency that is much easier to program.

In any discussion of concurrency, the iron law is Amdahl's Law. If you have a process in which some fraction, s, of the steps are sequential and the rest, 1-s, can be done in parallel, then the most you can gain from concurrency over N processors is to reduce the time to execute from 1 to s + (1-s)/N. So basically, the time to execute a program is limited by s as much as by N. The hardware determines N and the software determines s. So what determines s? Basically, s is the fraction of the algorithm that you spend queueing stuff. If I go to the grocery store, I can shop in parallel with everyone else. It takes a little longer to shop in a busy store, but only a little longer. But when it is time to check out, I have to queue up to pay.

What is exciting is that by using techniques like 'no shared state' and Actors for communication can greatly reduce s, so we can write software that efficiently tanks advantage of concurrent hardware (i.e., big N). If we look at a standard sort of web applications, the bottlenecks are things like writing to the database. The first thing to do is to replace a single server with a cluster of identical servers, so that a given client can access any one of the servers and get the job done.

But there turns out to be a better plan - sharding. In sharding, you still have a cluster of N servers, but you don't have each server having identical data. If I have a large database, I would shard it by creating N servers, but I would shard the data so that any given server has only a fraction of the data. This is much like partitioning data on a RAID array, and the more sophisticated designs take into account 'k safety". Reed-Solomon erasure codes exhibit this behavior, as do Tornado codes. This means that the data is divided in to n+k partitions so that any set of n partitions can reconstruct the entire data set. This means you can loose k partitions before you loose any data. This idea is central to VoltDB, but they implement it by having k+1 copies of each block. It seems that a more efficient design could be built using erasure codes (See exercise 11.9 in David MacKay's Information Theory, Inference and Learning Algorithms.)

Trends in programming

2010-06-02T19:53:00.001-07:00

There is a dizzying wealth of new options for web programmers. I am trying to understand why there are so many changes and which of those changes I should be choosing to learn. I have found that the answer to this sort of question often comes from looking backward and trying to judge current trends in an historical context.

Hardware: Multicore is the future

This is an inevitable trend, as carefully explained in The Future of Computing by Anant Agarwal to MIT freshman. The power consumption of a CPU scales as the cube of the voltage. So trying to improve performance by brute force is a loosing game. Instead, it is much better to build smaller, lower power cores. Prof. Agarwal is among those who are laying the foundation for a future of RISC cores arranged in a tiled multi core architecture. As a programmer, you can get much better performance if you can leverage concurrency. We can leave the hardware issues to the likes of MIT, but the programming challenges will affect everyone in who programs.

To better understand hardware and how it affects software, listen to A Crash Course in Modern Hardware by Cliff Click of Azul. Azul has systems with 864 processor cores, so they are on the front lines of this revolution. He explains how modern computers, with caches and pipelining, are not particularly well described by the von Neumann architecture. The entire talk seems like an explanation of how hardware engineers have been addressing the von Neumann bottleneck between CPU and memory. Even here, we find that each physical core is running concurrent processes.

So, we are seeing horizontal scaling of computers in clusters, with each computer having more cores and with each core handing more concurrent tasks. I can spot a trend - concurrent processing is the future. But I HATE threading and locks.

Functional Programming and Concurrency

Erlang was developed for high reliability in Ericsson telephone switches. Joe Armstrong describes how they tried to develop Systems that Never Stop and why that lead to Message Passing Concurrency in Erlang. Armstrong came up with a list of six laws for reliable systems. The laws/requirements are:

isolation (the probability of isolated systems failing is independent. So processes need to be isolated.)
concurrency (do you need to worry about how many concurrent processes can you support? You need at least two systems to make a non-stop system. Even a two computer system is concurrent. You can't avoid concurrency)
failure detection (if you can't detect a failure, you can't fix it. the thing that crashed can be the thing that detects failures.)
fault identification (You need to figure out why there was a failure if you want to fix it.)
live code upgrade (Systems that should never stop, need live code upgrade)
stable storage (no backups, implies multiple copies. must keep crash logs)

Erlang is a functional language, so there are no side effects to calculations, Joe Armstrong discusses the importance of Functional Programming in Functions + Messages + Concurrency = Erlang. There really is no point in arguing about if he is right. Erlang is the proof. It is reliable, Ericsson has the AXD 301 switch with two million lines of code: BT operates these switches at a 'nine-nines' reliability - it can fail for 32 ms each year. BT has run switches for six years without stopping. Erlang also allows for high concurrency, this is clearly illustrated in Apache vs. Yaws: while Apache and Yaws have similar throughputs for score of concurrent clients, Apache bogs down at around 4 thousand parallel sessions. By contrast, the Erlang powered Yaws can serve over 80 thousand parallel sessions on Linux with similar hardware. Apache is a well written server, Netcraft shows that Apache is easily the #1 server on the World Wide Web in May 2010. But Yaws beat it handily in this measure. In RESTful Services with Erlang and Yaws, Steve Vinoski examines Yaws in more detail.

But perhaps Erlang is hard to use and expensive to code? According to Ulf Wigner, Ericsson recorded a Four-fold increase in Productivity and Quality on the ADX 301 project. Joe Armstrong is very optimistic about the future of Erlang, and I think he is right.

When I looked into this some more, I realized that Erlang was not the 'bolt of the the blue' that is first seemed to me. Some of the greatest minds in programming have been noticing that imperative programming has been at something of a standstill. Read John Backur's 1977 ACM Turing Award Lecture, Can Programming Be Liberated from the von Neumann Style? A Functional Style and Its Algebra of Programs. This is a broadside attack on imperative programming is all the more remarkable because it is coming from the author of FORTRAN. He then goes on to explain what he sees are the advantages of functional programming. This must have been a controversial paper, Edsger Dijkstra critique of Backur's lecture ended with:

In short, the article is a progress report on a valid effort but suffers badly from agressive overselling of its significance, long before convincing results have been reached.

I tend to believe both Backur and Dijkstra. I resolve any apparent conflict by admitting that Backur was ahead of his time.

Gaelyk again

2010-05-10T17:06:00.000-07:00

Gaelyk is gaining ground as a Google AppEngine framework. Developer Works has Java Development 2.0: Gaelyk for Google App Engine by Andrew Glover. This title emphasizes that Java is more of a platform than a language. I tried this a few months ago, but was disappointed by the initial load times. The application runs fine once it loaded, but my application doesn't get many hits, so most users ended up hitting a cold site.

One solution is to regularly ping the site with a service like Pingdom with something like a 1 minute interval. This seems like overkill to me, but 43,200 pings a month is still well below the roughly 500,000 hits per month that triggers payments. A Basic Pingdom account is $9.95 per moth and there is also a free account that would be fine for a single site. In addition, Groovy 1.7 as improvements for faster compilation. So the performance issues are getting resolved.

The joy of Gaelyk is getting to use the AppEngine features with a minimum of Fuss. Pratik Patel's screencast shows how to build a basic CRUD application. This is also shown nicely in the Gaelyk tutorial. Recently, Gaelyk introduced plugins in version 0.4. There is also a great tutorial about adding REST via JSON and Jackson. It also seems like it should be reasonably easy to integrate with RESTClient.

Jeffrey Zeldman vs. Ed Bott

2010-03-20T19:46:00.000-07:00

In his ZD Net Blog, Ed Bott stated:

In one breath he said, “I’m not challenging the quality of the hardware and software improvements,” and then, in the very next breath, he criticized Microsoft’s “brilliant browser engineers” for “torturing the IE rendering engine every couple of years instead of putting it out of its misery.”

This certainly seems damning. But it isn't quite true. Each of the quotes can be pulled from Zeldman's blog, IE9 Preview, but Bott's juxtaposition of Zeldman's words are downright deceptive. The two statements were separated by four paragraphs, so perhaps Bott quite understands what 'very next breath' means. Or more likely, Bott is being intentionally being misleading.

It is clear that Jeffrey Zeldman does respect Microsoft Engineering, and that he considers at least some of the Microsoft engineers to be brilliant.

Jeffrey Zeldman clearly states that he is writing in response to Dean Hachamovitch's report, An Early Look at IE9 for Developers. Zeldman's blog is dated 9 AM Eastern on 16 Mar 2010. Bott then launched into a description of Hachamovitch's presentation at MIX at Las Vagas. The talk was live blogged by Sharon Chan at 9:51 AM Pacific on 16 March 2010. Or about 4 hours after Zeldman posted his blog.

For some reason, Bott is surprised that Zelman didn't report on the still 4 hour in the future MIX talk. This is even more surprising since Bott was at the talk, so I would have expected him to note who was present in the room.

What there the significant changes that occurred between Hachamovitch's blog and his presentation at MIX? From what I can tell, the major changes introduced at the talk were all known at PDC 2009 (18 November 2009):

html5
SVG 1.1
GPU rendering using Direct2D
CSS3
new JavaScript engine

These are good news. What has changed between November:

improving Acid3 (which did improve from 32 to 55)
support for CSS3 selectors (which improved from passing 574/578 tests to a perfect 578/578)

So, we are seeing IE9 making significant improvements in standards compliance. But this is just a start. To see how IE9 is faring on SVG 1.1, we can look at this recent blog at codedread. On the official SVG Test Suite, IE9 preview is scoring 28%, Firefox 3.7 scores 73%, Opera 10.5 scores 94%, Chrome 5 scores 87% and Safari 4 scores 82%. These numbers are quite different than the test results presented by Microsoft. Haavard noticed that before I did.

The Acid3 score of 55 is well below the scores of the production versions of Firefox 3.5 (94%), Opera (100%), Chrome (99%) and Safari (100%).

That still leaves JavaScript. In addition to speed, a a browser's JavaScript should conform to the ECMA standard. Google released the Sputnik JavaScript conformance test suite. The Chromium blog recently posted test results. The winner is Opera (78 failures) and the looser is IE8, with 463 failures. In terms of JavaScript performance, IE9 is now respectably fast. It is on par with the the other modern JavaScript engines.

Making the case for Scala

2010-03-04T08:41:00.000-08:00

At first glance, Scala seems like an odd choice for developing web applications. Scala is a JVM language created by Martin Odersky of EPFL seems like an academic's testbed for a 'grand unified language' combining functional programming and strict object-oriented design. Scala is clearly gaining in popularity, but the current TIOBE Software Index places Scala at #24 with a rank of only 0.459%, far behind the #1 language, Java, which has a a rank of 17.348%. So, if you want to be on a JVM, why not stay with the undisputed champion of the TIOBE ranking since June 2001?

The key reason is money. Developer time is money. Developer time is, to a remarkably good approximation, proportional to the number of lines of codes that need to be written. This holds over a wide range of languages. Rather than regurgitate an existing piece, I urge you to read Research: Programming Style and Productivity and Scala vs Java: Conciseness equals less bugs. Scala was found to reduce lines of code by 30-35%. This should lead to a commensurate increase in productivity.

The second reason is that functional programming looks like it is the future. Erlang proves that functional programming can lead to high reliability systems with massive concurrency. Computers are becoming highly concurrent, not faster.

The third reason is more prosaic. Liftweb gives many of the advantages of Ruby on Rails and it also provides type checking. Call me old fashioned, but I get warm fuzzy feelings knowing that a compiler is preventing wide classes of errors.

The last reason is still in the future. In 2002, Phil Bagwell wrote Fast Functional Lists, Hash-Lists, Deques and Variable Length Arrays. He introduced new data structures for functional programming. Bagwell is Swiss and is at EPFL, where Martin Odersky works. So EPFL is the center of Scala. I learned about these by watching Rich Hickey's QCon London 2010 talk, Persistent Data Structures and Managed References. The slides are available on Michael Galpin's blog entry Persistent Data Strictures. This excites me, because many of the ideas of state took me back to physics, especially statistical mechanics. The talk, and Clojure's approach to state just seems right. So why is this a reason to use Scala!? Why not cut to the chase and go right to Clojure? At this point, I don't know, that may be the right answer. But Scala is a bit of a chameleon, and it seems to me that the concepts of State and Identity, as well as the persistent data structures that make this efficient, will be implemented soon in Scala. Scala 2.8 is getting Bagwell's VLists.

Suddenly, this is becoming an interesting time to be a programmer, and the JVM seems to be the epicenter of these changes. The JVM is very robust - there has been a decade of work with theorem provers that demonstrate the robustness of the design. There has also been a decade to flush out the bugs. Now, Scala, Groovy and Clojure are leveraging the JVM, and the plethora of libraries on the JVM. Full speed ahead!

David Pollak's Web Framework Manifesto

2010-03-01T06:19:00.000-08:00

The best part of blogs is that is makes it possible for people with clear ideas, not just fame or access to money, to be published. David Pollak certainly falls within the category of clear thinking people, and his Web Framework Manifesto, from November 2006, is a great technical summary of what we all should expect of a web framework. It is fascinating to see the breadth of frameworks that he has examined.

Security figures predominantly in this manifesto. He states:

There should exist an orthogonal security layer such that objects that are not accessible to a user should never be returned in a query for the user and fields on an object that are not accessible should not be visible. The security and access control rules should be algebraic and demonstrable during a security audit. This means that neither the view nor the controller should have to test for access control. Objects and requests should be sanitized before they get to the “controller.”

At first, this seems like Spring Security, but the algebraic caught my eye. I don't know what this means. A Google search found Beyond Separation of Duty: An Algebra for Specifying High-level Security Policies, but this seems quite theoretical at this point. In the IEEE Security & Privacy Journal, the article, A Metrics Framework to Drive Application Security Improvement, describes a more pragmatic, but still quantitative approach to security.

David then went on to create and act as benevolent dictator for life of the Liftweb framework, or just Lift. As a benevolent dictator, it is no surprise that the key ideas of the Web Framework Manifesto are all present in Lift.

Liftweb 1.1 and Google AppEngine

2010-02-12T20:00:00.000-08:00

With only minor modifications, I have repeated Joe Kutner's installation of Liftweb on AppEngine. I'm running OS/X, so I started by using macports to install maven2. I saw that Java App-Engine was just updated to 1.3.1, so that was installed and I adjusted my APPENGINE_HOME and M2_HOME in .profile. The latest liftweb version is 1.1-M8, so I'll try that one.

Now that I know I have updated libraries, I generated a liftweb project with:

mvn archetype:generate -U -DarchetypeGroupId=net.liftweb

-DarchetypeArtifactId=lift-archetype-blank -DarchetypeVersion=1.1-M8

-DremoteRepositories=http://scala-tools.org/repo-releases

-DgroupId=com.folkertsfotografie. rlftest -DartifactId= rlftest

Rather than using -Dversion=1.0-SNAPSHOT, I entered a version of 1.1-M8 when prompted in maven. As usual, then next commands were cd rlftestl and a quick mvn jetty:run to see that http://localhost:8080/ was working. As an eclipse user, I added mvn eclipse:eclipse. So far, so good. The simple helloword application came up normally under the jetty installed with liftweb.

To integrate with AppEngine, I just added the boilerplate appengine-web.xml in my WEB-INF directory. The contents are:

<?xml version="1.0" encoding="utf-8"?>
 <appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
  <application>rlftest>
  <version>3</version>
  <system-properties>
    <property name="in.gae.j" value="true" />
  </system-properties>
  <sessions-enabled>true</sessions-enabled>
  <static-files>
  <exclude path="/**" />
  </static-files>
</appengine-web-app>

With that done, I built the war file with mvn package. To run that war file under AppEngine, just type

$APPENGINE_HOME/bin/dev_appserver.sh target/rlftest-1.1-M8.

That also worked, I could see http://localhost:8080/. Finally, I published this to appsite with $APPENGINE_HOME/bin/appcfg.sh update target/rlftest-1.1-M8.

With that done, I browsed my AppEngine control panel and tested that the hello world application. It worked without a hitch. Now, I can get back to learning liftweb and Scala. It isn't at all clear to me how liftweb and datastore will work together, but I'll just have to burn that bridge when I get to it. So thanks Joe, you have been a great help getting me started with Lift.

Google Analytics: integration with web apps.

2010-02-07T19:56:00.000-08:00

Google Analytics is a powerful, free tool to gather information on events in the click streams of your website's visitors. I'm trying to develop a web application for photographers, starting with my wife's site, folkertsfotografie.com. I'm tracking progress on fotositeproject and the new webapp is running on GAE, Google App Engine. This is a rather Google-centric project. I'm using Picasa to hold image galleries. The RSS from Blogger, Picasa and flickr are being integrated using Google's Feed API via the jQuery library jGFeed. So when we wanted to add analytics, Google Analytics was on the top our our list.

I had hoped to develop with Grails, but it has proven to be too slow. This is a known issue, and both Google and SpringSource are working on speed ups. I switch to Gaelyk, a lighter Groovy framework for GAE, which is resulting in reasonable performance. I looked at GAE sites using other Java frameworks, and was surprised how much faster they are. However, GAE also has similar issues with Rails on jRuby, presumably for the same architectural reasons. GAE performance with Groovy is still an issue, but one that I trust will have a happy ending in the next few months. GAE does a lot of validating of included libraries on startup, which is probably a really good thing for security. I wonder if they can have some standard pre-verified jar files for groovy, jruby, grails, sitemesh and so on. When I want to use one of these jars, I agree to use the 'Google compatible version'. When I load my app, they check the MD5 hash to make sure that I am still using an unadulterated copy & they can just copy their jar image into my application's memory without the expensive validation. Can't we pay this price once when we load the app, not every time we start it up? There is still activity in the Grails community to develop GAE integration, so I can't believe that this issue will remain for long.

Returning to my original topic: Google Analytics requires adding some code snippets to the bottom of each tracked 'page' or in the event handlers for tracked events. In a MVC design, the code snippets must resign in the views. However, the URLs are tied to controllers, so the keys for each URL need to be managed by the controller. I had assumed that a quick search would find others who had discussed this topic. Surprisingly, this is not the case. I have written a page that will read a Picasa RSS feed and then show the album in Galleriffic, a terrific jQuery image gallery. I want to have URLs like /gallery/show/sarah2009, so I don't want to place 'the key' in a file like /album/show.gsp, since the controller don't even expose that URL to the user. I certainly want to track patrick2009 with a different code that campingTripSpring2007, even if they both use the same view. So for each controller, I need a map of the view, the model, Google Analytics keys to use. This isn't quite model data, since the model should only have 'domain' data, which in this cased is information about the albums, media feeds, images, models and clients. It seems that only the controller is in a position to address this.

REST and application evolution

2010-02-03T06:39:00.000-08:00

REST turns out to be the key to allowing software to be upgraded without breaking applications. The idea is pretty simple,you agree on the small number of REST verbs. You then have interfaces that are specified by URL and a version number (perhaps part of URL). You then allow clients to specify their version number in a request.

This is discussed in a couple of InfoQ talks. Alex Antonov gave Case Study: RESTful Web Services at Orbitz. He shows how Protocol Buffers allow for efficient (he claims 7 times faster than SOAP) and flexible message passing using HTTP. Protocol Buffers is an open source HTTP messaging protocol developed by Google. The exchanged data uses protobuf.

Steve Vinoski gives a great talk on RPC and its fundamental flaws. He was an expert in CORBA in the 90's and he has recently moved to REST and Erlang. He is quite convinced that functional programming is the future and that imperative languages, such as Java and C# are based upon a the wrong model of distributed programming.

These talks dovetail nicely with Kirk Wylie's earlier InfoQ talk, Restful approaches to financial systems.

Google Sites - easy project web sites

2010-01-23T12:53:00.000-08:00

I'm working with my wife to develop a web site for her business, FolkertsFotografie. Currently, it is a Smug Mug site. She isn't a fan of selling images via shopping carts, so SmugMug is not a special advantage. I'm wanting to learn about Google AppEngine, so we set up a 'project' to make this happen.

This is where Google Sites fits in. I wanted to organize all of the brainstorming, designing and so forth. Google Sites makes it easy. See for yourself at Fotositeproject. There is easy integration with a host of Google tools, such as Google Docs, Calendar, Groups, Analytics and Webmaster Tools. Plus, I can rest knowing that I have offsite backup of everything and I do not have to worry about keeping the servers running 24/7.

Google Analytics & Webmaster Tools gives me as much monitoring data as my employer gets from IT. But unlike me, Google provides IT services for free.

Grails: By the numbers

2010-01-08T17:36:00.000-08:00

Everybody likes to talk about highly productive environments, but numbers are notoriously hard to come by. This makes alterlab's ALTERthoughts Blog, entitled Grails vs. Rails especially interesting. Using Java J2EE as a base, both Ruby on Rails and Grails were found to be twice as productive. Grails beat Rails, but by a margin so slim it seems a dead heat.

Erlang, Mochiweb and Webmachine

2009-10-01T13:52:00.000-07:00

Webmachine is amazing. Why doesn't .Net or Java have something this elegant?

Erlang Factory 2009 Palo Alto - Justin Sheehy - Webmachine @ Yahoo! Video

Have a look. I was impressed by the clean design, but I was blown away by the debugger described in the last ten minutes. This is an open source project in Erlang, the high concurrency functional language created by Joe Armstrong at Ericsson. As noted in the talk, Webmachine is a high quality component to include in a web framework, like Mochiweb. Mochiweb itself is interesting: it is an Erlang framework for high concurrency web applications. When serving static content, it has been compared with yaws and nginx in a deathmatch on Joes Blog! Richard Jones is contemplating a Million User Comet App. Erlang is turning into a pretty cool language for web apps with high concurrency.

I wonder if there is any attempt to write an equivalent in Scala, a high concurrency functional language for the JVM? I have got to get up to speed with Scala and Lift. There is a lot going on here...

Involve them in the same conspiracy

2009-09-26T09:29:00.000-07:00

As previously noted, I am listening to the lectures from Berkeley's CS 61A. In the fifth lecture, the class is listening to a video of Alan Kay. He ends the talk, "If you want others to go along with you, you have to involve them in the same conspiracy." Oddly, this made me think about spreadmarts. Wayne Eckerson at TDWI create this term. He described a spreadmart as:

renegade spreadsheets and desktop databases that are wreaking havoc on organizations. Since then, many people have adopted the term because it highlights a painful, yet largely ignored, problem that plagues organizations today.
Spreadmarts contain vital pieces of corporate data that are needed to run the business. But since spreadmarts are created by individuals at different times using different data sources and rules for defining metrics, they create a fractured view of the enterprise. Without a single version of corporate data and centrally defined metrics, employees can’t share a common understanding the business. With spreadmarts, each worker marches to the “beat of their own drummer” instead of marching together toward a common goal. In short, spreadmarts undermine corporate productivity and sabotage business alignment.

I think Mr. Eckerson needs to consider Alan Kay's words. Mr. Eckerson has just cast the creators of spreadmarts as corporate saboteurs that are destroying the One True Way™ that he and and upper management wish to bestow upon their ungrateful underlings. This is a very top down approach, that probably isn't going to work well. He does present a 'co-opt' solution where you agree to using CVS output so the spreadsheet jockeys can still use their beloved Excel. Bu with title like Reeling in Spreadmarts, In Search of a Single Version of Truth: Strategies for Consolidating Analytic Silos and Taming Spreadsheet Jockeys, I hear contempt for the existing business processes and the folks on the front line that do the business' work.

Consider how differently Kirk Wylie treats the traders in financial firms in RESTful Approaches to Financial Services. They are exactly the 'spreadsheet jockeys' derided by Eckerson. Wylie describes centralized systems as über-systems, clearly a derogatory term. Mr. Wylie also recognizes the proper desire for traders to keep control of their data processing. He notes that developers are only acquiring the needed business understanding to become productive after three to four months - he is looking for bottom up solutions that support the 'front line', not centralized solutions to give upper management a dashboard of global KPIs, key performance indicators. He accepts that "Any system that doesn't consider the traders pathological dependency on Excel is doomed to failure." In the rest of the talk, he describes how RESTful solutions allow him to roll out shared data across a variety of format, including Excel, to end users on demand.

Like Alan Kay, Kirk Wylie is involved in the same conspiracy as his users. I'm betting he has more success than Mr. Eckerson is rolling out working systems.

Comp Sci 61A - Online lectures

2009-09-19T06:40:00.000-07:00

This week, I am starting to follow one series of the online UC Berkeley lectures. The lectures are by Brian Harvey's CS 61A from the spring of 2008. As is always the case, following a lecture without trying to do it yourself is not going to get you far.

In the first two months of the class, Prof. Harvey is examining functional programming with LISP. To follow along, I am using Clojure. Installing on OSX amounts to 'sudo port install clojure'. The clojure command line is accessed by the clj command. As noted in the lecture, it is easy to write interpreters. The class uses scheme, so I need a scheme interpreter for clojure.

In order to describe the foundations of computer programming, Prof. Harvey wrote the following during his second lecture:

Physical foundation of computer science
Topic	studied in...
Application Program	CS 61A
High level language (Scheme)	CS 61B
Low Level Langauge (C)	CS 61C
Machine Langauge/architecture	...
circuit elements	EE
transistors	applied physics
solid state physics	Physics
Quantum Mechanics	Physics

This ties in rather naturally with the OSI Seven Layer Model.

This class is presenting a 'real world' view of computer science, not computer science as applied mathematics. As with the physical sciences and engineering, mathematics is the language, not the subject in CS. This is not meant to be a put down of math.

In the first lecture, he gave a concise explanation of 'function' vs. 'procedure'. I am going to enjoy these lectures...

Slowly Changing DImensions, Type 6 = Type 1 + Type 2 + Type 3

2009-09-06T10:23:00.000-07:00

Recently, I have designed a multi-tenant data warehouse for AdTrack clients. One key decision in building a data warehouse is choosing how to deal with slowly changing dimensions (SCD). In a standard data warehouse, there a two basic types of data tables: fact tables hold information about a particular event (e.g. a record of one sale) and the dimension tables describe how the facts can be organized (e.g. records about stores, customers, products, date of sale). Basically, you access the data by specifying a 'slice' of each dimension and you then have the database find all the facts that fall within these slices. The database then lists or summarizes (aggregates) these facts which are displayed in tables and charts which are reported to the users.

A major fly in the ointment is that the information in the dimension tables are slowly evolving. Customers move, change phone numbers, addresses, marital status, last names, jobs and income levels. Sometimes, you just want the current values. It is not obvious a priori if clients will care about current values or historical values. If I were selling hot tubs, does 'current' marital status or 'historic' (e.g. at the time of the sales) marital status matter? I can imagine an analyst wanting to track sales by marital status, sex and income, so I can see them caring about tracking the history of a client's marital status. However, if I were to make a report of top 10 married customers for a promotion, I would probably want to add in any sales to that customer when they were still single. So, unless you know the context, there is no general answer to 'historic' vs. 'current' value when building a data warehouse. This is doubly true for a mult-tenant warehouse where one client could be selling construction equipment b3b, another could be selling medical equipment to university hospitals and a third could be selling windows b2c. Any universal assumption as to what each of these clients need now and forever is going to be badly wrong for some users.

Data warehouse designers will recognize this as the Type 1 SCD (current value ) vs. Type 2 SCD (historic value) decision that needs to be addressed in design. There is also a Type 3 SCD that has current and 'some history'. The technorati, or anyone that reads the slowly changing dimensions article in Wikipeida, will know that there is type which can act like Type 1 SCD, Type 2 SCD or Type 3 SCD. Since 1+2+3=6, Ralph Kimball called this a Type 6 SCD. I have come up with a new implementation of a Type 6 SCD. My solution is basically a Type 2 SCD, but with the addition of a single column. I can then build views of this dimension table that function as Type 1, Type 2 or Type 3 SCD. This solution has now been tested, and the query times are roughly equal for all types of behavior, this is not true of older solutions which could really slow down if you asked for historic values. At least in the case of using Pentaho PDI, the changes to ETL are simple to implement and can be retrofitted to an existing Type 2 SCD. If you are interested in how I do this, I have a paper on Google Docs that you may read. If anyone is interested in implementing this, I would be eager to help or to hear if it works for you.

PosgreSQL: how to run on EC2 with EBS

2009-08-30T20:00:00.000-07:00

Ron Evans has recently blogged in great detail about getting PostgreSQL up and running on Ubuntu images on Amazon EC2 with EBS storage for the database files. Basically, you use EC2 to provide the server running PostgreSQL and you use EBS, elastic block storage to create a volume that you mount to hold the database files. Presumably, you can create a second file system to hold WAL segment files, but perhaps you just use S3 to store these.

The next step would be to learn how to create PostgreSQL clusters on EC2. This isn't much of post, but it is useful to me.

A virtual IT department: the database

2009-08-29T17:22:00.000-07:00

Software as a Service, SaaS, seems like it is about to revolutionize how small and medium companies run their IT. Today, many small and medium businesses have really kludgy IT. There are a bunch of Windows desktops, and a few Windows servers for Exchange, SQL Server, a File Server, a domain server and perhaps a company web site, but that is likely to be a hosted site for small business. I think we will see Microsoft's dominance fracture as a result of SaaS. Today, consider how SaaS may effect SQL Server.

Small companies now have the option of running database services in the cloud. Using Amazon EC2, they can have a hosted data center with a 99.95% uptime in the SLA. Of course, this only works when the company has a working internet connection. Since Amazon is using Xen, you can find tools that will convert between Xen images and Amazon AMI files. For Debian Linux, that maverick James Gardner has blogged about this. So, you could set up a local server to host Xen and you could then have a piece of the cloud in your data center.

Running SQL Server is a costly proposition. Enterprise Edition is $25K per CPU. Microsoft is planning to upgrade on a 2-3 year cycle and they don't intend to support servers more than two generations old. So your $25K/CPU is recurring fee that will average around $5K/year per CPU. If you want, you can run SQL Server on Amazon EC2. There is a premium for this. Lets assume that you will need a large server. For Windows alone, this is $0.50 per hour fee. But adding SQL Server Standard will increase this to $2.20 per hour. For one year, a hosted SQL Server 2008 Standard Server instance works out to about $19K. This isn't crazy for a managed server, but it isn't a cost savings for most companies over running a Windows Server in-house. So why is there so much buzz about the cloud?

With a few changes, we can save a great deal of money. For the database engine, consider PostgreSQL. It is fast and stable. It has a carefully reviewed code base. There are drivers for ODBC, JDBC and ADO.Net, so your applications can still access data. Because PostgreSQL is open source, you can license it for $0. On Amazon EC2, you can run a medium service instance with Linux for $0.40 per hour. You can also take advantage of Reserved Instances. For $1,400 you can reserve a server instance for three years. This drops the per hour fee to $0.12 per hour. Three years now costs 4, 555.76. So for one year, a hosted PostgreSQL instance works out to to just under $1,520. The cloud loves open source.

To be fair, SQL Server also include SSIS, which allows for a variety of ETL services to be built and run via a GUI. I have been using Pentaho's Kettle for this. Pentaho has renamed this as Pentaho Data Integration. PDI can be downloaded for free. So, with PDI and PostgreSQL, you have the same basic functionality as SQL Server, it just saves $17,766.61 per year for a database server. You still get 99.95% data center uptime.

Note: I have not added S3 storage for archiving databases or transaction logs. This will increase costs, but for both servers.

PostgreSQL: the EnterpriseDB webcasts

2009-08-16T17:51:00.000-07:00

PostgreSQL is becoming one of my favorite databases. Postgres is the direct descendent of Ingres, both are the creation of the Berkeley and were authored by Michael Stonebreaker and Eugine Wong. Other descendants of Ingres include Oracle, Sybase Enterprise Server and Microsoft SQL Server. PostgreSQL has the advantage of a very clean code base, as demonstrated by Coverity's open source scans. A limit for many open source project is getting access to paid support. In the case of PostgreSQL there is commercial support from EnterpriseDB.

One of the gems of the EnterpriseDB support is the wide range of webcasts about PostgreSQL. Because PostgreSQL is open source, the developers present detailed information about the internals of PostgreSQL and best practices for database administrators and developers.

SQL Server and clustered indexes

2009-08-16T06:03:00.000-07:00

When a database reads a range of values, a very common case, you may to read many values and IO usually becomes performance limiting. These values reside on your data storage, which is almost always a hard drive or a RAID array of hard drives. To make the IO faster, there are some basic strategies.

First you can make the data more compact. As an extreme example, consider the case of Decisionmark's Census Counts. This product provided a database with US Census Bureau Data distributed on a CD. Watcom SQL Anywhere, now Sybase SQL Anywhere, was used in part because we could compress each row. Compared with a hard drive, a CD has very slow data reads. So it was faster to read data from compressed rows and to unzip the data than it would have been to just read the same data uncompressed. Since SQL Anywhere's census data was read-only, we didn't need to worry about the write performance.

If you read about some modern database research, you will see that compressing data is one significant strategy in improving performance. For example, in MonetDB/X100 - A DBMS In The CPU Cache Zukowski and co-workers report

Figure 5 show the speedup of the decompressed queries to be close to the compression ratio, which in case of TPC-H allows for a bandwidth (and performance) increase of a factor 3.

So this is still an area of ongoing work at the frontier of research. The goal is maximizing the amount of data in each block of storage, so that IO is minimized and performance is maximized.

A second strategy is to use indexes. An index is a small bit of data that helps you quickly find the larger data recorded in your row. An index is build by looking at the values of some subset of your row, e.g., one or a few columns. For each value of the indexed value, you have a pointer to the row that holds the data. If you have a hash index, the pointer is directly to the row, and if you have a b-tree index, you may have to traverse a few nodes in the tree before you actually get a reference to the data row. If there is an index that is relevant for a query, it can speed that query significantly. It can do this by either reducing the IO to find the data (the database engine might otherwise have to read the entire table) or, in many cases, allow the database to avoid reading the table and simply use the index. Careful choices of indexes can vastly improve performance

A third strategy to improve read performance is physically order the data so that the data for your queries is adjacent on the hard drive. Unless you are using a solid state drive, you need to be moving the head and rotating the platter to read data. If the database can read adjacent blocks, it will read faster. In a typical query, the database engine will use the index to build a list of blocks that need to be read. If blocks are sequential, they can be read more quickly.

After laying this groundwork, I will now discuss what seems to be a fundamental problem with SQL Servers implementation of table ordering. In many databases, there is a command that allows the records in a table to be ordered by an index. To have a concrete example, consider the case of a department store chain. The chain has a computer system that records each sale in a purchase table. The chain has several stores, several departments, many sales representatives and many clients. To maximize the performance of our purchase table, you may want to order the records by (store, department, sales rep). For example, in PostgreSQL, you can create an index with create index index_purchase_store_dept on purchase (store_id, department_id, order_id). You can then order the table records with the command cluster index_purchase_store_dept on purchase. The database engine will then order the data in the purchase table using the index. This could significantly speed up reports of store or department. DB2 has a similar concept, where you can define an index to be cluster index for a table, you can then reorganize the table using the REORG utility. Oracle has a slightly different tactic: a cluster is defined in a schema. You can include up to 32 tables in the cluster and you can define an index to order the tables in the cluster. This means that not only are the rows in a given table close together, but rows from these tables will be close together. This takes the idea of clustering to a new level. Donald Burleson, author of Oracle Tuning, describes how these clusters are used. When records are added to any table in the cluster, they are placed into overflow blocks if there is no more room in the block specified by the index. When the overflow blocks reach as percent specified in PCTFREE, the records added to the tables get reorganized by the index.

SQL Server, as well as MySQL, makes clustering more convenient by defining an index to be clustered. You may declare one index per table to be the clustered index. To maximize the performance of our purchase table, you may want to order the records by either (store, department, sales rep) or perhaps by (customer). When records are inserted, they are insert in the order specified by the index. As carefully described in a blog by Sunil Agarwal, a Program Manager in the SQL Server Storage Engine Group at Microsoft, this will result in a severely fragmented table.

Instead, Microsoft urges the user to define the primary key to be an auto-generated integer and to cluster the table on the primary key's index. In other words, order the table chronologically. This seems to fundamentally contradict the concept of clustering. The records are 'clustered' in groups of one and they groups must be in chronological order. I checked some tables that followed this rule, and they all had 0% (or at least under 1%) fragmentation on the clustered index, even after a year of operation without maintenance.

Recently, I was asked to improve the performance of set of queries that were run against a SQL Server 2000. A review of the index statistics showed severe fragmentation, for the reasons outlined by Sunil Agarwal. To see if defragmenting would matter, I followed the advice in Microsoft SQL Server 2000 Index Defragmentation Best Practices. I defragmented the tables used in one particularly slow query. I was able to reduce the query time from over 4 minutes to just under 10 seconds. With a rewrite of the query, the query time dropped to under a second. However, after only two days of transaction processing, the clustered index had a fragmentation of over 40%. The query times were not back to the 4 minute level, but the formerly sub-second query was taking more than 10 seconds. In this case, the indexes are fragmenting faster than we could reasonably defragment them, since we cannot take this server offline daily to reindex the tables and using DBCC INDEXDEFRAG is simply too slow.

My conclusion is that in order to make clustered indexes useful, you need to be able to append a significant number of rows to a table and then order those records using the clustered index. Microsoft's solution of inserting each record at the location specified by the clustered index can rapidly cause severe fragmentation. The solutions of PostgreSQL, Oracle and DB2 avoid this issue, but at the cost of additional maintenance.

Microformats: Reuse, don't reinvent

2009-07-10T13:44:00.000-07:00

Orbitz engineer Mark Meeker has an introduction to microformats with examples

This is one of the best introductions I have found.

I found this Even Bill Gates wants you to use Microformats.

After some reflection, it seems that building an open source object model based upon hCard for people and organizations, hCalendar for events and rel-tags could be useful for many projects. How often do you end up reinventing classes for people, addresses and so on? With rel tags, you could define all sorts of relationships between people and institutions: you can add 'subsidiary', 'employee', 'retailer' and build all sorts of models for a wide range of business needs. If there was a simple object model, with some basic implementations in a few key frameworks, you could start coding from that rather than from scratch. This would also be useful for learning and comparing frameworks. Presumably, you could get a REST implementation, an XHTML view, and a WS-* web services almost free. You would also get the services in a format that would help others build mash-ups with your data.

The Story of REST: Representational State Transfer

2009-07-06T19:50:00.000-07:00

Joe Gregario has provided a lucid description of REST at YouTube.

He has also provided a companion video on the Atom Publishing Protocol, which is described in the Wikipedia article on Rest is described as a canonical RESTful protocol. So Atom provides a great example of a real-world system that uses REST and Joe describes it clearly.

If you want a clear understanding of how to use Rest, this may be your best use of half an hour. If all you want to do is to grok Rest, Ryan Tomayako's How I Explained RESR To My Wife is a classic, and it only takes five to ten minutes.

If you have more time and really want to understand in greater detail, there is Roy Thomas Fielding's dissertation, Architectural Styles and
the Design of Network-based Software Architectures.

For an actual implementation of a REST service, I am keen to use Grails. Grails has build-in support for REST. Grails also offers content negotiation. In HTTP, part of the request is a specification of the media type. Using content negotiation, the user agent can specify which format it prefers. This means that a URL for a person could return a portrait as image/jpeg, a hcard page via text/html, the vcard data as text/json, text/plain, text/xml, or text-plain. You could also use content negotiation to specify the language for the response. Suddenly URL seems to be the locator for a universal resource, as well as being the universal locator of a resource.

While it may be feasible to support multiple mime types at a single URL, it is not trivial to provide multiple representations of the same item. Even converting between two apparently similar data format has surprising complexity, as discussed in
Convert Atom documents to JSON at IBM Developer Works. The Developer Works also has a series of papers on Grails, which has specific examples of using Grails with Atom syndication.

What HTTP reveals about your browser.

2009-07-02T18:13:00.000-07:00

When surfing the web, is easy to feel that you are a relatively anonymous consumer of content. However, the HTTP traffic between your browser and the web server is a two way street. Henrik Gemal has provided browserspy.dk which has a series of queries that find our more about your browser than you probably know. Much of this information is potentially quite useful to the web site. The classic example is the HTTP_ACCEPT_ENCODING header that tells the server if your browser can accept compressed data. This can significantly reduce the size of a page. The other classic use is to identify Internet Explorer, the bane of JavaScript and CSS authors. But this is just the start. By knowing which version of Flash is installed, YouTube can warn you if you need to upgrade to view their video content. By sensing color depth and window size, a web site could determine an optimum image for me. This would be especially useful on a mobile device, where throughput and CPU limit battery life.

I would really like it if the geolocation information could be used to set the default country, state and city in web forms. In my case, the geolocation would have gotten me to Iowa, but would have placed me in Hiawatha rather than Marion. There is another geolocation demo that gets closer, but is still off by about 3 miles. I would like to be able to set a location, address and hcard info and have the option of using that on web forms. I would encourage more browser providers to support the navigator.geolocation object in the W3C Geolocation API.

But in my opinion, the scary information is from a CSS Exploit page. This exploit has been covered today in Slashdot. Web 2.o Collage will produce a collection of favicons of sites you have visited. What is most surprising to me is that this exploit by Brendon Boshell doesn't even require JavaScript. He has a Javascript version as well, which he describes in detail. So, unless you use one of the 'stealth modes' that don't record history, anyone can be checking to see if you have visited a particular site. Think about how that could facilitate a phishing attack.

V8 - the extremes

2009-06-30T17:24:00.001-07:00

When it comes to the V8 Benchmarks, Chrome is still king, even on a Mac. Despite the fact that this is running under Parallels, Chrome 2 is almost twice as fast on the iMac as it is on the Dell at work.

Here is the real surprise. While every other browser ran faster on the iMac than the Dell, IE8 actually slowed down. But the slow down is almost entirely due to the final test, Splay. Google describes this test:

Data manipulation benchmark that deals with splay trees and exercises the automatic memory management subsystem (378 lines).

So it seems that the VM on the Mac can have performance issues with memory management.

If there is a problem with memory management, perhaps I should shut down all of the other applications and see if the numbers improve. Yup. Look at Splay: it went from 0.397 to 143.

So, running Parallels usually works fine, but in this case, it seems important to help the memory management along by closing down other applications. Still, it is IE8 which is far behind the other browsers in terms of JavaScript performance. Hey Microsoft, any chance you can keep up with the pack?

More V8 - on an iMac

2009-06-30T16:55:00.000-07:00

After running the V8 benchmark suite at work, I am repeating at home. Home means a iMac with a 2.93 GHz Intel Core 3 Duo with 4 GB. I'm running Leopard (10.5.7). This time, I was more interested in looking at the improvements in Firefox 3.5 relative to Firefox 3.

But first, here are the results for Safari 2, this time running on OS/X.

For some reason, Blogger reverses the order of the images, so here we have the results for Firefox 3.5. There are two runs, one under OS/X and the other on XP. The XP version is running under Parallels. It sure looks like the VM running XP is very efficient: the Firefox numbers are within a few percent. They are also twice as fast the Dell Optiplex at work, even if the memory and clock speeds are quite similar.

Finally, we see the results of the old Firefox 3. The upgrade almost doubled the speed. So kudos to the Firefox team for the improvement in JavaScript performance. But we still have to recognize that the Webkit-based browsers are really dominating the JavaScript performance numbers.

V8 Benchmarks

2009-06-30T15:09:00.001-07:00

Google's V8 Benchmark Suite is easy to run. Here are the results for several new browsers on my workstation, which is a Dell Optiplex GX620 with 2.79 GHz Pentium D and 3.49 GB of RAM. Wow. I didn't expect to see Webkit being this much faster. If my brand new Firefox 3.5 is given a relative score of 1, Safari 4 on Windows has a score of 7.5, Google Chrome has a relative score of 8.6. Internet Explore 8 has a score of 0.21. Webkit browsers have an order of magnitude better performance than Microsoft's flagship browser: IE8.

So, is Microsoft that bad a writing a JavaScript interpreter or are they trying to move us away from web standards like JavaScript and toward Silverlight?

Jef Jarvis and what to do next

2009-06-19T18:04:00.000-07:00

Jeff Jarvis has more gray hair than me, but he really seems to understand the Internet. He may be best know as the author of What Would Google Do? You can read it online. Fora.tv has some lectures of his:

Future of Media and the Prospects for Brands
What Would Google Do? (the Video)

After listening to him, I am getting jazzed to try some Web 2.0 projects. He advocates being small, but being part of something big. So, what are some big things that we can expect to see? Here is something that comes to mind:

We all like images, and its probably the case that we have pictures all over the place that we like to access. I have pictures on Picasa, a friend has images on MobileMe, my wife has images on SmugMug and Flickr. My kids have images on FaceBook. This is probably pretty common. I don't want more ways to store images, I want a way to reference and search these images.

For images, I would like to have a linker site that lets me reference feeds, on line albums or even individual images on the Internet. Many of these sources provide titles, descriptions and other image metadata. Many images also have EXIF or IPTC metadata. What would be great would be a simple web interface to allow me to build a tool to subscript to several different sources and then subscribe to unified feed. Give the feed a URL and share with friends and family.

Even better, allow images to be linked to urls that identify people - hcard descriptions, home pages, FaceBook pages and so on. Then, you tag images (or better yet, some section of each image) with links to a person. The tags should have some 'types' like 'photographer/owner', 'model/subject' that show the relationship between the person and the image. Some key words links to images would also be great. They I could build feeds for all images linked to family in the last 6 months. Each of these dynamic albums could be given a URL on PubSubHubBub and expose the results has Media RSS. Then, anyone with a Media RSS viewer could see the feed in a browser. Open Iris and Slideshow Pro have inovative Media RSS viewers.

Software Tutorials on Google Wave: Eclipsy

2009-06-17T15:30:00.000-07:00

Google Wave absolutely floors me, but it isn't available yet. I think that Wave is going to revolutionize teaching software. Consider software tutorials. First, we need something like an Eclipsy, a hypothetical Wave agent/Eclipse plug-in that allows Eclipse to be integrated into a Wave. As one developer goes through the steps in a tutorial, a second developer (or even the same developer with a microphone), could be giving a verbal description of the actions.

Once the expert has successfully performed the tutorial, a student else could play it back and see each step in the processes. I see major advantages of using Wave. Perhaps you could play back the development in your Eclipse while listening to the process, with Eclipsy in a playback-mode. At the end of this, you would know that the process worked with your configuration. Assuming that it did work, you could then go through the steps yourself while listening to the audio & perhaps watching the steps in Eclipse running on some virtual machine that you would watch as your copy the steps at home. (Two monitors would be nice for this.) The ability to pause during the playback is a big advantage of a Wave solution.

With many Web 2.0 tools like Grails, you are iteratively developing you application. For example, you might start by defining all of your domain objects and use scaffolding to build the controllers. You would then define the relationships between all of the objects. Next, you might add more attributes. To complete the domain objects, you could go in and add the constraints. This sort of iterative approach seems perfect for the as-yet-hypothetical Eclipsy.

Typically, you would also follow an iterative approach to evolving the views and the controllers. Much of the skill in this approach is having a feel to know what to do in each iteration. Eclipsy would be a good way to developing this.

In a complementary role, Eclipsy seems like it could be used for version control. At the very least, Eclipsy would be tracking the state of each source code file and your project configuration. This means that it could also function as a version control system that you could replay to get to any state in your development.

IE8: on Acid3

2009-06-16T07:15:00.000-07:00

Just for fun, I tested Internet Explorer 8 on the Acid3 test. For completeness, the test was run on the morning of June 16, 2009. Today, Opera 10 was released. Opera 10, along with the recently released Safari 4, have attained a score of 100% on the Acid3 test. I also tested Chrome, release 2.0.172.31, and it also scored 100%. Is browser conformance breaking out? Helas, no. Just to prove what I say, I'm including bitmaps of 'About Internet Explorer at the time of the test. As you can read, I am using version 8.0.6001.18702. This is running on XP, as you can probably guess from the title bar.
During the test, I was asked if I wanted to let an ActiveX component run, I believe that it was for XML processing, but I foolishly clicked OK before recording the component name. The test looked like it completed at 12%. I was surprised to see what appears to be an HTML text area suddenly pop up.

After several seconds, the score started to creep up, finally reaching 20%. This is consistent with the scores reported at Anomalous Anomaly, which has much more complete Acid3 test results. Finally, I tested Firefox 3.0.10, which produced a score of 71%. As expected, this is also consistent with Steve Noonan's results at Anomalous Anomaly.

I also checked Acid2, all the browsers scored 100%. Perhaps there is still hope that bothIE and Firefox will be made more standard compliant so that there Acid3 scores can match their Acid2 scores.
[followup on July 20, 2009] Firefox 3.5 is making rapid progress. With Firefox 3.5.1, I have an Acid3 score of 93. My IE scores have remained the same.

Richer web interfaces

2009-06-13T13:41:00.000-07:00

Web interfaces are improving all around. With tools like GWT and OpenLaszlo, it is possible to build rich web applications with GUIs that at least match stand-alone applications. Some of the applications that I am finding innovative include:

Pentaho BI Server. Their new Mantle UI is build with GWT and is a huge jump beyond the old interface. It just behaves like a stand-alone application. Some of the components, especially jPivot, could use an update, but the Server platform is in a great place to organize all of these new features. With the modular design of BI Server 3.0, Pentaho users will be seeing a great deal of new reporting and analytic tools they can plug in.
G.ho.st, the globally hosted operating system, is a virtual computer that you can access from anywhere. You have a full GUI, built in OpenLaszlo, apparently by their team of only 30-40 staff. Ghost seeks to provide a free, web-based virtual computer for anyone in the world. The UI is fast and simply doesn't feel like a browser. Have a look.
LZPIX is another OpenLaszlo application for viewing photos on Flickr. The link is to the Flash version, there is also a DHTML version and you can see the source code as well. On a side note, open the application and search for 'toureiffel' , the photo Paris s'éveille is magnificent!
Maple is another interesting tool to view photos, but it is designed for multimedia slide shows. This can run as either a Java application or a a Java Applet. It uses Java 6, and my shiny new Mac only has Java 5, so the only way for me to view this is running a VM with either Windows or Linux so I can install Java 6, which was released in December 2006. Come on Apple, is it not possible to get the bugs out in 2 1/2 years!? Oh well, one more reason to hope for Snow Leopard. I just hope that they have a Java 6 upgrade for Leopard, since I can't run Snow Leopard on the older G4/G5 Macs. I was going to ask why Java applets didn't catch on like Flash applets, but I guess when you have trouble accessing a consistent Java platform across OS/X and Windows, the answer is clear. This is unfortunate, as Maple is a great slide show viewer. Java really does work, it is just a shame that it has been held captive by an Apple or Microsoft.
Ben Fry's zipdecode. Try this and tell me why this shouldn't be part of any application that needs a zip code. It should be easy to gather the data needed to make this international. If I am filling out a web form, why do I have to type in 52302 and then choose Marion, Iowa? There are some zipcodes that server multiple communities, but in that case, I would just need one extra click to pick the city. The only thing I would add to this applet would be a semi-transparent zip code on top of the map.

What we need to do is develop simple, rest-based tools that we can easily drop into web applications. If your application is residing on the web, you should be able to simply reference well designed tools. The current practice of installing web apps on your application's server seems to undercut the promise of the web.

For a coherent theory of how software should be designed for people, I recommend Bret Victor's Magic Ink. He has thought deeply about this, while I am just providing examples.

In some ways, it is possible to outdo stand-alone applications.

Google I/O viewed from IOwa.

2009-06-05T13:23:00.000-07:00

With some software presentations, you are overwhelmed by the show but afterwards the gee-wiz wears off as you start to piece together what is going on. But the more I think about Google Wave, the more the ideas are growing on me. At this time, hardly anyone has heard about Google Wave, but that is going to change. Within a few years, Google Wave could be recognized one of the few paradigm changing applications, much like hypertext going mainstream was to the growing of the web in the 1990's.

In the developer preview, there were no lack of eye-popping features:

Drag and drop photos from iPhoto into the browser. Really cool use of Gears and GWT. Watching the photo upload automatically and appear on the other browsers in seconds was a direct reminder to me that we are only beginning to understand how the web can connect us.
Watching real-time translation between French and English shows how we are going to be able to interact more freely in wider communities.
The multiple, concurrent editing of a single document shows how much power there is concurrent versioning systems.
The use of associative memory, which resides not only on the server, but is shared with each participant is part of the secret sauce that makes is seem that everyone on the wave is 'together'.

This just seems like it is the next phase in the evolution of the world wide web. Before this, the web was a bunch of places to go. With Wave, it will become a bunch of events to join, review and create.

Innovation in the computer industry

2009-05-15T14:59:00.000-07:00

Most innovations are combinations of existing ideas, this is how innovation works. In a recent ZD Net blog, Larry Dignan examined how effective Microsoft, IBM and others are at profiting from their R&D spending. The article seems quite reasonable and the claims seem well supported.

But in the comments, there was a recurring meme that claimed that there is no innovation in FLOSS software. For example, mikefarinha claims that "all of the big name OSS projects" exist to steal market share from Microsoft. His list of FLOSS projects is Firefox, Open Office, Samba, WINE and Lindows. For starters, Lindows is hardly a major FLOSS project, I would list it well behind Apache, any of the BSDs, Linux and OpenJDK, none of which made his list of 'big name projects'.

Firefox comes from Mozilla. Mozilla comes from Netscape. Apparently, Firefox exists to take market share away from Internet Explorer. If you look at the User-Agent string from Internet Explorer, you will read

Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 6.0)

Interesting, it looks like Internet Explorer is emulating Mozilla. If you really want to look for who invented the web browser, you will discover ViolaWWW. Rather that running on Windows, ViolaWWW ran on Unix and X Window. You may recognize X Window, since X.org is a 'big name' FOSS projects

OpenOffice.org is simply the open source version of Star Office. StarOffice began with StarWriter, just like Microsoft Office began with Microsoft Word. But StarWriter was originally a German word processor for Zilog Z80 and CP/M. CP/M is an operating system that was originally developed by Gary Kindall at DEC. CP/M is the ancestor of several DOS systems, including MS-DOS. In Microprocessor Report (Vol 8, No. 13, October 3, 1994), John Warton concluded "The Origins of DOS" with:

The strong impression I drew 13 years ago was that Microsoft programmers were untrained, undisciplined, and content merely to replicate other people’s ideas, and that they did not seem to appreciate the importance of deﬁning operating systems and user interfaces with an eye to the future. In the end it was this latter vision, I feel, that set Gary Kildall so far apart from his peers.

Not exactly a rousing defense of software innovation at Microsoft. So, we find that StarWriter was developed for a OS that predates MS-DOS. If I recall correctly, the GUI version of MS Word was actually developed for the Macintosh. So it is not clear to me how OpenOffice.org is an example of a Microsoft innovation that others copied, it can trace its code base back further than MS Word can.

Next, we have the case of Samba. Samba implements the SMB protocol, which Barry Feigenbaum at IBM. Microsoft implemented a heavily modified version of SMB. Andrew Tidgell, and the Samba team, worked to create a version of SMB that worked with DEC Pathworks. It is only later that they tried to understand Microsoft's undocumented modifications so their SMB Server could work with more of the computers in their employer's network. It seems that trying to achieve file sharing between Windows and Unix is just an attempt to steal market share. To my way of thinking, when Microsoft added undocumented changes to an otherwise open protocol, it was an attempt to achieve vendor lock-in and steal market share from the rest of the industry.

That leaves only Wine, which is an emulator that allows *nix computers to run Windows applications. Personally, I haven't had much luck with Wine, so I don't see how Microsoft has much to worry about from Wine.

In addition to mikefarinha, we have Rick S._z who states

But here's a TECHNICAL creation which changed the computing world, and was almost totally invented by Microsoft: truetype fonts. Before MS built Windows 3.1 around them, no one had thought to use the SAME fonts for your printers and your screens. Fantastic idea, and implemented beautifully.

The time is about right, but the source of the innovation is wrong. TrueType Fonts were developed by Apple, as noted by Microsoft.

I am not actually just an 'anything but Microsoft' Zealot, but it is just wrong to state that FOSS does nothing other than 'steal market share from Microsoft.' For example, Microsoft did release NT, which does represent several significant innovations over DOS. According to the New York Times, the original name for NT was OS/2 3.0. OS/2 was an operating system developed by IBM. Microsoft did a great deal to improve the network stack. Namely, they used the BSD Unix network stack, see the Defcon archives for evidence to support this claim. There is nothing wrong with this, but you have to admit that the innovations for the network stack came from those open source Unix developers at Berkeley. Just for giggles, open ftp.exe, its in your C:/Windows/System32 directory with Notepad. You will see the BSD license. Microsoft is not amazing because they invented everything, they are amazing because they integrated innovations from all of the world.

Remember Pablo Picasso's observation, "Bad artists copy. Good artists steal." If you want to be creative, you have to steal the good ideas.

Choosing SCT Type at Run Time

2009-05-05T17:48:00.000-07:00

When a data warehouse is designed, the architect must choose a dimension type for each dimension in the warehouse. The two must common types identified by Ralph Kimball are the Type 1 SCD, in which changes in dimensional attributes are overwritten. In the Type 2 SCD, changes are not overwritten, but are recorded in multiple records for each object in the dimension table.

After using Pentaho's PDI, it become clear that you can specify the dimension type at the attribute level as long as the underlying table is a Type 2 SCD. Still these decisions need to be made when the dimension table's ETL is being designed.

What happens if you would rather delay the decision until run-time? This flexibility can be provided with an auxiliary table for each dimension table. The auxiliary table links each record in the dimension table with the most recent record for that object. With that view defined, you can easily set up a view for a Type 2 dimension table that behaves like a Type 1 dimension.

I am surprised that I don't find any references to this in standard data warehouse references or in online articles. If anyone can point out a reference to this, I would be grateful,.

Learning to program on OS/X

2009-04-17T15:05:00.000-07:00

Ages ago, I used HyperCard on a Mac SE. Now, I'm trying to get back into programming on an iMac running Leopard. My first observation, Holy Cow! This is a fast workstation. This absolutely blows away my workstation at work. At work, I'm (still) running XP with Office 2003. Outlook is a complete hog: between Outlook, Windows checking for updates, and virus scanning, just logging up takes a few minutes. All day long, I am listening to my hard drive chatter as I work on as Pentaho BI Server site, so once I get Tomcat, Eclipse and a browser or two running. I boot up, log in, and go get coffee.

But on my Mac, Netbeans 6.5 comes up faster (under 15 seconds). Learning Grails is a joy with a workstation that responds quickly. I am quite happy to pay my 'Apple Tax', I get OS/X, iLife and iWork. I also get a faster JVM. The biggest downer is that I can't run IE, so its hard to tell if my valid web pages will break on the world's dominant browser.

To get started, I followed these instructions at ShiftEleven to get up and running with Apple's developer tools, the MacPorts tools and PostgreSQL. However, I didn't want to start PostgreSQL by default. So rather than using lauchctl, I'm just going to type

sudo -u postgres/opt/local/lib/postgresql83/bin/postgres -D /opt/local/var/db/postgresql83/defaultdb

in iTerm.

Dimension tables in data warehouses: Type 2+1

2009-03-28T09:39:00.000-07:00

Introduction

In condensed matter physics, a system is often characterized by the dimensionality of the crystal lattice. A bulk crystal has a 3D lattice, while a surface has a 2D lattice. Crystal growth occurs on the surface. There are times where surface growth is described as being 2+1D. I used to research this.

In data warehouses, data is stored using a data cube. When a data cube is stored in a relational database, a star schema is build from fact tables and dimension tables. In a fact table, the facts are stable and do not have any time dependence: facts are facts. All of the time dependence is recorded in the dimension tables.

The problem: with example

To make this more concrete, consider a simple data cube for a swim conference. The facts will be the race results. The dimensions will be a swimmer dimension, an event dimension and a meet dimension. Race results include the time recorded. The results are linked to the dimension tables. So, we can build reports on a given swimmer, even if they change teams. Swimmers have a name, photo and birth date. Swimmers also have a link to their current team. Teams have a name, coach, city and league. (This example gives us a simple 'snowflake dimension'). Meets have a location, date, league and season. Events include a stroke (backstroke, butterfly, breaststroke and freestyle) and distance. For now, we will ignore team events, so the schema is quite simple.

This is for a Web 2.0 swim league, so we also have to create an iPhone web app to record all of the results during a meet. We can even be cool and have a URL to a 'photo finish' image for each race, pictures for each swimmer, and a Google map showing the location of each meet. After each meet, we can then run a Kettle job to move all of the data into our data warehouse running on Amazon EC2 at 10 cents an hour.

In the Data Warehouse Toolkit, Ralph Kimball has described three types of time dependence for dimension data

Type 1 - the past doesn't matter. When data in the dimension tables changes, we just throw away the old data and use the new data. This is appropriate for correcting errors. If we find that we should have spelled Vicky rather than Vickie, we just change the name in production database for our application. The ETL process will then change the swimmer's name in the data warehouse.
Type 2 - slowly changing dimension. In this case, a single entity in the production system gets mapped to multiple records in the dimension table. At any moment in time, there is only one record that is valid for the entity. When we change the description in the production system, the ETL job will create a new record in the dimension table. If our iPhone toting parents take a new photo of Vicky every month, Vicky will will get a new record in the swimmer dimension table. When we look at the results of a race where Vicky was swimming, we can see what she looked like at the time of the race.
Type 3 - keep a few versions. In this design, we end up with multiple columns for a given attribute in the dimension table. Usually, this is used so we can have a 'current' value and a 'last' value. This is used for something like an company reorganization. Vicky is a superstar and in the middle of the season she is forced to move to a really weak team to make the league more equal. The parents are upset, so they track Vicky with the current team assignments, but they also track her beginning-of-season team. This example may seem silly, but similar reorganizations happens all the time within companies.

As the database architect, what are you to do? The first thing to realize is that you have to decide on the time dependence for each field in the database. In our case, we want a Swimmer dimension with name as a type 1 dimension attribute, a 'photo URL' as a type 2 dimension attribute and a type three dimension for the 'team' attribute. To further complicate matters, we didn't even know that the team should be 'type 3' before the Reorganization Controversy.

In physics and chemistry, we talk about reversible and irreversible transformations. In a computer system, erasing data is irreversible. From this perspective, we see that only the type 2 time dependence does not erase data: type 2 is reversible in the sense that even after we change the data in production, we can still reproduce the old data exactly - if a report was run last year, we can get exactly the same results running the report this year. A type 1 dimension throws data away immediately and a type 3 dimension will throw data away eventually (after two team changes in the example). Since a data warehouse is supposed to warehouse data, I have an immediate preference for the type 2 time dependence.

But as a practical matter, people often prefer the 'curent' results, not the results 'at that point in history'. For example, lets say that we have a swim team dashboard that lets you click on a photo of each swimmer to see their stats. Almost certainly, you want to have only one picture of Vicky that links to her entire history, not simply her history since that picture was take. When designing the warehouses, I am constantly having to choose between 'current' and 'historical' dimension data.

The New Solution: Type 2+1 dimension

In order to explain this, I need to define the structure of the production database and the data warehouse for the swimmer. In the production database, the swimmer table will have the following columns:

GUID, an id for each swimmer that is unique for every swimmer in every league that is using my Swim 2.0 web application. This is the primary key of the production table.
Name, a string with the swimmer's name. There could be separate fields for first and last names, but this doesn't matter for this discussion
Birthdate, a date field.
Photo URL, preferably a URL class
Team GUID, a foreign key tying each swimmer to their current team

In our data warehouse, our swimmer dimension is also represented by a database table. In a table designed for type 2 data, we would have the following columns:

GUID (same as above, the G is for global, and the data warehouse is part of our world of data)
ID, a unique integer determined by the database engine. This is the primary key of the swimmer dimension table.
version, an integer that starts at 1 and increments by 1 each time a new record is added for a given GUID. There is a 1-1 mapping between (GUID, version) and ID
start date and end date. These two columns define when a given version is valid.
name, birthdate and photo URL fields, just like in production
Team ID. Foreign keys reference primary keys, so rather than recording the Team GUID, we will record the Team ID. This may be a bad idea, if you have arguments about when to use the GUID and when to use the ID when dimensions reference each other, I would be happy to hear from you.

The rule for a data change is fairly simple. In ETL, any changes in the production data are detected. When a change is found, the current version of the entity (e.g., with the same GUID) is end dated and the current version is read and the end time is recorded. A new record is inserted with the GUI, the attributes from production, a new ID , the next version number, the start date is set to the end date of the previous record and the end date is usually sent to 'the end of time'.

In order to identify which version is current, the data warehouse architects at ETL-Tools recommend adding a current field, a Boolean set to true for the current value and false for all others. This field is could be determine by a simple rule, such as now() between swimmer_dim.date_from and swimmer_dim.date_to or swimmer_dim.version = ( select max(version) from swimmer_dim sd where sd.GUID = swimmer_dim.GUID) (I use the default field names from Pentaho's Kettle, and the date functions from PostgreSQL, but this is easy to rewrite for other environments.) As a practical matter, it is more efficient to add a current field so you can apply the rule once during ETL rather than each time the you run a query that needs the current values.

I argued that a 'proper' design should be reversible and that no data should be lost. Doesn't adding a current field that I am willing to 'throw away' each time I run ETL violate my theoretical argument? No. The reason is that the information in the current field is redundant. Since I can figure out which record would have been current at any time in the past, changing this field does not irreversibly lose data.

Rather than just recording a Boolean as to which record is current, I add a field called current_id to my dimension tables. If you want a Type 2 time dependence, you just ignore the current_id since the rest of the table was set up to support Type 2. So, if you want a dimension with Type 2 time dependence, you can write

create view swimmer_dim_type2 as
select GUID, ID, version, date_from, date_to
, name, birth_date, photo_url, team_id
from swimmer_dim

If you want a Type 1 time dependence to return the current values for any historical id, you need to make a single self join

create view swimmer_dim_type1
select any.GUID, any.ID, curr.name
, curr.birth_date, curr.photo_url, curr.team_id
from swimmer_dim any
join swimmer_dim curr on any.current_id = curr.id

This ability to act like a dimension of Type 1 or Type 2 is why I call this Type 2+1. But, we know that 2+1=3, so is there a way to get the Type 3 time dependence as well? Yes, but we will need to be able to choose a time when we want to choose the additional columns. In the swimmer example, we want the current team and the team at the beginning of the season. Let's assume that the season began on June 1, 2008. We can create our Type 3 time dependence with:

create view swimmer_dim_type3 as
select any.GUID, any.ID
, curr.name, curr.birth_date, curr.photo_url
, any.team_id, june.team_id as starting_team_id
from swimmer_dim any
join swimmer_dim curr on any.current_id = curr.id
join swimmer_dim june on any.current_id = june.current_id
where cast('2009-06-01' as date) between june.date_from and june.date_to

So, Type 2+1 can emulate Type 1, Type 2 or Type 3 time dependence. As I noted earlier, the time dependence should be determined at the field level, not the table level. If we want to always use the current name, the photo that was current at the time of each race, and the team as of June 1, 2008, we can write a view to do exactly that without the need to alter the ETL or the table structures.

I have tested this idea on larger data sets, and each view is almost as fast a query of the underlying dimension table. In most queries of a data warehouse, the query performance is dominated by the reads and aggregates of the data within the fact tables. This technique should be valuable to nearly any data warehouse architect, so I would be pleased to hear from anyone who finds this useful.

Bayes' Theorem and Taleb's Silent Evidence

2009-03-21T08:34:00.000-07:00

In chapter Eight of the Black Swan, Taleb recounts Cicero's Story of the Drowned Worshippers:

One Diagoras, a non believer in the gods, was shown painted tablets bearing the portraits of some worshippers who prayed, then survived, a subsequent shipwreck. The implication was that praying protected you from drowning. Diagoras asked, "Where were those who prayed, then drowned?"
The drowned workshippers, being dead, would have a lot of trouble advertising their experiences from the bottom of the sea. This can fool the casual observer into believing in miracles.

Taleb calls this as the problem of silent evidence. We a drawn to success, and we shun failure, so, as Taleb notes, nobody ever writes 'How I failed to Make a Million Dollars on Wall Street'.

Suppose that one of our Wall Street success stories claims that in order to succeed in Wall Street, you need to be a Harvard graduate; this author cites himself and and his college roommate who have both made a million dollars on Wall Street. If you are in high school (or have a child in school), you need to choose a collage. You are wanting to know if a Harvard education is worth its rather considerable cost. If the author is right, then the decision is easy. You will simply replay your student loans with the millions you make on Wall Street.

What information do you need to assess this claim? We could begin by looking for particular cases. Finding one black swan may prove that not all swans are white, likewise finding one millionaire from a state university will disprove a claim that a Harvard education is a requirement for success on Wall Street. Similarly, finding an unsuccessful Wall Street trader from Harvard proves that a Harvard education is not sufficient for success on Wall Street. Likewise, we can find examples of Harvard graduates that failed on Wall Street. So being a Harvard Alumni is not a guarantee of success of Wall Street.

Life is not certain, so we need to make a gamble. How can we to play the odds odds intelligently? The sort of data that would be helpful would include the resumes of both successful and failed Wall Street investors. As a practical matter, getting the list of unsuccessful investors would be something of a trick. Taleb is right about silent evidence: we remember the winners and forget the losers. The winners advertise and the losers go on to something else.

In the age of data mining and databases, it should be possible to build such as list.

If we can gather the data, what framework should we use to decide if going to Harvard would be a rational risk? The correct framework to assess the validity of a claim is to use Bayes' Theorem from statistics. If you haven't heard about Bayesian statistics, a good starting point is An Intuitive Explanation of Bayes' Theorem by Eliezer S. Yudkowsky. In fact, Yudkowsky's explanation is so good that there really isn't any point in me writing more on this subject. There are also on-line course materials. Currently, I'm trying to work through Jeff Grynaviski's (at the University of Chicago) has provided his course materials. In order to learn more, there is David McKay's Information Theory, Inference and Learning Algorithms. I especially like McKay's book because it is able to unite Bayesian methods with Claude Shanon's Information Theory and even to include elements of AI.

If you still have some time left after reading about Bayesian methods, read Yudkowsky's article Cognitive biases potentially affecting judgement of global risk. His assessment seems to be in general agreement with Taleb. Since I live next to Cedar Rapids, which has just had a major flood, Yudkowsky's observation about flood damage was particularly revealing:

Burton et. al. (1978) report that when dams and levees are built, they reduce the frequency of floods, and thus apparently create a false sense of security, leading to reduced precautions. While building dams decreases the frequency of floods, damage per flood is so much greater afterward that the average yearly damage increases.

Wow, this is an extraordinary claim. If true, the much of the work done to protect people in the Mississippi/Missouri River basins is not just useless but is actually counterproductive. I will remain skeptical on this claim, but it does seem worthy of investigation before Cedar Rapids, Linn County, the State of Iowa and Federal agencies sink more money into flood control on the Cedar River. We need to find a more rational way to understand and manage risk. If we can improve that, we will have learned an important lesson from the catastrophic failures of 9/11, the Indian Ocean Tsunami of 2004, the flooding of New Orleans following Hurricane Katrina, and the current financial collapse. We need Taleb's empiricism. When we have enough data to analyze, we should be using Bayesian methods for that analysis.

Why Swamp Fox Analyst?

2009-03-08T07:44:00.001-07:00

First, I live in Marion, Iowa, home of the Swamp Fox Festival. I was was raised largely in Marion County, Iowa. Both are named after Francis Marion. Of course, there are hundreds of places in the US named after Francis Marion. I am no historian, but even a quick read of his Wikipedia article or the Smithsonian's recent article reveal a fascinating character. He was a small child, being something of the family runt. He was a good student and fluent in French. At age 15, he survived a shipwreck when a whale rammed the schooner he was sailing. At 25, he became a military officer in the French-Indian Wars. The following quote

The next morning we proceeded by order of Colonel James Grant, to burn down the Indians' cabins. Some of our men seemed to enjoy this cruel work, laughing very heartily at the curling flames, as they mounted loud crackling over the tops of the huts. But to me it appeared a shocking sight. Poor creatures! thought I, we surely need not grudge you such miserable habitations. But, when we came, according to orders, to cut down the fields of corn, I could scarcely refrain from tears. For who could see the stalks that stood so stately with broad green leaves and gaily tasseled shocks, filled with sweet milky fluid and flour, the staff of life; who, I say, without grief, could see these sacred plants sinking under our swords with all their precious load, to wither and rot untasted in their mourning fields.

suggests to me that he was a sensitive and compassionate man, but still a man that did his duty. He is most famous for his leadership in the American Revolutionary War. Following the Waxhaw Massacre , his band of about fifty irregulars was the only force opposing the British in South Carolina. He earned his nickname, The Swamp Fox, by outfoxing Col. Banastre Tarleton and his British forces. His tactics are predecessors of modern guerrilla warfare. In short, he was able to accomplish a great deal with very little. In short, he used intelligence to succeed. I cannot make claims that I am following in his footsteps (I failed my military physical for acne scars on my chest and back), I can certainly take inspiration from his success.

There is also a cautionary note that I associate with Francis Marion. While we Americans, and even Mel Gibson, regard Marion as hero, he was clearly viewed as a terrorist by the British. The historian Christopher Hibbert described him as a racist and rapist for his treatment of the Cherokee. There is evidence that both views are correct. One man's terrorist is another man's freedom fighter.

Francis Marion was also alive during the Age of Enlightenment. His lifetime (1732-1795) overlaps the lives of the skeptic David Hume (1711-1776), the Reverend Thomas Bayes (1702-1761), and the French aristocrat Pierre-Simon, marquis de Laplace (1749-1827). As Nassim Nicholas Taleb notes in The Black Swan, Hume was an influential empiricist for the English-speaking world. For all of the reasons Taleb outlines, we need empiricism and some of the intellectual humility of a real skeptic, to counter the hubris of modern economists.

Taleb speaks a great deal about The Problem of Silent Evidence. In a nutshell, if we try to understand the unusual, we try to look for characteristics that we feel were a cause of the event. For example, we have all been taught that World War I was caused by 'entangling alliances' between the major powers and the 'powderkeg' in the Balkans. But, identifying characteristics that were true before WWI is not the same as identifying causes. To have a useful knowledge, we need to be able to predict, not create an after-the-fact narrative. There were many times in history where there were alliances and many places where angry young men plotted to be revolutionaries. Does the presence of these conditions have any predictive value? That is what we need to know if we want to plan.

It seems to me that some of Taleb's criticism of classical statistical analysis can be addressed by using Bayesian methods. This is the tie to the Reverend Bayes and Laplace, they founded what we now called Bayesian statistics. Over the last twenty years, Bayesian methods have been rapidly evolving. In particular, Bayes theorem provides a mathematical framework to discuss Taleb's Problem of Silent Evidence. Of course, a framework is not a 'solution' and this only addresses one of Taleb's issues. More to come ...