And the Winner is: eXist-db.org My Award for the Most Innovative Product of 2007

As the year starts to come to a close we often take some time to look back at the year and look at the innovations that we have seen that have changed our worldview. I can think of many things that have had a large impact on me: The FireFox XForms extension, discovering the beauty of a well-designed REST interface, the mule enterprise service bus, Yahoo pipes and microformats all come to mind.

But after careful consideration I have to hand my award out to the most innovative application of the year to me to the eXist database/web server. This application constantly amazes me with what I can do with it. If you have a Java JVM on your desktop it only take a few minutes to get it running. Hit the startup script and you have a web server running on localhost:8080. Open a WebDAV browser and drag-and-drop you files and folders and away you go. With eXist I can easily setup a full enterprise metadata registry on a laptop in under five minutes.

The power of eXist comes from its use of the XQuery engine. Any well-formed XML file that is added to the system gets instantly indexed and can instantly be searched. That means that 10ms after you hit the "Save" on your XForms you can see the data appear in XQuery reports.

But the "Secret Sauce" of eXist rests in it's use of a lightweight Jetty web server with a remarkably integrated REST and WebDAV interface. This means that every XML file…is a static web service. This means that every XQuery that returns XML…is a dynamic web service that you can parameterize. This means that you can be writing your first web services in fifteen minutes that can be grabbing data from a dozen different XML files and quickly serializing the results out the wire. Enterprise mash-ups at your fingertips. Very little programming is done other then selecting data. Just like it should be.

eXist technologies are going to start to have a big impact once XForms/REST and XQuery (XRX) web development matures. I predict this is going to happen in the next three years. An although IBM, Microsoft, Oracle and many others are supporting XQuery in a big way, they don't yet have the really smooth integration with next-generation XForms-driven clients. You still need teams of AJAX, JavaScript, Java, .Net and SQL programmers to build simple web services for rich client interfaces.

I must admit it took me a little while to really understand how the simplicity of the interface really rocked my world. Just swap out the world "rest" in the URL and replace it with the word "webdav" and you go from a data browser to a file system that works with every copy/cut and paste operation of the file system.

It is also interesting to note that this product did not come out of a Bay Areas startup, it didn't come out of Google Labs, it didn't come from Microsoft or IBM. From what I can tell the initial version was pretty much written by a single guy in Germany, Wolfgang Meier. It think this shows that there is still room for true innovation by a single individual in this world. Although eXist now appears to have a great team of people behind it, it shows that a single person with a clear architectural can put the right pieces together really can make a difference in the world. Our teams could not have been as productive with XForms if we didn't have a clean and elegant tool like eXist behind our forms. I know that XRX innovators will no be able to stand on Wolfgang's shoulders and build incredible applications.

Thanks Wolfgang! Your labor of love is growing up!

Introducing the XRX Architecture: XForms/REST/XQuery

At the XML 2007 conference there were many people that seemed to have independently discovered that if you combine XForms/REST and XQuery you can create a software development environment that circumvents the need for middle tier objects and conversion to and from relational databases.

The conference also had a great deal of discussion of how XForms is a great architecture but as Elliot Rusty Harold pointed out, we need something to happen for XForms to replace things like AJAX. Although Rusty did propose a few interesting ideas (like getting XForms built into FireFox) I feel what we really need is a contagious meme. And I think it goes beyond XForms. What we need is an easy-to-remember and easy-to-communicate name for a collection of great ideas. For example there are few books that have the word XMLHttpRequest in it. But there are hundreds of books with AJAX in the title. Once the word AJAX was coined the meme started to spread. The world was waiting for the "sticky" label behind the concepts.

What XForms needs is a label around the collection of ideas that makes XForms really give you an order-of-magnitude improvement over other web development architectures. I would like to suggest that we get behind one "label" that brings many of use under one big tent. I would like to suggest we use "XRX".

XRX to me would stand for the ideas behind XForms/REST/XQuery web applications. But if people are using XHTML as a container for XForms you could use XHTML/REST/XQuery. Just spare use the "J" which is associated with a hard-to-use language for über geeks.

You could also use XRX for the format of the data in the client and on the server. In this case it would be XML on the client (in the XForms Model), and in a native XML database that supports a REST interface.

We can still build awesome tools and frameworks that make the XRX architecture really fly. IBM Workplace Forms Development tools, Orbeon Presentation Server, Kurt Cagle's new frameworks that use the eXist native XML database, and the hundreds of other XForms development tools all fit into the XRX strategy.

But if we make XRX into a "big tent" strategy we can also include XQuery on the server as part of the architecture. This would include people like Jason Hunter at MarkLogic. Jason is an experienced Java developer that has started to move away from Java on the server and toward XQuery because of its richness. And I could not agree more.

I have been reading the book The Tipping Point: How Little Things Can Make a Big Difference by Malcolm Gladwell and trying to apply the ideas in it to the adoption of XForms and XQuery. I think that the book has many example of how a careful packaging of the ideas behind a contagious meme can make ideas spread like widefire.

So when we hear the term "XRX" we want to have the following concepts come to mind:

1. A web development architecture with a 10x productivity improvement over traditional JavaScript/OO/RDBMS methods
2. A development architecture based on international standards that is designed to minimize the probability of vendor-lockin
3. An architecture that gives a rich user experience without creating mountains of spaghetti procedural code on either the client or the server
4. A system that leverages the REST architecture to take advantage of high-performance and simple interfaces using web standards
5. Portability on both the client and the server using a variety of forms players and XQuery databases
6. The option of avoiding costly shredding (and reconstitution) of complex XML documents into RDBMS tables
7. A community of standards/tools and a "complete solution" ecosystem that can give you a proven ROI on your IT investment

I don't expect everyone to buy into the XRX meme ideas. Everyone has their own favorite tools/standards and acronyms. But everyone that was at the XML 2007 conference could feel the excitement in the room at the XML presentations. Now we need to give that excitement a name and see if it sticks. Hopefully a few years we will be able to walk into a bookstore and see a bunch of books with "XRX" on the cover.

What I now need is for people to start to test the XRX meme and see if it slicks. Let me know what works and what does work. Send me your 30-second elevator pitch. How big should the idea tent be? What do we have to do to get more people in our tent? What evidence (case studies), example programs and resource do we need to convince the world we have seen the light …and its so elegant and beauty to will keep us from ever going back to the stone age of web development.

Dynamic Graphs for K-12 Educators

At the recent XML 2007 conference in Boston several of us met to discuss what we could do to promote FireFox and some of the important standards built into FireFox (or almost built into FireFox). Specifically several of us were using XForms and SVG and thought that the combination offered some great opportunities.

It was pointed out that the combination of the XForms range control (a slider control) and SVG allowed developers to create dynamic graphs for teaching educational concepts. There is an example here:

http://en.wikibooks.org/wiki/XForms/Pie_Chart

Note there is a link to the actual program here which you will need to use the FireFox XForms extension to view.:

http://xforms-examples.googlecode.com/svn/trunk/10-full-examples/86-piechart-range-controls/pie-chart.xhtml

So several of us are interested in putting together a grant to develop SmartBoard enabled educational content for the K-12 market using a combination of FireFox, XForms and SVG. The goal would be to create a "Killer App" for FireFox in the K-12 space. Ideally we will build 100 demos of some of the most useful dynamic graphs and then build a framework for teachers to extend these applications.

I also have a few Supply and Demand examples that run under IE but require the SVG add-on and just just plain JavaScript for the range-controls: http://www.danmccreary.com/svg/ You will note that visualizing the maximum profit by the area of the square under the triangle is much easier when you add the motion.

It was also pointed out that the Mozilla foundation has the funds to support this type of development and that they might fund a grant to build the top 100 useful dynamic graphs. We could then start building a toolkit to extend these dynamic graphs using some of the advanced drag-and-drop connection-based programming parts of SVG and FireFox. The kit should shield the average middle school physics teacher from haveing to write and debug JavaScript.

Let me know what you think - Dan

Impressions of XML 2007 in Boston

I returned from the XML 2007 conference in Boston with my head spinning about all the new developments with XForms and XQuery. The conference was three days long and had between three and five tracks going concurrently. My impression was that there were about 400 people there.

Here are some of the highlights for me.

XForms appears to be getting lots of traction. The XForms sessions were all packed with standing-room only in most of them. The presenters we passionate that the combination of XForms, REST and XQuery has had a huge impact on their projects and may soon impact the rest of the development world.

Microformats are definitely starting to make huge strides where Metcalf's Law will start to kick in. The existence of 450 million tags and five FireFox microformat extensions indicate that within a year it will be just part of web publishing best practices. Even Microsoft is said to be supporting Microformats in IE 8, just a few years from now.

Taylor Cowan (who works for Sabre) gave an excellent presentation on how Microformats can be extended into the travel space with an Atom based format for publishing trip ideas. His use of Atom and semantic web technologies was right in line with my vision of how personalized agents are going to leverage microformats.

John Boyer's demo of IBM's Workplace Manager showed that you can now purchase top-shelf XForms GUI design tools that does not lock you into a proprietary vendor solution. I hope the people from Microsoft, Adobe, FairIsaac, Altova, Google GWT and other vendors take notice. Locking your users into a Java or JavaScript client is no longer an acceptable strategic option for developing web applications. Wake up everyone! Are you listening? We have a standard that works and lets now start building more great tools that use this standard. XForms is the only system that is creating as John puts it "Order of Magnitude" increases in developer productivity.

Mark Birbeck's Sidewinder demo showed that XForms is breaking out of the browser an on to the desktop. Showing web and iPhone web pages directly in desktop widgets was a great demonstration of the power of XForms.

Kurt Cagle's presentation about problems with programming with the fragile DOM model in JavaScript was interesting in its format. The presentation was actually an XForms application running on eXist. Kurt "eats his own dog food" on the client and the server! I am humbled by the man.

Jason Hunter from MarkLogic (an XML database vendor) gave an nice demo of the MarkMail.org application and showed that, yes, XQuery databases really do scale to the terabyte range. He has about 4 million e-mail messages indexed and you can do combinations of database and text searches in a few seconds. In other words a demonstration of how real-world analytics comes to XQuery. I also found out that MarkLogic is free if your database is under 100MB. Perfect for 10MB of metadata in a metadata registry.

The people from IBM research labs appear to be starting to take the XForms MVC architecture and really run with it. We saw some very interesting presentations on how future standards for data driven XML Application Components (XACs) and State Chart XML (SCXML) might be used in the future to make browser-based mash-ups easier. Componentization, composition, customization and reuse are all part of this architecture and it would not be possible without XForms MVC architecture. Hopefully Charles Weicha from IBM research will post his slides so you can see more.

Norm Walsh also talked about how the new XML Pipeline standard (XProc) is starting to take shape. Even though the standard is still in working draft there are already at least five implementations of the standard. This single standard could have more impact on the shape of server-side processing then any other standard coming out of the w3c. Integrating this into XQuery-based application servers seems to be the next step.

The eXist database also seemed to clean up all the open source native XML database talks. eXist was mentioned in over a half-dozen presentations and there were no other native XML databases even mentioned in the presentations I attended. It is not surprising that eXist is now the top hit on Google when you search for "XML database".

One of the most interesting presentations that I did not get a chance to see (but I did talk to the presenter afterwards) was Thomas White's presentation on using XSLT in the browser to get a 10x speedup in form rendering. It turns out that implementations of XForms players like FromFaces use heavyweight JavaScript objects in the browser. Thomas has found that a donkey cart can be turned into a sports car by implementing an in-browser XSLT centric event scheduler. If his architecture is implemented we could see vastly better XForms players in the future. Lets all hope that Thomas's architecture comes to the XForms world sooner than later.

I did not attend any of the XML publishing tracks. But from the titles it appears that the Darwin Information Typing Architecture (DITA) standard is a hot topic. Presentations on using XSL-FO and CSS for book and other print publishing were also on the agenda.

I also did not attend too many of the XML in the Enterprise tracks. There were good presentations on XML accelerators, security and mining XML Schemas.

I am disappointed that the conference organizers did not do anything to require that people post their presentation on the web site before their talk. There were many very interesting presentations that could have a larger impact the world and the sloppiness of people not publishing their slides seems to be a little unprofessional. There were some people that just brought their digital cameras and took pictures of each slide. Smart.

In summary I think that the conference had two types of people:

First there were people that had not done extensive work with XForms. The first group I call the "evolutionary proceduralists". Theses were people that were trying to beat the dead horse by patching the old-style procedural web application architecture with JavaScript, JSON and AJAX hacks. They seemed to spend a lot of time arguing about the merits of JSON vs. XML as a data transport syntax.

The other type were "the enlightened innovators". People that have seen the real world benefits of dumping object middle-tier stacks and relational databases and going with a pure declarative approach to solving business problems based around XForms/REST and XQuery. These were the people that are going to lead innovations in application development.

I was also impressed how a large percentage of XML innovations came from outside the US. I attribute the fact that some of the best presentations came from the UK to be the "Michael Kay" effect.

Beauty is in the Eye of the Beholder At one of the XForms sessions a young Ruby advocate stated that he thought his Ruby code was "beautiful" but he did not himself think that XForms code was "beautiful". Most of the people in the audience agreed that XForms was beautiful but like any new language, it takes getting used to. Beauty is the process of your visual cortex recognizing familiar patterns and signaling your pleasure centers. If you have never read Japanese then the most profound Haiku written in the most beautiful script will not have beauty.

XForms/REST/XQuery to me is indeed beautiful. Its beauty comes from its ability to quickly map real-world requirements into working systems with high fidelity. No other system in the world approaches its elegant architecture. And after XML 2007 I realized I was not the only one that had independently reached this conclusion.