XForms, Dyslexia and the Right Brain

Those of you that have worked with me know that I am a little bit of an odd-duck. I have dyslexia. If you ever see me attempt to write on a whiteboard my spelling is at the ninth grade level. My left brain, used for phoneme recognition, never really developed like the rest of you. My right brain had to be co-opted to help out. But one of the skills I seemed to have picked up due to my over-exercised right brain is the ability to visualize multiple complex application architectures and quickly understand architectural tradeoffs. I am one of the few people that seem to be interested in discussing how XForms, metadata registries, ontologies, the semantic web, OWL, RDF, graphs, business rules, BPM and Kimball conformed dimensions all can work together to deliver elegant and cost-effective enterprise-scale solutions. It seems easy for me to simultaneously visualize two or more architectures and it constantly challenges my patience when I have to explain over and over why architecture alternatives will not meet a business requirement.

It turns out that many people that have dyslexia also have the gift of being able to visualize complex systems. Albert Einstein, Thomas Edison, Jackie Steward and Charles Schwab are good examples of dyslexic people that have used the strengths of the right brain to do things that left-brain thinkers could not.

As a right-brain centric person I need to also tell you that I really love the XForms architecture. The magic of a declarative language, MVC, bindings and a dependency graph makes XForms development 10 years more advanced than anything else I have worked with. I think it is beautiful and elegant. It is everything that AJAX and JavaScript application are not. Clean, simple and easy to visualize (for me at least). When someone asks me if I can create an XForms application to do something, I create a mental image in my mind of the model, the view and how events will update instance data in the model using inserts or external submission results. I can easily visualize the bindings of view controls to data elements in the model. Once I can visualize the application clearly, writing the application is just a matter of typing in the code.

I think that my right-brain is also a reason I detest JavaScript and AJAX. It is far too much code to read and trying to visualize how 300 lines of JavaScript enables me to do a drag-and-drop. I want to just add an attribute to an element like "drag-source" and "drop-target" and I want it to just work.

What triggered this posting is that I have been reading Proust and the Squid by Maryanne Wolf. This is a book about how the brain's circuits are used in the reading process. She has a wonderful explanation of how the dyslexic brain co-opts the right brain for reading and enhances it functionality. I didn't really understand the relationship between my defects and my gifts .

So how about you and your development team? Do you have a dyslexic right-brained person on your team? Can they quickly visualize architectural tradeoffs? Have they tried XForms? And if they do, will you be willing to tolerate their disgust of AJAX and JavaScript after they have built their first XForms applications?

For more information about dyslexia check out these two Wikipedia entries: http://en.wikipedia.org/wiki/List_of_notable_people_diagnosed_with_dyslexia http://en.wikipedia.org/wiki/Dyslexia

Great Example of Multi-dimensional Bubble Chart

Here is a beautiful example of a bubble chart display using global population statistics. The example on carbon emissions are very interesting. Note the dimensions

1. Population of County (size of bubble)
2. Continent (color of bubble)
3. Live Expectancy (vertical axis)
4. Income (horizontal)
5. Time (the play button)

It is interesting to see the huge impact AIDS has had on the life expectancy in African counties. Imagine if you could see your organizations product sales using this type of graph. This application was done with a software system called Trendalyzer. It was initially developed by Hans Rosling's Gapminder Foundation in Sweden and acquired by Google Inc. in March 2007. This version is a Flash application. Does anyone know of any open-source software that could do this?

Metadata Repositories vs. Metadata Registries

For several years people have been using the terms metadata Registry and Repository inconstantly, imprecisely and almost interchangeably and I would like to weigh in as to how these terms could be used more precisely to allow organizations to effectively to manage metadata processes.

First lets take the definition of a Repository. Webster defines a repository as …a place, room, or container where something is deposited or stored.. Note that here is nothing in this definition about the quality of the things being stored or the process to check to see if new incoming items are duplicates of things already in the repository. If I have 100 users they could each define "Customer" as the see fit and put their own definition into the metadata repository as their own definition. No problems.

On the other had lets take the word Registry. A Registry has the connotation of more than just a shared dumping ground. Registries have the additional capability to create workflow processes to check that new metadata is not a duplicate (for a given namespace). One of the definitions from Webster is an official record book. Note the word official.

A Repository is similar to a front-porch of a house. No locks prevent new things from landing there. But a Registry is a protected back room where human-centric workflow processes are used ensure that metadata items are non-duplicates, precise, consistent, concise, distinct, approved and unencumbered with business rules that prevent reuse across an enterprise. These registries have become the central foundation that agility can be baked-in to many enterprise process. The latest version of the Kimball's Data Warehouse Lifecycle Toolkit (which is actually a very good read) even goes as far as to call their process "metadata-driven". Not different the model-driven development world.

Registries have the implicit connotation of trust behind them. They now serve a a central process for the creation of shared meaning across the enterprise. Definitions in a registry have been vetted by an enterprise-level organization that has the responsibility of enterprise data stewardship. They have a high probability of being consistent with industry best-practices and vertical industry standards. Registries are the go-to source for creating canonical XML schemas, enterprise ontologies or conformed dimensions in a OLAP cube. Repositories are personal or small departmental definitions of an isolated view of the world.

None of these ideas are really new. They are at the core of the ISO/IEC 11179 metadata registry standard. Note that they don't call it a repository standard! People are just now starting to understand how important Registries are in most enterprise-wide systems. The growth of Business Intelligence and Enterprise Data Warehouse terminology and Service Oriented Architectures is a good place to see the rise of repositories and registries. We now see service registries, portlet registries, model registries...the list goes on-and-on.

Much of the background on the differences between the use of repositories and registries can be traced way back to the early days of object-oriented systems in the 1995 book Succeeding with Objects by Adele Goldberg and Kenneth Rubin. This was one of the first books on enterprise reuse strategies and they defined the concept enterprise asset reuse and the need for a trust-driven repository as a basis for reusing assets. They identified a multi-step process for reviewing new submissions to determine if the submission duplicated existing assets. They showed how critical it was to classify items in a registry and search an existing registry for duplicates before new items are added. If you can get a copy of the book I would suggest you read the section on "Set Up a Process for Maintaining Reusable Assets" on page 245.

The book then goes on to show how organizations can and should be structured to reuse these assets and gives the pros and cons of the differing organization structures and their impact on reuse. This is the basis for the data governance and data stewardship movement in many organizations today.

So the next time someone uses the word registry or repository in a conversation, ask them if they are using the definition of the word that is consistent with the corporate business term registry or is their own private definition from their own repository of imprecisely used buzzwords.

XForms Tutorial and Cookbook Voted Featured Wikibook!

I am happy to announce that the XForms Tutorial and Cookbook that I have been working on for over a year been voted a "Featured Book" on the wikibooks web site. A quote from the award:

XForms is a featured book on Wikibooks because it contains substantial content, it is well-formatted, and the Wikibooks community has decided to feature it on the main page or in other places. Please continue to improve it and thanks for the great work so far!

Here are some stats on the book so far: 90 sample programs
116 chapters
626 edits
28,811 words
29 registered authors

One of the big tasks is to start to move some of the advanced XForms examples that require siginficant server-side logic to a seperate server-specific book. As a pilot I have started an XRX cookbook for people using REST interfaces and the eXist server.

I would like to thank the over 30 others that have contributed ideas and content to this wikibook. We still have lots of work to do to cleanup the example programs, make them more consistent and add additional examples for new XForms students. But I believe it is one of the best examples of collaborative training that I have worked on in the last few years.

XRX and MDA

Summary

This posting describes how using three technologies (XForms, REST and XQuery or XRX) you can dramatically transform your organizations development methodology. You can move true model-driven architecture (MDA) that dramatically reduces the temptation to duplicate code.

In January of 2007 I was working on a complex real-estate forms project for a state agency in Minnesota. Our job was to get 87 counties across the state of Minnesota to agree on the over 250 data elements that describe real estate transactions. We had spend months developing a complex XML Schema and web forms with over 50 one-to-many relationships. And out next task was to be able to save the form

data into a structure that could be quickly queried by any of these 250 data elements. But the prospect of "shredding" the documents up into 50 distinct SQL INSERT statements and then reconstituting the documents with 50 distinct SQL SELECT statements was going to require doubling our small team of four to at least eight developers. We didn't have the budget or schedule to do this. After chatting with Kurt Cagle he suggested saving the data to a database that supported XQuery. Both the open source eXist native XML database and our corporate standard, DB2 supported XQuery. It turned out that we could indeed save the entire document with a single command and still perform complex queries on any element in any document. Our project could proceed with a small team and stay on schedule.

And then an interesting thing happened. Our team started to understand that if we saved additional metadata in these native XML data stores we could accelerate our project even further. Every pick-list in every form could dynamically call REST-enabled web services to get their values. XML Schemas could be stored in the databases and be queried. We had found a simple and elegant solution to the pervasive problem of where to store the model in a model-driven development project.

In the past I had been on large teams of developers that attempted to use model-driven development. But although we started out each project with idealistic goals, each time we were faces with a classic option: copy, paste and edit the model or write a transform. The copy and paste solution was quick but it would have to be redone each time the model changes. The transform took longer to write but then could be rerun each time the model changed.

Like most developers, I always was over-optimistic that I had done my homework and done accurate and detailed modeling of the problem. I thought the models were stable and would not change very often. And I was almost always wrong. Models do change and sometimes in totally unexpected ways.

But by storing our models in a native XML database an interesting started to happen. We could quickly transform the models (stored in XML) into other artifacts with simple XQueries. It became far less tempting to commit the sins of copy/paste/edit when it only took five minutes to create another transform. And most remarkably, each of these transforms was another REST enabled web service that could be reused by many web clients or other development tools.

So how do you get started? You might try the following steps.

1. Download the exist database from http://www.exist-db.org
2. Use a WebDAV interface and copy some XML Schema documents into a collection
3. Write a few small XQuerys that pull metadata out of your XML Schemas. For example pull out all the enumeration values out of an element.
4. Use those XQueries to drive some aspects of your development such as an XForms selection list.
5. Compare the amount of code you wrote with any other MDA system. If you don't have at least a 10 to 1 savings I would be very surprised.

What do you think? Do you have a more efficient way to query your model? Let me know!

- Dan

And the Winner is: eXist-db.org My Award for the Most Innovative Product of 2007

As the year starts to come to a close we often take some time to look back at the year and look at the innovations that we have seen that have changed our worldview. I can think of many things that have had a large impact on me: The FireFox XForms extension, discovering the beauty of a well-designed REST interface, the mule enterprise service bus, Yahoo pipes and microformats all come to mind.

But after careful consideration I have to hand my award out to the most innovative application of the year to me to the eXist database/web server. This application constantly amazes me with what I can do with it. If you have a Java JVM on your desktop it only take a few minutes to get it running. Hit the startup script and you have a web server running on localhost:8080. Open a WebDAV browser and drag-and-drop you files and folders and away you go. With eXist I can easily setup a full enterprise metadata registry on a laptop in under five minutes.

The power of eXist comes from its use of the XQuery engine. Any well-formed XML file that is added to the system gets instantly indexed and can instantly be searched. That means that 10ms after you hit the "Save" on your XForms you can see the data appear in XQuery reports.

But the "Secret Sauce" of eXist rests in it's use of a lightweight Jetty web server with a remarkably integrated REST and WebDAV interface. This means that every XML file…is a static web service. This means that every XQuery that returns XML…is a dynamic web service that you can parameterize. This means that you can be writing your first web services in fifteen minutes that can be grabbing data from a dozen different XML files and quickly serializing the results out the wire. Enterprise mash-ups at your fingertips. Very little programming is done other then selecting data. Just like it should be.

eXist technologies are going to start to have a big impact once XForms/REST and XQuery (XRX) web development matures. I predict this is going to happen in the next three years. An although IBM, Microsoft, Oracle and many others are supporting XQuery in a big way, they don't yet have the really smooth integration with next-generation XForms-driven clients. You still need teams of AJAX, JavaScript, Java, .Net and SQL programmers to build simple web services for rich client interfaces.

I must admit it took me a little while to really understand how the simplicity of the interface really rocked my world. Just swap out the world "rest" in the URL and replace it with the word "webdav" and you go from a data browser to a file system that works with every copy/cut and paste operation of the file system.

It is also interesting to note that this product did not come out of a Bay Areas startup, it didn't come out of Google Labs, it didn't come from Microsoft or IBM. From what I can tell the initial version was pretty much written by a single guy in Germany, Wolfgang Meier. It think this shows that there is still room for true innovation by a single individual in this world. Although eXist now appears to have a great team of people behind it, it shows that a single person with a clear architectural can put the right pieces together really can make a difference in the world. Our teams could not have been as productive with XForms if we didn't have a clean and elegant tool like eXist behind our forms. I know that XRX innovators will no be able to stand on Wolfgang's shoulders and build incredible applications.

Thanks Wolfgang! Your labor of love is growing up!

Introducing the XRX Architecture: XForms/REST/XQuery

At the XML 2007 conference there were many people that seemed to have independently discovered that if you combine XForms/REST and XQuery you can create a software development environment that circumvents the need for middle tier objects and conversion to and from relational databases.

The conference also had a great deal of discussion of how XForms is a great architecture but as Elliot Rusty Harold pointed out, we need something to happen for XForms to replace things like AJAX. Although Rusty did propose a few interesting ideas (like getting XForms built into FireFox) I feel what we really need is a contagious meme. And I think it goes beyond XForms. What we need is an easy-to-remember and easy-to-communicate name for a collection of great ideas. For example there are few books that have the word XMLHttpRequest in it. But there are hundreds of books with AJAX in the title. Once the word AJAX was coined the meme started to spread. The world was waiting for the "sticky" label behind the concepts.

What XForms needs is a label around the collection of ideas that makes XForms really give you an order-of-magnitude improvement over other web development architectures. I would like to suggest that we get behind one "label" that brings many of use under one big tent. I would like to suggest we use "XRX".

XRX to me would stand for the ideas behind XForms/REST/XQuery web applications. But if people are using XHTML as a container for XForms you could use XHTML/REST/XQuery. Just spare use the "J" which is associated with a hard-to-use language for über geeks.

You could also use XRX for the format of the data in the client and on the server. In this case it would be XML on the client (in the XForms Model), and in a native XML database that supports a REST interface.

We can still build awesome tools and frameworks that make the XRX architecture really fly. IBM Workplace Forms Development tools, Orbeon Presentation Server, Kurt Cagle's new frameworks that use the eXist native XML database, and the hundreds of other XForms development tools all fit into the XRX strategy.

But if we make XRX into a "big tent" strategy we can also include XQuery on the server as part of the architecture. This would include people like Jason Hunter at MarkLogic. Jason is an experienced Java developer that has started to move away from Java on the server and toward XQuery because of its richness. And I could not agree more.

I have been reading the book The Tipping Point: How Little Things Can Make a Big Difference by Malcolm Gladwell and trying to apply the ideas in it to the adoption of XForms and XQuery. I think that the book has many example of how a careful packaging of the ideas behind a contagious meme can make ideas spread like widefire.

So when we hear the term "XRX" we want to have the following concepts come to mind:

1. A web development architecture with a 10x productivity improvement over traditional JavaScript/OO/RDBMS methods
2. A development architecture based on international standards that is designed to minimize the probability of vendor-lockin
3. An architecture that gives a rich user experience without creating mountains of spaghetti procedural code on either the client or the server
4. A system that leverages the REST architecture to take advantage of high-performance and simple interfaces using web standards
5. Portability on both the client and the server using a variety of forms players and XQuery databases
6. The option of avoiding costly shredding (and reconstitution) of complex XML documents into RDBMS tables
7. A community of standards/tools and a "complete solution" ecosystem that can give you a proven ROI on your IT investment

I don't expect everyone to buy into the XRX meme ideas. Everyone has their own favorite tools/standards and acronyms. But everyone that was at the XML 2007 conference could feel the excitement in the room at the XML presentations. Now we need to give that excitement a name and see if it sticks. Hopefully a few years we will be able to walk into a bookstore and see a bunch of books with "XRX" on the cover.

What I now need is for people to start to test the XRX meme and see if it slicks. Let me know what works and what does work. Send me your 30-second elevator pitch. How big should the idea tent be? What do we have to do to get more people in our tent? What evidence (case studies), example programs and resource do we need to convince the world we have seen the light …and its so elegant and beauty to will keep us from ever going back to the stone age of web development.

Dynamic Graphs for K-12 Educators

At the recent XML 2007 conference in Boston several of us met to discuss what we could do to promote FireFox and some of the important standards built into FireFox (or almost built into FireFox). Specifically several of us were using XForms and SVG and thought that the combination offered some great opportunities.

It was pointed out that the combination of the XForms range control (a slider control) and SVG allowed developers to create dynamic graphs for teaching educational concepts. There is an example here:

http://en.wikibooks.org/wiki/XForms/Pie_Chart

Note there is a link to the actual program here which you will need to use the FireFox XForms extension to view.:

http://xforms-examples.googlecode.com/svn/trunk/10-full-examples/86-piechart-range-controls/pie-chart.xhtml

So several of us are interested in putting together a grant to develop SmartBoard enabled educational content for the K-12 market using a combination of FireFox, XForms and SVG. The goal would be to create a "Killer App" for FireFox in the K-12 space. Ideally we will build 100 demos of some of the most useful dynamic graphs and then build a framework for teachers to extend these applications.

I also have a few Supply and Demand examples that run under IE but require the SVG add-on and just just plain JavaScript for the range-controls: http://www.danmccreary.com/svg/ You will note that visualizing the maximum profit by the area of the square under the triangle is much easier when you add the motion.

It was also pointed out that the Mozilla foundation has the funds to support this type of development and that they might fund a grant to build the top 100 useful dynamic graphs. We could then start building a toolkit to extend these dynamic graphs using some of the advanced drag-and-drop connection-based programming parts of SVG and FireFox. The kit should shield the average middle school physics teacher from haveing to write and debug JavaScript.

Let me know what you think - Dan