AnyChart Case Study

I just released a case study (click on the blog post title or click here) using the commercial software charting package called "AnyChart". After using over a dozen charting solutions including Google Charts and FreeCharts I really feel that AnyChart has the best solution that really scales up.  Not only does AnyChart have a large number of charts but it is also very reasonably priced at around $500USD per server. Because I store all the chart specification and implementations in XML and I use eXist to manage the charts it is easy to manage the full lifecycle from requirements to testing all within a single Lucene-search powered environment.

The latest release of AnyChart also generates SVG and HTML5 compliant charts that run on the iPhone and iPad. This really shows how "declarative" systems are much more portable than any flash-based implementations.

If you are looking for an inexpensive but powerful tool for building charts and dashboards I would defiantly give them a try.

You can get samples at their web site at anychart.com

The "Pathification" of NoSQL

In my consulting work I try to focus on the empowerment of the non-programmer to build and maintain their own web applications. This is not an easy task. There are many barriers, some political and some technical. But when a new Business Analysts or Project Manager can create their own metadata management tools and begin to tell others their story I have great job satisfaction.

My current best-practice is to teach non-programmers how to write XPath expressions into their data sets. I tell them that if they are committed to "knowing their data" that I can teach them how to write web-applications in a week or so. The trick is to give them a "template" CRUDS application (Create, Read, Update, Delete, Search) that they can just plug in their own XPath expressions to get the application to work. They don't need to know the thousands of features of XQuery and native XML databases, they just need to know their data, how to write some simple XPath expressions and where to put these expressions into the templates.

I call this process "Pathification", a word that is similar to "Simplification" and "Standardization" but it has a focus on only teaching a very small subset of the many possible tools and a focus on cross database standards like XPath. Once they feel confident they can create simple CRUDS applications in a few hours they are then highly motivated to continue learning XQuery, but on their own pace and balanced with their other job responsibilities like data governance and semantics.

Pathification is something that can happen because native XML database have many enabling technologies. They have WebDAV drivers that makes you database look like folders on your desktop. We have tools that automatically convert XML into HTML and XForms. We have subversion libraries that allow your form "save" to go directly into a version control system. And these tools continually get better and make it easier to empower novice users.

The NoSQL movement is a huge and diverse community. And many people have focused on areas such as scalability and the integration of different types of data into a single XML development environment. These are all great efforts and really move us in the right direction. But I think for us to make NoSQL into a true multi-vendor, multi-product development platform is to create more tools that empower the non-programmer. For people in the native XML world we have a great start. Standards like EXPath, new application packaging tools and XRX application repositories will continue to make it easier for non-programmers to reuse and share their applications.

What I would like to see (but I am not sure is possible) is both XML and JSON-centric system to start to come together and share their ideas on how to make this happen. Can JSON data-stores like MongoDB and CouchDB be "Pathified"? Can we create shared applications that work with both types of back-end systems?

If we do this right, I think the world will be a much better place. People will have more choices for managing both structured and semi-structured data and non-programmers will be able to take a spreadsheet or new data feed in a standard format and build a new web application that manages this data in just a few hours without having to depend on other programmers. I think of this as a system as easy to use as Microsoft Access(TM) but without the chaos: All data can be stored on centralized servers with REST service-oriented interfaces for every data set.

Please let me know your thoughts on these ideas.

McCreary's Law

I am just returning from a three-day conference on Enterprise Information Management. I was very pleasantly surprised to meet many other very smart and warm independent consultants that do metadata and EIM consulting. More posts about what they taught me will be coming.

But these conferences for me to constantly answer the same question over and over again. What do I do an why don't I use traditional RDBMS systems to store metadata? What I need is some strong statement that explains the core principal of the need for agility in metadata management. So in that context, I would like to suggest a law of agile software development. To give it a label, I will just use McCreary's Law for now.

McCreary's Law states the following:

The agility of any software project in inversely proportional to the square of the number of data transformations in the developer stack.

The number of data transformation in the traditional three tier-architecture (Web to Objects to RDBMS to Objects and back to web) is four.

The number of data transformation in the traditional three-tier plus XML web services is six, since web services are usually created by one transform from objects to XML and one from XML to objects.

The number of data transformations in XRX is approximately zero, since we could make an argument that XML to HTML is a 1/2 translation since the HTML and XML tags are different. Of course if you are not changing the order all XML can be styled with CSS which is not really a transformation. This gets around the problem of infinite developer agility if the denominator is zero.

This formula is not perfect and I am sure their will be modifications to it like adding a constant for training and IT resistance to change, but I hope people can use it to predict or explain their current development agility.

Thanks to everyone I met at the EIM conference that listened to my rants and have not discarded me as a complete metadata radical.

XLucene

For the past two years I have been quietly observing the growth of something called Structured Search. This is the use of document structure to aid in search and retrieval of documents or sub-documents in large document collections.

I have also seen a common solution to this problem. This is to use a combination of native XML database and the Lucene keyword indexing problem. But people doing this are doing far more than just simple keyword search. So calling this new approach simply “Lucene” keyword search does not really describe what is going on. So I am proposing an new meme: XLucene.

I define XLucene as using a combination of document structure and keyword searches together to create very precise search ranking or search hit scores. The key is that the design must be able to store each branch in the document tree as a separate subdocument.

The key factor in widespread adoption is that almost anyone that can identify the tag names in their document structures will soon be able to create their own XLucene search and retrieval systems.

What first got me interested in “Structure Retrieval” was the book “Introduction to Information Retrieval” by Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze. In this book they present the following table:

	RDB search	unstructured retrieval	structured retrieval
objects	records	unstructured documents	trees with text at leaves
model	relational model	vector space & others	?
main data structure	table	inverted index	?
queries	SQL	free text queries	?

The following quote follows this table: There is no consensus yet as to which methods work best for structured retrieval although many researchers believe that XQuery will become the standard for structured queries.

I think that the jury has returned their verdict. XQuery wins! Here is that final table with the following properties:

Objects: Trees with text at leaves
Model: Hierarchical documents
Main-data-structure: trees and inverted indexes tied to node-ids
Queries: W3C XQuery with fulltext extensions

The design of using variable-length node-ids as the document identifiers was done by Wolfgang Meier of the eXist-db.org project. There are many people now using this design today. I will be following up this posting with some case studies of how people are using this in several areas, both in the government sector, library metadata and document search and retrieval.

Since there are so many people using this same problem-solution pair, that like all good patterns, it deserves its own name. XLucene (with no embedded dash) can be picked up by search engines and we can all start to share design experience and solutions. This is why the Open Source model really is superior to closed systems. Knowledge just travels faster once the memes are created.

I will also be presenting more details on the general topic of Structured Search and XLucene at the Enterprise Information Management Conference in Toronto next Tuesday, Sept 21st.

What Makes a Good Solution Architect?

A recruiter friend of mine was looking for a senior "Solution Architect" and needed some help setting up a screening interview. Here is what I told him.

I look for solid experience in a diversity of solution. There are just too many people that think that HTML/Objects and RDBMS systems are the only tools in your architect's toolkit.

Here is what I suggested the interview questions would be like:

Consider the following application architectures:

1. Ruby on Rails on a relational database
2. XRX on a native XML database
3. A Java Client on J2EE app server using JMS
4. A JQuery client on an OLAP database

A senior "Solution Architect" should be able to compare the pros and cons of each system and describe how they have used each of these architectures to build solutions and give organizations a competitive advantage. I also drill down into their understanding of search and retrieval and natural language processing.

As a bonus question, I would ask how each architecture would impact an organization's business strategy and to what extent business units could be empowered to build their own applications with minimal training and little or no need for IT involvement.

Most good solution architects should have experience with 3 out of 4 solution architectures but very few usually have a strong MBA-type background to understand how solution architectures impact business strategy. And if you find anyone that has built systems with all for and has an MBA you have a serious senior solution architect. Hire them!

Impressions of SemTech 2010

Two weeks ago I attended my fifth Semantic Technology conference in San Francisco. It was a great conference! This is my fifth time attending the conference and I plan to attend in the future! My biggest dilemma was which session to attend. At times there were as many as eight concurrent sessions going on.

There were a few big trends that I spotted.

The use of RDFa to annotate web pages using semantically precise elements was clearly a big trend. Jay Myer of BestBuy described how the BestBuy sales went up 30% when they added RDFa tags to their product pages. Although many search engines are not transparent about their use of RDFa tags in page rankings, Jay’s results should make it clear that this strategy works.

Jay told a great story about how hard it was to find a fridge that met his specific criteria: black with a specific size etc. using a keyword-based search engine. The semantic web will change all of this!

The big factor here is Martin Hepps GoodRelations ontology for products and services. This can really bring the Semantic Web to the masses. Finally a way to code the hours of operation for your store so that you can use Siri to ask "What Sushi Restrauants are Open at 10pm Near Here"

There were also a lot of sessions on LinkedData and specifically on LinkedData in the government area. Jim Heldler and his students at RPI are scraping every data set on data.gov and converting them all to RDF and storing them in a huge triple store.

Jim also provided one of the best sound-bites from the conference:

“Get AWAY from the Table”

This goes far beyond the NOSQL movement that we are seeing. It gets to the core of the problems with innovation in many organizations.

Jim is a consultant to the US data.gov transparency process and a key advocate of open linked data standards. It is interesting to see a friendly rivalry between Jim and Tim Berners-Lee who is a consultant for the UK data transparency movement. Both are using RDF to convert data but each is using slightly different strategic approaches.

He was referring to the fact that many organizations over-use the relational model and they need to understand that there are many alternatives, especially when doing mashups of data sets from many sources.

There were also many presentations on natural language processing and finding the true “meaning” of words within unstructured and semi-structured data sets.

I always attend Brian Sletton’s session on REST. Many people do not understand the relationship between REST, URI’s and the semantic web stack. Joe Wicentowski’s work at the US Department of State on the URL rewriting frameworks within eXist has really reinforced how this can be done easily in a single XQuery module.

I presented a 3.5 hour tutorial session on Entity Extraction. Since it was scheduled for the last session of the last day of the conference I though the attendance would be very low. But the room was packed and most of the people stayed till the very end.

It was wonderful to finally meet Marie Wallace and DJ McCloskey from the LanguageWare team at IBM. Their support of the Apache UIMA standards will be a great step forward to the creation of interoperable language analysis piplelines. Marie and DJ introduced me to the Millennium restaurant just a few blocks from the hotel. My wife Ann and I went there over the weekend and we now own one of their cookbooks.

By the way Google is now also support recipes in their rich snippets in their search results so you will soon be able to find "only recipes that take under 30 minutes" in a Google search engine.

There were also many session on the need for good taxonomies and ontologies in the enterprise. The use of SKOS for controlled vocabularies was a very hot topic and there were dozens of presentation on how OWL is being used to capture and exchange business rules.

It was also really great to finally meet Jeni Tennison after reading her books and following her Tweets for a long time. She is deeply involved in the federal data transparency projects in the UK. Here use of RDF is breaking new best practices for the entire community.

It was also great to catch up with Mark Birbeck, one of the chief architects of the Ubiquity XForms libraries. I am looking forward to trying out some of the new tools based on his backplane JavaScript libraries.

The people from Facebook also gave a presentation on how they are adding the ability for people to add a few lines to each web site to allow users to add a “like” button on a web site using a variation of RDF. Many people were a little disappointed to hear that they will not be using namespaces in their interfaces. Facebook felt that host HTML coders could only handle a single namespace. Many people agreed with their findings and thought that until 90% of HTML coders knew what namespaces were that organizations like Facebook were stuck just adding new data to HTML meta elements.

One of the most interesting discussions I had was with the people from Cray Computer. Apparently a large unnamed government agency has given Cray Research a very large contract to build customized ASIC (FPGA) chips to do graph analysis. Their ThreadStorm XMT architecture has 128 register sets and allows the CPUs to get continuous feeds of graph queries without waiting for memory. Their claim is that the federal agency has told them they are getting a 100X improvement in complex graph queries. The challenge is that the API is currently a low-level C interface and they have not yet put a SPARQL complier in front of an XML. So on good SPARQL benchmarks are yet available to compare the results of this hardware with a typical triple store. But even if it is fast, the price would be in the six digits for one of the XMP systems. So only very large organizations and government agencies might use this unless it was provided as a service.

One of the other things I found was how many people are using OWL and the OWL reasonerers like Pellet as replacements for traditional rules engines from companies like FairIsaac. There are two reasons for this. First is that many rules engine companies only talk about their engine performance but not on the need for precise semantics in the data elements. With OWL and the methods behind the semantic web stack we attempt to put semantics higher in the requirements of a system a use pre-built and pretested ontologies.

The second big reason that people are use OWL is that the rules are created using open standards. That means you are not locked into any one vendors rule format. The new W3C standards for rule interchange (RIF) will also start to break down many of the barriers to the creation and exchange of large industry vertical rule sets to that each organization does not have to start from scratch with a rule base. This will be big for industries like insurance where claims processing ontologies are just being created. Thanks to Kendal Clark for good information on this topic.

I also had a great time talking to many people in the publishing area. Seth Maislin gave a great presentation on Taxonomies. Seth and Marlene Rockmore joined me for a lunch at nearby Indian restaurant. Marlene is an expert on Taxonomy development. She was working on a book with O’Relly on taxonomies that is on hold for now but we hope to see one in the future from her.

It was also nice to see how many federal agencies are now starting to use SPARQL in the intelligence community. I saw a good presentation by Dennis Wisnosky with the US DOD talk about how they plan to use NIEM and semantic technologies to cut their integration costs. Yeah NIEM! Most people don’t understand that the NEIM is based on the RDF model, it just uses XML syntax to store the relationships. See my web-cast on the SemanticUniverse.com web site for more details. Hopefully his slides will be available in the future. Many people were taking photos of the slides since the DOD is not great at getting their slides out. Wonder why?

I also had lunch with several people from within the intelligence community that had great discussions about the pros and cost of pulling assertions out of NLP-analyzed annotated XML documents and the challenges relating to this. Keeping the links back to the original XML documents to is critical to verify the results of an assertion in context. Traceability, linage, provenance and time-domain representations in RDF that don’t cause a 10X growth is the number of triples is a difficult problem.

If there was some way to relate RDF assertion to a source document XML Node ID (the fourth column question) is a very difficult problem and one that needs close research with native XML and RDF systems. One of the things I learned about the eXist-Lucene integration done by Wolfgang Meier is that keeping the node-id as the document ID in Lucene allows non-programmers the ability to configure customized search rank rules past on the context of the keyword in a document. This is where highly customizable structured search rocks. I think this innovation needs to be added to RDF triples projects. Keeping the XML node-id of an assertion with an RDF triple context could be a great way to prevent RDF triple bloom for context.

Finally, I felt that David Wood and James Leigh's presentation on Callimacus project had the most potential to have a "big impact" on the adoption of Semantic Web tools. This new open source project allows users to specify a simple HTML template with special XML tags that allows them to bring triples directly into a web application. This has the potential to allow far more non-SPARQL programmers to integrate RDF data directly into a web page much like XQuery does today. With some work it might be possible to auto-generate the XForms bind elements for form rules. This would be a huge win for the integration of rules into web forms without needing to write JavaScript.

You can also see my tweets on the conference here: http://twitter.com/dmccreary

I hope to see more of you in San Francisco next year. Let me know if you are interested in co-presenting any papers!

Problems with WebDAV on Vista 64bit

Does anyone have any suggestions on a good reliable WebDAV product for Vista 64bit? I can not get the one that is bundled with Vista to work. Every time I try to create a remote connection I get the following error message:

"The folder you entered does not appear to be valid"

I have run the Vista WebDAV patches:

KB907306

I have also tried the following Hotfix:

Windows Vista WebDAV Hotfix

I have also tried to get various 3rd party packages working such as Independent DAV, BitKinex and WebDrive. All of the WebDAV products seem to have problems with the Windows Vista 64 bit systems. Is there any way to get Vista to use the Windows XP WebDAV "drivers"? I am using both eXist and MarkLogic native XML databases. Thanks - Dan

How to compile the eXist 1.3 build

I have been learning how to use higher-order functions in XQuery. I am using the eXist system to test this. To do this you will need to get the 1.3 build that there is no download for (yet). Here are the steps I used:

1. Download fresh copy of eXist from svn via TortoiseSVN or Eclipse subversion subclipse plugin from the eXist SVN sourceforge repository:

   https://exist.svn.sourceforge.net/svnroot/exist

2. (Confirm environmental variables are set correctly for JAVA_HOME and EXIST_HOME) To do this I use the Windows/Computer/Properties/Advanced Settings and check it with the SET command at the CMD prompt. The result looks like this:

   JAVA_HOME=C:\Program Files\Java\jdk1.6.0_10
   

   My Eclipse workspace is just "C:\ws" and I created a project called eXist-1.3 so the path name I used for EXIST_HOME is:

   EXIST_HOME=C:\ws\eXist-1.3dev
   

3. Open the DOS cmd prompt, cd to eXist directory
4. type in "build.bat"
5. Type in "build.bat -f build\scripts\jarsigner.xml"

It took just 1 minute and 6 seconds for the main build to run on my 4CPU system with 8GB of RAM even though it is running Vista. Then I just ran the start.bat in the bin directory. eXist was then running on http://localhost:8080/exist. I had to use the new WebStart admin tool on the admin section of the eXist web page to change the admin password. Thanks to Joe Wicentowski at the US Dept of State for helping out!

The Finch and the Raccoon

The blogs on xml.com are down so I have reposed this here. Have you ever wondered if the laws of evolution apply to computer languages? When you walk down the isle at your favorite bookstore, does it seam like there are actually more computer languages than last year? What forces are driving each of these new languages to evolve? In 1835 Charles Darwin visited the Galapagos Islands. There he collected what he thought were about a dozen distinct species of birds. Upon returning to England he discovered that each of these species had evolved from a single species of finches. On the various Galapagos Islands the requirements for food gathering was different, but consistent over hundreds of thousands of years. Enough time for a single species to adapt to meet consistent requirements. Consider the Raccoon: omnivores that have proved to be one of the most adaptable mammals on Earth. The Raccoon’s range has rapidly expanded into urban areas due to their ability to quickly adapt to new requirements before other animals have had time for the wheels of evolution to turn. So goes it with computer languages. Some procedural languages can be quickly adapted to fill in the needs for a new niche. When the web was young, procedural languages like Java and JavaScript quickly filled in the need for a variety of tasks. As the requirements for building web applications stabilized, declarative systems like CSS, XForms and XQuery started to push procedural languages back into niche-areas. As these declarative languages stabilize and become worldwide standards, graphical tools are being created to allow non-programmers to create, manipulate and extend these systems. This is why many of us believe their will always be some need for procedural programming, but certainly not for building standard web applications that are controlled by style sheets and user interaction forms. Like the finch, declarative languages need a little longer to evolve. It sometimes takes years for a small vocabulary of functional specification patterns to emerge and be given labels. Additionally, it can takes years for the standards bodies to agree on the best way to deliver these new languages in a set of semantically precise data elements that have unambiguous interpretations. Finally, it may take another few years for IT managers to realized that they really do lower costs if they avoid vendor-specific implementations and adopt worldwide standards. When CSS first came out you may have been a little reluctant to let web designers play with a rules engine. As XForms becomes ubiquitous you may be resisting change because you have invested so much time and energy learning how to debug JavaScript (without a debugger). You can not hold back the forces of evolution…and now we all need to adapt to the declarative world or risk our own extinction. If you are interested in more on this topic see my Presentation from the 2007 Semantic Technology Conference The Semantics of Declarative Systems

XRX: Simple, Elegant, Disruptive

I recently started writing for O’Reilly Media. I posted an article on XRX. Here is the link:
http://www.oreillynet.com/xml/blog/2008/05/xrx_a_simple_elegant_disruptiv_1.html
We are just getting started moving to a new MoveableType (MT) system so not all of the features (like keywords and feedback) are working.
If you have problems commenting on the site, please feel free to post your comments here.
Here is a comment from Arun Batchu:
An elegant introduction to an elegant architecture, Dan. Thank you. The essential takeaway from what you describe is the exploitation of XML from one end to the other end - especially from the Developer's perspective, for, how XRX actually manifests in runtime could be left to implementation technologies. Thus the logical architecture of XRX could be realized by a few variations of concrete technology - which is great. The XForms could be realized by XForms server technology (such as the excellent Orbeon stack), the ReST could be realized by any middle tier and the XQuery could be realized by an XQuery engine (such as Data Direct) that may actually be driving any one or a combination of datastores (XML, SQL, file system or ...) . The symmetrical and consistent leverage of XML as a data model from creation to transport to rest and back eliminates a whole lot of wasteful work. Like you point out, XPath is one of the most powerful query systems I have encountered; you can pack so much in so little and reuse it across the board with little if any change from the drawing board to production. In such a system as you describe, a business rule expressed once can be reused anywhere - from one end-to-end , however long the travel, as long as the architecture is XRX, like you have described. Thanks for expressing it so well. A few of your readers will not get it - it is one of those things that once you experience it, you are left wondering why this did not happen before. Oh, well!

You are welcome! Thank for your feedback Arun!

XForms, Dyslexia and the Right Brain

Those of you that have worked with me know that I am a little bit of an odd-duck. I have dyslexia. If you ever see me attempt to write on a whiteboard my spelling is at the ninth grade level. My left brain, used for phoneme recognition, never really developed like the rest of you. My right brain had to be co-opted to help out. But one of the skills I seemed to have picked up due to my over-exercised right brain is the ability to visualize multiple complex application architectures and quickly understand architectural tradeoffs. I am one of the few people that seem to be interested in discussing how XForms, metadata registries, ontologies, the semantic web, OWL, RDF, graphs, business rules, BPM and Kimball conformed dimensions all can work together to deliver elegant and cost-effective enterprise-scale solutions. It seems easy for me to simultaneously visualize two or more architectures and it constantly challenges my patience when I have to explain over and over why architecture alternatives will not meet a business requirement.

It turns out that many people that have dyslexia also have the gift of being able to visualize complex systems. Albert Einstein, Thomas Edison, Jackie Steward and Charles Schwab are good examples of dyslexic people that have used the strengths of the right brain to do things that left-brain thinkers could not.

As a right-brain centric person I need to also tell you that I really love the XForms architecture. The magic of a declarative language, MVC, bindings and a dependency graph makes XForms development 10 years more advanced than anything else I have worked with. I think it is beautiful and elegant. It is everything that AJAX and JavaScript application are not. Clean, simple and easy to visualize (for me at least). When someone asks me if I can create an XForms application to do something, I create a mental image in my mind of the model, the view and how events will update instance data in the model using inserts or external submission results. I can easily visualize the bindings of view controls to data elements in the model. Once I can visualize the application clearly, writing the application is just a matter of typing in the code.

I think that my right-brain is also a reason I detest JavaScript and AJAX. It is far too much code to read and trying to visualize how 300 lines of JavaScript enables me to do a drag-and-drop. I want to just add an attribute to an element like "drag-source" and "drop-target" and I want it to just work.

What triggered this posting is that I have been reading Proust and the Squid by Maryanne Wolf. This is a book about how the brain's circuits are used in the reading process. She has a wonderful explanation of how the dyslexic brain co-opts the right brain for reading and enhances it functionality. I didn't really understand the relationship between my defects and my gifts .

So how about you and your development team? Do you have a dyslexic right-brained person on your team? Can they quickly visualize architectural tradeoffs? Have they tried XForms? And if they do, will you be willing to tolerate their disgust of AJAX and JavaScript after they have built their first XForms applications?

For more information about dyslexia check out these two Wikipedia entries: http://en.wikipedia.org/wiki/List_of_notable_people_diagnosed_with_dyslexia http://en.wikipedia.org/wiki/Dyslexia

Great Example of Multi-dimensional Bubble Chart

Here is a beautiful example of a bubble chart display using global population statistics. The example on carbon emissions are very interesting. Note the dimensions

1. Population of County (size of bubble)
2. Continent (color of bubble)
3. Live Expectancy (vertical axis)
4. Income (horizontal)
5. Time (the play button)

It is interesting to see the huge impact AIDS has had on the life expectancy in African counties. Imagine if you could see your organizations product sales using this type of graph. This application was done with a software system called Trendalyzer. It was initially developed by Hans Rosling's Gapminder Foundation in Sweden and acquired by Google Inc. in March 2007. This version is a Flash application. Does anyone know of any open-source software that could do this?

Metadata Repositories vs. Metadata Registries

For several years people have been using the terms metadata Registry and Repository inconstantly, imprecisely and almost interchangeably and I would like to weigh in as to how these terms could be used more precisely to allow organizations to effectively to manage metadata processes.

First lets take the definition of a Repository. Webster defines a repository as …a place, room, or container where something is deposited or stored.. Note that here is nothing in this definition about the quality of the things being stored or the process to check to see if new incoming items are duplicates of things already in the repository. If I have 100 users they could each define "Customer" as the see fit and put their own definition into the metadata repository as their own definition. No problems.

On the other had lets take the word Registry. A Registry has the connotation of more than just a shared dumping ground. Registries have the additional capability to create workflow processes to check that new metadata is not a duplicate (for a given namespace). One of the definitions from Webster is an official record book. Note the word official.

A Repository is similar to a front-porch of a house. No locks prevent new things from landing there. But a Registry is a protected back room where human-centric workflow processes are used ensure that metadata items are non-duplicates, precise, consistent, concise, distinct, approved and unencumbered with business rules that prevent reuse across an enterprise. These registries have become the central foundation that agility can be baked-in to many enterprise process. The latest version of the Kimball's Data Warehouse Lifecycle Toolkit (which is actually a very good read) even goes as far as to call their process "metadata-driven". Not different the model-driven development world.

Registries have the implicit connotation of trust behind them. They now serve a a central process for the creation of shared meaning across the enterprise. Definitions in a registry have been vetted by an enterprise-level organization that has the responsibility of enterprise data stewardship. They have a high probability of being consistent with industry best-practices and vertical industry standards. Registries are the go-to source for creating canonical XML schemas, enterprise ontologies or conformed dimensions in a OLAP cube. Repositories are personal or small departmental definitions of an isolated view of the world.

None of these ideas are really new. They are at the core of the ISO/IEC 11179 metadata registry standard. Note that they don't call it a repository standard! People are just now starting to understand how important Registries are in most enterprise-wide systems. The growth of Business Intelligence and Enterprise Data Warehouse terminology and Service Oriented Architectures is a good place to see the rise of repositories and registries. We now see service registries, portlet registries, model registries...the list goes on-and-on.

Much of the background on the differences between the use of repositories and registries can be traced way back to the early days of object-oriented systems in the 1995 book Succeeding with Objects by Adele Goldberg and Kenneth Rubin. This was one of the first books on enterprise reuse strategies and they defined the concept enterprise asset reuse and the need for a trust-driven repository as a basis for reusing assets. They identified a multi-step process for reviewing new submissions to determine if the submission duplicated existing assets. They showed how critical it was to classify items in a registry and search an existing registry for duplicates before new items are added. If you can get a copy of the book I would suggest you read the section on "Set Up a Process for Maintaining Reusable Assets" on page 245.

The book then goes on to show how organizations can and should be structured to reuse these assets and gives the pros and cons of the differing organization structures and their impact on reuse. This is the basis for the data governance and data stewardship movement in many organizations today.

So the next time someone uses the word registry or repository in a conversation, ask them if they are using the definition of the word that is consistent with the corporate business term registry or is their own private definition from their own repository of imprecisely used buzzwords.

XForms Tutorial and Cookbook Voted Featured Wikibook!

I am happy to announce that the XForms Tutorial and Cookbook that I have been working on for over a year been voted a "Featured Book" on the wikibooks web site. A quote from the award:

XForms is a featured book on Wikibooks because it contains substantial content, it is well-formatted, and the Wikibooks community has decided to feature it on the main page or in other places. Please continue to improve it and thanks for the great work so far!

Here are some stats on the book so far: 90 sample programs
116 chapters
626 edits
28,811 words
29 registered authors

One of the big tasks is to start to move some of the advanced XForms examples that require siginficant server-side logic to a seperate server-specific book. As a pilot I have started an XRX cookbook for people using REST interfaces and the eXist server.

I would like to thank the over 30 others that have contributed ideas and content to this wikibook. We still have lots of work to do to cleanup the example programs, make them more consistent and add additional examples for new XForms students. But I believe it is one of the best examples of collaborative training that I have worked on in the last few years.

XRX and MDA

Summary

This posting describes how using three technologies (XForms, REST and XQuery or XRX) you can dramatically transform your organizations development methodology. You can move true model-driven architecture (MDA) that dramatically reduces the temptation to duplicate code.

In January of 2007 I was working on a complex real-estate forms project for a state agency in Minnesota. Our job was to get 87 counties across the state of Minnesota to agree on the over 250 data elements that describe real estate transactions. We had spend months developing a complex XML Schema and web forms with over 50 one-to-many relationships. And out next task was to be able to save the form

data into a structure that could be quickly queried by any of these 250 data elements. But the prospect of "shredding" the documents up into 50 distinct SQL INSERT statements and then reconstituting the documents with 50 distinct SQL SELECT statements was going to require doubling our small team of four to at least eight developers. We didn't have the budget or schedule to do this. After chatting with Kurt Cagle he suggested saving the data to a database that supported XQuery. Both the open source eXist native XML database and our corporate standard, DB2 supported XQuery. It turned out that we could indeed save the entire document with a single command and still perform complex queries on any element in any document. Our project could proceed with a small team and stay on schedule.

And then an interesting thing happened. Our team started to understand that if we saved additional metadata in these native XML data stores we could accelerate our project even further. Every pick-list in every form could dynamically call REST-enabled web services to get their values. XML Schemas could be stored in the databases and be queried. We had found a simple and elegant solution to the pervasive problem of where to store the model in a model-driven development project.

In the past I had been on large teams of developers that attempted to use model-driven development. But although we started out each project with idealistic goals, each time we were faces with a classic option: copy, paste and edit the model or write a transform. The copy and paste solution was quick but it would have to be redone each time the model changes. The transform took longer to write but then could be rerun each time the model changed.

Like most developers, I always was over-optimistic that I had done my homework and done accurate and detailed modeling of the problem. I thought the models were stable and would not change very often. And I was almost always wrong. Models do change and sometimes in totally unexpected ways.

But by storing our models in a native XML database an interesting started to happen. We could quickly transform the models (stored in XML) into other artifacts with simple XQueries. It became far less tempting to commit the sins of copy/paste/edit when it only took five minutes to create another transform. And most remarkably, each of these transforms was another REST enabled web service that could be reused by many web clients or other development tools.

So how do you get started? You might try the following steps.

1. Download the exist database from http://www.exist-db.org
2. Use a WebDAV interface and copy some XML Schema documents into a collection
3. Write a few small XQuerys that pull metadata out of your XML Schemas. For example pull out all the enumeration values out of an element.
4. Use those XQueries to drive some aspects of your development such as an XForms selection list.
5. Compare the amount of code you wrote with any other MDA system. If you don't have at least a 10 to 1 savings I would be very surprised.

What do you think? Do you have a more efficient way to query your model? Let me know!

- Dan

And the Winner is: eXist-db.org My Award for the Most Innovative Product of 2007

As the year starts to come to a close we often take some time to look back at the year and look at the innovations that we have seen that have changed our worldview. I can think of many things that have had a large impact on me: The FireFox XForms extension, discovering the beauty of a well-designed REST interface, the mule enterprise service bus, Yahoo pipes and microformats all come to mind.

But after careful consideration I have to hand my award out to the most innovative application of the year to me to the eXist database/web server. This application constantly amazes me with what I can do with it. If you have a Java JVM on your desktop it only take a few minutes to get it running. Hit the startup script and you have a web server running on localhost:8080. Open a WebDAV browser and drag-and-drop you files and folders and away you go. With eXist I can easily setup a full enterprise metadata registry on a laptop in under five minutes.

The power of eXist comes from its use of the XQuery engine. Any well-formed XML file that is added to the system gets instantly indexed and can instantly be searched. That means that 10ms after you hit the "Save" on your XForms you can see the data appear in XQuery reports.

But the "Secret Sauce" of eXist rests in it's use of a lightweight Jetty web server with a remarkably integrated REST and WebDAV interface. This means that every XML file…is a static web service. This means that every XQuery that returns XML…is a dynamic web service that you can parameterize. This means that you can be writing your first web services in fifteen minutes that can be grabbing data from a dozen different XML files and quickly serializing the results out the wire. Enterprise mash-ups at your fingertips. Very little programming is done other then selecting data. Just like it should be.

eXist technologies are going to start to have a big impact once XForms/REST and XQuery (XRX) web development matures. I predict this is going to happen in the next three years. An although IBM, Microsoft, Oracle and many others are supporting XQuery in a big way, they don't yet have the really smooth integration with next-generation XForms-driven clients. You still need teams of AJAX, JavaScript, Java, .Net and SQL programmers to build simple web services for rich client interfaces.

I must admit it took me a little while to really understand how the simplicity of the interface really rocked my world. Just swap out the world "rest" in the URL and replace it with the word "webdav" and you go from a data browser to a file system that works with every copy/cut and paste operation of the file system.

It is also interesting to note that this product did not come out of a Bay Areas startup, it didn't come out of Google Labs, it didn't come from Microsoft or IBM. From what I can tell the initial version was pretty much written by a single guy in Germany, Wolfgang Meier. It think this shows that there is still room for true innovation by a single individual in this world. Although eXist now appears to have a great team of people behind it, it shows that a single person with a clear architectural can put the right pieces together really can make a difference in the world. Our teams could not have been as productive with XForms if we didn't have a clean and elegant tool like eXist behind our forms. I know that XRX innovators will no be able to stand on Wolfgang's shoulders and build incredible applications.

Thanks Wolfgang! Your labor of love is growing up!

Introducing the XRX Architecture: XForms/REST/XQuery

At the XML 2007 conference there were many people that seemed to have independently discovered that if you combine XForms/REST and XQuery you can create a software development environment that circumvents the need for middle tier objects and conversion to and from relational databases.

The conference also had a great deal of discussion of how XForms is a great architecture but as Elliot Rusty Harold pointed out, we need something to happen for XForms to replace things like AJAX. Although Rusty did propose a few interesting ideas (like getting XForms built into FireFox) I feel what we really need is a contagious meme. And I think it goes beyond XForms. What we need is an easy-to-remember and easy-to-communicate name for a collection of great ideas. For example there are few books that have the word XMLHttpRequest in it. But there are hundreds of books with AJAX in the title. Once the word AJAX was coined the meme started to spread. The world was waiting for the "sticky" label behind the concepts.

What XForms needs is a label around the collection of ideas that makes XForms really give you an order-of-magnitude improvement over other web development architectures. I would like to suggest that we get behind one "label" that brings many of use under one big tent. I would like to suggest we use "XRX".

XRX to me would stand for the ideas behind XForms/REST/XQuery web applications. But if people are using XHTML as a container for XForms you could use XHTML/REST/XQuery. Just spare use the "J" which is associated with a hard-to-use language for über geeks.

You could also use XRX for the format of the data in the client and on the server. In this case it would be XML on the client (in the XForms Model), and in a native XML database that supports a REST interface.

We can still build awesome tools and frameworks that make the XRX architecture really fly. IBM Workplace Forms Development tools, Orbeon Presentation Server, Kurt Cagle's new frameworks that use the eXist native XML database, and the hundreds of other XForms development tools all fit into the XRX strategy.

But if we make XRX into a "big tent" strategy we can also include XQuery on the server as part of the architecture. This would include people like Jason Hunter at MarkLogic. Jason is an experienced Java developer that has started to move away from Java on the server and toward XQuery because of its richness. And I could not agree more.

I have been reading the book The Tipping Point: How Little Things Can Make a Big Difference by Malcolm Gladwell and trying to apply the ideas in it to the adoption of XForms and XQuery. I think that the book has many example of how a careful packaging of the ideas behind a contagious meme can make ideas spread like widefire.

So when we hear the term "XRX" we want to have the following concepts come to mind:

1. A web development architecture with a 10x productivity improvement over traditional JavaScript/OO/RDBMS methods
2. A development architecture based on international standards that is designed to minimize the probability of vendor-lockin
3. An architecture that gives a rich user experience without creating mountains of spaghetti procedural code on either the client or the server
4. A system that leverages the REST architecture to take advantage of high-performance and simple interfaces using web standards
5. Portability on both the client and the server using a variety of forms players and XQuery databases
6. The option of avoiding costly shredding (and reconstitution) of complex XML documents into RDBMS tables
7. A community of standards/tools and a "complete solution" ecosystem that can give you a proven ROI on your IT investment

I don't expect everyone to buy into the XRX meme ideas. Everyone has their own favorite tools/standards and acronyms. But everyone that was at the XML 2007 conference could feel the excitement in the room at the XML presentations. Now we need to give that excitement a name and see if it sticks. Hopefully a few years we will be able to walk into a bookstore and see a bunch of books with "XRX" on the cover.

What I now need is for people to start to test the XRX meme and see if it slicks. Let me know what works and what does work. Send me your 30-second elevator pitch. How big should the idea tent be? What do we have to do to get more people in our tent? What evidence (case studies), example programs and resource do we need to convince the world we have seen the light …and its so elegant and beauty to will keep us from ever going back to the stone age of web development.

Dynamic Graphs for K-12 Educators

At the recent XML 2007 conference in Boston several of us met to discuss what we could do to promote FireFox and some of the important standards built into FireFox (or almost built into FireFox). Specifically several of us were using XForms and SVG and thought that the combination offered some great opportunities.

It was pointed out that the combination of the XForms range control (a slider control) and SVG allowed developers to create dynamic graphs for teaching educational concepts. There is an example here:

http://en.wikibooks.org/wiki/XForms/Pie_Chart

Note there is a link to the actual program here which you will need to use the FireFox XForms extension to view.:

http://xforms-examples.googlecode.com/svn/trunk/10-full-examples/86-piechart-range-controls/pie-chart.xhtml

So several of us are interested in putting together a grant to develop SmartBoard enabled educational content for the K-12 market using a combination of FireFox, XForms and SVG. The goal would be to create a "Killer App" for FireFox in the K-12 space. Ideally we will build 100 demos of some of the most useful dynamic graphs and then build a framework for teachers to extend these applications.

I also have a few Supply and Demand examples that run under IE but require the SVG add-on and just just plain JavaScript for the range-controls: http://www.danmccreary.com/svg/ You will note that visualizing the maximum profit by the area of the square under the triangle is much easier when you add the motion.

It was also pointed out that the Mozilla foundation has the funds to support this type of development and that they might fund a grant to build the top 100 useful dynamic graphs. We could then start building a toolkit to extend these dynamic graphs using some of the advanced drag-and-drop connection-based programming parts of SVG and FireFox. The kit should shield the average middle school physics teacher from haveing to write and debug JavaScript.

Let me know what you think - Dan

Impressions of XML 2007 in Boston

I returned from the XML 2007 conference in Boston with my head spinning about all the new developments with XForms and XQuery. The conference was three days long and had between three and five tracks going concurrently. My impression was that there were about 400 people there.

Here are some of the highlights for me.

XForms appears to be getting lots of traction. The XForms sessions were all packed with standing-room only in most of them. The presenters we passionate that the combination of XForms, REST and XQuery has had a huge impact on their projects and may soon impact the rest of the development world.

Microformats are definitely starting to make huge strides where Metcalf's Law will start to kick in. The existence of 450 million tags and five FireFox microformat extensions indicate that within a year it will be just part of web publishing best practices. Even Microsoft is said to be supporting Microformats in IE 8, just a few years from now.

Taylor Cowan (who works for Sabre) gave an excellent presentation on how Microformats can be extended into the travel space with an Atom based format for publishing trip ideas. His use of Atom and semantic web technologies was right in line with my vision of how personalized agents are going to leverage microformats.

John Boyer's demo of IBM's Workplace Manager showed that you can now purchase top-shelf XForms GUI design tools that does not lock you into a proprietary vendor solution. I hope the people from Microsoft, Adobe, FairIsaac, Altova, Google GWT and other vendors take notice. Locking your users into a Java or JavaScript client is no longer an acceptable strategic option for developing web applications. Wake up everyone! Are you listening? We have a standard that works and lets now start building more great tools that use this standard. XForms is the only system that is creating as John puts it "Order of Magnitude" increases in developer productivity.

Mark Birbeck's Sidewinder demo showed that XForms is breaking out of the browser an on to the desktop. Showing web and iPhone web pages directly in desktop widgets was a great demonstration of the power of XForms.

Kurt Cagle's presentation about problems with programming with the fragile DOM model in JavaScript was interesting in its format. The presentation was actually an XForms application running on eXist. Kurt "eats his own dog food" on the client and the server! I am humbled by the man.

Jason Hunter from MarkLogic (an XML database vendor) gave an nice demo of the MarkMail.org application and showed that, yes, XQuery databases really do scale to the terabyte range. He has about 4 million e-mail messages indexed and you can do combinations of database and text searches in a few seconds. In other words a demonstration of how real-world analytics comes to XQuery. I also found out that MarkLogic is free if your database is under 100MB. Perfect for 10MB of metadata in a metadata registry.

The people from IBM research labs appear to be starting to take the XForms MVC architecture and really run with it. We saw some very interesting presentations on how future standards for data driven XML Application Components (XACs) and State Chart XML (SCXML) might be used in the future to make browser-based mash-ups easier. Componentization, composition, customization and reuse are all part of this architecture and it would not be possible without XForms MVC architecture. Hopefully Charles Weicha from IBM research will post his slides so you can see more.

Norm Walsh also talked about how the new XML Pipeline standard (XProc) is starting to take shape. Even though the standard is still in working draft there are already at least five implementations of the standard. This single standard could have more impact on the shape of server-side processing then any other standard coming out of the w3c. Integrating this into XQuery-based application servers seems to be the next step.

The eXist database also seemed to clean up all the open source native XML database talks. eXist was mentioned in over a half-dozen presentations and there were no other native XML databases even mentioned in the presentations I attended. It is not surprising that eXist is now the top hit on Google when you search for "XML database".

One of the most interesting presentations that I did not get a chance to see (but I did talk to the presenter afterwards) was Thomas White's presentation on using XSLT in the browser to get a 10x speedup in form rendering. It turns out that implementations of XForms players like FromFaces use heavyweight JavaScript objects in the browser. Thomas has found that a donkey cart can be turned into a sports car by implementing an in-browser XSLT centric event scheduler. If his architecture is implemented we could see vastly better XForms players in the future. Lets all hope that Thomas's architecture comes to the XForms world sooner than later.

I did not attend any of the XML publishing tracks. But from the titles it appears that the Darwin Information Typing Architecture (DITA) standard is a hot topic. Presentations on using XSL-FO and CSS for book and other print publishing were also on the agenda.

I also did not attend too many of the XML in the Enterprise tracks. There were good presentations on XML accelerators, security and mining XML Schemas.

I am disappointed that the conference organizers did not do anything to require that people post their presentation on the web site before their talk. There were many very interesting presentations that could have a larger impact the world and the sloppiness of people not publishing their slides seems to be a little unprofessional. There were some people that just brought their digital cameras and took pictures of each slide. Smart.

In summary I think that the conference had two types of people:

First there were people that had not done extensive work with XForms. The first group I call the "evolutionary proceduralists". Theses were people that were trying to beat the dead horse by patching the old-style procedural web application architecture with JavaScript, JSON and AJAX hacks. They seemed to spend a lot of time arguing about the merits of JSON vs. XML as a data transport syntax.

The other type were "the enlightened innovators". People that have seen the real world benefits of dumping object middle-tier stacks and relational databases and going with a pure declarative approach to solving business problems based around XForms/REST and XQuery. These were the people that are going to lead innovations in application development.

I was also impressed how a large percentage of XML innovations came from outside the US. I attribute the fact that some of the best presentations came from the UK to be the "Michael Kay" effect.

Beauty is in the Eye of the Beholder At one of the XForms sessions a young Ruby advocate stated that he thought his Ruby code was "beautiful" but he did not himself think that XForms code was "beautiful". Most of the people in the audience agreed that XForms was beautiful but like any new language, it takes getting used to. Beauty is the process of your visual cortex recognizing familiar patterns and signaling your pleasure centers. If you have never read Japanese then the most profound Haiku written in the most beautiful script will not have beauty.

XForms/REST/XQuery to me is indeed beautiful. Its beauty comes from its ability to quickly map real-world requirements into working systems with high fidelity. No other system in the world approaches its elegant architecture. And after XML 2007 I realized I was not the only one that had independently reached this conclusion.

Full text Search Standards for XQuery

I have been using the eXist (http:www.exist-db.org) native XML database for almost a year now and I am constantly impressed by the power and depth of the eXist community. XQuery is a great way to search XML documents and most of my XSLTs that manage metadata are now being transferred to XQuery. Although there are still some places that XSLTs are superior, XQuery is much easier to use when I am building RESTful web services for metadata. My XForms are frequently generated directly from XQuery programs.

It is interesting to note that eXist is now dominating all the Google search results for "XML database". Leading the pack and pulling away!

From my days doing IT strategy I recall a great deal of analysis done by Adele Goldberg and Kenneth S. Rubin and documented in their excellent book Succeeding With Objects: Decision Frameworks for Project Management. This is an excellent book based on over 50 case studies on reuse and top-notch analysis of the role of trust in any high-reuse culture.

One of the key points in the Goldberg/Rubin analysis was the factors that cause people to be able to reuse assets. In their case the assists were programming objects. I my case I am concerned with the reuse of data objects. From simple data types that are part of the XML Schema standards to registered data elements in a metadata registry. Registered data elements have an approval process and have semantics that span a group, project, program or enterprise.

But the pattern that reoccurs is that people need to be able to quickly find assets before they can reuse them. One of my main points in my work is that you can not reuse what you can not find. The corollary is that the longer it takes to find an asset the higher the temptation is to recreate the asset. That is where the search functions of a metadata registry needs to come in. It must be fast and accurate.

The quality of a metadata registry also depends on strategies about informing users when two data elements have the same semantics and are potentially duplicates. To find this we need semantic nearness searches for data element definitions.

Today I am providing simple substring searches. But I know that text mining can provide semantic nearness searches. I was glad to see that the world-wide web is working on full-text search standards for XQuery. The w3c http://www.w3.org/TR/xquery-full-text/standard has now reached the working draft last-call stage and I hope that we soon start to see implementations that take advantage of these features. I know that Google's support of this project for eXist (see http://exist.sourceforge.net/soc-ft.html) is a very good sign.

I was also fortunate to work with Gary Berosik on the Advocate Agents project. Gary works for Thomson/West Publishing that has a large group of people doing text mining. Gary introduced my to the Text Mining Application Programming book by Manu Konchandy. This book has excellent sections on the decomposition of free-form text into parts of speech for information extraction, clustering, categorization and search.

I hope that eventually the w3c will approve the full text extensions to XQuery and that they will make it into the eXist system. Based on this link http://exist.sourceforge.net/soc-ft.html I am hoping that it will be in the next few years.