When XML Schemas Imports Become XQueries

For the last five years I have been using a XML Schema design process that I learned from the people at Georgia Tech Research Institute as part of my work with GJXDM and the NIEM. This work has strongly been influenced by Jim Heldler who co-wrote the Semantic Web article with Tim Berners-Lee. Although GTRI wanted to use RDF (graphs) to represent the exchange data they had to settle on using XML and XML Schemas since those technologies were widely used by most criminal justice agencies already. They compromised by still using a striped model within the XML Schema driven structures.

Creating an exchange documents starts with creating a metadata registry of all the data elements used in a family of XML Schemas. A shopping-cart-like tool is used to select a subset of the data elements. These data elements form a "wantlist" of data elements. The checkout process creates a zip file that when uncompressed creates a set of XML Schema file that you import into your constraint schema.

This process is an order-of-magnitude to use than before the subschema generation tools were created by GTRI. The subschema generated by the checkout process not only included the XML Schema type definitions but also all the data types that they depended on. The metadata registry had an internal dependency list that it maintained and so if you needed just ten data elements but they depended on then other data types you got all 20. you didn't have to manually figure out what additional data types to include.

I adopted the GTRI shopping-cart process to include adding a "subscribers list" to each of my data elements used in local metadata registries and local XML Schema creation. I used a large, complex and brittle ant script to update all the imported files for each XML Schema we were developing. But this could not easily be done during a requirements gathering meeting.

In the past we have viewed this subschema generation as a process of interacting with a web application that creates a set of files. One of our first insights is that this process is in fact just a transformation of the metadata registry based on the rules created by a wantlist and the dependency graph. It does not have to be a manual "batch" process.

In a recent blog (Metadata Web Services) I pointed out that tools like eXist and XForms allow us to store metadata in XML format and allow it to be easily updated by BAs and SMEs and queried using languages like XQuery. One of the great realizations I had on my last project was that XQuery has a simple construct that allows you to "chain" web services. The argument to the doc() function is just a URL. If that URL points to a web service that returns XML you can easily build composite web services. Web services that are transformations can be chained together. This allows you to start using enterprise integration patterns to build reusable services at a very fine grain.

Putting these facts together it has occurred to me that the ultimate goal of the metadata registry process is to allows new data element definitions and enumerated values to be changed during a XML Schema design process. We should be able to change a definition, hit the "save" on the XForms application that edits data elements and then just refresh the generated subschema. The refresh process must call a XQuery that calls web services that analyzes the dependency graph and updates new data elements imported into the XML Schema.

What this does is helps us get to the goal of updating the a model quickly and generating new artifacts directly from the model. This is part of the overall model-driven development process. The artifacts we create include XML Schemas, instance documents generated from the XML Schemas and XForms applications to view and edit the documents. This faster turn-around time allows our users to quickly see if their definitions and enumerated values are being precisely captured and used to create actual systems.

XForms Tutorial and Cookbook Being Translated to French and Used in China

I got a nice notes from people both in France and China that our XForms Tutorial and Cookbook are now being used in both France and China.

The French version is being hosted on a version of the MediaWiki server that also has a new macro for code syntax highlighting.

The URL for that version is here: http://xforms.free-web-hosting.biz/mediawiki-1.9.3/index.php/Wikilivre I have not figured out if this will work in the Wikibooks yet but I have my doubts.

Hints of the Chinese version were posted by a zhouliyi from ZheJiang China. These were posted in the Cookbook Guest Registry: http://en.wikibooks.org/wiki/XForms/Guest_Registry

Please let me know if anyone else is using XForms so we can maintain a list of users.

Metadata Web Services

I have been using the eXist native XML database/web server for about nine months now and it is starting to change the way I think about metadata management.

My latest project for a financial institution requires us to quickly build XForms to manage various metadata as well as data. What I am finding is that my old method of storing metadata in XML files on a file system and then transforming the metadata using Apache Ant was a complex process. My new approach is to store the XML directly in eXist. This used to be a little bit hard since I thought that you had to use the eXist web interface to upload each XML file and ant scripts to backup your eXist database.

This all changed when I was shown how to use the Microsoft Windows WebDAV tool. Copying files to eXist and backing up the entire data store is just a drag and drop using Windows.

Now an entire new set of metadata web services are becoming much easier to build. Take the simple task of building a pick list of enumerate values for a form. XForms allows you to use the select1 control and specify an itemset using an XPath expression. I can now just load the data elements into an instance and grab the values and labels directly from the enumerations in the metadata registry files.

The only drawback to this is the fact that you load more metadata (like the full definitions) then you need in building the form. But once again eXist comes to the rescue. It is just a few lines of code to create a little web service (using XQuery) that you pass a code table to that returns just the label/value pairs. Using this method the selection list are always up to date and don't require any "batch" updates.

What I am learning from this is that in the past, metadata management was usually an after thought. Something that the coding standards people used to enforce database column naming conventions. But with metadata being stored in eXist metadata becomes part of application services. Building apps is just assembling forms that pull metadata from the registry in real time.

If you are concerned that the metadata registry server will be overloaded with requests for information each time forms load, we should remember that these services are RESTful. The results can also be cached so they don't have to be regenerated. I still have more to learn about how to make these services fast but since metadata is small it can usually always exist in RAM and disk I/O is very limited.

All of these developments are just small pieces of the puzzle at putting well-managed metadata at the core of your enterprise development methodologies. It is really the heart of the model-driven enterprise.

Let me know if you are creating metadata web services. I would like to know what things you feel are useful to your users.

- Dan

Impressions of Sem-Tech -07

Impressions of Sem-Tech -07

I just returned from the 2007 Semantic Technology Conference in San Jose California. It was a great conference and opened my mind to several new ideas. Well worth the time!

The conference was held over four days and had over around 125 presentations including tutorials and research projects. There were almost 800 attendees. This is the third semantic technology conference that I attended and the second that I presented a paper at.

Here are some high-level observations and some patterns that I detected.

The Semantic Web gets the “Web 3.0” Label

Most people at the conference have tried to embrace the idea that the semantic web will be adopting the popular culture label “Web 3.0”. The final straw was the Nov 2006 NYT article by John Markoff which set the blogosphere is a buzz. This was a web that includes technologies to enable intelligent reuse of data. Wikipedia, after a long pro-active discussion about if “Web 3.0” deserved an entry, finally undeleted the page and let is stand. See http://en.wikipedia.org/wiki/Web_3.0 and check out the discussion page on the article for more.

From If to When to How

Eric Miller (now at Zepheira) calls some of the new technologies “recombinant data. Eric spent about five years with the w3c. He is perhaps the most well-connected person in the world with accurate knowledge of who is using semantic web technologies to solve real business problems today. His observation was that three years ago at the first semantic technology we were wondering if the semantic web would take off. Last year the speculation was when the semantic web technologies would start to become common place. This year the focus was debates on how the semantic web should be implemented.

Venture Capitalists are Becoming Educated on the Semantic Web

One example of this is the fact that last year, most semantic-web startup companies had to carefully explain what the semantic web was to venture capital companies when trying to get their initial rounds of funding. This year, many venture capital companies not-only had some level of understanding of the semantic web but were asking each of their potential startups how their technologies fit into the semantic web. Other companies that did not have a semantic-web focus were not coming to the conference to get educated.

Some of the companies that got VC last year have already been purchased and absorbed by larger firms. These companies were replaced by new venture-funded companies.

Consensus on the RDF/SPARQL Foundation

One of the first things that struck me was the consistent use of RDF and triple-stored to solve many hard problems. The use of RDF and SPARQL seemed to be the primary distinguishing factor about if people thought you were really using semantic web technologies or not. If you we not using RDF, you were not really in the club, just and outsider looking in.

OWL Fragmentation

Another thing that also surprised me was the discord about the use of OWL and its relative sub-functions. Central to this was the rise of use-cases of simple things examples of things that OWL could not do. Much of this centered around DLP (Data Log Programming).

A good example of a non-OWL solution was the use of SKOS to store things typically stored in a metadata registry. SKOS is a great example of a simple standard, built on top of RDF that attempts to solve common problems without getting overly complex. See SKOS in Wikipedia.

We are also starting to see that rules must also be exchanged between systems in semantically precise ways. The need for a Rule Interchange Format (RIF). Nice to see vendors like FairIssac supporting complex business rules running INSIDE the web browser using XForms. They rock!

REST is Deep

Perhaps my favorite presentation was given by David Wood and Brian Sletten from Zepheria. In this presentation, David and Brian gave a demo of the NetKernel system. They demonstrated how NetKernel embraces REST at a much deeper level then I had previously anticipated. Now they were not yet generating XForms from an XML Schema but it showed a great example of convergent evolution of my ideas and theirs.

Case Studies

This year also started to show examples of new startup companies actually using semantic web technologies to differentiate themselves in the marketplace. Although because they are all trying to differentiate themselves, much of the actual technologies they were using was not disclosed.

RDF Taggers, Harvesters, Linkers and Analyzers

The conference seemed to have three sets of problems that everyone agreed on. First was how do you harvest RDF from a web page or any other resource. Most of these presentations related to getting RDF out of un-structured and structured data. Lots of discussion of microformats (pros and cons).

iReader Rocks

The coolest demo I saw was the iReader demo from http://www.syntactica.com. This is an awesome FireFox (and IE) extension that does concept mapping from unstructured text. The people behind this have been doing research on linguistics for about 40 years and have only recently got a round of venture capital to start to publicize this tool. But they are not yet converting to RDF for storage other systems.

URL Design Patterns

One common themes that came up was the need for best practices about good URL design. Everyone said a few things: The design of URLs is very important, most people screw it up the first time and then have to redo the designs, there are not a lot of good documents out there and the ones that are available are at least five years old. The people from the w3c did take notes on this.

Reception of My Paper

My presentation was titled “The Semantics of Declarative Systems”. This talk covered how by using a set of small languages with precise semantics, you can build entire applications that allow non-programmers to draw pictures of their business requirements and generate working Apps. The purpose of this talk was really to test the metaphors on how I could explain these concepts to non-computer scientists. I got some positive feedback and was happy to see convergent evolution of our designs with other organizations.

The URL to the paper is here:

http://www.danmccreary.com/presentations/sem-web-07

The Semantics of Declarative Systems

I put my presentation that I did for the 2007 Semantic Technology Conference on my web site. The link is here:

http://www.danmccreary.com/presentations/sem-web-07

Please give me feedback on both the PowerPoint presentation (note the builds and notes pages) as well as the paper.

I am specifically interested if the metaphors worked for a non-technical audience.

- Dan

Declarative Systems and the Cambrian Explosion

I have been reading Carl Zimmer's book Evolution, The Triumph of an Idea and I have been startled about the similarities between the the Cambrian Explosion and the growth of declarative languages.

Just to recap (if you have not read the Wikipedia article yet) here is a quick summary. For the first 3.5 billions years, life on earth was mostly single celled animals. Then about 550 million years ago, the number of new multicellular complex-body-plan animals exploded. There are two salient reasons. First, we needed oxygen to breath. Before there was oxygen in the atmosphere respiration was not easy. Oxygen was a foundation for complex body plans on land. Second, life needed a genetic "tool kit" to build complex animal shapes. We needed the HOX genes for building complex body plans. After that happened, complex life forms did explode.

I am seeing the same thing appear in declarative languages. Declarative languages need foundations also. We needed ways for people to quickly create new languages (e.g. GUI XML Schema editors like XMLSpy) and ways to build consensus around the semantics of these new languages (wiki's, social networks, folkonomies and voting).

And next we need the declarative systems tool kits. Right now we have started to develop tools to allow XML Schemas to be quickly transformed into GUI XForms editors and we have XML databases that eliminated the need for complex middle tiers. We can put many of these tools in the hands of non-programmers and allow them to create domain-specific vocabularies and rule-based systems with GUI editors.

And when those tools mature we will see an explosion of declarative languages similar to the explosion of life forms 550 million years ago.

So what is your prediction? 1 year, 3 years, 5 years or 10 years from now? I think a small group of people could build this toolkit without millions of dollars of venture capital. There may even be open-source tools that do this in a few years. But I think the key is to combine the best of metadata registries, XML Schemas, transforms, XForms, XQuery and XML databases into a holistic declarative system.

And from here the many NEW ideas will evolve.

XForms and the Chasm: What XForms Need to Become a Mainstream Technology

Geoff Moore wrote a very insightful book on technology adoption called “Crossing the Chasm”. If you study the psychology of technology adoption, you find that people and organization fall in a bell curve. About 2% are innovators, 14% are early adopters, 33% early majority, 33% late majority and 15% are laggards. This is all standard stuff out of basic marketing-psychology textbooks. Each group has different reasons for using (or not using) a specific product.

Moore's insight was that many new products never move from early adopters to early majority. He called this region “The Chasm”. I have found that the same rules applyto new standards and new languages.

Moore did a lot of analysis on why some new technologies don’t make it across the chasm despite their merits or why some standards are adopted over others. Hisbiggest finding is that they were just too hard to use by non-techies. They required too much work to setup, install or learn. Ease-of-use was a principal factor in many technologies not being quickly adopted by organizations that did not want to use technology to differentiate their organization. Early Majority customers also tend to purchase new technology not based on how they work but who is using them in their industry. They purchase on references.

New technologies usually need a focused approach in a specific industry to be adopted by an early majority. If an organization can target a specific vertical industry and provide a more complete solution for that industry, once a few reference accounts are set up the rest of the industry can see there are proven case studies.

Another factor is the need of the early majority for complete solutions. Innovators and Early adopters are willing to glue together new technologies into their existing fabric of solutions. They dedicate time, staff and money to do this. If their XForms sends XML data over REST interfaces and their databases use JDBC they hire programmers to write middle-tier applications.

But early majority buyers can’t justify that additional cost. They just want a solution to work out-of-the-box. They are OK paying for a week for training for their BAs, but they don’t want to write and maintain complex applications.

Another thing that I have been searching for are metaphors that are understandable by non-programmers. Since the high-level decision makers are usually non-technical their ability to make a decision is usually based on their trust of the developers or their understanding based on metaphors.

The Language Translation Metaphor I find that if you can send your data from an XForms application directly to a native XML database in XML format that there is no need for middle-tier objects or shredding into relational databases. No translation is involved. One of the ways to demonstrate how machine-based language translation is not always semantically precise is to try an automatic translation program. Convert a statement from a native language into a foreign language and then translate it back. For example try this using Google Language Tools at http://www.google.com/language_tools

Original Phrase in English: The quick brown fox jumped over the lazy dog.

Google Translation to Spanish El zorro marrón rápido saltó sobre el perro perezoso.

Spanish Translation Back to English The fast brown fox jumped on the sluggish dog.

Although Google translate does a pretty good job, quick was changed to fast, over was changed to on, and lazy was changed to sluggish. If you study semantics, you can find lots of problems trying to communicate with close approximations. Note the original implicit business rule of don’t repeat a letter was also lost. We have many implicit business rules in complex systems and translation of these to other formats is always problematic for software developers.

Saving XML using an object-middle tier is one translation. Saving objects to a relational database is a second translation. Fetching data from a database using SQL is a third translation. Converting tabular data into objects is a fourth translation and converting objects back to XML is a fifth transformation. There are five translations, not just two.

Not that XForms solutions today are not without translation. In our process we are planning to convert XML Schemas into XForms as well as initial instance data. We need to also import leaf-level definitions from a metadata registry. But the nice thing about this process is that we can validate the submitted data directly against the original XML Schema that was used to generate the XForms. An elegant self-checking pattern.

All of these transforms are possible. But all have to be maintained each time you change your XML Schema. Right now we also need great tools that translate XML Schemas directly into XForms. And these need to retain the fidelity of the original intent of the subject matter experts.

So perhaps deploying applications with native XML databases is a critical part of the XForms value proposition.

Moore also noted that both early late majority buyers would be more willing to use a technology if it was buried deeply in a complete solution. That way the IT managers don’t have to sell the technology to management, just the solution. Similarly, once third party developers start building accounting or complete solutions for vertical industry using XForms the buyer will not need to know or care how the solution is being delivered.

Metaphors for Declarative Systems

I am working on my presentation for the 2007 semantic technology conference. My topic is the semantics of declarative languages.

I am attempting to present to business strategists that may not appreciate how powerful the new XForms and XQuery standards are and how entire rich user experience web 2.0 applications can be built using these tools as long as the tools have RESTful interfaces.

My first consideration is to find the right metaphors. I am working with the evolution metaphor and the puzzle metaphor right now and it seems to be working well.

I have also done some research on Domain Specific Languages and I see that they have also used the evolution metaphor but I find that they are focused on building small languages that are specific to a group or project and they don't really look at the external semantics of the problem.

Please write me if you are also trying to explain the business benefits of either declarative languages to a non-technical audience and if you have found metaphors that get the key points across.


Thanks! - Dan

Doing Zero Fill with XSLT 2.0

I was scratching my head, trying to find a simple way to do a zero fill padding with XSLT and I could not find it in my Michael Kay XPath 2.0 book. It turns out he put in in the XSLT 2.0 book. I am not sure how the index missed it. Anyway, searching for "Zero Fill XPath" turned up nothing on Google. Here is how it works. Just use the following:

format-number(number(.), '00000').

Here is an example of the input and output:

Input	Output
1	00001
12	00012
123	00123
1234	01234
12345	12345

Hope this helps. - Dan

The Penguin, The Fox, The Weasel and the Elephant

Once upon a time there were four animals that lived in a magical “Land of Bew”: The Penguin, the Fox, the Weasel and the Elephant.

Each animal had their own way of looking at the world. Penguin and Fox loved to share their ideas.

Weasel also had lots of interesting ideas but he was afraid that if he shared too many ideas his owners would be worried.

Elephant always seemed to be around everywhere but didn’t pick up new ideas very fast.

In the Land of Bew most of the animals got along pretty well with each other. Fox played Penguin’s house and Fox played at Weasel’s house.

Unfortunately the Elephant was not allowed to play at the Penguin’s house. It was not that Elephant was mean, its just that Elephant’s owners were somewhat controlling.

The Land of Bew was blessed with special magic books. These books that allowed the reader to press on special words and pictures in the books and suddenly new pages would magically appear. The children of Bew loved the books and always wanted more.

But some of the books were very hard to make. Especially if you wanted nice moving and dancing pictures that children could change.

To make dancing pictures you had to use special type of black magic that sometimes got the animals in trouble. Goblins would sometimes used the black magic and cause havoc in the Land of Bew. ( I would tell you more about the horrible things Goblins did with the black magic but this IS a children’s story.)

One day Penguin and Fox found a new way of making dancing books that animals could use in magical ways without using the dangerous black magic. Fox tried it out and it worked very well.

But when they suggested their ideas to Elephant, Elephant said no. Elephant was worried that even kids would be able to make the dancing books. And she worried that all her friends would stop playing with Weasel and spend all their time over at Penguins house.

And although the story is not over…this is where our story ends. The ending is being written by you.

Want to try to make some dancing books?

Try XForms today and see if you can make your web applications dance without using any nasty JavaScript! And try to keep those Goblins out. OK?

The eXist OpenSource Native XML Database

After a suggestion from Kurt Cagle, we have been using the eXist OpenSource XML Database for a week now and I must say that it has exceeded our expectations.
It was a snap to download a set up. It was easy to get all the XQuery demos running and it was almost trivial to administer the little but annoying things like adding collection and users.
Unfortunately I did have to actual read the documentation to figure out how to get our XForms and XMLSpy to work with it. But once you know the secret, it is easy.
The big secret is that by just adding the words "rest" or "webdav" before the "db" in the URL you automatically and to HTTP GETs and HTTP PUTs directly to your data.
You can also actually edit the data directly in the eXist database DIRECTLY from within XMLSpy by just using the "Open URL" option of the XMLSpy editor and then just slip the "webdav" string into your URL path. Hopefully XMLSpy will allow you to use the XQuery editor in future versions of the database.
eXist had given our entire group a large boost of confidence. We can now build a Ruby-on-rails like environment just by drawing XML Schemas, generating XForms from the XML Schemas and storing the form data directly into eXist. A full application development stack without a single like of procedural Java or JavaScript!!!
I posted a short article on how to get XForms working with eXist in the XForms WikiBook
I have a FireFox T-shirt on order for Kurt and I am also buying beers at the first XForms conference...which I may have to organize.
Thanks Kurt! Your advise was very sound.

Computers Don't Create Models (Today) – People Do

I had several good comments about my posting about the increasing role of semantics in the "Era of Tera". I did not mention some of the key concepts that the Intel paper discussed and the impact these concepts will have on IT strategy. Here is an excerpt:

The problem is ordinary computers don't model things. Aside from supercomputers, today's computers aren't capable of developing mathematical models of complex objects, systems or processes. Nor are they powerful or fast enough to perform such tasks at speeds people demand. We can't plug in a statistical model for a rare malignant tumor or the behavioral pattern of a shoplifting employee and search for similar instances of the model in a data set. To benefit from the wealth of data building up in the world, we need to be able to communicate with computers in more abstract terms (high-level concepts or semantics). We need to speak in terms of models.

I believe that the core problem is that there will be a HUGE demand for highly-skilled data-modelers/ontologists in five years. But there will not be a large supply since this skill is not something that can be learned from a single class in college. Rather it is more of a tacit skill that is not easily codifiable.

That being said, there are many best practices that ARE codifiable. You can already purchase books on OWL/RDF and metadata registries, although most of them are mired in relational database models or written by academics with little real-world experience. You can read the Wikipedia articles on “Data Stewardship”. What is still needed is a single set of best practices and tools for creating families of machine-readable data models that can be use as a basis for creating exchange models. And we need to do this without having to learn how to represent all 12 types of UML diagrams in XMI and transform them. The NIEM subset generator is a good example of the solid XML-Schema-driven front-end of this process of selecting elements from a metadata registry and putting them in your shopping cart.

The bottom line is that this is really about empowerment. And unless organizations introduce semantic web technologies into their organizations at a grass-root level and support them at the CEO level, many organizations will be left behind in the Era of the Tera.

Era of the Tera

I read in the New York Times yesterday that Intel has produced a chip with 80 core processors giving it a total computing power of 1.3 teraflops. So the natural question is what could you use 80 CPUs for? The article referred to a 2005 Intel Paper titled Recognition, Mining and Synthesis Moves Computers to the Era of Tera by Pradeep Dubey. This article opened with the following quote: “The great strength of computers is that they can reliably manipulate vast amounts of data very quickly. Their great weakness is that they don’t have a clue as to what any of that data actually means.” Stephen Cass, “A Fountain of Knowledge,” IEEE Spectrum, January 2004. When I saw this quote, I realized even the hardware engineers agree semantics is now the critical factor limiting our ability to effectively use computers I realized that we must continue our mission to promote semantic mapping concepts. So let me summarize my feelings after reading this article:

If want to be able to leverage the power coming in the Era of the Tera, you must start with semantics – recoding the meaning of your data.

And if you don’t yet have a metadata registry…get one!

Right next to the CEO, CIO, and CFO in the board room should be your CDO (Chief Data Officer).

Whuffie, Social Networks and Metadata Semantics

I was reading Ambient Findabilty and I saw a reference to a concept called “whuffie”. After checking out the Wikipedia article on whuffie, I discovered that whuffie was reputation-based currency used in Cory Doctorow's sci-fi novel, Down and Out in the Magic Kingdom. This future history book describes a post-scarcity economy where people purchase goods and services based on their reputation, not just cash.

I was interested in this topic because I think that in the future new keywords with precise semantic mappings will be introduced into declarative systems based on social networks. Right now it just takes too long for new tags to appear in an XML standard. Just think of how long it takes for the w3c to create a new XML tag for something like a menu tab! Up to 10 years based on the rate that the XForms standard is being adopted.

Anyway the book was a great mind-opener. One of the first science fictions books that takes nanotechnology and social networking into account. I have always loved cyber-punk: Neil Stephenson’s Diamond Age being one of my favorites. No time-travel or spaceships: just highly imaginative extrapolations of the net and Moore’s Law.

I like the concept of the post-economic society. When all of our economic needs are met what do we strive for? Why don’t they ever complain about their raises in Star Trek? The book also discusses the concept of an adhocracy: groups that quickly band together to solve problems. Doctorow also looks at the concepts of death, backing up your brain to a persistence store (the immortality SAN) and filtering restores to change values.

What does this matter to the metadata architect? It matters because our march away from procedural programming toward declarative systems is totally dependant on the creation of shared semantics. When you need to pick an XML tag how do you pick the tag that will have the highest probability to have semantic precision in the future? Social networks will help us and when we build systems to vote on which tags get approved not all votes should count the same. Votes by people with high-whuffie should get counted a lot more. The concept of one-person, one vote will be modified in the post-economic society.

My visit to the metadata library

Last night I had the strangest dream…

I was looking for a good mystery, perhaps a Sherlock Holmes book. I went to the library and an elderly lady at the front desk said “Welcome to Roy G. Biv Library. How can I help you?”

“I am looking for a Sherlock Holmes mystery. Do you have one?” I asked.

“What color of book are you looking for?” She asked.

“Well I don’t really care what color it is.” I replied.

“Well in this library, our books are sorted by color of the cover. The red covers are over there, the orange covers over there and the yellow ones right over here. You get the idea. Shelving books by color makes it very easy for us to shelve a book because the color of the cover is obvious. It also makes it easy to see when a book is miss-shelved. You can see the miss-shelved books right away.”

“Yes, I can see how that would make your work much easier. Do you know where I can find the Sherlock Holmes book?”

“Well, it would help us a lot if you could tell us what color the cover is.” She said firmly.

“I don’t know what the color of the cover is…, I just want a mystery book about Sherlock Holmes!” I said exasperated.

“Well, why don’t you just start looking at the reds, then orange, yellow, green, blue, indigo and finally violet. I am sure you will find it in a few days…”

Then I woke up and when to work. There the staff was attempting to store all the data elements in the enterprise sorted by the system name that each data element was first recorded. Then my real nightmare began.

We Declare!

Many of you may be wondering why a data architect has been spending so much time talking about XForms. I have been using XSL for many years to transform XML and I have loved the declarative aspects of XSL. My friend Arun has been challenging me to articulate the key aspects of declarative programming in the context of our join efforts to promote metadata registries. Here is my first version of a Delarative Manifesto.

1. Declarative development is about the precise association of a programming symbol with the semantics of a design pattern
2. Declarative approaches leverage the inherent pattern-matching structures of the human brain to allow developers to quickly recognize the underlying patterns used in a system
3. The rapid association of a symbol in a program to previously encountered problem by a wide audience is essential for developer productivity
4. Developer productivity is dependant on the efficiency of matching a symbol to a previously encountered approach to a problem
5. Shared meaning happens when a community agrees that a symbol should be associated with design pattern
6. Tools that help us quickly promote shared meaning make us all more productive
7. Published and carefully annotated XML Schemas and formal metadata registries are ways of associating symbols with precise semantics
8. Developers need tools that allows a new tag to be quickly associated with the semantics of the tag, history of the tag, usage of a tag and similar tags
9. Declarative programming makes development more accessible to people that don't have a computer science background
10. Advanced collaborative computing tools, a focus on architectures of participation, and social networking software will dramatically impact our ability to rapidly create precise semantics

The Power of Declarative Programming

This presentation by Steve Pemberton of the W3C on declarative programming is very insightful. Thanks goes to Mark Birbeck for pointing it out. I hope we can get more students exposed to technologies like XForms much earlier in their programming careers. Take this use case. Today we give 10 students a simple assignment to create a tab menu in JavaScript we would get 10 very different programs. But if we gave them the same assignment XForms they would hopefully all use similar toggle-switch structures. This has big implications for maintenance. Hopefully there would be less temptation to recode an XForms application. Since over 50% of software is maintenance this could have long-term impacts on developer productivity.

XForms Tutorial and Cookbook

I am speaking tomorrow at the Minnesota Open Source User group (see http://www.josug.org) and I thought I would put some of my examples in Wikibook format. Here is a link: http://en.wikibooks.org/wiki/XForms Let me know if you have any feedback. Thanks - Dan

Firefox XForms Extension Rocks!

For the past several months I have been building XFroms prototypes on my current project. I ran into the XForms Firefox plugin two months ago and have been using happily using it ever since. Although the plug-in version number is only "0.6", I have found for my use, it works perfectly. It allows me to quickly test XForms on all the computers I use and I don’t need to setup and configure a web server. The extension takes under 30 seconds to install, but you will have to reboot FireFox once after you download the extension.

Now biggest thing that I am searching for are XForms example programs. The examples provided with some of the books are not very good. I would like to use these samples XForms and teaching XForms to others. I would also like to create XForms directly from the XML Schemas I am building and so I can integrate the hints, help and metadata registry links for each field on the forms I am developing. I am also trying to figure out how to make CSS work correctly across many browsers since many of the XForms CSS templates in the books use CSS3 and CSS3 only works correctly with FireFox.

If you know of any nice collections of XForms for learning XForms, please let me know. I have set up a Wikibook to start collecting tutorials and examples. The link is http://en.wikibooks.org/wiki/XForms

UNIX Shell Script to Transfer Files via FTP

I recently needed a UNIX shell script to transfer all my files to the web collaboration server running plone. Although you can do this with Ant's FTP command, my remote system just had UNIX. Here is the script I wrote.

#!/bin/sh
# This UNIX shell script FTPs all the files in the input directory to a remote directory
# It uses a UNIX Shell "hear document" to read the commands to the FTP program from the shell file
# You have to use the -n option to the FTP command to disable the prompting for the password
# the quote in front of the ftp user and pass are also necessary to disable reading from stdin
# the prompt must be turned off for the multi-put to not prompt the user
LOCALDIR=mylocaldir
REMOTESERVER=remote.server
REMOTEPATH=/my/remote/path
LOGIN=myloginname
PASSWORD=mypassword

cd $LOCALDIR
ftp -n $REMOTESERVER <<INPUT_END
quote user $LOGIN
quote pass $PASSWORD
cd $REMOTEPATH
prompt off
mput *.*
exit
INPUT_END