Doing Zero Fill with XSLT 2.0

I was scratching my head, trying to find a simple way to do a zero fill padding with XSLT and I could not find it in my Michael Kay XPath 2.0 book. It turns out he put in in the XSLT 2.0 book. I am not sure how the index missed it. Anyway, searching for "Zero Fill XPath" turned up nothing on Google. Here is how it works. Just use the following:

format-number(number(.), '00000').

Here is an example of the input and output:

Input	Output
1	00001
12	00012
123	00123
1234	01234
12345	12345

Hope this helps. - Dan

The Penguin, The Fox, The Weasel and the Elephant

Once upon a time there were four animals that lived in a magical “Land of Bew”: The Penguin, the Fox, the Weasel and the Elephant.

Each animal had their own way of looking at the world. Penguin and Fox loved to share their ideas.

Weasel also had lots of interesting ideas but he was afraid that if he shared too many ideas his owners would be worried.

Elephant always seemed to be around everywhere but didn’t pick up new ideas very fast.

In the Land of Bew most of the animals got along pretty well with each other. Fox played Penguin’s house and Fox played at Weasel’s house.

Unfortunately the Elephant was not allowed to play at the Penguin’s house. It was not that Elephant was mean, its just that Elephant’s owners were somewhat controlling.

The Land of Bew was blessed with special magic books. These books that allowed the reader to press on special words and pictures in the books and suddenly new pages would magically appear. The children of Bew loved the books and always wanted more.

But some of the books were very hard to make. Especially if you wanted nice moving and dancing pictures that children could change.

To make dancing pictures you had to use special type of black magic that sometimes got the animals in trouble. Goblins would sometimes used the black magic and cause havoc in the Land of Bew. ( I would tell you more about the horrible things Goblins did with the black magic but this IS a children’s story.)

One day Penguin and Fox found a new way of making dancing books that animals could use in magical ways without using the dangerous black magic. Fox tried it out and it worked very well.

But when they suggested their ideas to Elephant, Elephant said no. Elephant was worried that even kids would be able to make the dancing books. And she worried that all her friends would stop playing with Weasel and spend all their time over at Penguins house.

And although the story is not over…this is where our story ends. The ending is being written by you.

Want to try to make some dancing books?

Try XForms today and see if you can make your web applications dance without using any nasty JavaScript! And try to keep those Goblins out. OK?

The eXist OpenSource Native XML Database

After a suggestion from Kurt Cagle, we have been using the eXist OpenSource XML Database for a week now and I must say that it has exceeded our expectations.
It was a snap to download a set up. It was easy to get all the XQuery demos running and it was almost trivial to administer the little but annoying things like adding collection and users.
Unfortunately I did have to actual read the documentation to figure out how to get our XForms and XMLSpy to work with it. But once you know the secret, it is easy.
The big secret is that by just adding the words "rest" or "webdav" before the "db" in the URL you automatically and to HTTP GETs and HTTP PUTs directly to your data.
You can also actually edit the data directly in the eXist database DIRECTLY from within XMLSpy by just using the "Open URL" option of the XMLSpy editor and then just slip the "webdav" string into your URL path. Hopefully XMLSpy will allow you to use the XQuery editor in future versions of the database.
eXist had given our entire group a large boost of confidence. We can now build a Ruby-on-rails like environment just by drawing XML Schemas, generating XForms from the XML Schemas and storing the form data directly into eXist. A full application development stack without a single like of procedural Java or JavaScript!!!
I posted a short article on how to get XForms working with eXist in the XForms WikiBook
I have a FireFox T-shirt on order for Kurt and I am also buying beers at the first XForms conference...which I may have to organize.
Thanks Kurt! Your advise was very sound.

Computers Don't Create Models (Today) – People Do

I had several good comments about my posting about the increasing role of semantics in the "Era of Tera". I did not mention some of the key concepts that the Intel paper discussed and the impact these concepts will have on IT strategy. Here is an excerpt:

The problem is ordinary computers don't model things. Aside from supercomputers, today's computers aren't capable of developing mathematical models of complex objects, systems or processes. Nor are they powerful or fast enough to perform such tasks at speeds people demand. We can't plug in a statistical model for a rare malignant tumor or the behavioral pattern of a shoplifting employee and search for similar instances of the model in a data set. To benefit from the wealth of data building up in the world, we need to be able to communicate with computers in more abstract terms (high-level concepts or semantics). We need to speak in terms of models.

I believe that the core problem is that there will be a HUGE demand for highly-skilled data-modelers/ontologists in five years. But there will not be a large supply since this skill is not something that can be learned from a single class in college. Rather it is more of a tacit skill that is not easily codifiable.

That being said, there are many best practices that ARE codifiable. You can already purchase books on OWL/RDF and metadata registries, although most of them are mired in relational database models or written by academics with little real-world experience. You can read the Wikipedia articles on “Data Stewardship”. What is still needed is a single set of best practices and tools for creating families of machine-readable data models that can be use as a basis for creating exchange models. And we need to do this without having to learn how to represent all 12 types of UML diagrams in XMI and transform them. The NIEM subset generator is a good example of the solid XML-Schema-driven front-end of this process of selecting elements from a metadata registry and putting them in your shopping cart.

The bottom line is that this is really about empowerment. And unless organizations introduce semantic web technologies into their organizations at a grass-root level and support them at the CEO level, many organizations will be left behind in the Era of the Tera.

Era of the Tera

I read in the New York Times yesterday that Intel has produced a chip with 80 core processors giving it a total computing power of 1.3 teraflops. So the natural question is what could you use 80 CPUs for? The article referred to a 2005 Intel Paper titled Recognition, Mining and Synthesis Moves Computers to the Era of Tera by Pradeep Dubey. This article opened with the following quote: “The great strength of computers is that they can reliably manipulate vast amounts of data very quickly. Their great weakness is that they don’t have a clue as to what any of that data actually means.” Stephen Cass, “A Fountain of Knowledge,” IEEE Spectrum, January 2004. When I saw this quote, I realized even the hardware engineers agree semantics is now the critical factor limiting our ability to effectively use computers I realized that we must continue our mission to promote semantic mapping concepts. So let me summarize my feelings after reading this article:

If want to be able to leverage the power coming in the Era of the Tera, you must start with semantics – recoding the meaning of your data.

And if you don’t yet have a metadata registry…get one!

Right next to the CEO, CIO, and CFO in the board room should be your CDO (Chief Data Officer).

Whuffie, Social Networks and Metadata Semantics

I was reading Ambient Findabilty and I saw a reference to a concept called “whuffie”. After checking out the Wikipedia article on whuffie, I discovered that whuffie was reputation-based currency used in Cory Doctorow's sci-fi novel, Down and Out in the Magic Kingdom. This future history book describes a post-scarcity economy where people purchase goods and services based on their reputation, not just cash.

I was interested in this topic because I think that in the future new keywords with precise semantic mappings will be introduced into declarative systems based on social networks. Right now it just takes too long for new tags to appear in an XML standard. Just think of how long it takes for the w3c to create a new XML tag for something like a menu tab! Up to 10 years based on the rate that the XForms standard is being adopted.

Anyway the book was a great mind-opener. One of the first science fictions books that takes nanotechnology and social networking into account. I have always loved cyber-punk: Neil Stephenson’s Diamond Age being one of my favorites. No time-travel or spaceships: just highly imaginative extrapolations of the net and Moore’s Law.

I like the concept of the post-economic society. When all of our economic needs are met what do we strive for? Why don’t they ever complain about their raises in Star Trek? The book also discusses the concept of an adhocracy: groups that quickly band together to solve problems. Doctorow also looks at the concepts of death, backing up your brain to a persistence store (the immortality SAN) and filtering restores to change values.

What does this matter to the metadata architect? It matters because our march away from procedural programming toward declarative systems is totally dependant on the creation of shared semantics. When you need to pick an XML tag how do you pick the tag that will have the highest probability to have semantic precision in the future? Social networks will help us and when we build systems to vote on which tags get approved not all votes should count the same. Votes by people with high-whuffie should get counted a lot more. The concept of one-person, one vote will be modified in the post-economic society.

My visit to the metadata library

Last night I had the strangest dream…

I was looking for a good mystery, perhaps a Sherlock Holmes book. I went to the library and an elderly lady at the front desk said “Welcome to Roy G. Biv Library. How can I help you?”

“I am looking for a Sherlock Holmes mystery. Do you have one?” I asked.

“What color of book are you looking for?” She asked.

“Well I don’t really care what color it is.” I replied.

“Well in this library, our books are sorted by color of the cover. The red covers are over there, the orange covers over there and the yellow ones right over here. You get the idea. Shelving books by color makes it very easy for us to shelve a book because the color of the cover is obvious. It also makes it easy to see when a book is miss-shelved. You can see the miss-shelved books right away.”

“Yes, I can see how that would make your work much easier. Do you know where I can find the Sherlock Holmes book?”

“Well, it would help us a lot if you could tell us what color the cover is.” She said firmly.

“I don’t know what the color of the cover is…, I just want a mystery book about Sherlock Holmes!” I said exasperated.

“Well, why don’t you just start looking at the reds, then orange, yellow, green, blue, indigo and finally violet. I am sure you will find it in a few days…”

Then I woke up and when to work. There the staff was attempting to store all the data elements in the enterprise sorted by the system name that each data element was first recorded. Then my real nightmare began.

We Declare!

Many of you may be wondering why a data architect has been spending so much time talking about XForms. I have been using XSL for many years to transform XML and I have loved the declarative aspects of XSL. My friend Arun has been challenging me to articulate the key aspects of declarative programming in the context of our join efforts to promote metadata registries. Here is my first version of a Delarative Manifesto.

1. Declarative development is about the precise association of a programming symbol with the semantics of a design pattern
2. Declarative approaches leverage the inherent pattern-matching structures of the human brain to allow developers to quickly recognize the underlying patterns used in a system
3. The rapid association of a symbol in a program to previously encountered problem by a wide audience is essential for developer productivity
4. Developer productivity is dependant on the efficiency of matching a symbol to a previously encountered approach to a problem
5. Shared meaning happens when a community agrees that a symbol should be associated with design pattern
6. Tools that help us quickly promote shared meaning make us all more productive
7. Published and carefully annotated XML Schemas and formal metadata registries are ways of associating symbols with precise semantics
8. Developers need tools that allows a new tag to be quickly associated with the semantics of the tag, history of the tag, usage of a tag and similar tags
9. Declarative programming makes development more accessible to people that don't have a computer science background
10. Advanced collaborative computing tools, a focus on architectures of participation, and social networking software will dramatically impact our ability to rapidly create precise semantics

The Power of Declarative Programming

This presentation by Steve Pemberton of the W3C on declarative programming is very insightful. Thanks goes to Mark Birbeck for pointing it out. I hope we can get more students exposed to technologies like XForms much earlier in their programming careers. Take this use case. Today we give 10 students a simple assignment to create a tab menu in JavaScript we would get 10 very different programs. But if we gave them the same assignment XForms they would hopefully all use similar toggle-switch structures. This has big implications for maintenance. Hopefully there would be less temptation to recode an XForms application. Since over 50% of software is maintenance this could have long-term impacts on developer productivity.

XForms Tutorial and Cookbook

I am speaking tomorrow at the Minnesota Open Source User group (see http://www.josug.org) and I thought I would put some of my examples in Wikibook format. Here is a link: http://en.wikibooks.org/wiki/XForms Let me know if you have any feedback. Thanks - Dan

Firefox XForms Extension Rocks!

For the past several months I have been building XFroms prototypes on my current project. I ran into the XForms Firefox plugin two months ago and have been using happily using it ever since. Although the plug-in version number is only "0.6", I have found for my use, it works perfectly. It allows me to quickly test XForms on all the computers I use and I don’t need to setup and configure a web server. The extension takes under 30 seconds to install, but you will have to reboot FireFox once after you download the extension.

Now biggest thing that I am searching for are XForms example programs. The examples provided with some of the books are not very good. I would like to use these samples XForms and teaching XForms to others. I would also like to create XForms directly from the XML Schemas I am building and so I can integrate the hints, help and metadata registry links for each field on the forms I am developing. I am also trying to figure out how to make CSS work correctly across many browsers since many of the XForms CSS templates in the books use CSS3 and CSS3 only works correctly with FireFox.

If you know of any nice collections of XForms for learning XForms, please let me know. I have set up a Wikibook to start collecting tutorials and examples. The link is http://en.wikibooks.org/wiki/XForms

UNIX Shell Script to Transfer Files via FTP

I recently needed a UNIX shell script to transfer all my files to the web collaboration server running plone. Although you can do this with Ant's FTP command, my remote system just had UNIX. Here is the script I wrote.

#!/bin/sh
# This UNIX shell script FTPs all the files in the input directory to a remote directory
# It uses a UNIX Shell "hear document" to read the commands to the FTP program from the shell file
# You have to use the -n option to the FTP command to disable the prompting for the password
# the quote in front of the ftp user and pass are also necessary to disable reading from stdin
# the prompt must be turned off for the multi-put to not prompt the user
LOCALDIR=mylocaldir
REMOTESERVER=remote.server
REMOTEPATH=/my/remote/path
LOGIN=myloginname
PASSWORD=mypassword

cd $LOCALDIR
ftp -n $REMOTESERVER <<INPUT_END
quote user $LOGIN
quote pass $PASSWORD
cd $REMOTEPATH
prompt off
mput *.*
exit
INPUT_END

XForms, metadata and MDA

For the last several weeks I have been looking at the XForms standard. This has been related to my contract at the Minnesota Department of Revenue to replace many of their paper forms with electronic versions. Although my contact is to design XML Schemas, ultimately they need to be entered through a web form. I started to build some mock-ups of web forms to clarify requirements but found the older HTLM forms lacking in many areas. I also wanted to be able to validate the data using the XML Schemas that I was building.

What I found was that XForms is a wonderfully designed specification. It integrates XML Schemas, XML data-types and XPath expressions directly into the standard and also adds the new XML Events standard. XForms also faithfully uses both the Model-View-Controller pattern and several separation-of-concerns issues. In short, XForms is an excellent design and appears to be very well thought out. My congrats to the XForms team.

My other worry was that it would be hard to setup and test XForms. I have been happy to find out that FireFox has a new plug-in that supports XForms and there is a plug-in free solution using the FormFaces javascript tools. Many of the examples just run without modification. Just build your XForms and out pops complex XML documents.

What I am most interested in is that there appears to be several things that XForms demonstrates that will make software engineering better: the drive away from the complex procedural world and toward a more model-driven declarative style of development that allows non-programmers to maintain complex systems. This means less complex javascript and complex server-side coding in Java. I hope to have a sample PowerPoint presentation on my web site soon that goes through these arguments. Look for an XForms user interface to data element entry in the near future.

Another last point is that if a large audience starts to build XForms you are going to start to have fully-formed XML documents flying around. This means you will need more XML Schemas being generated and web services to accept them and respond with XML documents and XML services. This will make the creation of new XForms go even faster. Positive feedback cycles. All the more reason to build a metadata registry!

I have been thinking about creating an transform from a constraint schema directly into an XForm. This would be an nice MDA-type approach.

Let me know your thoughts. - Dan

Photo gallery Software

I have been looking into OpenSource photo gallery software. I am a little frustrated by the limitations of FrontPage for publishing my photographs. I would like to just tag my images from the Windows browser and say "publish". I want a thumbnail-drive front page for each section with a variety of viewers (matrix, filmstrip, montage) that can be customized with the FireFox View/Page Style menu. My friend Liela Tite recommended Coppermine. I also ran into these two: Gallery Project - based on the Drupel web content management system. 4Image - which is a system written in Germany. Since the 4Image site was written in German I had a more difficult time trying to find a list of all the features. Both the gallery and Coppermine systems seemed very full-featured. Let me know if anyone has suggestions. A Wikipedia feature-comparison would be very helpful. Perhaps I should start one? :-) OK, OK, I will...

Thanks Arun!

I recently started a project at the Minnesota Department of Revenue. I was happy to find a kindred spirit in the IT department that also in interested in using Wiki's to manage IT data. Arun has got me to think about the implications of Wiki publishing and how I can integrate metadata into a ESB framework. Check out his web site. It has lots of interesting things. http://netrii.com/display/ABB/Home I was also interested to find that he uses a Wiki for his own personal web site. - Dan

Ant Image Tasks

I was pleasantly surprised to find that Apache Ant has a large library of image processing tasks under the optional Ant tasks. For example, the following will create a folder of thumbnail images for all the images in a folder:

This creates thumbnails of the images and make sure they all fit within the 160x160 size whether the image is portrait or landscape. Very cool! Check out the Ant manual for more information: http://ant.apache.org/manual (Ant tasks/Optional Tasks/Image) This makes me think that you could use Apache ant to do basic photo publishing. It would make thumbnails, a layout page that has links to the thumbnails and ftp the folder's web-resized images to a web server. I have also found that there are some nice photo-layout CSS pages. It would seem that it should be easy to add a set of CSS styles that could be changed using the FireFox "Page Style" option. I would be happy to write a basic ant task and an XML transform to create an HTML page of all the images in an image set if anyone wants to help. What about image keywords and captions? Should each photo have its own photo metadata? What if you want to change the file name of a photo? How can the metadata file stay in sync with the file system? Could you use a subversion-tortoiseSVN-like http://tortoisesvn.tigris.org/ Any ideas?

Article on Common Metadata

Here is an interesting article on the importance of common metadata: Common Metadata The author appears to understand the importance of trust and metadata but appear to not be familiar with metadata standards such as ISO/IEC 11179 He also did not reference the work of Goldberg and Rubin in their 1995 book: Succeeding With Objects: Decision Frameworks for Project Management Adele Goldberg, Kenneth S. Rubin Addison-Wesley Professional; May 1995 See my href="http://www.danmccreary.com">book review for more. But all-in-all, it was a very good article.

Natural Language Interface to Wikipedia

I have been saying for a while that because wikipedia is a database it is much easier to link it into a natual language seach system. The Wikipedia search function is also very limted and does not allow you to type in complete questions like ask.com. I reciently found out that MIT is already building a very powerful natuaral language interface in their START project. http://start.csail.mit.edu I did a search for "Who was Tesla?" and got a link right to the Wikipedia article. Very cool! Although there seems to be very limited documentation on START and I do not believe that there is an OpenSource version, I believe this points to the direction that wiki software should be moving. Could your relational database bennefit from this? When I was at the Minnesota Department of Education I wanted to build a natural language interface to their "cubes" of student test data. For example "What school district has the highest highest 8th grade math scores for girls?" Give it a try and let me know what you think.

A Little Bit of Semantics Goes a Long Way

There are now over 20 versions of semantic wikis that are being developed. For example see here. What is interesting is the large number of ways that people are approaching this process. One of the phrases that I heard a lot of at the Semantic Technology Conference was A Little Bit of Semantics Goes a Long Way. An although I admire all the innovations that people are putting into their semantic wiki projects, to me I think that just adding attributes to a page and adding types to links is all that we really need to test the concepts out. It was also pointed out to me that several other wikis do have strong ACL features and you can "lock-down" pages using tools such as Apache administration tools. There also seems to be a lot of discussion about implementing semantic wikis using triple stores. This seems to me a little ahead of its time. If using a triple store makes a system more flexible, but it appears to me that the current structure is more than flexible enough to meet most business requirements. If people wanted to work on something useful, I would suggest something like a natural language front-end to Wikipedia so people could type in a written language questions like "What is the population of Minneapolis?". That would be useful! - Dan

Impresssions of Semantic Technology Conference 2006

I just returned form the 2006 Semantic Technology Conference in San Jose CA. This was a fantastic conference with over 600 people attending 72 presentations over four days. This was over twice what the attendance was last year. It is showing that semantics are really becoming a hot topic, although there is still plenty of room for discussion on when and how this will impact the industry.  The biggest problem for me was that with up to six concurrent sessions running it was very hard to decide what sessions to attend.

The composition of the conference consisted mostly of vendors of semantic technology products, large customers from government agencies like the DOD, NASA, some academics working on semantic research topics, a few venture capital firms and a handful of independent consultants like myself.  Some of the people were from the "old AI school" and were promoting the use of description logic (DL) within an ontology. Others like me, were simply searching for semantic integration techniques for the deployment of intelligent agents.

I think one of the most important presentations was the keynote given by Jim Hendler and Ora Lassila, two of the three people that wrote the original article in Scientific American with Tim Berners-Lee.  The presentation discussed some of the linkages between traditional AI (with complex ontologies) and the current document-centric web with simple un-typed link between documents.  My favorite sound bite was "Linking is Power".  The metaphor was that semantics are the "plumbing" necessary to allow intelligent agents will to perform interesting work over the world wide web. If agents don't know how to access distributed data they will be restricted to local databases, not a great prospect.  And the way to promote agent interoperability is to link ontologies, something I have been advocating for a long time.  They also pointed out that google now has around 12K ontologies (search google for ontology filetype:owl).  But the problem is that each ontology is an "island" and few people are linking ontologies.  Looking for ontologies with lots of equivalentClass statements leads to less than 40 ontologies.

This leads to the natural questions, how do we encourage ontology-linking.  One standard proposal is to encourage everyone to link to a standard reference upper ontology such as SUMO or CYC.  The problem is that few people can agree on what this upper ontology should be.

I have been trying to champion the application of some basic economic theory to the publishing on ontology links.  If we do supply-and-demand analysis on ontologies links we see that there are few economic incentives to publish inter-ontology data element links.  If people did this then agents could take advantage of them and automatically perform semantic translation.  Jim Hendler and Ora Lassila both though this was a worthy idea and indicated they would support this.  I hope to kick this off as some type of reward/recognition for next year's conference.  Tony Shaw from Wilshire Conferences has also been very supportive of the idea.

Semantic Wiki's were also a very hot topic.  Any presentation with the word Wiki in the title were packed full of people.  The topic of using Wiki's to build controlled vocabularies was mentioned in several presentations.  Problems with locking down approved terms was also mentioned.  The topic of semantic wiki's was also discussed.  Ideally a metadata wiki would have different access control for different sections of a single page.  The approved definitions would require stakeholder team approval for changes.

I advocated for first adding simple typed links into MediaWiki for obvious relationships like subClassOf, instanceOf, partOf and basic GIS-type things like insideOf, capitalOf.  Note that by adding a prefix to the current MediaWiki could allow this to be incrementally added to WikiPedia.  I think this goes along the lines of "A little semantics goes a long way".  Typed links are an awesome way to add semantics to any system and I predict that there were be dozens of semantic wiki's in the near future.

For some time I have been suggesting that OpenCYC and Wikipedia will eventually merge into a single system.  It would be very easy to add simple link types to WikiPedia.  We have thousands of volunteers waiting in the wings.  But after talking to Doug Lenat he indicated that CYC has over 16,000 types of relationships.  Yikes!  We would almost need an mini-expert system to figure out the link type.  The training for adding consistent links might be challenging.  Nonetheless the prospect of WikiPedia evolving into HAL over the next ten years is exciting.

I always enjoy going to Doug Lenat's presentations.  He indicated that over half of the new rules added to CYC in the last year were done by reading natural language text.  Something that is VERY exciting.  Automated machine learning.  We just hope it is not another false peak.  I also asked Doug how I could add my K-12 ontology into CYC.  He indicated that I would have to learn about CYC's relationships types to do this effectively.  But he also agreed that ontology linking was the Rosetta Stone for enabling web-wide intelligent agents.

For all the advanced topics, I was impressed by the lack of the use of structured XML Schemas to capture semantics using controlled vocabularies and importing semantic XML schemas created by subschema generators from metadata registries.  I was also very impressed by Contivo's new Builder product which is going for an incredibly low $500.

There also appears to be good movement toward incrementally adding semantics to HTML documents using RDF/A.  I attended an excellent presentation by a person from MIT that was working on these standards.  Although his arguments were not all clear, the one point he did make was that adding RDF in the "class" attributes would have minimal negative impact to the current web.

I was also glad to hear there is more interest in the semantic-web in Minnesota.  Apparently Lockheed-Martin is starting a new logistics agent project that will be done in Eagan Minnesota.  They indicated that much of this would be based on semantic technologies.  Lockheed sent seven people to the conference at the last minute so at least I know they have a budget for training.  Logistics problems like those from Katrina could clearly benefit from semantic agents.  Lockheed has been promoting use of semantic web in their literature:  See the article on page 14.

My only real complaint with the conference is that the conference organizers allow people to present without sharing any of their slides even though they are supposedly "due" almost two months before the conference is scheduled.   We are forced to take notes and ask for e-mailed copies of the slides.  My success rate has been very low about getting copies mail out after a conference.  <shameless bragging> I believe I was also one of the few presenters that wrote the supplementary paper for the conference.  Providing only the PDF versions of the slides also prevents you from getting builds and speakers notes.  For an example see my full presentation web site.</shameless bragging>

The Semantic Technology conference next year will also be in San Jose next year but has been moved to early April.