Saturday, April 14, 2007

XForms and the Chasm: What XForms Need to Become a Mainstream Technology

Geoff Moore wrote a very insightful book on technology adoption called “Crossing the Chasm”. If you study the psychology of technology adoption, you find that people and organization fall in a bell curve. About 2% are innovators, 14% are early adopters, 33% early majority, 33% late majority and 15% are laggards. This is all standard stuff out of basic marketing-psychology textbooks. Each group has different reasons for using (or not using) a specific product.

Moore's insight was that many new products never move from early adopters to early majority. He called this region “The Chasm”. I have found that the same rules applyto new standards and new languages.

Moore did a lot of analysis on why some new technologies don’t make it across the chasm despite their merits or why some standards are adopted over others. Hisbiggest finding is that they were just too hard to use by non-techies. They required too much work to setup, install or learn. Ease-of-use was a principal factor in many technologies not being quickly adopted by organizations that did not want to use technology to differentiate their organization. Early Majority customers also tend to purchase new technology not based on how they work but who is using them in their industry. They purchase on references.

New technologies usually need a focused approach in a specific industry to be adopted by an early majority. If an organization can target a specific vertical industry and provide a more complete solution for that industry, once a few reference accounts are set up the rest of the industry can see there are proven case studies.

Another factor is the need of the early majority for complete solutions. Innovators and Early adopters are willing to glue together new technologies into their existing fabric of solutions. They dedicate time, staff and money to do this. If their XForms sends XML data over REST interfaces and their databases use JDBC they hire programmers to write middle-tier applications.

But early majority buyers can’t justify that additional cost. They just want a solution to work out-of-the-box. They are OK paying for a week for training for their BAs, but they don’t want to write and maintain complex applications.

Another thing that I have been searching for are metaphors that are understandable by non-programmers. Since the high-level decision makers are usually non-technical their ability to make a decision is usually based on their trust of the developers or their understanding based on metaphors.

The Language Translation Metaphor I find that if you can send your data from an XForms application directly to a native XML database in XML format that there is no need for middle-tier objects or shredding into relational databases. No translation is involved. One of the ways to demonstrate how machine-based language translation is not always semantically precise is to try an automatic translation program. Convert a statement from a native language into a foreign language and then translate it back. For example try this using Google Language Tools at http://www.google.com/language_tools

Original Phrase in English: The quick brown fox jumped over the lazy dog.

Google Translation to Spanish El zorro marrón rápido saltó sobre el perro perezoso.

Spanish Translation Back to English The fast brown fox jumped on the sluggish dog.

Although Google translate does a pretty good job, quick was changed to fast, over was changed to on, and lazy was changed to sluggish. If you study semantics, you can find lots of problems trying to communicate with close approximations. Note the original implicit business rule of don’t repeat a letter was also lost. We have many implicit business rules in complex systems and translation of these to other formats is always problematic for software developers.

Saving XML using an object-middle tier is one translation. Saving objects to a relational database is a second translation. Fetching data from a database using SQL is a third translation. Converting tabular data into objects is a fourth translation and converting objects back to XML is a fifth transformation. There are five translations, not just two.

Not that XForms solutions today are not without translation. In our process we are planning to convert XML Schemas into XForms as well as initial instance data. We need to also import leaf-level definitions from a metadata registry. But the nice thing about this process is that we can validate the submitted data directly against the original XML Schema that was used to generate the XForms. An elegant self-checking pattern.

All of these transforms are possible. But all have to be maintained each time you change your XML Schema. Right now we also need great tools that translate XML Schemas directly into XForms. And these need to retain the fidelity of the original intent of the subject matter experts.

So perhaps deploying applications with native XML databases is a critical part of the XForms value proposition.

Moore also noted that both early late majority buyers would be more willing to use a technology if it was buried deeply in a complete solution. That way the IT managers don’t have to sell the technology to management, just the solution. Similarly, once third party developers start building accounting or complete solutions for vertical industry using XForms the buyer will not need to know or care how the solution is being delivered.

Friday, April 06, 2007

Metaphors for Declarative Systems

I am working on my presentation for the 2007 semantic technology conference. My topic is the semantics of declarative languages.

I am attempting to present to business strategists that may not appreciate how powerful the new XForms and XQuery standards are and how entire rich user experience web 2.0 applications can be built using these tools as long as the tools have RESTful interfaces.

My first consideration is to find the right metaphors. I am working with the evolution metaphor and the puzzle metaphor right now and it seems to be working well.

I have also done some research on Domain Specific Languages and I see that they have also used the evolution metaphor but I find that they are focused on building small languages that are specific to a group or project and they don't really look at the external semantics of the problem.

Please write me if you are also trying to explain the business benefits of either declarative languages to a non-technical audience and if you have found metaphors that get the key points across.


Thanks! - Dan

Monday, February 26, 2007

Doing Zero Fill with XSLT 2.0

I was scratching my head, trying to find a simple way to do a zero fill padding with XSLT and I could not find it in my Michael Kay XPath 2.0 book. It turns out he put in in the XSLT 2.0 book. I am not sure how the index missed it. Anyway, searching for "Zero Fill XPath" turned up nothing on Google. Here is how it works. Just use the following:

format-number(number(.), '00000').

Here is an example of the input and output:
Input Output
1 00001
12 00012
123 00123
1234 01234
12345 12345
Hope this helps. - Dan

Thursday, February 22, 2007

The Penguin, The Fox, The Weasel and the Elephant

Once upon a time there were four animals that lived in a magical “Land of Bew”: The Penguin, the Fox, the Weasel and the Elephant.

Each animal had their own way of looking at the world. Penguin and Fox loved to share their ideas.

Weasel also had lots of interesting ideas but he was afraid that if he shared too many ideas his owners would be worried.

Elephant always seemed to be around everywhere but didn’t pick up new ideas very fast.

In the Land of Bew most of the animals got along pretty well with each other. Fox played Penguin’s house and Fox played at Weasel’s house.

Unfortunately the Elephant was not allowed to play at the Penguin’s house. It was not that Elephant was mean, its just that Elephant’s owners were somewhat controlling.

The Land of Bew was blessed with special magic books. These books that allowed the reader to press on special words and pictures in the books and suddenly new pages would magically appear. The children of Bew loved the books and always wanted more.

But some of the books were very hard to make. Especially if you wanted nice moving and dancing pictures that children could change.

To make dancing pictures you had to use special type of black magic that sometimes got the animals in trouble. Goblins would sometimes used the black magic and cause havoc in the Land of Bew. ( I would tell you more about the horrible things Goblins did with the black magic but this IS a children’s story.)

One day Penguin and Fox found a new way of making dancing books that animals could use in magical ways without using the dangerous black magic. Fox tried it out and it worked very well.

But when they suggested their ideas to Elephant, Elephant said no. Elephant was worried that even kids would be able to make the dancing books. And she worried that all her friends would stop playing with Weasel and spend all their time over at Penguins house.

And although the story is not over…this is where our story ends. The ending is being written by you.

Want to try to make some dancing books?

Try XForms today and see if you can make your web applications dance without using any nasty JavaScript! And try to keep those Goblins out. OK?

Thursday, February 15, 2007

The eXist OpenSource Native XML Database

After a suggestion from Kurt Cagle, we have been using the eXist OpenSource XML Database for a week now and I must say that it has exceeded our expectations.
It was a snap to download a set up. It was easy to get all the XQuery demos running and it was almost trivial to administer the little but annoying things like adding collection and users.
Unfortunately I did have to actual read the documentation to figure out how to get our XForms and XMLSpy to work with it. But once you know the secret, it is easy.
The big secret is that by just adding the words "rest" or "webdav" before the "db" in the URL you automatically and to HTTP GETs and HTTP PUTs directly to your data.
You can also actually edit the data directly in the eXist database DIRECTLY from within XMLSpy by just using the "Open URL" option of the XMLSpy editor and then just slip the "webdav" string into your URL path. Hopefully XMLSpy will allow you to use the XQuery editor in future versions of the database.
eXist had given our entire group a large boost of confidence. We can now build a Ruby-on-rails like environment just by drawing XML Schemas, generating XForms from the XML Schemas and storing the form data directly into eXist. A full application development stack without a single like of procedural Java or JavaScript!!!
I posted a short article on how to get XForms working with eXist in the XForms WikiBook
I have a FireFox T-shirt on order for Kurt and I am also buying beers at the first XForms conference...which I may have to organize.
Thanks Kurt! Your advise was very sound.

Computers Don't Create Models (Today) – People Do

I had several good comments about my posting about the increasing role of semantics in the "Era of Tera". I did not mention some of the key concepts that the Intel paper discussed and the impact these concepts will have on IT strategy. Here is an excerpt:

The problem is ordinary computers don't model things. Aside from supercomputers, today's computers aren't capable of developing mathematical models of complex objects, systems or processes. Nor are they powerful or fast enough to perform such tasks at speeds people demand. We can't plug in a statistical model for a rare malignant tumor or the behavioral pattern of a shoplifting employee and search for similar instances of the model in a data set. To benefit from the wealth of data building up in the world, we need to be able to communicate with computers in more abstract terms (high-level concepts or semantics). We need to speak in terms of models.

I believe that the core problem is that there will be a HUGE demand for highly-skilled data-modelers/ontologists in five years. But there will not be a large supply since this skill is not something that can be learned from a single class in college. Rather it is more of a tacit skill that is not easily codifiable.

That being said, there are many best practices that ARE codifiable. You can already purchase books on OWL/RDF and metadata registries, although most of them are mired in relational database models or written by academics with little real-world experience. You can read the Wikipedia articles on “Data Stewardship”. What is still needed is a single set of best practices and tools for creating families of machine-readable data models that can be use as a basis for creating exchange models. And we need to do this without having to learn how to represent all 12 types of UML diagrams in XMI and transform them. The NIEM subset generator is a good example of the solid XML-Schema-driven front-end of this process of selecting elements from a metadata registry and putting them in your shopping cart.

The bottom line is that this is really about empowerment. And unless organizations introduce semantic web technologies into their organizations at a grass-root level and support them at the CEO level, many organizations will be left behind in the Era of the Tera.

Tuesday, February 13, 2007

Era of the Tera

I read in the New York Times yesterday that Intel has produced a chip with 80 core processors giving it a total computing power of 1.3 teraflops. So the natural question is what could you use 80 CPUs for? The article referred to a 2005 Intel Paper titled Recognition, Mining and Synthesis Moves Computers to the Era of Tera by Pradeep Dubey. This article opened with the following quote: “The great strength of computers is that they can reliably manipulate vast amounts of data very quickly. Their great weakness is that they don’t have a clue as to what any of that data actually means.” Stephen Cass, “A Fountain of Knowledge,” IEEE Spectrum, January 2004. When I saw this quote, I realized even the hardware engineers agree semantics is now the critical factor limiting our ability to effectively use computers I realized that we must continue our mission to promote semantic mapping concepts. So let me summarize my feelings after reading this article:

If want to be able to leverage the power coming in the Era of the Tera, you must start with semantics – recoding the meaning of your data.

And if you don’t yet have a metadata registry…get one!

Right next to the CEO, CIO, and CFO in the board room should be your CDO (Chief Data Officer).

Friday, November 24, 2006

Whuffie, Social Networks and Metadata Semantics

I was reading Ambient Findabilty and I saw a reference to a concept called “whuffie”. After checking out the Wikipedia article on whuffie, I discovered that whuffie was reputation-based currency used in Cory Doctorow's sci-fi novel, Down and Out in the Magic Kingdom. This future history book describes a post-scarcity economy where people purchase goods and services based on their reputation, not just cash.

I was interested in this topic because I think that in the future new keywords with precise semantic mappings will be introduced into declarative systems based on social networks. Right now it just takes too long for new tags to appear in an XML standard. Just think of how long it takes for the w3c to create a new XML tag for something like a menu tab! Up to 10 years based on the rate that the XForms standard is being adopted.

Anyway the book was a great mind-opener. One of the first science fictions books that takes nanotechnology and social networking into account. I have always loved cyber-punk: Neil Stephenson’s Diamond Age being one of my favorites. No time-travel or spaceships: just highly imaginative extrapolations of the net and Moore’s Law.

I like the concept of the post-economic society. When all of our economic needs are met what do we strive for? Why don’t they ever complain about their raises in Star Trek? The book also discusses the concept of an adhocracy: groups that quickly band together to solve problems. Doctorow also looks at the concepts of death, backing up your brain to a persistence store (the immortality SAN) and filtering restores to change values.

What does this matter to the metadata architect? It matters because our march away from procedural programming toward declarative systems is totally dependant on the creation of shared semantics. When you need to pick an XML tag how do you pick the tag that will have the highest probability to have semantic precision in the future? Social networks will help us and when we build systems to vote on which tags get approved not all votes should count the same. Votes by people with high-whuffie should get counted a lot more. The concept of one-person, one vote will be modified in the post-economic society.

Sunday, October 29, 2006

My visit to the metadata library

Last night I had the strangest dream…

I was looking for a good mystery, perhaps a Sherlock Holmes book. I went to the library and an elderly lady at the front desk said “Welcome to Roy G. Biv Library. How can I help you?”

“I am looking for a Sherlock Holmes mystery. Do you have one?” I asked.

“What color of book are you looking for?” She asked.

“Well I don’t really care what color it is.” I replied.

“Well in this library, our books are sorted by color of the cover. The red covers are over there, the orange covers over there and the yellow ones right over here. You get the idea. Shelving books by color makes it very easy for us to shelve a book because the color of the cover is obvious. It also makes it easy to see when a book is miss-shelved. You can see the miss-shelved books right away.”

“Yes, I can see how that would make your work much easier. Do you know where I can find the Sherlock Holmes book?”

“Well, it would help us a lot if you could tell us what color the cover is.” She said firmly.

“I don’t know what the color of the cover is…, I just want a mystery book about Sherlock Holmes!” I said exasperated.

“Well, why don’t you just start looking at the reds, then orange, yellow, green, blue, indigo and finally violet. I am sure you will find it in a few days…”

Then I woke up and when to work. There the staff was attempting to store all the data elements in the enterprise sorted by the system name that each data element was first recorded. Then my real nightmare began.

Monday, October 23, 2006

We Declare!

Many of you may be wondering why a data architect has been spending so much time talking about XForms. I have been using XSL for many years to transform XML and I have loved the declarative aspects of XSL. My friend Arun has been challenging me to articulate the key aspects of declarative programming in the context of our join efforts to promote metadata registries. Here is my first version of a Delarative Manifesto.
  1. Declarative development is about the precise association of a programming symbol with the semantics of a design pattern
  2. Declarative approaches leverage the inherent pattern-matching structures of the human brain to allow developers to quickly recognize the underlying patterns used in a system
  3. The rapid association of a symbol in a program to previously encountered problem by a wide audience is essential for developer productivity
  4. Developer productivity is dependant on the efficiency of matching a symbol to a previously encountered approach to a problem
  5. Shared meaning happens when a community agrees that a symbol should be associated with design pattern
  6. Tools that help us quickly promote shared meaning make us all more productive
  7. Published and carefully annotated XML Schemas and formal metadata registries are ways of associating symbols with precise semantics
  8. Developers need tools that allows a new tag to be quickly associated with the semantics of the tag, history of the tag, usage of a tag and similar tags
  9. Declarative programming makes development more accessible to people that don't have a computer science background
  10. Advanced collaborative computing tools, a focus on architectures of participation, and social networking software will dramatically impact our ability to rapidly create precise semantics

Friday, October 13, 2006

The Power of Declarative Programming

This presentation by Steve Pemberton of the W3C on declarative programming is very insightful. Thanks goes to Mark Birbeck for pointing it out. I hope we can get more students exposed to technologies like XForms much earlier in their programming careers. Take this use case. Today we give 10 students a simple assignment to create a tab menu in JavaScript we would get 10 very different programs. But if we gave them the same assignment XForms they would hopefully all use similar toggle-switch structures. This has big implications for maintenance. Hopefully there would be less temptation to recode an XForms application. Since over 50% of software is maintenance this could have long-term impacts on developer productivity.

Wednesday, September 13, 2006

XForms Tutorial and Cookbook

I am speaking tomorrow at the Minnesota Open Source User group (see http://www.josug.org) and I thought I would put some of my examples in Wikibook format. Here is a link: http://en.wikibooks.org/wiki/XForms Let me know if you have any feedback. Thanks - Dan

Tuesday, September 05, 2006

Firefox XForms Extension Rocks!

For the past several months I have been building XFroms prototypes on my current project. I ran into the XForms Firefox plugin two months ago and have been using happily using it ever since. Although the plug-in version number is only "0.6", I have found for my use, it works perfectly. It allows me to quickly test XForms on all the computers I use and I don’t need to setup and configure a web server. The extension takes under 30 seconds to install, but you will have to reboot FireFox once after you download the extension.

Now biggest thing that I am searching for are XForms example programs. The examples provided with some of the books are not very good. I would like to use these samples XForms and teaching XForms to others. I would also like to create XForms directly from the XML Schemas I am building and so I can integrate the hints, help and metadata registry links for each field on the forms I am developing. I am also trying to figure out how to make CSS work correctly across many browsers since many of the XForms CSS templates in the books use CSS3 and CSS3 only works correctly with FireFox.

If you know of any nice collections of XForms for learning XForms, please let me know. I have set up a Wikibook to start collecting tutorials and examples. The link is http://en.wikibooks.org/wiki/XForms

Thursday, August 17, 2006

UNIX Shell Script to Transfer Files via FTP

I recently needed a UNIX shell script to transfer all my files to the web collaboration server running plone. Although you can do this with Ant's FTP command, my remote system just had UNIX. Here is the script I wrote.
#!/bin/sh
# This UNIX shell script FTPs all the files in the input directory to a remote directory
# It uses a UNIX Shell "hear document" to read the commands to the FTP program from the shell file
# You have to use the -n option to the FTP command to disable the prompting for the password
# the quote in front of the ftp user and pass are also necessary to disable reading from stdin
# the prompt must be turned off for the multi-put to not prompt the user
LOCALDIR=mylocaldir
REMOTESERVER=remote.server
REMOTEPATH=/my/remote/path
LOGIN=myloginname
PASSWORD=mypassword

cd $LOCALDIR
ftp -n $REMOTESERVER <<INPUT_END
quote user $LOGIN
quote pass $PASSWORD
cd $REMOTEPATH
prompt off
mput *.*
exit
INPUT_END

Sunday, July 30, 2006

XForms, metadata and MDA

For the last several weeks I have been looking at the XForms standard. This has been related to my contract at the Minnesota Department of Revenue to replace many of their paper forms with electronic versions. Although my contact is to design XML Schemas, ultimately they need to be entered through a web form. I started to build some mock-ups of web forms to clarify requirements but found the older HTLM forms lacking in many areas. I also wanted to be able to validate the data using the XML Schemas that I was building.

What I found was that XForms is a wonderfully designed specification. It integrates XML Schemas, XML data-types and XPath expressions directly into the standard and also adds the new XML Events standard. XForms also faithfully uses both the Model-View-Controller pattern and several separation-of-concerns issues. In short, XForms is an excellent design and appears to be very well thought out. My congrats to the XForms team.

My other worry was that it would be hard to setup and test XForms. I have been happy to find out that FireFox has a new plug-in that supports XForms and there is a plug-in free solution using the FormFaces javascript tools. Many of the examples just run without modification. Just build your XForms and out pops complex XML documents.

What I am most interested in is that there appears to be several things that XForms demonstrates that will make software engineering better: the drive away from the complex procedural world and toward a more model-driven declarative style of development that allows non-programmers to maintain complex systems. This means less complex javascript and complex server-side coding in Java. I hope to have a sample PowerPoint presentation on my web site soon that goes through these arguments. Look for an XForms user interface to data element entry in the near future.

Another last point is that if a large audience starts to build XForms you are going to start to have fully-formed XML documents flying around. This means you will need more XML Schemas being generated and web services to accept them and respond with XML documents and XML services. This will make the creation of new XForms go even faster. Positive feedback cycles. All the more reason to build a metadata registry!

I have been thinking about creating an transform from a constraint schema directly into an XForm. This would be an nice MDA-type approach.

Let me know your thoughts. - Dan

Friday, July 07, 2006

Photo gallery Software

I have been looking into OpenSource photo gallery software. I am a little frustrated by the limitations of FrontPage for publishing my photographs. I would like to just tag my images from the Windows browser and say "publish". I want a thumbnail-drive front page for each section with a variety of viewers (matrix, filmstrip, montage) that can be customized with the FireFox View/Page Style menu. My friend Liela Tite recommended Coppermine. I also ran into these two: Gallery Project - based on the Drupel web content management system. 4Image - which is a system written in Germany. Since the 4Image site was written in German I had a more difficult time trying to find a list of all the features. Both the gallery and Coppermine systems seemed very full-featured. Let me know if anyone has suggestions. A Wikipedia feature-comparison would be very helpful. Perhaps I should start one? :-) OK, OK, I will...

Thursday, July 06, 2006

Thanks Arun!

I recently started a project at the Minnesota Department of Revenue. I was happy to find a kindred spirit in the IT department that also in interested in using Wiki's to manage IT data. Arun has got me to think about the implications of Wiki publishing and how I can integrate metadata into a ESB framework. Check out his web site. It has lots of interesting things. http://netrii.com/display/ABB/Home I was also interested to find that he uses a Wiki for his own personal web site. - Dan

Ant Image Tasks

I was pleasantly surprised to find that Apache Ant has a large library of image processing tasks under the optional Ant tasks. For example, the following will create a folder of thumbnail images for all the images in a folder:

This creates thumbnails of the images and make sure they all fit within the 160x160 size whether the image is portrait or landscape. Very cool! Check out the Ant manual for more information: http://ant.apache.org/manual (Ant tasks/Optional Tasks/Image) This makes me think that you could use Apache ant to do basic photo publishing. It would make thumbnails, a layout page that has links to the thumbnails and ftp the folder's web-resized images to a web server. I have also found that there are some nice photo-layout CSS pages. It would seem that it should be easy to add a set of CSS styles that could be changed using the FireFox "Page Style" option. I would be happy to write a basic ant task and an XML transform to create an HTML page of all the images in an image set if anyone wants to help. What about image keywords and captions? Should each photo have its own photo metadata? What if you want to change the file name of a photo? How can the metadata file stay in sync with the file system? Could you use a subversion-tortoiseSVN-like http://tortoisesvn.tigris.org/ Any ideas?

Tuesday, June 27, 2006

Article on Common Metadata

Here is an interesting article on the importance of common metadata: Common Metadata The author appears to understand the importance of trust and metadata but appear to not be familiar with metadata standards such as ISO/IEC 11179 He also did not reference the work of Goldberg and Rubin in their 1995 book: Succeeding With Objects: Decision Frameworks for Project Management Adele Goldberg, Kenneth S. Rubin Addison-Wesley Professional; May 1995 See my href="http://www.danmccreary.com">book review for more. But all-in-all, it was a very good article.

Saturday, April 22, 2006

Natural Language Interface to Wikipedia

I have been saying for a while that because wikipedia is a database it is much easier to link it into a natual language seach system. The Wikipedia search function is also very limted and does not allow you to type in complete questions like ask.com. I reciently found out that MIT is already building a very powerful natuaral language interface in their START project. http://start.csail.mit.edu I did a search for "Who was Tesla?" and got a link right to the Wikipedia article. Very cool! Although there seems to be very limited documentation on START and I do not believe that there is an OpenSource version, I believe this points to the direction that wiki software should be moving. Could your relational database bennefit from this? When I was at the Minnesota Department of Education I wanted to build a natural language interface to their "cubes" of student test data. For example "What school district has the highest highest 8th grade math scores for girls?" Give it a try and let me know what you think.