New Arduino Moving Rainbow Lab for Kids for Under $10

After participating as a mentor in the local Twin Cities CoderDojo  in Minneapolis at the University of Minnesota I found that there are two things that kids love: motion and color.  My prior experience with kids is that they love to take projects home with them to show their friends.  So here is the question - could we design an Arduino kit that kids could come to Maker events and actually take a working project home that has both color and motion?

At our local CoderDojo we wisely use the SparkFun Inventor's Kit.  If you have not seen this I encourage you to take a look.  Lots of nice components and a great guide (PDF). Many people undervalue the amount of work going into the guides for these kits.  Good writing is hard to find, are rarely included in many lower-cost projects.  However, since the cost of full Inventors Kit is still around $100 (without shipping) it is beyond the budget of most Maker events to let kids take these home.  So here is my answer: lets find a way to use the low-cost Arduino Compatible Pro Mini combined with a short strip of addressable LEDs in a kit that we could sell for around $10.

Here is a picture of my initial design:

Arduino Moving Rainbow Kit

I think this kit is perfect for kids since we have the wonderful colors of these amazing individually addressable LEDs as well as the ability to create "motion" by programs such as running lights.  The total cost of this should be around $10, depending on the options we use.

Now lets go through the components to verify that we can do this in the $10 price range.

Here is a place on eBay that sells the Pro Mini Arduino Compatible for around $2.00. Although I suspect that we could get them in quantity for less.  Note that these do not have a USB port on them.  More about that later.

Here is an addressable LED strip with 60 LEDs that we can find on eBay for around $17.00. You can cut these up into 6 strips of 10 LEDs each which gives you a price per kit of around $2.80. What is wonderful about these LED strips is that they only take two power pins and a single data pin to hook up!  I use the standard WS2811B which you can program with many libraries.  I have some sample code on github that uses the Adafruit NeoPixel libraries, however I have found other libraries work well also.  Keeping wiring simple is important for kids that don't yet have the fine motor skills (and patience) to wire up complex projects.

Now the last two components are the mini soderless breadboad (around $.70 in quantity), a switch and a power source.  In the kit in the photo above I am using a 3.7volt Lithium Poly battery taken from an old RC helicopter that no longer works. We could also use an external 3 AA or AAA battery holder for around $1.

I also added a power switch and a clear plastic polystyrene box I purchased at the Container Store, although a small plastic leftover box will also work well.

Arduino Pro Mini Rainbow Kit Parts List

1. Arduino Compatible Pro Mini $2.00
2. 10 element LED strip $2.70
3. Solderless mini breadboard $0.80
4. Battery holder $0.70
5. Plastic Polystyrene Box $2.50
6. Wire $.50
7. On-off switch $1.00

Total: $11.40 (with 3 AA batteries)  I have a detailed parts list here

I also suspect that if we purchase these in quantities of 20 or more the prices would come down a bit and we could get in under $10.

The biggest challenge is that each station must be equipped with a laptop with a FTDI basic breakout programmer with the Arduino software pre-loaded. The Arduino Uno (which runs about $25) does not need the FTDI programmer, but would be 10x more expensive for each Rainbow kit.  Getting these setup and configured is not easy.

I should also acknowledge that when we purchase items off of eBay or other sources, we are not using the official mini supplied by the fine people at Arduino.  However at over $20 these devices would not fit into our under $10 budget.  I still think that for other projects we should encourage people to use original Arduino hardware since the costs go to promote further open Arduino projects and quality educational materials.

Let me know your thoughts on this.  Getting high-quality clear plastic boxes that kids could take home, throw in their back and show their friends is still something I am working on.  I found one source here that I might try but any suggestions you have would be great.  I know other projects use simple baggies for their parts but these seem hard to use and show to others.

There are also other options is to provide a "night light" mode that you could plug into a wall outlet and not have to use batteries.  I have seen 5v USB "wall warts" for under $1.

Although I know that this is a very small project to get kids started in Arduino, I think it might be the platform that other projects could be added.  Creating a clock (12 LEDs required), putting sensors on to display temperature or adding accelerometers and magnetometers might be the next step for these kids.  The key is that they could take them home and show their friends and that keeps them interested and motivate to continue their work.

I also want to thank Gerd Knops for helping me get started on using the LED strips.  He is a brilliant engineer, a great friend and wonderful mentor to me.  I hope that what I learn from Gerd I can use to get more kids interested in science, technology, engineering and math.  We don't really have much time left to do this.

Agile Transformation in the Post NoSQL Era

Over the past four years we have seen the NoSQL movement grow from a small "Meetup" in the Bay Area to a technology that is touching all corners of the database world.  Each year NoSQL software becomes more capable, lower cost and easier to use. Document stores, in particular, make the process of doing object-relational mapping unnecessary allowing anyone with god metadata (JSON and XML) to simply drag-and-drop their files into a centralized corporate data store.  There are still a few challenges left.  For example using statistical analysis of inserts and record counts to optimized indexes and putting good query languages (like JSONiq) on top of these data stores.  But in large, these features are just polish on systems that are optimized to scale and be highly-available.  The hard work seems like it has been done and we are now in the stage of refinement, not revolution.

We documented the emergence of the NoSQL database patterns in our book, Making Sense of NoSQL, which now available through Manning Publications.  If you read this book you know that NoSQL systems have a diverse set of architectural patterns and different patterns apply to different problems.  Selecting the right database architecture is a complex process of carefully understanding the subtleties of requirements and weighing the alternatives.  Yet they do work well and once they are setup and configured they make data persistence a straightforward process.

So whats next?

Now that saving data has shifted from a project in its own to a smaller part of the application developer project we see the skills needed to build applications starting to shift.  The need to model your data with ER modeling tools is getting less.  The need to write complex joins with SQL is no longer needed.  The next major skill set we would like to address is the the movement to agile transformation.  How do you get you data out of your database and how do you transform it into the many formats that your application needs?

We think that the answer to this question is clear.  Organizations need to be better at transforming the data in their database to other forms.  This is the shift in skill sets from persistence centric to transformation centric.  And it is not just the software developers that need to be able to transform data.  Everyone on your team including non-programmers can play a role.  They all need skills to quickly transform data from one form to another.

We call this shift the movement to toward Agile Transformation.  We hope to document how organizations are waking up to this new movement and understand the tools and processes they are adopting to empower everyone on their team to quickly transform data.

This process of data extraction and transformation used to be a two-step process.  SQL developers might create a series of tabular reports.  These reports were then converted into the medium needed, HTML, XML, JSON, or even CSV files and other structures needed by other tools.  Now the extract and transformation process can be done with a single step.  Query results are no longer restricted to tabular formats.

Strategies for Agile Transformation

Over the next few months (or perhaps years), we hope to document many of the ways that organizations are attacking the agility challenges.  Here are just a few strategies to get us started.

Single Source Canonical Data Models

If you are in the content management business you know that the concept of single-source publishing is central to your productivity.  Using a single format to store content gets around the many-to-many transform problems that can drag down a teams productivity.  We see the same principals also applying to web applications in general.  Getting many data sources into a single format and then transforming this single format into many forms is the key to organizational productivity.  We call these models "Canonical" since they are the standards that organization can build publish/subscribe web services around.

Flexible Query Languages

If you have every worked with tools like XQuery and JSONiq you know that they are the most flexible query languages around.  These languages have benefited from years of work combining the best features of SQL, XSLT, XPath and dozens of other advanced query languages into a grammar that is designed to transform a variety of use cases.

Reusable Transformation Libraries

One of the first strategies that companies find is that many transformations are similar and can benefit from reusable code.  Languages like SQL do offer a wide variety of non-portable stored procedures.  Yet most of these languages limit your ability to build reusable transformation functions and modules.  Modern languages need to be close enough to your data to understand how queries use indexes but abstract enough to be reused in new applications.

Using Great Tools: IDEs and Report Writers

SQL GUI Report Writers were one of the first tools that tool the complexity out of transforming tabular data.  And we need more tools like these to make NoSQL reporting accessible to non-programmers.  Some NoSQL products like HBASE already have SQL-like query tools.  From our other blog posts you may know that we are big fans of the the oXygen IDE for managing JSON and XML data.  oXygen makes the process of learning how to write XPath expressions easy for even the non-programmer.  Even if their data is complex.  These tools are complex in themselves and require hands-on training if non-programmers are going to get the most out of them.

Simplicity for Non-Programmers

One of the core foundations of agility is not have all your transformation be controlled by a small group of overworked developers.  We learned that simple tools like GUI-based report writers or simple XPath templates can empower a non-programmer, with a bit of training, to build and maintain their own data transformations.  Not needing to understand database joins is a big step in empowerment.  Getting a good foundation library is another great step.  Setting up small but easy to use templates is another good strategy.  Building a search system to find the right library and templates also helps empower new staff and lowers the training burden on existing staff.  In general, we feel that if a user "knows their data" that they should be given the tools to transform their data.

So what is the best practices to build an organization that has agile transformation competency? We would love to know your ideas.  Please send us email or tweet us at @dmccreary on Twitter.

Rounding error in Java when converting strings to doubles

I came across a rounding error when I was running an XQuery.

Here is the error:

xquery version "1.0";
number(3.1) + number(3.2)

which returned:

        6.300000000000001

Not the expected value of 6.3.  Note that the XQuery function number() returns double precision data.

After a note from Eric Bruchez he suggested I cast the numbers to decimal:

xquery version "1.0";
xs:decimal(3.1) + xs:decimal(3.2)

and the problem seems to go away.  He also showed that he could reproduce this error in other Java JVM languages like Scala.

Let me know if anyone else has seen this problem before and has any other suggestions for a fix.

Thanks! - Dan

Analytical Reports for Technical Books

We are wrapping up our work on our book "Makings Sense of NoSQL" and I have created a series of XQuery reports on our book that we have found useful.

Our book is stored in XML DocBook format, which for those of you that have not used it, is ideal for technical books. DocBook contains elements for almost everything you need in a technical book including chapters, sections, paragraphs, figures, glossary terms, bibliographic references, and index terms. In short DocBook is is the perfect fit for most technical books and it can easily be extended. One key aspect about DocBook is that it is easy to transform into multiple formats such as HTML, PDF or ePub. There are many open source transforms available for DocBook. DocBook is at the heart of the movement into single-source publishing for technical publications.

In additional to the standard DocBook transforms we also created a series of reports on the book and I thought they might be of interested to others.  Here is a summary of some of these reports.

Chapter length report

When we started writing our book our goal was to produce a 310 page book with 12 chapters, each with approximately equal length.  But logically we wanted our fist chapter to be a brief overview and we found that the chapter that described the core NoSQL patterns had more content.

This is a horizontal bar chart that shows the length of each chapter. The goal is that all chapters be roughly the same size in length.

Chapter Length Report

You can see that one chapter in our book (chapter 4)  is a bit longer then the rest of the chapters.  We knew this up front after running this report and were prepared to defined our decisions with our editors.

The "Naughty words" report

We quickly found out that editors have some words that they don't want to see in a technical book. Words such as "just" or "very" should be used with great caution. There were also words that were not allowed "vs.", "e.g.", "etc.") according to the style guide.  This report shows you how often you use these words in each chapter.

Report counting specific words in each chapter

Book metrics report

This is just a raw counts of elements such as book parts, chapters, sections, figures and tables etc.

Book Metrics Report

Chapter metrics report

For each chapter I created a detailed report of the content. As you are writing each chapter, other people can view your progress by running this report.  I tend to put in outlines first, then figures and then the actual text of each chapter.

List of figures and tables by chapter

These reports shows a listing of figures and tables sorted by chapter and location in each chapter. It also shows the caption for each figure and a thumbnail of the image. There are versions that show the type of figure (line-art vs. bitmapped) the sources of each figure.

List of Figures Report

Figure and table captions reports

Editors want to make sure a book is "browsable" which means that every other page has an interesting figure that people will see when they flip through the book.  Each of these reports can list the length of the figures and table captions sorted by the length of the caption.  The captions that are too short will need further work.

Paragraph and sentence length histogram reports

There are guidelines that should be used when writing technical books on sentence and paragraph length. This report shows the distribution of paragraph lengths in each chapter. You should work with your editor to find reasonable guidelines for your audience.

Paragraph Size Distribution Report

Table of contents reports

For reviewing the book structure it is always nice to have reports that list the book's structure by chapter, section, sub-section and sub-sub-section. These reports come with several parameters that limit the depth of the table of content and can also be modified to create mind maps using open source mind mapping tools.

Table of contents showing Parts, Chapters, Section 1 and Section 2

Once I wrote the basic structure of the table-of-contents report it was easy to create other output formats for different functions.  Here is an example of a GraphViz output.

Book Outline using GraphViz format.

This example used the DOTML markup format and an external transform using the Chris Wallace graphviz XQuery module.

You can also convert the table of contents into an Mindmap file and open the file in FreeMind or XMind.

Here is an example of the book rendered in XMind.  NoSQL Book MindMap

Glossary of terms reports

Our book puts a focus on the terminology used in the NoSQL movement. We try to create precise definitions of all the terms we use and discuss the variations in definitions in different NoSQL communities.  I use these reports to list each time a term is first introduced and make sure that we have a formal definition in the Glossary Appendix at the end of the book.

Glossary Term Report Showing Glossterm IDs and Definition Status

There are also other miscellaneous reports listing the introductory chapter quotes, lists of comments by reviewer (extracted from PDF using Apache Tika), and various tools to help us gauge the completeness of each chapter.

Report Strategy

We created a central XQuery book module that had all the common functions such as $book:chapters, or  that returned a sequence of all book chapters or book:word-count($node) that returned a word count of a node.  I used the eXist-db database to store and execute the transforms.  After we created the module the templates could be quickly customized for each report.  We also used the oXygen XML IDE extensively and we want to thank George Bina for his support of our project.

I think that DocBook, oXygen, XQuery and eXist are ideal tools for managing the book creation process.

Cognitive Bias in NoSQL System Selection

After attending the Saturn 2013 Conference I was exposed to the use of "Cognitive Bias" in software architecture.

Here are some examples of cognitive bias I have seen as applied to the world of NoSQL database selection.

Anchoring bias - the tendency to produce an estimate near a cue amount - "Our managers were expecting an RDBMS solution so that’s what we gave them."

Availability heuristic - the tendency to estimate that what is easily remembered is more likely than that which is not. - "I hear that NoSQL does not support ACID." or "I hear that XML is verbose?"

Bandwagon effect - the tendency to do or believe what others do or believe - "Everyone else at this company and in our local area uses RDBMSs."

Confirmation bias - the tendency to seek out only that information that supports one's preconceptions – "We only read posts from the Oracle|Microsoft|IBM groups."

Framing effect - the tendency to react to how information is framed, beyond its factual content "We know of some NoSQL projects that failed."

Gambler's fallacy (aka sunk cost bias) the failure to reset one's expectations based on one's current situation – "We already paid for our Oracle|Microsoft|IBM license so why spend more money?"

Hindsight bias - the tendency to assess one's previous decisions as more efficacious than they were – "Our last five systems worked on RDBMS solutions".

Halo effect - the tendency to attribute unverified capabilities in a person based on an observed capability. – "Oracle|Microsoft|IBM sells billions of dollars of licenses each year, how could so many people be wrong". 

Representativeness heuristic - the tendency to judge something as belonging to a class based on a few salient characteristics  - "Our accounting systems work on RDBMS so why not our product search?"

Thanks to everyone at CMU/SEI and the SATURN Conference for the exposure to these ideas.

How to pass oXygen XML IDE Editor variables into an XQuery script.

I am a big fan of the oXygen XML Integrated Development Environment.  It has many powerful features and I learn about new ones the more I use it.  I recently started a project that required me to run XQuery transform on hard-drive files using Saxon HE.  Since I normally run XQuery from within the eXist-db this was a new challenge for me.

I have many XML files and many different XQueries and I wanted to run them directly from test cases in an oXygen project.  But unlike XSLT, XQuery does not have a concept of "standard input".  You need to specify a path to the document within the XQuery file itself.  So how to pass the file name from the project directly to Saxon?  Luckily, the guys from Syncro Soft (the authors of oXygen) thought about this.  You can use oXygen "Editor Variables" to be parameters to your XQuery.

Here is a sample program that reads two external variables, one for the project director and one for the current file name (with extension).

XQuery Program

xquery version "1.0";

(: Example of how to access external "Editor Variable" variables passed

  by oXygen. This one example gets us to a full file path within a

  project.  See:

http://www.oxygenxml.com/doc/ug-editor/topics/editor-variables.html 

  for other editor variables.
:)

(: oXygen project directory :)
declare variable $pd as xs:string external;

(: file name with extension :)
declare variable $cfne as xs:string external;

(: note the triple forward slashes before the file. :)
let $path := concat('file:///', $pd, '\', $cfne)

return
<results>
   <project-dir>{$pd}</project-dir>
   <file-name>{$cfne}</file-name>
   <file-path>{$path}</file-path>
   <doc-available>{doc-available($path)}</doc-available>
</results>

Note that we put the project and file together to get the path to the file and use the doc-available() function to verify that the file is there.

Next, we set up a transformation scenario and put the file above in the transform URL.  We have to add the parameters to the transform.  oXygen parses your XQuery for all external variables and puts them in each line for you.  For each variable put in the same variable with the ${pd} notation where the dollar sign is outside the curly braces (opposite of XQuery).

Here is a screen image of these parameters and their values.

Results

When we run this transform on any file on Windows we get the following:

<results>
   <project-dir>D:\ws\project\subdir</project-dir>
   <file-name>my-input-file.xml</file-name>
   <file-path>file:///D:\ws\project\subdir\my-input-file.xml</file-path>
   <doc-available>true</doc-available>
</results>

On Mac and UNIX systems we would not use the file:/// prefix and the separators will be forward slashes.

NoSQL book now available in MEAP

The book that Ann Kelly (my wife) and I have been working on for the last year is now available on the Manning Early Access Program (MEAP). Here is the link:

http://manning.com/mccreary

The final title of the book will be:

Making Sense of NoSQL: A guide for managers and the rest of us

You can download chapter 1 for free but to get the other three chapters you will need to be part of the MEAP program which is priced at $23.99 for the Ebook only and $29.99 if you would also like a printed copy sent to you when the book is done.

We are looking for reviewers of the book. We are especially interested in finding anyone that has an interest in how to explain NoSQL concepts to non-programmers. This includes Managers, Project Managers, Business Analysts and anyone else that is interested in using NoSQL.

I believe this this is one of the first NoSQL books that is targeting managers and other non-programmers. In order to write to a general audience we are trying to use extensive tables and annotated graphics to make a highly readable book that give people an overall understanding of the NoSQL movement. Much of these tools have been developed by Ann and myself over the last several years trying to explain NoSQL concepts as part of our consulting business.

If you know of good NoSQL metaphors or even NoSQL jokes that we can include in the book please let us know! We will be very involved in the Manning Author feedback program.

AnyChart Case Study

I just released a case study (click on the blog post title or click here) using the commercial software charting package called "AnyChart". After using over a dozen charting solutions including Google Charts and FreeCharts I really feel that AnyChart has the best solution that really scales up.  Not only does AnyChart have a large number of charts but it is also very reasonably priced at around $500USD per server. Because I store all the chart specification and implementations in XML and I use eXist to manage the charts it is easy to manage the full lifecycle from requirements to testing all within a single Lucene-search powered environment.

The latest release of AnyChart also generates SVG and HTML5 compliant charts that run on the iPhone and iPad. This really shows how "declarative" systems are much more portable than any flash-based implementations.

If you are looking for an inexpensive but powerful tool for building charts and dashboards I would defiantly give them a try.

You can get samples at their web site at anychart.com

The "Pathification" of NoSQL

In my consulting work I try to focus on the empowerment of the non-programmer to build and maintain their own web applications. This is not an easy task. There are many barriers, some political and some technical. But when a new Business Analysts or Project Manager can create their own metadata management tools and begin to tell others their story I have great job satisfaction.

My current best-practice is to teach non-programmers how to write XPath expressions into their data sets. I tell them that if they are committed to "knowing their data" that I can teach them how to write web-applications in a week or so. The trick is to give them a "template" CRUDS application (Create, Read, Update, Delete, Search) that they can just plug in their own XPath expressions to get the application to work. They don't need to know the thousands of features of XQuery and native XML databases, they just need to know their data, how to write some simple XPath expressions and where to put these expressions into the templates.

I call this process "Pathification", a word that is similar to "Simplification" and "Standardization" but it has a focus on only teaching a very small subset of the many possible tools and a focus on cross database standards like XPath. Once they feel confident they can create simple CRUDS applications in a few hours they are then highly motivated to continue learning XQuery, but on their own pace and balanced with their other job responsibilities like data governance and semantics.

Pathification is something that can happen because native XML database have many enabling technologies. They have WebDAV drivers that makes you database look like folders on your desktop. We have tools that automatically convert XML into HTML and XForms. We have subversion libraries that allow your form "save" to go directly into a version control system. And these tools continually get better and make it easier to empower novice users.

The NoSQL movement is a huge and diverse community. And many people have focused on areas such as scalability and the integration of different types of data into a single XML development environment. These are all great efforts and really move us in the right direction. But I think for us to make NoSQL into a true multi-vendor, multi-product development platform is to create more tools that empower the non-programmer. For people in the native XML world we have a great start. Standards like EXPath, new application packaging tools and XRX application repositories will continue to make it easier for non-programmers to reuse and share their applications.

What I would like to see (but I am not sure is possible) is both XML and JSON-centric system to start to come together and share their ideas on how to make this happen. Can JSON data-stores like MongoDB and CouchDB be "Pathified"? Can we create shared applications that work with both types of back-end systems?

If we do this right, I think the world will be a much better place. People will have more choices for managing both structured and semi-structured data and non-programmers will be able to take a spreadsheet or new data feed in a standard format and build a new web application that manages this data in just a few hours without having to depend on other programmers. I think of this as a system as easy to use as Microsoft Access(TM) but without the chaos: All data can be stored on centralized servers with REST service-oriented interfaces for every data set.

Please let me know your thoughts on these ideas.

McCreary's Law

I am just returning from a three-day conference on Enterprise Information Management. I was very pleasantly surprised to meet many other very smart and warm independent consultants that do metadata and EIM consulting. More posts about what they taught me will be coming.

But these conferences for me to constantly answer the same question over and over again. What do I do an why don't I use traditional RDBMS systems to store metadata? What I need is some strong statement that explains the core principal of the need for agility in metadata management. So in that context, I would like to suggest a law of agile software development. To give it a label, I will just use McCreary's Law for now.

McCreary's Law states the following:

The agility of any software project in inversely proportional to the square of the number of data transformations in the developer stack.

The number of data transformation in the traditional three tier-architecture (Web to Objects to RDBMS to Objects and back to web) is four.

The number of data transformation in the traditional three-tier plus XML web services is six, since web services are usually created by one transform from objects to XML and one from XML to objects.

The number of data transformations in XRX is approximately zero, since we could make an argument that XML to HTML is a 1/2 translation since the HTML and XML tags are different. Of course if you are not changing the order all XML can be styled with CSS which is not really a transformation. This gets around the problem of infinite developer agility if the denominator is zero.

This formula is not perfect and I am sure their will be modifications to it like adding a constant for training and IT resistance to change, but I hope people can use it to predict or explain their current development agility.

Thanks to everyone I met at the EIM conference that listened to my rants and have not discarded me as a complete metadata radical.

XLucene

For the past two years I have been quietly observing the growth of something called Structured Search. This is the use of document structure to aid in search and retrieval of documents or sub-documents in large document collections.

I have also seen a common solution to this problem. This is to use a combination of native XML database and the Lucene keyword indexing problem. But people doing this are doing far more than just simple keyword search. So calling this new approach simply “Lucene” keyword search does not really describe what is going on. So I am proposing an new meme: XLucene.

I define XLucene as using a combination of document structure and keyword searches together to create very precise search ranking or search hit scores. The key is that the design must be able to store each branch in the document tree as a separate subdocument.

The key factor in widespread adoption is that almost anyone that can identify the tag names in their document structures will soon be able to create their own XLucene search and retrieval systems.

What first got me interested in “Structure Retrieval” was the book “Introduction to Information Retrieval” by Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze. In this book they present the following table:

	RDB search	unstructured retrieval	structured retrieval
objects	records	unstructured documents	trees with text at leaves
model	relational model	vector space & others	?
main data structure	table	inverted index	?
queries	SQL	free text queries	?

The following quote follows this table: There is no consensus yet as to which methods work best for structured retrieval although many researchers believe that XQuery will become the standard for structured queries.

I think that the jury has returned their verdict. XQuery wins! Here is that final table with the following properties:

Objects: Trees with text at leaves
Model: Hierarchical documents
Main-data-structure: trees and inverted indexes tied to node-ids
Queries: W3C XQuery with fulltext extensions

The design of using variable-length node-ids as the document identifiers was done by Wolfgang Meier of the eXist-db.org project. There are many people now using this design today. I will be following up this posting with some case studies of how people are using this in several areas, both in the government sector, library metadata and document search and retrieval.

Since there are so many people using this same problem-solution pair, that like all good patterns, it deserves its own name. XLucene (with no embedded dash) can be picked up by search engines and we can all start to share design experience and solutions. This is why the Open Source model really is superior to closed systems. Knowledge just travels faster once the memes are created.

I will also be presenting more details on the general topic of Structured Search and XLucene at the Enterprise Information Management Conference in Toronto next Tuesday, Sept 21st.

What Makes a Good Solution Architect?

A recruiter friend of mine was looking for a senior "Solution Architect" and needed some help setting up a screening interview. Here is what I told him.

I look for solid experience in a diversity of solution. There are just too many people that think that HTML/Objects and RDBMS systems are the only tools in your architect's toolkit.

Here is what I suggested the interview questions would be like:

Consider the following application architectures:

1. Ruby on Rails on a relational database
2. XRX on a native XML database
3. A Java Client on J2EE app server using JMS
4. A JQuery client on an OLAP database

A senior "Solution Architect" should be able to compare the pros and cons of each system and describe how they have used each of these architectures to build solutions and give organizations a competitive advantage. I also drill down into their understanding of search and retrieval and natural language processing.

As a bonus question, I would ask how each architecture would impact an organization's business strategy and to what extent business units could be empowered to build their own applications with minimal training and little or no need for IT involvement.

Most good solution architects should have experience with 3 out of 4 solution architectures but very few usually have a strong MBA-type background to understand how solution architectures impact business strategy. And if you find anyone that has built systems with all for and has an MBA you have a serious senior solution architect. Hire them!

Impressions of SemTech 2010

Two weeks ago I attended my fifth Semantic Technology conference in San Francisco. It was a great conference! This is my fifth time attending the conference and I plan to attend in the future! My biggest dilemma was which session to attend. At times there were as many as eight concurrent sessions going on.

There were a few big trends that I spotted.

The use of RDFa to annotate web pages using semantically precise elements was clearly a big trend. Jay Myer of BestBuy described how the BestBuy sales went up 30% when they added RDFa tags to their product pages. Although many search engines are not transparent about their use of RDFa tags in page rankings, Jay’s results should make it clear that this strategy works.

Jay told a great story about how hard it was to find a fridge that met his specific criteria: black with a specific size etc. using a keyword-based search engine. The semantic web will change all of this!

The big factor here is Martin Hepps GoodRelations ontology for products and services. This can really bring the Semantic Web to the masses. Finally a way to code the hours of operation for your store so that you can use Siri to ask "What Sushi Restrauants are Open at 10pm Near Here"

There were also a lot of sessions on LinkedData and specifically on LinkedData in the government area. Jim Heldler and his students at RPI are scraping every data set on data.gov and converting them all to RDF and storing them in a huge triple store.

Jim also provided one of the best sound-bites from the conference:

“Get AWAY from the Table”

This goes far beyond the NOSQL movement that we are seeing. It gets to the core of the problems with innovation in many organizations.

Jim is a consultant to the US data.gov transparency process and a key advocate of open linked data standards. It is interesting to see a friendly rivalry between Jim and Tim Berners-Lee who is a consultant for the UK data transparency movement. Both are using RDF to convert data but each is using slightly different strategic approaches.

He was referring to the fact that many organizations over-use the relational model and they need to understand that there are many alternatives, especially when doing mashups of data sets from many sources.

There were also many presentations on natural language processing and finding the true “meaning” of words within unstructured and semi-structured data sets.

I always attend Brian Sletton’s session on REST. Many people do not understand the relationship between REST, URI’s and the semantic web stack. Joe Wicentowski’s work at the US Department of State on the URL rewriting frameworks within eXist has really reinforced how this can be done easily in a single XQuery module.

I presented a 3.5 hour tutorial session on Entity Extraction. Since it was scheduled for the last session of the last day of the conference I though the attendance would be very low. But the room was packed and most of the people stayed till the very end.

It was wonderful to finally meet Marie Wallace and DJ McCloskey from the LanguageWare team at IBM. Their support of the Apache UIMA standards will be a great step forward to the creation of interoperable language analysis piplelines. Marie and DJ introduced me to the Millennium restaurant just a few blocks from the hotel. My wife Ann and I went there over the weekend and we now own one of their cookbooks.

By the way Google is now also support recipes in their rich snippets in their search results so you will soon be able to find "only recipes that take under 30 minutes" in a Google search engine.

There were also many session on the need for good taxonomies and ontologies in the enterprise. The use of SKOS for controlled vocabularies was a very hot topic and there were dozens of presentation on how OWL is being used to capture and exchange business rules.

It was also really great to finally meet Jeni Tennison after reading her books and following her Tweets for a long time. She is deeply involved in the federal data transparency projects in the UK. Here use of RDF is breaking new best practices for the entire community.

It was also great to catch up with Mark Birbeck, one of the chief architects of the Ubiquity XForms libraries. I am looking forward to trying out some of the new tools based on his backplane JavaScript libraries.

The people from Facebook also gave a presentation on how they are adding the ability for people to add a few lines to each web site to allow users to add a “like” button on a web site using a variation of RDF. Many people were a little disappointed to hear that they will not be using namespaces in their interfaces. Facebook felt that host HTML coders could only handle a single namespace. Many people agreed with their findings and thought that until 90% of HTML coders knew what namespaces were that organizations like Facebook were stuck just adding new data to HTML meta elements.

One of the most interesting discussions I had was with the people from Cray Computer. Apparently a large unnamed government agency has given Cray Research a very large contract to build customized ASIC (FPGA) chips to do graph analysis. Their ThreadStorm XMT architecture has 128 register sets and allows the CPUs to get continuous feeds of graph queries without waiting for memory. Their claim is that the federal agency has told them they are getting a 100X improvement in complex graph queries. The challenge is that the API is currently a low-level C interface and they have not yet put a SPARQL complier in front of an XML. So on good SPARQL benchmarks are yet available to compare the results of this hardware with a typical triple store. But even if it is fast, the price would be in the six digits for one of the XMP systems. So only very large organizations and government agencies might use this unless it was provided as a service.

One of the other things I found was how many people are using OWL and the OWL reasonerers like Pellet as replacements for traditional rules engines from companies like FairIsaac. There are two reasons for this. First is that many rules engine companies only talk about their engine performance but not on the need for precise semantics in the data elements. With OWL and the methods behind the semantic web stack we attempt to put semantics higher in the requirements of a system a use pre-built and pretested ontologies.

The second big reason that people are use OWL is that the rules are created using open standards. That means you are not locked into any one vendors rule format. The new W3C standards for rule interchange (RIF) will also start to break down many of the barriers to the creation and exchange of large industry vertical rule sets to that each organization does not have to start from scratch with a rule base. This will be big for industries like insurance where claims processing ontologies are just being created. Thanks to Kendal Clark for good information on this topic.

I also had a great time talking to many people in the publishing area. Seth Maislin gave a great presentation on Taxonomies. Seth and Marlene Rockmore joined me for a lunch at nearby Indian restaurant. Marlene is an expert on Taxonomy development. She was working on a book with O’Relly on taxonomies that is on hold for now but we hope to see one in the future from her.

It was also nice to see how many federal agencies are now starting to use SPARQL in the intelligence community. I saw a good presentation by Dennis Wisnosky with the US DOD talk about how they plan to use NIEM and semantic technologies to cut their integration costs. Yeah NIEM! Most people don’t understand that the NEIM is based on the RDF model, it just uses XML syntax to store the relationships. See my web-cast on the SemanticUniverse.com web site for more details. Hopefully his slides will be available in the future. Many people were taking photos of the slides since the DOD is not great at getting their slides out. Wonder why?

I also had lunch with several people from within the intelligence community that had great discussions about the pros and cost of pulling assertions out of NLP-analyzed annotated XML documents and the challenges relating to this. Keeping the links back to the original XML documents to is critical to verify the results of an assertion in context. Traceability, linage, provenance and time-domain representations in RDF that don’t cause a 10X growth is the number of triples is a difficult problem.

If there was some way to relate RDF assertion to a source document XML Node ID (the fourth column question) is a very difficult problem and one that needs close research with native XML and RDF systems. One of the things I learned about the eXist-Lucene integration done by Wolfgang Meier is that keeping the node-id as the document ID in Lucene allows non-programmers the ability to configure customized search rank rules past on the context of the keyword in a document. This is where highly customizable structured search rocks. I think this innovation needs to be added to RDF triples projects. Keeping the XML node-id of an assertion with an RDF triple context could be a great way to prevent RDF triple bloom for context.

Finally, I felt that David Wood and James Leigh's presentation on Callimacus project had the most potential to have a "big impact" on the adoption of Semantic Web tools. This new open source project allows users to specify a simple HTML template with special XML tags that allows them to bring triples directly into a web application. This has the potential to allow far more non-SPARQL programmers to integrate RDF data directly into a web page much like XQuery does today. With some work it might be possible to auto-generate the XForms bind elements for form rules. This would be a huge win for the integration of rules into web forms without needing to write JavaScript.

You can also see my tweets on the conference here: http://twitter.com/dmccreary

I hope to see more of you in San Francisco next year. Let me know if you are interested in co-presenting any papers!

Problems with WebDAV on Vista 64bit

Does anyone have any suggestions on a good reliable WebDAV product for Vista 64bit? I can not get the one that is bundled with Vista to work. Every time I try to create a remote connection I get the following error message:

"The folder you entered does not appear to be valid"

I have run the Vista WebDAV patches:

KB907306

I have also tried the following Hotfix:

Windows Vista WebDAV Hotfix

I have also tried to get various 3rd party packages working such as Independent DAV, BitKinex and WebDrive. All of the WebDAV products seem to have problems with the Windows Vista 64 bit systems. Is there any way to get Vista to use the Windows XP WebDAV "drivers"? I am using both eXist and MarkLogic native XML databases. Thanks - Dan

How to compile the eXist 1.3 build

I have been learning how to use higher-order functions in XQuery. I am using the eXist system to test this. To do this you will need to get the 1.3 build that there is no download for (yet). Here are the steps I used:

1. Download fresh copy of eXist from svn via TortoiseSVN or Eclipse subversion subclipse plugin from the eXist SVN sourceforge repository:

   https://exist.svn.sourceforge.net/svnroot/exist

2. (Confirm environmental variables are set correctly for JAVA_HOME and EXIST_HOME) To do this I use the Windows/Computer/Properties/Advanced Settings and check it with the SET command at the CMD prompt. The result looks like this:

   JAVA_HOME=C:\Program Files\Java\jdk1.6.0_10
   

   My Eclipse workspace is just "C:\ws" and I created a project called eXist-1.3 so the path name I used for EXIST_HOME is:

   EXIST_HOME=C:\ws\eXist-1.3dev
   

3. Open the DOS cmd prompt, cd to eXist directory
4. type in "build.bat"
5. Type in "build.bat -f build\scripts\jarsigner.xml"

It took just 1 minute and 6 seconds for the main build to run on my 4CPU system with 8GB of RAM even though it is running Vista. Then I just ran the start.bat in the bin directory. eXist was then running on http://localhost:8080/exist. I had to use the new WebStart admin tool on the admin section of the eXist web page to change the admin password. Thanks to Joe Wicentowski at the US Dept of State for helping out!

The Finch and the Raccoon

The blogs on xml.com are down so I have reposed this here. Have you ever wondered if the laws of evolution apply to computer languages? When you walk down the isle at your favorite bookstore, does it seam like there are actually more computer languages than last year? What forces are driving each of these new languages to evolve? In 1835 Charles Darwin visited the Galapagos Islands. There he collected what he thought were about a dozen distinct species of birds. Upon returning to England he discovered that each of these species had evolved from a single species of finches. On the various Galapagos Islands the requirements for food gathering was different, but consistent over hundreds of thousands of years. Enough time for a single species to adapt to meet consistent requirements. Consider the Raccoon: omnivores that have proved to be one of the most adaptable mammals on Earth. The Raccoon’s range has rapidly expanded into urban areas due to their ability to quickly adapt to new requirements before other animals have had time for the wheels of evolution to turn. So goes it with computer languages. Some procedural languages can be quickly adapted to fill in the needs for a new niche. When the web was young, procedural languages like Java and JavaScript quickly filled in the need for a variety of tasks. As the requirements for building web applications stabilized, declarative systems like CSS, XForms and XQuery started to push procedural languages back into niche-areas. As these declarative languages stabilize and become worldwide standards, graphical tools are being created to allow non-programmers to create, manipulate and extend these systems. This is why many of us believe their will always be some need for procedural programming, but certainly not for building standard web applications that are controlled by style sheets and user interaction forms. Like the finch, declarative languages need a little longer to evolve. It sometimes takes years for a small vocabulary of functional specification patterns to emerge and be given labels. Additionally, it can takes years for the standards bodies to agree on the best way to deliver these new languages in a set of semantically precise data elements that have unambiguous interpretations. Finally, it may take another few years for IT managers to realized that they really do lower costs if they avoid vendor-specific implementations and adopt worldwide standards. When CSS first came out you may have been a little reluctant to let web designers play with a rules engine. As XForms becomes ubiquitous you may be resisting change because you have invested so much time and energy learning how to debug JavaScript (without a debugger). You can not hold back the forces of evolution…and now we all need to adapt to the declarative world or risk our own extinction. If you are interested in more on this topic see my Presentation from the 2007 Semantic Technology Conference The Semantics of Declarative Systems

XRX: Simple, Elegant, Disruptive

I recently started writing for O’Reilly Media. I posted an article on XRX. Here is the link:
http://www.oreillynet.com/xml/blog/2008/05/xrx_a_simple_elegant_disruptiv_1.html
We are just getting started moving to a new MoveableType (MT) system so not all of the features (like keywords and feedback) are working.
If you have problems commenting on the site, please feel free to post your comments here.
Here is a comment from Arun Batchu:
An elegant introduction to an elegant architecture, Dan. Thank you. The essential takeaway from what you describe is the exploitation of XML from one end to the other end - especially from the Developer's perspective, for, how XRX actually manifests in runtime could be left to implementation technologies. Thus the logical architecture of XRX could be realized by a few variations of concrete technology - which is great. The XForms could be realized by XForms server technology (such as the excellent Orbeon stack), the ReST could be realized by any middle tier and the XQuery could be realized by an XQuery engine (such as Data Direct) that may actually be driving any one or a combination of datastores (XML, SQL, file system or ...) . The symmetrical and consistent leverage of XML as a data model from creation to transport to rest and back eliminates a whole lot of wasteful work. Like you point out, XPath is one of the most powerful query systems I have encountered; you can pack so much in so little and reuse it across the board with little if any change from the drawing board to production. In such a system as you describe, a business rule expressed once can be reused anywhere - from one end-to-end , however long the travel, as long as the architecture is XRX, like you have described. Thanks for expressing it so well. A few of your readers will not get it - it is one of those things that once you experience it, you are left wondering why this did not happen before. Oh, well!

You are welcome! Thank for your feedback Arun!

XForms, Dyslexia and the Right Brain

Those of you that have worked with me know that I am a little bit of an odd-duck. I have dyslexia. If you ever see me attempt to write on a whiteboard my spelling is at the ninth grade level. My left brain, used for phoneme recognition, never really developed like the rest of you. My right brain had to be co-opted to help out. But one of the skills I seemed to have picked up due to my over-exercised right brain is the ability to visualize multiple complex application architectures and quickly understand architectural tradeoffs. I am one of the few people that seem to be interested in discussing how XForms, metadata registries, ontologies, the semantic web, OWL, RDF, graphs, business rules, BPM and Kimball conformed dimensions all can work together to deliver elegant and cost-effective enterprise-scale solutions. It seems easy for me to simultaneously visualize two or more architectures and it constantly challenges my patience when I have to explain over and over why architecture alternatives will not meet a business requirement.

It turns out that many people that have dyslexia also have the gift of being able to visualize complex systems. Albert Einstein, Thomas Edison, Jackie Steward and Charles Schwab are good examples of dyslexic people that have used the strengths of the right brain to do things that left-brain thinkers could not.

As a right-brain centric person I need to also tell you that I really love the XForms architecture. The magic of a declarative language, MVC, bindings and a dependency graph makes XForms development 10 years more advanced than anything else I have worked with. I think it is beautiful and elegant. It is everything that AJAX and JavaScript application are not. Clean, simple and easy to visualize (for me at least). When someone asks me if I can create an XForms application to do something, I create a mental image in my mind of the model, the view and how events will update instance data in the model using inserts or external submission results. I can easily visualize the bindings of view controls to data elements in the model. Once I can visualize the application clearly, writing the application is just a matter of typing in the code.

I think that my right-brain is also a reason I detest JavaScript and AJAX. It is far too much code to read and trying to visualize how 300 lines of JavaScript enables me to do a drag-and-drop. I want to just add an attribute to an element like "drag-source" and "drop-target" and I want it to just work.

What triggered this posting is that I have been reading Proust and the Squid by Maryanne Wolf. This is a book about how the brain's circuits are used in the reading process. She has a wonderful explanation of how the dyslexic brain co-opts the right brain for reading and enhances it functionality. I didn't really understand the relationship between my defects and my gifts .

So how about you and your development team? Do you have a dyslexic right-brained person on your team? Can they quickly visualize architectural tradeoffs? Have they tried XForms? And if they do, will you be willing to tolerate their disgust of AJAX and JavaScript after they have built their first XForms applications?

For more information about dyslexia check out these two Wikipedia entries: http://en.wikipedia.org/wiki/List_of_notable_people_diagnosed_with_dyslexia http://en.wikipedia.org/wiki/Dyslexia

Great Example of Multi-dimensional Bubble Chart

Here is a beautiful example of a bubble chart display using global population statistics. The example on carbon emissions are very interesting. Note the dimensions

1. Population of County (size of bubble)
2. Continent (color of bubble)
3. Live Expectancy (vertical axis)
4. Income (horizontal)
5. Time (the play button)

It is interesting to see the huge impact AIDS has had on the life expectancy in African counties. Imagine if you could see your organizations product sales using this type of graph. This application was done with a software system called Trendalyzer. It was initially developed by Hans Rosling's Gapminder Foundation in Sweden and acquired by Google Inc. in March 2007. This version is a Flash application. Does anyone know of any open-source software that could do this?

Metadata Repositories vs. Metadata Registries

For several years people have been using the terms metadata Registry and Repository inconstantly, imprecisely and almost interchangeably and I would like to weigh in as to how these terms could be used more precisely to allow organizations to effectively to manage metadata processes.

First lets take the definition of a Repository. Webster defines a repository as …a place, room, or container where something is deposited or stored.. Note that here is nothing in this definition about the quality of the things being stored or the process to check to see if new incoming items are duplicates of things already in the repository. If I have 100 users they could each define "Customer" as the see fit and put their own definition into the metadata repository as their own definition. No problems.

On the other had lets take the word Registry. A Registry has the connotation of more than just a shared dumping ground. Registries have the additional capability to create workflow processes to check that new metadata is not a duplicate (for a given namespace). One of the definitions from Webster is an official record book. Note the word official.

A Repository is similar to a front-porch of a house. No locks prevent new things from landing there. But a Registry is a protected back room where human-centric workflow processes are used ensure that metadata items are non-duplicates, precise, consistent, concise, distinct, approved and unencumbered with business rules that prevent reuse across an enterprise. These registries have become the central foundation that agility can be baked-in to many enterprise process. The latest version of the Kimball's Data Warehouse Lifecycle Toolkit (which is actually a very good read) even goes as far as to call their process "metadata-driven". Not different the model-driven development world.

Registries have the implicit connotation of trust behind them. They now serve a a central process for the creation of shared meaning across the enterprise. Definitions in a registry have been vetted by an enterprise-level organization that has the responsibility of enterprise data stewardship. They have a high probability of being consistent with industry best-practices and vertical industry standards. Registries are the go-to source for creating canonical XML schemas, enterprise ontologies or conformed dimensions in a OLAP cube. Repositories are personal or small departmental definitions of an isolated view of the world.

None of these ideas are really new. They are at the core of the ISO/IEC 11179 metadata registry standard. Note that they don't call it a repository standard! People are just now starting to understand how important Registries are in most enterprise-wide systems. The growth of Business Intelligence and Enterprise Data Warehouse terminology and Service Oriented Architectures is a good place to see the rise of repositories and registries. We now see service registries, portlet registries, model registries...the list goes on-and-on.

Much of the background on the differences between the use of repositories and registries can be traced way back to the early days of object-oriented systems in the 1995 book Succeeding with Objects by Adele Goldberg and Kenneth Rubin. This was one of the first books on enterprise reuse strategies and they defined the concept enterprise asset reuse and the need for a trust-driven repository as a basis for reusing assets. They identified a multi-step process for reviewing new submissions to determine if the submission duplicated existing assets. They showed how critical it was to classify items in a registry and search an existing registry for duplicates before new items are added. If you can get a copy of the book I would suggest you read the section on "Set Up a Process for Maintaining Reusable Assets" on page 245.

The book then goes on to show how organizations can and should be structured to reuse these assets and gives the pros and cons of the differing organization structures and their impact on reuse. This is the basis for the data governance and data stewardship movement in many organizations today.

So the next time someone uses the word registry or repository in a conversation, ask them if they are using the definition of the word that is consistent with the corporate business term registry or is their own private definition from their own repository of imprecisely used buzzwords.