Motors for Arduino Labs

This week I went through many of the SparkFun Inventor's Kits that we use in our CoderDojo Arduino labs and realized that the kids have been loving our little DC mini-motors to death.  As you can see in Figure 1 over half of the wires had broken off!

Figure 1: DC Motors with most connector broken off

The ones supplied in the SparkFun Inventors kits have small delicate wires on them that break off easily. Since the motor circuit lab is a popular lab we need reliable DC motors!

It was pretty easy to solder the wires back on.  However, I wanted to make sure they didn't get pulled off again.  So I have replaced the old thin wires with some thinker 22 gauge stranded wire. I then use heat shrink and some cable ties to bind the wires to the case of the motors. You can see the heat shrink just before I applied the heat in Figure 2.

Figure 2: DC motors with heat shrink

Next I used some small cable ties (see Figure 3) to firmly secure the wires to the motors.  So if the kids pick up the motor by the wires it will not strain the solder joints.

Figure 3: DC with cable tie

Viewing Motor Direction (Optional)

I also realized that some of the motors don't have gears on them so it is difficult to tell what direction they are spinning.  In our labs (not in the Sparkfun Inventors Kit) we need to show both clockwise and counterclockwise directions in the labs. The direction of rotation is used in the motor labs that use the motor driver H-bridge chips like the popular L293D. So I have added a small drop of hot-glue on the end of the motor spindle and then added a Golden Spiral sticker to help the students see the direction of rotation. I also put a ring of hot-blue around the base and then used a felt pad so that the motor could stand upright.

Figure 4: Motor on base with Golden Spiral label on motor spindle.  The wires are multi-stranded 22 gauge but have a 1/2 inch solid gage at each end to work with the breadboard.

Figure 5 has the artwork from the Golden Spiral Label. This is one of may favorite designs.

Figure 5: Golden Spiral Label to indicate rotation

As an aside, some of you may know that spirals are painted on jet engines like in Figure 6 to indicate that the engine is rotating and give the viewer some indication of direction.

Figure 6: Golden Spiral Graphic painted on jet engine to show motion

A discussion of the golden spiral is also a change to discuss the beauty of mathematics. A PDF file that can be used print out the designs on sheet of labels is here. Let me know if you want the source PPT file. I am not artist and others might have a better spiral design. An SVG image might be a bit smoother.

So some of you might be thinking that this is a lot of detail for a just one lab! You would be right! However, I think that the care and attention to detail we give each of these labs help the kids get quickly to the next level. This principal helps us apply the theory of Constructivism to our labs. Once kids know they can change motor direction directly from within their Arduino code they can then make the logical next step to seeing how they can turn a robot car direction by having one wheel motor go backwards. Each of these are small logical learning steps we need to keep kids coming back to our STEM classes.

In a busy noisy class with 20 students jumping around we don't want to have to search through 20 kits looking for a motor that has the wires attached. This takes time away from our learning objectives.

With a good solid supply of motors that will last through a year of labs like in Figure 7 you will be ready to help all your students get to the really fun parts - building their own Arduino powered robot!

Figure 7: Motors ready for the kids!

We will hear more about this and the learning steps to building robots in future blog posts.

References

• SparkFun Inventors Kit ($99.00)
• Replacement Motors on SparkFun ($1.95 + shipping) with jumper pins.
• Replacement Motors on ebay (5 for $7.85 - no jumper pins. They usually take two weeks for shipping)
• Heat shrink tubing on ebay (280 pieces for $8.79 with multiple sizes and a nice box)

Spark.io Core Evaluation

Last week at the Minnesota Arduino meeting I met the team from Spark.io.  They make a low cost (under $15 in quantity) device that you can program like an Arduino, but it has a full WiFi stack running on it and works with an optional cloud-based service.  They gave me (free) a device to evaluate.  So here is a first look.

The device came in a small box with a half breadboard and a USB connector.  You plug it in and it starts flashing a color that indicates it is "listening for a WiFi login".  If I had a smartphone it would have been easy to give it my WiFi ssid and password, however I did figure out how to give it the credentials using a Windows serial port (putty).  The serial port is not a full unix-like shell, but it got the job done.

Once it got on the local WiFi I went to the spark.io web site and "claimed" my device by putting in its device ID (which I also got via the Windows serial interface).  I could then use a web-based IDE to download the standard Arduino "Blink" app.  I also found the NeoPixel library and ported over some of my Moving Rainbow demos. Here is the spark core running one of the rainbow patterns:

It is interesting to note that you can't use many Arduino Libraries alone.  Someone needs to port them.  However I think that most of the common libraries that students would need are there.

What I really liked about the device is that the cloud interface give each device a REST interface.  For example if I put the following into my browser (or use the UNIX curl), I can get my device status:

https://api.spark.io/v1/devices?access_token=1234…

We can get the status of our device in a JSON format:

[
 {
   "id": "1234...",
   "name": "dan-test-1",
   "last_app": null,
   "last_heard": "2014-10-10T04:19:08.528Z",
   "connected": true
 }
]

If you have a smartphone there is a nice app you can use to setup the spark core (wifi and password) and do some basic programming.  However I also want to make sure that I can figure out how to make the spark core work in a teaching setting similar to the CoderDojo meetings.  This is where kids come in with Windows PCs and hook up the Arduino to the USB and fire up the Arduino IDE.  I also think that schools may purchase Chromebooks that they also want to use in the labs.  So I still have a bit more learning to do to see if I can create a full Internet-of-things lab that kids with a low-cost Windows PC can really use.

I like the fact that the spark.io system seems pretty open and all the code is on github.  The ARM processor also is a LOT more powerful than the Arduino controller.  The cost is also very low when you consider that a WiFi Arduino shield alone is $90 which is 3x more expensive than the entire device.  The spark.io Internet of Things "starter kit" is currently $39.00.  They have another version with many sensors for $99.00 which seems like a good deal.

My initial impressions are that if you have a Linux or Mac and can run node.js from the command line that you should have very few problems.  That is clearly their target developer audience. However many kids don't have access to a smartphone and a $1,000 Mac.  However, having a low cost Windows computer and using an simple Eclipse-like IDE is still something I hope we can have in the future.

I have started to create a simple web-front end and have some basic unit tests running.  I am using Bootstrap 3 and eXist (my favorite NoSQL database) with a few simple XQuery functions to build the URLs. My initial tests allow you to store the device ID and Access codes in a config file and I have a set of functions that convert the JSON responses into pretty HTML pages.  Next I want to be able to change the colors and light patterns using a web form.  I hope that I have something to demo for the Luminarie team at the hackfest a week from Saturday!

Overall rating 5 out of 5 stars!

New Arduino Moving Rainbow Lab for Kids for Under $10

After participating as a mentor in the local Twin Cities CoderDojo  in Minneapolis at the University of Minnesota I found that there are two things that kids love: motion and color.  My prior experience with kids is that they love to take projects home with them to show their friends.  So here is the question - could we design an Arduino kit that kids could come to Maker events and actually take a working project home that has both color and motion?

At our local CoderDojo we wisely use the SparkFun Inventor's Kit.  If you have not seen this I encourage you to take a look.  Lots of nice components and a great guide (PDF). Many people undervalue the amount of work going into the guides for these kits.  Good writing is hard to find, are rarely included in many lower-cost projects.  However, since the cost of full Inventors Kit is still around $100 (without shipping) it is beyond the budget of most Maker events to let kids take these home.  So here is my answer: lets find a way to use the low-cost Arduino Compatible Pro Mini combined with a short strip of addressable LEDs in a kit that we could sell for around $10.

Here is a picture of my initial design:

Arduino Moving Rainbow Kit

I think this kit is perfect for kids since we have the wonderful colors of these amazing individually addressable LEDs as well as the ability to create "motion" by programs such as running lights.  The total cost of this should be around $10, depending on the options we use.

Now lets go through the components to verify that we can do this in the $10 price range.

Here is a place on eBay that sells the Pro Mini Arduino Compatible for around $2.00. Although I suspect that we could get them in quantity for less.  Note that these do not have a USB port on them.  More about that later.

Here is an addressable LED strip with 60 LEDs that we can find on eBay for around $17.00. You can cut these up into 6 strips of 10 LEDs each which gives you a price per kit of around $2.80. What is wonderful about these LED strips is that they only take two power pins and a single data pin to hook up!  I use the standard WS2811B which you can program with many libraries.  I have some sample code on github that uses the Adafruit NeoPixel libraries, however I have found other libraries work well also.  Keeping wiring simple is important for kids that don't yet have the fine motor skills (and patience) to wire up complex projects.

Now the last two components are the mini soderless breadboad (around $.70 in quantity), a switch and a power source.  In the kit in the photo above I am using a 3.7volt Lithium Poly battery taken from an old RC helicopter that no longer works. We could also use an external 3 AA or AAA battery holder for around $1.

I also added a power switch and a clear plastic polystyrene box I purchased at the Container Store, although a small plastic leftover box will also work well.

Arduino Pro Mini Rainbow Kit Parts List

1. Arduino Compatible Pro Mini $2.00
2. 10 element LED strip $2.70
3. Solderless mini breadboard $0.80
4. Battery holder $0.70
5. Plastic Polystyrene Box $2.50
6. Wire $.50
7. On-off switch $1.00

Total: $11.40 (with 3 AA batteries)  I have a detailed parts list here

I also suspect that if we purchase these in quantities of 20 or more the prices would come down a bit and we could get in under $10.

The biggest challenge is that each station must be equipped with a laptop with a FTDI basic breakout programmer with the Arduino software pre-loaded. The Arduino Uno (which runs about $25) does not need the FTDI programmer, but would be 10x more expensive for each Rainbow kit.  Getting these setup and configured is not easy.

I should also acknowledge that when we purchase items off of eBay or other sources, we are not using the official mini supplied by the fine people at Arduino.  However at over $20 these devices would not fit into our under $10 budget.  I still think that for other projects we should encourage people to use original Arduino hardware since the costs go to promote further open Arduino projects and quality educational materials.

Let me know your thoughts on this.  Getting high-quality clear plastic boxes that kids could take home, throw in their back and show their friends is still something I am working on.  I found one source here that I might try but any suggestions you have would be great.  I know other projects use simple baggies for their parts but these seem hard to use and show to others.

There are also other options is to provide a "night light" mode that you could plug into a wall outlet and not have to use batteries.  I have seen 5v USB "wall warts" for under $1.

Although I know that this is a very small project to get kids started in Arduino, I think it might be the platform that other projects could be added.  Creating a clock (12 LEDs required), putting sensors on to display temperature or adding accelerometers and magnetometers might be the next step for these kids.  The key is that they could take them home and show their friends and that keeps them interested and motivate to continue their work.

I also want to thank Gerd Knops for helping me get started on using the LED strips.  He is a brilliant engineer, a great friend and wonderful mentor to me.  I hope that what I learn from Gerd I can use to get more kids interested in science, technology, engineering and math.  We don't really have much time left to do this.

Agile Transformation in the Post NoSQL Era

Over the past four years we have seen the NoSQL movement grow from a small "Meetup" in the Bay Area to a technology that is touching all corners of the database world.  Each year NoSQL software becomes more capable, lower cost and easier to use. Document stores, in particular, make the process of doing object-relational mapping unnecessary allowing anyone with god metadata (JSON and XML) to simply drag-and-drop their files into a centralized corporate data store.  There are still a few challenges left.  For example using statistical analysis of inserts and record counts to optimized indexes and putting good query languages (like JSONiq) on top of these data stores.  But in large, these features are just polish on systems that are optimized to scale and be highly-available.  The hard work seems like it has been done and we are now in the stage of refinement, not revolution.

We documented the emergence of the NoSQL database patterns in our book, Making Sense of NoSQL, which now available through Manning Publications.  If you read this book you know that NoSQL systems have a diverse set of architectural patterns and different patterns apply to different problems.  Selecting the right database architecture is a complex process of carefully understanding the subtleties of requirements and weighing the alternatives.  Yet they do work well and once they are setup and configured they make data persistence a straightforward process.

So whats next?

Now that saving data has shifted from a project in its own to a smaller part of the application developer project we see the skills needed to build applications starting to shift.  The need to model your data with ER modeling tools is getting less.  The need to write complex joins with SQL is no longer needed.  The next major skill set we would like to address is the the movement to agile transformation.  How do you get you data out of your database and how do you transform it into the many formats that your application needs?

We think that the answer to this question is clear.  Organizations need to be better at transforming the data in their database to other forms.  This is the shift in skill sets from persistence centric to transformation centric.  And it is not just the software developers that need to be able to transform data.  Everyone on your team including non-programmers can play a role.  They all need skills to quickly transform data from one form to another.

We call this shift the movement to toward Agile Transformation.  We hope to document how organizations are waking up to this new movement and understand the tools and processes they are adopting to empower everyone on their team to quickly transform data.

This process of data extraction and transformation used to be a two-step process.  SQL developers might create a series of tabular reports.  These reports were then converted into the medium needed, HTML, XML, JSON, or even CSV files and other structures needed by other tools.  Now the extract and transformation process can be done with a single step.  Query results are no longer restricted to tabular formats.

Strategies for Agile Transformation

Over the next few months (or perhaps years), we hope to document many of the ways that organizations are attacking the agility challenges.  Here are just a few strategies to get us started.

Single Source Canonical Data Models

If you are in the content management business you know that the concept of single-source publishing is central to your productivity.  Using a single format to store content gets around the many-to-many transform problems that can drag down a teams productivity.  We see the same principals also applying to web applications in general.  Getting many data sources into a single format and then transforming this single format into many forms is the key to organizational productivity.  We call these models "Canonical" since they are the standards that organization can build publish/subscribe web services around.

Flexible Query Languages

If you have every worked with tools like XQuery and JSONiq you know that they are the most flexible query languages around.  These languages have benefited from years of work combining the best features of SQL, XSLT, XPath and dozens of other advanced query languages into a grammar that is designed to transform a variety of use cases.

Reusable Transformation Libraries

One of the first strategies that companies find is that many transformations are similar and can benefit from reusable code.  Languages like SQL do offer a wide variety of non-portable stored procedures.  Yet most of these languages limit your ability to build reusable transformation functions and modules.  Modern languages need to be close enough to your data to understand how queries use indexes but abstract enough to be reused in new applications.

Using Great Tools: IDEs and Report Writers

SQL GUI Report Writers were one of the first tools that tool the complexity out of transforming tabular data.  And we need more tools like these to make NoSQL reporting accessible to non-programmers.  Some NoSQL products like HBASE already have SQL-like query tools.  From our other blog posts you may know that we are big fans of the the oXygen IDE for managing JSON and XML data.  oXygen makes the process of learning how to write XPath expressions easy for even the non-programmer.  Even if their data is complex.  These tools are complex in themselves and require hands-on training if non-programmers are going to get the most out of them.

Simplicity for Non-Programmers

One of the core foundations of agility is not have all your transformation be controlled by a small group of overworked developers.  We learned that simple tools like GUI-based report writers or simple XPath templates can empower a non-programmer, with a bit of training, to build and maintain their own data transformations.  Not needing to understand database joins is a big step in empowerment.  Getting a good foundation library is another great step.  Setting up small but easy to use templates is another good strategy.  Building a search system to find the right library and templates also helps empower new staff and lowers the training burden on existing staff.  In general, we feel that if a user "knows their data" that they should be given the tools to transform their data.

So what is the best practices to build an organization that has agile transformation competency? We would love to know your ideas.  Please send us email or tweet us at @dmccreary on Twitter.

Rounding error in Java when converting strings to doubles

I came across a rounding error when I was running an XQuery.

Here is the error:

xquery version "1.0";
number(3.1) + number(3.2)

which returned:

        6.300000000000001

Not the expected value of 6.3.  Note that the XQuery function number() returns double precision data.

After a note from Eric Bruchez he suggested I cast the numbers to decimal:

xquery version "1.0";
xs:decimal(3.1) + xs:decimal(3.2)

and the problem seems to go away.  He also showed that he could reproduce this error in other Java JVM languages like Scala.

Let me know if anyone else has seen this problem before and has any other suggestions for a fix.

Thanks! - Dan

Analytical Reports for Technical Books

We are wrapping up our work on our book "Makings Sense of NoSQL" and I have created a series of XQuery reports on our book that we have found useful.

Our book is stored in XML DocBook format, which for those of you that have not used it, is ideal for technical books. DocBook contains elements for almost everything you need in a technical book including chapters, sections, paragraphs, figures, glossary terms, bibliographic references, and index terms. In short DocBook is is the perfect fit for most technical books and it can easily be extended. One key aspect about DocBook is that it is easy to transform into multiple formats such as HTML, PDF or ePub. There are many open source transforms available for DocBook. DocBook is at the heart of the movement into single-source publishing for technical publications.

In additional to the standard DocBook transforms we also created a series of reports on the book and I thought they might be of interested to others.  Here is a summary of some of these reports.

Chapter length report

When we started writing our book our goal was to produce a 310 page book with 12 chapters, each with approximately equal length.  But logically we wanted our fist chapter to be a brief overview and we found that the chapter that described the core NoSQL patterns had more content.

This is a horizontal bar chart that shows the length of each chapter. The goal is that all chapters be roughly the same size in length.

Chapter Length Report

You can see that one chapter in our book (chapter 4)  is a bit longer then the rest of the chapters.  We knew this up front after running this report and were prepared to defined our decisions with our editors.

The "Naughty words" report

We quickly found out that editors have some words that they don't want to see in a technical book. Words such as "just" or "very" should be used with great caution. There were also words that were not allowed "vs.", "e.g.", "etc.") according to the style guide.  This report shows you how often you use these words in each chapter.

Report counting specific words in each chapter

Book metrics report

This is just a raw counts of elements such as book parts, chapters, sections, figures and tables etc.

Book Metrics Report

Chapter metrics report

For each chapter I created a detailed report of the content. As you are writing each chapter, other people can view your progress by running this report.  I tend to put in outlines first, then figures and then the actual text of each chapter.

List of figures and tables by chapter

These reports shows a listing of figures and tables sorted by chapter and location in each chapter. It also shows the caption for each figure and a thumbnail of the image. There are versions that show the type of figure (line-art vs. bitmapped) the sources of each figure.

List of Figures Report

Figure and table captions reports

Editors want to make sure a book is "browsable" which means that every other page has an interesting figure that people will see when they flip through the book.  Each of these reports can list the length of the figures and table captions sorted by the length of the caption.  The captions that are too short will need further work.

Paragraph and sentence length histogram reports

There are guidelines that should be used when writing technical books on sentence and paragraph length. This report shows the distribution of paragraph lengths in each chapter. You should work with your editor to find reasonable guidelines for your audience.

Paragraph Size Distribution Report

Table of contents reports

For reviewing the book structure it is always nice to have reports that list the book's structure by chapter, section, sub-section and sub-sub-section. These reports come with several parameters that limit the depth of the table of content and can also be modified to create mind maps using open source mind mapping tools.

Table of contents showing Parts, Chapters, Section 1 and Section 2

Once I wrote the basic structure of the table-of-contents report it was easy to create other output formats for different functions.  Here is an example of a GraphViz output.

Book Outline using GraphViz format.

This example used the DOTML markup format and an external transform using the Chris Wallace graphviz XQuery module.

You can also convert the table of contents into an Mindmap file and open the file in FreeMind or XMind.

Here is an example of the book rendered in XMind.  NoSQL Book MindMap

Glossary of terms reports

Our book puts a focus on the terminology used in the NoSQL movement. We try to create precise definitions of all the terms we use and discuss the variations in definitions in different NoSQL communities.  I use these reports to list each time a term is first introduced and make sure that we have a formal definition in the Glossary Appendix at the end of the book.

Glossary Term Report Showing Glossterm IDs and Definition Status

There are also other miscellaneous reports listing the introductory chapter quotes, lists of comments by reviewer (extracted from PDF using Apache Tika), and various tools to help us gauge the completeness of each chapter.

Report Strategy

We created a central XQuery book module that had all the common functions such as $book:chapters, or  that returned a sequence of all book chapters or book:word-count($node) that returned a word count of a node.  I used the eXist-db database to store and execute the transforms.  After we created the module the templates could be quickly customized for each report.  We also used the oXygen XML IDE extensively and we want to thank George Bina for his support of our project.

I think that DocBook, oXygen, XQuery and eXist are ideal tools for managing the book creation process.

Cognitive Bias in NoSQL System Selection

After attending the Saturn 2013 Conference I was exposed to the use of "Cognitive Bias" in software architecture.

Here are some examples of cognitive bias I have seen as applied to the world of NoSQL database selection.

Anchoring bias - the tendency to produce an estimate near a cue amount - "Our managers were expecting an RDBMS solution so that’s what we gave them."

Availability heuristic - the tendency to estimate that what is easily remembered is more likely than that which is not. - "I hear that NoSQL does not support ACID." or "I hear that XML is verbose?"

Bandwagon effect - the tendency to do or believe what others do or believe - "Everyone else at this company and in our local area uses RDBMSs."

Confirmation bias - the tendency to seek out only that information that supports one's preconceptions – "We only read posts from the Oracle|Microsoft|IBM groups."

Framing effect - the tendency to react to how information is framed, beyond its factual content "We know of some NoSQL projects that failed."

Gambler's fallacy (aka sunk cost bias) the failure to reset one's expectations based on one's current situation – "We already paid for our Oracle|Microsoft|IBM license so why spend more money?"

Hindsight bias - the tendency to assess one's previous decisions as more efficacious than they were – "Our last five systems worked on RDBMS solutions".

Halo effect - the tendency to attribute unverified capabilities in a person based on an observed capability. – "Oracle|Microsoft|IBM sells billions of dollars of licenses each year, how could so many people be wrong". 

Representativeness heuristic - the tendency to judge something as belonging to a class based on a few salient characteristics  - "Our accounting systems work on RDBMS so why not our product search?"

Thanks to everyone at CMU/SEI and the SATURN Conference for the exposure to these ideas.

How to pass oXygen XML IDE Editor variables into an XQuery script.

I am a big fan of the oXygen XML Integrated Development Environment.  It has many powerful features and I learn about new ones the more I use it.  I recently started a project that required me to run XQuery transform on hard-drive files using Saxon HE.  Since I normally run XQuery from within the eXist-db this was a new challenge for me.

I have many XML files and many different XQueries and I wanted to run them directly from test cases in an oXygen project.  But unlike XSLT, XQuery does not have a concept of "standard input".  You need to specify a path to the document within the XQuery file itself.  So how to pass the file name from the project directly to Saxon?  Luckily, the guys from Syncro Soft (the authors of oXygen) thought about this.  You can use oXygen "Editor Variables" to be parameters to your XQuery.

Here is a sample program that reads two external variables, one for the project director and one for the current file name (with extension).

XQuery Program

xquery version "1.0";

(: Example of how to access external "Editor Variable" variables passed

  by oXygen. This one example gets us to a full file path within a

  project.  See:

http://www.oxygenxml.com/doc/ug-editor/topics/editor-variables.html 

  for other editor variables.
:)

(: oXygen project directory :)
declare variable $pd as xs:string external;

(: file name with extension :)
declare variable $cfne as xs:string external;

(: note the triple forward slashes before the file. :)
let $path := concat('file:///', $pd, '\', $cfne)

return
<results>
   <project-dir>{$pd}</project-dir>
   <file-name>{$cfne}</file-name>
   <file-path>{$path}</file-path>
   <doc-available>{doc-available($path)}</doc-available>
</results>

Note that we put the project and file together to get the path to the file and use the doc-available() function to verify that the file is there.

Next, we set up a transformation scenario and put the file above in the transform URL.  We have to add the parameters to the transform.  oXygen parses your XQuery for all external variables and puts them in each line for you.  For each variable put in the same variable with the ${pd} notation where the dollar sign is outside the curly braces (opposite of XQuery).

Here is a screen image of these parameters and their values.

Results

When we run this transform on any file on Windows we get the following:

<results>
   <project-dir>D:\ws\project\subdir</project-dir>
   <file-name>my-input-file.xml</file-name>
   <file-path>file:///D:\ws\project\subdir\my-input-file.xml</file-path>
   <doc-available>true</doc-available>
</results>

On Mac and UNIX systems we would not use the file:/// prefix and the separators will be forward slashes.

NoSQL book now available in MEAP

The book that Ann Kelly (my wife) and I have been working on for the last year is now available on the Manning Early Access Program (MEAP). Here is the link:

http://manning.com/mccreary

The final title of the book will be:

Making Sense of NoSQL: A guide for managers and the rest of us

You can download chapter 1 for free but to get the other three chapters you will need to be part of the MEAP program which is priced at $23.99 for the Ebook only and $29.99 if you would also like a printed copy sent to you when the book is done.

We are looking for reviewers of the book. We are especially interested in finding anyone that has an interest in how to explain NoSQL concepts to non-programmers. This includes Managers, Project Managers, Business Analysts and anyone else that is interested in using NoSQL.

I believe this this is one of the first NoSQL books that is targeting managers and other non-programmers. In order to write to a general audience we are trying to use extensive tables and annotated graphics to make a highly readable book that give people an overall understanding of the NoSQL movement. Much of these tools have been developed by Ann and myself over the last several years trying to explain NoSQL concepts as part of our consulting business.

If you know of good NoSQL metaphors or even NoSQL jokes that we can include in the book please let us know! We will be very involved in the Manning Author feedback program.

AnyChart Case Study

I just released a case study (click on the blog post title or click here) using the commercial software charting package called "AnyChart". After using over a dozen charting solutions including Google Charts and FreeCharts I really feel that AnyChart has the best solution that really scales up.  Not only does AnyChart have a large number of charts but it is also very reasonably priced at around $500USD per server. Because I store all the chart specification and implementations in XML and I use eXist to manage the charts it is easy to manage the full lifecycle from requirements to testing all within a single Lucene-search powered environment.

The latest release of AnyChart also generates SVG and HTML5 compliant charts that run on the iPhone and iPad. This really shows how "declarative" systems are much more portable than any flash-based implementations.

If you are looking for an inexpensive but powerful tool for building charts and dashboards I would defiantly give them a try.

You can get samples at their web site at anychart.com

The "Pathification" of NoSQL

In my consulting work I try to focus on the empowerment of the non-programmer to build and maintain their own web applications. This is not an easy task. There are many barriers, some political and some technical. But when a new Business Analysts or Project Manager can create their own metadata management tools and begin to tell others their story I have great job satisfaction.

My current best-practice is to teach non-programmers how to write XPath expressions into their data sets. I tell them that if they are committed to "knowing their data" that I can teach them how to write web-applications in a week or so. The trick is to give them a "template" CRUDS application (Create, Read, Update, Delete, Search) that they can just plug in their own XPath expressions to get the application to work. They don't need to know the thousands of features of XQuery and native XML databases, they just need to know their data, how to write some simple XPath expressions and where to put these expressions into the templates.

I call this process "Pathification", a word that is similar to "Simplification" and "Standardization" but it has a focus on only teaching a very small subset of the many possible tools and a focus on cross database standards like XPath. Once they feel confident they can create simple CRUDS applications in a few hours they are then highly motivated to continue learning XQuery, but on their own pace and balanced with their other job responsibilities like data governance and semantics.

Pathification is something that can happen because native XML database have many enabling technologies. They have WebDAV drivers that makes you database look like folders on your desktop. We have tools that automatically convert XML into HTML and XForms. We have subversion libraries that allow your form "save" to go directly into a version control system. And these tools continually get better and make it easier to empower novice users.

The NoSQL movement is a huge and diverse community. And many people have focused on areas such as scalability and the integration of different types of data into a single XML development environment. These are all great efforts and really move us in the right direction. But I think for us to make NoSQL into a true multi-vendor, multi-product development platform is to create more tools that empower the non-programmer. For people in the native XML world we have a great start. Standards like EXPath, new application packaging tools and XRX application repositories will continue to make it easier for non-programmers to reuse and share their applications.

What I would like to see (but I am not sure is possible) is both XML and JSON-centric system to start to come together and share their ideas on how to make this happen. Can JSON data-stores like MongoDB and CouchDB be "Pathified"? Can we create shared applications that work with both types of back-end systems?

If we do this right, I think the world will be a much better place. People will have more choices for managing both structured and semi-structured data and non-programmers will be able to take a spreadsheet or new data feed in a standard format and build a new web application that manages this data in just a few hours without having to depend on other programmers. I think of this as a system as easy to use as Microsoft Access(TM) but without the chaos: All data can be stored on centralized servers with REST service-oriented interfaces for every data set.

Please let me know your thoughts on these ideas.

McCreary's Law

I am just returning from a three-day conference on Enterprise Information Management. I was very pleasantly surprised to meet many other very smart and warm independent consultants that do metadata and EIM consulting. More posts about what they taught me will be coming.

But these conferences for me to constantly answer the same question over and over again. What do I do an why don't I use traditional RDBMS systems to store metadata? What I need is some strong statement that explains the core principal of the need for agility in metadata management. So in that context, I would like to suggest a law of agile software development. To give it a label, I will just use McCreary's Law for now.

McCreary's Law states the following:

The agility of any software project in inversely proportional to the square of the number of data transformations in the developer stack.

The number of data transformation in the traditional three tier-architecture (Web to Objects to RDBMS to Objects and back to web) is four.

The number of data transformation in the traditional three-tier plus XML web services is six, since web services are usually created by one transform from objects to XML and one from XML to objects.

The number of data transformations in XRX is approximately zero, since we could make an argument that XML to HTML is a 1/2 translation since the HTML and XML tags are different. Of course if you are not changing the order all XML can be styled with CSS which is not really a transformation. This gets around the problem of infinite developer agility if the denominator is zero.

This formula is not perfect and I am sure their will be modifications to it like adding a constant for training and IT resistance to change, but I hope people can use it to predict or explain their current development agility.

Thanks to everyone I met at the EIM conference that listened to my rants and have not discarded me as a complete metadata radical.

XLucene

For the past two years I have been quietly observing the growth of something called Structured Search. This is the use of document structure to aid in search and retrieval of documents or sub-documents in large document collections.

I have also seen a common solution to this problem. This is to use a combination of native XML database and the Lucene keyword indexing problem. But people doing this are doing far more than just simple keyword search. So calling this new approach simply “Lucene” keyword search does not really describe what is going on. So I am proposing an new meme: XLucene.

I define XLucene as using a combination of document structure and keyword searches together to create very precise search ranking or search hit scores. The key is that the design must be able to store each branch in the document tree as a separate subdocument.

The key factor in widespread adoption is that almost anyone that can identify the tag names in their document structures will soon be able to create their own XLucene search and retrieval systems.

What first got me interested in “Structure Retrieval” was the book “Introduction to Information Retrieval” by Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze. In this book they present the following table:

	RDB search	unstructured retrieval	structured retrieval
objects	records	unstructured documents	trees with text at leaves
model	relational model	vector space & others	?
main data structure	table	inverted index	?
queries	SQL	free text queries	?

The following quote follows this table: There is no consensus yet as to which methods work best for structured retrieval although many researchers believe that XQuery will become the standard for structured queries.

I think that the jury has returned their verdict. XQuery wins! Here is that final table with the following properties:

Objects: Trees with text at leaves
Model: Hierarchical documents
Main-data-structure: trees and inverted indexes tied to node-ids
Queries: W3C XQuery with fulltext extensions

The design of using variable-length node-ids as the document identifiers was done by Wolfgang Meier of the eXist-db.org project. There are many people now using this design today. I will be following up this posting with some case studies of how people are using this in several areas, both in the government sector, library metadata and document search and retrieval.

Since there are so many people using this same problem-solution pair, that like all good patterns, it deserves its own name. XLucene (with no embedded dash) can be picked up by search engines and we can all start to share design experience and solutions. This is why the Open Source model really is superior to closed systems. Knowledge just travels faster once the memes are created.

I will also be presenting more details on the general topic of Structured Search and XLucene at the Enterprise Information Management Conference in Toronto next Tuesday, Sept 21st.

What Makes a Good Solution Architect?

A recruiter friend of mine was looking for a senior "Solution Architect" and needed some help setting up a screening interview. Here is what I told him.

I look for solid experience in a diversity of solution. There are just too many people that think that HTML/Objects and RDBMS systems are the only tools in your architect's toolkit.

Here is what I suggested the interview questions would be like:

Consider the following application architectures:

1. Ruby on Rails on a relational database
2. XRX on a native XML database
3. A Java Client on J2EE app server using JMS
4. A JQuery client on an OLAP database

A senior "Solution Architect" should be able to compare the pros and cons of each system and describe how they have used each of these architectures to build solutions and give organizations a competitive advantage. I also drill down into their understanding of search and retrieval and natural language processing.

As a bonus question, I would ask how each architecture would impact an organization's business strategy and to what extent business units could be empowered to build their own applications with minimal training and little or no need for IT involvement.

Most good solution architects should have experience with 3 out of 4 solution architectures but very few usually have a strong MBA-type background to understand how solution architectures impact business strategy. And if you find anyone that has built systems with all for and has an MBA you have a serious senior solution architect. Hire them!

Impressions of SemTech 2010

Two weeks ago I attended my fifth Semantic Technology conference in San Francisco. It was a great conference! This is my fifth time attending the conference and I plan to attend in the future! My biggest dilemma was which session to attend. At times there were as many as eight concurrent sessions going on.

There were a few big trends that I spotted.

The use of RDFa to annotate web pages using semantically precise elements was clearly a big trend. Jay Myer of BestBuy described how the BestBuy sales went up 30% when they added RDFa tags to their product pages. Although many search engines are not transparent about their use of RDFa tags in page rankings, Jay’s results should make it clear that this strategy works.

Jay told a great story about how hard it was to find a fridge that met his specific criteria: black with a specific size etc. using a keyword-based search engine. The semantic web will change all of this!

The big factor here is Martin Hepps GoodRelations ontology for products and services. This can really bring the Semantic Web to the masses. Finally a way to code the hours of operation for your store so that you can use Siri to ask "What Sushi Restrauants are Open at 10pm Near Here"

There were also a lot of sessions on LinkedData and specifically on LinkedData in the government area. Jim Heldler and his students at RPI are scraping every data set on data.gov and converting them all to RDF and storing them in a huge triple store.

Jim also provided one of the best sound-bites from the conference:

“Get AWAY from the Table”

This goes far beyond the NOSQL movement that we are seeing. It gets to the core of the problems with innovation in many organizations.

Jim is a consultant to the US data.gov transparency process and a key advocate of open linked data standards. It is interesting to see a friendly rivalry between Jim and Tim Berners-Lee who is a consultant for the UK data transparency movement. Both are using RDF to convert data but each is using slightly different strategic approaches.

He was referring to the fact that many organizations over-use the relational model and they need to understand that there are many alternatives, especially when doing mashups of data sets from many sources.

There were also many presentations on natural language processing and finding the true “meaning” of words within unstructured and semi-structured data sets.

I always attend Brian Sletton’s session on REST. Many people do not understand the relationship between REST, URI’s and the semantic web stack. Joe Wicentowski’s work at the US Department of State on the URL rewriting frameworks within eXist has really reinforced how this can be done easily in a single XQuery module.

I presented a 3.5 hour tutorial session on Entity Extraction. Since it was scheduled for the last session of the last day of the conference I though the attendance would be very low. But the room was packed and most of the people stayed till the very end.

It was wonderful to finally meet Marie Wallace and DJ McCloskey from the LanguageWare team at IBM. Their support of the Apache UIMA standards will be a great step forward to the creation of interoperable language analysis piplelines. Marie and DJ introduced me to the Millennium restaurant just a few blocks from the hotel. My wife Ann and I went there over the weekend and we now own one of their cookbooks.

By the way Google is now also support recipes in their rich snippets in their search results so you will soon be able to find "only recipes that take under 30 minutes" in a Google search engine.

There were also many session on the need for good taxonomies and ontologies in the enterprise. The use of SKOS for controlled vocabularies was a very hot topic and there were dozens of presentation on how OWL is being used to capture and exchange business rules.

It was also really great to finally meet Jeni Tennison after reading her books and following her Tweets for a long time. She is deeply involved in the federal data transparency projects in the UK. Here use of RDF is breaking new best practices for the entire community.

It was also great to catch up with Mark Birbeck, one of the chief architects of the Ubiquity XForms libraries. I am looking forward to trying out some of the new tools based on his backplane JavaScript libraries.

The people from Facebook also gave a presentation on how they are adding the ability for people to add a few lines to each web site to allow users to add a “like” button on a web site using a variation of RDF. Many people were a little disappointed to hear that they will not be using namespaces in their interfaces. Facebook felt that host HTML coders could only handle a single namespace. Many people agreed with their findings and thought that until 90% of HTML coders knew what namespaces were that organizations like Facebook were stuck just adding new data to HTML meta elements.

One of the most interesting discussions I had was with the people from Cray Computer. Apparently a large unnamed government agency has given Cray Research a very large contract to build customized ASIC (FPGA) chips to do graph analysis. Their ThreadStorm XMT architecture has 128 register sets and allows the CPUs to get continuous feeds of graph queries without waiting for memory. Their claim is that the federal agency has told them they are getting a 100X improvement in complex graph queries. The challenge is that the API is currently a low-level C interface and they have not yet put a SPARQL complier in front of an XML. So on good SPARQL benchmarks are yet available to compare the results of this hardware with a typical triple store. But even if it is fast, the price would be in the six digits for one of the XMP systems. So only very large organizations and government agencies might use this unless it was provided as a service.

One of the other things I found was how many people are using OWL and the OWL reasonerers like Pellet as replacements for traditional rules engines from companies like FairIsaac. There are two reasons for this. First is that many rules engine companies only talk about their engine performance but not on the need for precise semantics in the data elements. With OWL and the methods behind the semantic web stack we attempt to put semantics higher in the requirements of a system a use pre-built and pretested ontologies.

The second big reason that people are use OWL is that the rules are created using open standards. That means you are not locked into any one vendors rule format. The new W3C standards for rule interchange (RIF) will also start to break down many of the barriers to the creation and exchange of large industry vertical rule sets to that each organization does not have to start from scratch with a rule base. This will be big for industries like insurance where claims processing ontologies are just being created. Thanks to Kendal Clark for good information on this topic.

I also had a great time talking to many people in the publishing area. Seth Maislin gave a great presentation on Taxonomies. Seth and Marlene Rockmore joined me for a lunch at nearby Indian restaurant. Marlene is an expert on Taxonomy development. She was working on a book with O’Relly on taxonomies that is on hold for now but we hope to see one in the future from her.

It was also nice to see how many federal agencies are now starting to use SPARQL in the intelligence community. I saw a good presentation by Dennis Wisnosky with the US DOD talk about how they plan to use NIEM and semantic technologies to cut their integration costs. Yeah NIEM! Most people don’t understand that the NEIM is based on the RDF model, it just uses XML syntax to store the relationships. See my web-cast on the SemanticUniverse.com web site for more details. Hopefully his slides will be available in the future. Many people were taking photos of the slides since the DOD is not great at getting their slides out. Wonder why?

I also had lunch with several people from within the intelligence community that had great discussions about the pros and cost of pulling assertions out of NLP-analyzed annotated XML documents and the challenges relating to this. Keeping the links back to the original XML documents to is critical to verify the results of an assertion in context. Traceability, linage, provenance and time-domain representations in RDF that don’t cause a 10X growth is the number of triples is a difficult problem.

If there was some way to relate RDF assertion to a source document XML Node ID (the fourth column question) is a very difficult problem and one that needs close research with native XML and RDF systems. One of the things I learned about the eXist-Lucene integration done by Wolfgang Meier is that keeping the node-id as the document ID in Lucene allows non-programmers the ability to configure customized search rank rules past on the context of the keyword in a document. This is where highly customizable structured search rocks. I think this innovation needs to be added to RDF triples projects. Keeping the XML node-id of an assertion with an RDF triple context could be a great way to prevent RDF triple bloom for context.

Finally, I felt that David Wood and James Leigh's presentation on Callimacus project had the most potential to have a "big impact" on the adoption of Semantic Web tools. This new open source project allows users to specify a simple HTML template with special XML tags that allows them to bring triples directly into a web application. This has the potential to allow far more non-SPARQL programmers to integrate RDF data directly into a web page much like XQuery does today. With some work it might be possible to auto-generate the XForms bind elements for form rules. This would be a huge win for the integration of rules into web forms without needing to write JavaScript.

You can also see my tweets on the conference here: http://twitter.com/dmccreary

I hope to see more of you in San Francisco next year. Let me know if you are interested in co-presenting any papers!

Problems with WebDAV on Vista 64bit

Does anyone have any suggestions on a good reliable WebDAV product for Vista 64bit? I can not get the one that is bundled with Vista to work. Every time I try to create a remote connection I get the following error message:

"The folder you entered does not appear to be valid"

I have run the Vista WebDAV patches:

KB907306

I have also tried the following Hotfix:

Windows Vista WebDAV Hotfix

I have also tried to get various 3rd party packages working such as Independent DAV, BitKinex and WebDrive. All of the WebDAV products seem to have problems with the Windows Vista 64 bit systems. Is there any way to get Vista to use the Windows XP WebDAV "drivers"? I am using both eXist and MarkLogic native XML databases. Thanks - Dan

How to compile the eXist 1.3 build

I have been learning how to use higher-order functions in XQuery. I am using the eXist system to test this. To do this you will need to get the 1.3 build that there is no download for (yet). Here are the steps I used:

1. Download fresh copy of eXist from svn via TortoiseSVN or Eclipse subversion subclipse plugin from the eXist SVN sourceforge repository:

   https://exist.svn.sourceforge.net/svnroot/exist

2. (Confirm environmental variables are set correctly for JAVA_HOME and EXIST_HOME) To do this I use the Windows/Computer/Properties/Advanced Settings and check it with the SET command at the CMD prompt. The result looks like this:

   JAVA_HOME=C:\Program Files\Java\jdk1.6.0_10
   

   My Eclipse workspace is just "C:\ws" and I created a project called eXist-1.3 so the path name I used for EXIST_HOME is:

   EXIST_HOME=C:\ws\eXist-1.3dev
   

3. Open the DOS cmd prompt, cd to eXist directory
4. type in "build.bat"
5. Type in "build.bat -f build\scripts\jarsigner.xml"

It took just 1 minute and 6 seconds for the main build to run on my 4CPU system with 8GB of RAM even though it is running Vista. Then I just ran the start.bat in the bin directory. eXist was then running on http://localhost:8080/exist. I had to use the new WebStart admin tool on the admin section of the eXist web page to change the admin password. Thanks to Joe Wicentowski at the US Dept of State for helping out!

The Finch and the Raccoon

The blogs on xml.com are down so I have reposed this here. Have you ever wondered if the laws of evolution apply to computer languages? When you walk down the isle at your favorite bookstore, does it seam like there are actually more computer languages than last year? What forces are driving each of these new languages to evolve? In 1835 Charles Darwin visited the Galapagos Islands. There he collected what he thought were about a dozen distinct species of birds. Upon returning to England he discovered that each of these species had evolved from a single species of finches. On the various Galapagos Islands the requirements for food gathering was different, but consistent over hundreds of thousands of years. Enough time for a single species to adapt to meet consistent requirements. Consider the Raccoon: omnivores that have proved to be one of the most adaptable mammals on Earth. The Raccoon’s range has rapidly expanded into urban areas due to their ability to quickly adapt to new requirements before other animals have had time for the wheels of evolution to turn. So goes it with computer languages. Some procedural languages can be quickly adapted to fill in the needs for a new niche. When the web was young, procedural languages like Java and JavaScript quickly filled in the need for a variety of tasks. As the requirements for building web applications stabilized, declarative systems like CSS, XForms and XQuery started to push procedural languages back into niche-areas. As these declarative languages stabilize and become worldwide standards, graphical tools are being created to allow non-programmers to create, manipulate and extend these systems. This is why many of us believe their will always be some need for procedural programming, but certainly not for building standard web applications that are controlled by style sheets and user interaction forms. Like the finch, declarative languages need a little longer to evolve. It sometimes takes years for a small vocabulary of functional specification patterns to emerge and be given labels. Additionally, it can takes years for the standards bodies to agree on the best way to deliver these new languages in a set of semantically precise data elements that have unambiguous interpretations. Finally, it may take another few years for IT managers to realized that they really do lower costs if they avoid vendor-specific implementations and adopt worldwide standards. When CSS first came out you may have been a little reluctant to let web designers play with a rules engine. As XForms becomes ubiquitous you may be resisting change because you have invested so much time and energy learning how to debug JavaScript (without a debugger). You can not hold back the forces of evolution…and now we all need to adapt to the declarative world or risk our own extinction. If you are interested in more on this topic see my Presentation from the 2007 Semantic Technology Conference The Semantics of Declarative Systems

XRX: Simple, Elegant, Disruptive

I recently started writing for O’Reilly Media. I posted an article on XRX. Here is the link:
http://www.oreillynet.com/xml/blog/2008/05/xrx_a_simple_elegant_disruptiv_1.html
We are just getting started moving to a new MoveableType (MT) system so not all of the features (like keywords and feedback) are working.
If you have problems commenting on the site, please feel free to post your comments here.
Here is a comment from Arun Batchu:
An elegant introduction to an elegant architecture, Dan. Thank you. The essential takeaway from what you describe is the exploitation of XML from one end to the other end - especially from the Developer's perspective, for, how XRX actually manifests in runtime could be left to implementation technologies. Thus the logical architecture of XRX could be realized by a few variations of concrete technology - which is great. The XForms could be realized by XForms server technology (such as the excellent Orbeon stack), the ReST could be realized by any middle tier and the XQuery could be realized by an XQuery engine (such as Data Direct) that may actually be driving any one or a combination of datastores (XML, SQL, file system or ...) . The symmetrical and consistent leverage of XML as a data model from creation to transport to rest and back eliminates a whole lot of wasteful work. Like you point out, XPath is one of the most powerful query systems I have encountered; you can pack so much in so little and reuse it across the board with little if any change from the drawing board to production. In such a system as you describe, a business rule expressed once can be reused anywhere - from one end-to-end , however long the travel, as long as the architecture is XRX, like you have described. Thanks for expressing it so well. A few of your readers will not get it - it is one of those things that once you experience it, you are left wondering why this did not happen before. Oh, well!

You are welcome! Thank for your feedback Arun!