Saturday, September 24, 2011

The "Pathification" of NoSQL

In my consulting work I try to focus on the empowerment of the non-programmer to build and maintain their own web applications. This is not an easy task. There are many barriers, some political and some technical. But when a new Business Analysts or Project Manager can create their own metadata management tools and begin to tell others their story I have great job satisfaction.

My current best-practice is to teach non-programmers how to write XPath expressions into their data sets. I tell them that if they are committed to "knowing their data" that I can teach them how to write web-applications in a week or so. The trick is to give them a "template" CRUDS application (Create, Read, Update, Delete, Search) that they can just plug in their own XPath expressions to get the application to work. They don't need to know the thousands of features of XQuery and native XML databases, they just need to know their data, how to write some simple XPath expressions and where to put these expressions into the templates.

I call this process "Pathification", a word that is similar to "Simplification" and "Standardization" but it has a focus on only teaching a very small subset of the many possible tools and a focus on cross database standards like XPath. Once they feel confident they can create simple CRUDS applications in a few hours they are then highly motivated to continue learning XQuery, but on their own pace and balanced with their other job responsibilities like data governance and semantics.

Pathification is something that can happen because native XML database have many enabling technologies. They have WebDAV drivers that makes you database look like folders on your desktop. We have tools that automatically convert XML into HTML and XForms. We have subversion libraries that allow your form "save" to go directly into a version control system. And these tools continually get better and make it easier to empower novice users.

The NoSQL movement is a huge and diverse community. And many people have focused on areas such as scalability and the integration of different types of data into a single XML development environment. These are all great efforts and really move us in the right direction. But I think for us to make NoSQL into a true multi-vendor, multi-product development platform is to create more tools that empower the non-programmer. For people in the native XML world we have a great start. Standards like EXPath, new application packaging tools and XRX application repositories will continue to make it easier for non-programmers to reuse and share their applications.

What I would like to see (but I am not sure is possible) is both XML and JSON-centric system to start to come together and share their ideas on how to make this happen. Can JSON data-stores like MongoDB and CouchDB be "Pathified"? Can we create shared applications that work with both types of back-end systems?

If we do this right, I think the world will be a much better place. People will have more choices for managing both structured and semi-structured data and non-programmers will be able to take a spreadsheet or new data feed in a standard format and build a new web application that manages this data in just a few hours without having to depend on other programmers. I think of this as a system as easy to use as Microsoft Access(TM) but without the chaos: All data can be stored on centralized servers with REST service-oriented interfaces for every data set.

Please let me know your thoughts on these ideas.


DaveP said...

Makes sense to me!
More please?
What's hard/easy in your weeks tutorials?

Alain Couthures said...

XForms can work with JSON instances so interfacing a client XForms implementation such as XSLTForms with a JSON database with a REST interface such as couchDB is already possible!

Dr. Data Dictionary said...


Thanks for the feedback. Glad you liked the post. What seems to be easy is actually teaching basic use of sequences and path expressions using a "books" example. Almost every can do this with some practice and using oXygen's XPath builder. You get immediate feedback so learning happens quickly. I usually now teach sequences first and XPath second.

What is hard depends on the class and the topics they are interested in. One topic that is very challenging for non-programmers is basic tail recursion using templates. I don't ever teach XSLT any more. Just XQuery typeswitch document transforms. But people need to understand some basics of recursion when the use this. The TEI class is full of typeswitch transforms to HTML, PDF and soon e-pub.