Friday, January 09, 2015

$30 CoderDojo Arduino Robot Version 3

CoderDojo Robot Version 3 Top

CoderDojo Robot Version 3 Bottom

I was happy to get a warm reception last night at our local Arduino.mn user group when we presented version 3 of our $30 Arduino CoderDojo robot. We got great feedback and many request for the bill of materials for people that wanted to try to build their own robot.

Our goal is to keep kids engaged in math and and science and one of the best ways to do this is to create engaging yet low-cost kits that use light, colors and motion. We place a premium on keeping costs low so that students, can take them home and show their friends. We also want to make the cost effective so that schools, libraries and STEM organizations can purchase them in bulk. This robot fits that mission.

Just to review, the goal of this kit is to build on the 20 SparkFun Inventors Kit labs. We want to have a low cost, open platform, extensible robot that teaches the following concepts:

  1. Lab 1: Turn a motor forward and backward using the L293D motor controller.
  2. Lab 2: Change the speed of a one motor using PWM
  3. Lab 3: Moving Forward: Make both motors in the same direction for 1 second and then stops for 1 second.
  4. Lab 4: Turning: Write a function that moves motors in opposite directions for 1 second.
  5. Lab 5: Measuring distance - using an ultrasonic sensor to measure distance.
  6. Lab 6: Display distance - use the LED strip to show the distance to an object in front of the robot.
  7. Lab 7: Avoiding obstacles - move forward till you get close to something in front and then turn.

Because the Arduino.MN group is full of Arduino makers that want to build their own robots I got lots of request for the hardware I am using. So here it is the Bill of Materials. There are links to eBay however you may find these parts on many other sites. Note that many parts come from China and typically have a 3-5 week delivery time when they come via China Post. However this is the best way to keep the costs down.

Although the links may not work in the long run, we have also provided some sample keywords to find similar items.

Part Price Keywords Sample Links Notes
Chasis $13.00 Arduino Robot Chasis Kit
link1

link2

There are many similar kits or you can just get the motors and build your own chassis out of wood. Make sure your kit has a battery pack.
Motor Drive $2.33 Arduino Motor Drive L293D link This is a great little board at a great price. The L293D drives up to 600ma. The small DC motors only need about 100ma.
Arduino Nano Compatible $4.39 Arduino Nano link Note this come with a VERY short USB cable. You may want to pick up a longer one.
Ultrasonic distance sensor $1.10 Arduino Ultrasonic Sensor HC-SR04 link Try other distance sensors if you would want.
LED Strip $1.10 LED RGB Addressable WS2812B 5V link Getting feedback on the ping sensor is key to learning how the work.
Main Breadboard $1.74 Solderless Breadboard 400 tie points link Look for 10 pcs to save on shipping.
Sensor Breadboard $1.00 Solderless Breadboard 170 tie points link Many colors available.
Jumper Wire $1.28 dupont wire jumper male female 20cm link We only need 6 but they come in handy.
On/Off Switch $0.30 10PCS mini toggle switch spdt MTS-102 link The packages of 10 that sell for $2.70 are many pages in. The keyword "MTS-102" works!

A few hints about searching for parts on eBay. The search engine is not very good and the sort-by-price rankings are often wrong. Remember to use the "Buy it Now" button. Also watch for the shipping prices. You may have to try many different combinations of keywords to find the right component and the right quantity.

If you are not finding a low-priced item, just keep scrolling. The right price might be buried 10 pages in and eBay's "sort by price" seldom works and had no cost or price per item.

I also tend to purchase generic components like jumpers and wire in batches of ten or more since I know I can reuse them and I get breaks on shipping costs.

As pointed out in the meeting, the Arduino Nano uses a driver that does not work on the latest version of the Mac OS. We hope they fix the driver problems soon.

My Powerpoint/KeyNote slides are also available if you would like them.

The sample code is on github:
Github Code

Many thanks to Doug (who has been really helpful helping me find low-cost parts on eBay) and Gerd, who has been a great mentor to me and Eric Palmer, who keeps me motivated.

Saturday, November 08, 2014

Robot Kits for CoderDojo Labs

For a few months our CoderDojo project in Minnesota had access to several finch robots.  These were wonderful introductory systems with many different programming languages (C, JavaScript, Python, Snap, etc.), however they were expensive (about $100), and lacked extensibility.  There was no way to enhance the robots with new sensors or change the motors.  They were theather by the USB which limited their motion.  And in an ironic twist of fate, we found the finch robots simply can't evolve with our kids as they learn and grow.

The SparkFun Inventor's Kit that we use in our current Arduino labs do have the basics for robot control.  There are labs for controlling a servo and a useful one-way rotation motor lab.  However the kit does not have a basic H-bridge lab so I often bring some basic L293D chips and have the kids google "L239D Arduino" and show them this page which has a nice lab on the Instructables site.  In this lab they learn how to make a motor turn both forward and backward, a key skill for controlling the robot motion.

Working with kids, I know how important it is to keep them motivated to lean more and find the hooks that keep them interested and want to come back for more.  So I have taken it upon myself to try to replace the finch robots with an more open and cost effective platform. 

Here are some of my selection criteria.
  1. Extend the current CoderDojo Arduino Labs
  2. Use the same Arduino IDE to program the robot (more on this later)
  3. Keep the total price under $100 and under $75 if possible
In doing my research I came across the book "Make an Arduino-Controlled Robot" by Michael Margolis.   This book which has a list price of around $25 (Amazon has it for $17) has most of the things that we are looking for.  Low cost and extensibility.  The book is not perfect (see the support questions) and does have examples from an older version of a motor shield from Adafruit.  We hope a revised version comes out soon.  However, the book does have some good ideas and we have a good general instructors guide to start with.

The next step is to find a basic robot kit for a reasonable price.  I came across a kit on ebay called UNO R3 2WD smarter Car Speed Encoder L298N HC-SR04 Sensor Kit Robot for Arduino that lists for $47.99:

Here are the parts that the kit comes with:

1 Car chassis
2 x Gear motor, tire, encoders
1 Universal wheel
1 Battery box (batteries not included)
1 BUONO UNO R3  
1 L298N Motor Driver        
1 HC-SR04 Ultrasonic Ranging Sensor    
1 Sensor Kit V05    
1 female to female cable
1 male to female cable
1 male to male cable

I should also note that the motor drive is a bit different then the Adafruit driver.  The L297N we can get for about $5.00 on eBay.  The Adafruit version 2 motor shield is about $20.00 + shipping.

I have one of these kits on order and I will see how it goes to get it working.  I think the key is that the kits are reasonably priced and barring any barring any integration issues we should be able to use the parts to get us started on the basic forward, reverse, turn left and turn right labs.  Getting all the sensors the to work will still require some work on our part.  And if we get approval to purchase the kits I would want to get at least five of the mentors some training on how to use them.  We will also need to purchase four, AA batteries for each of the robots.

I should also mention that I asked everyone at the Arduino.mn meeting for suggestion and I got many of them.  There are many more options for kits from sites like DFRobot.   There are literally 100s of kits and options in the $100 to $200 range. The basic Arduino Robot from Radio Shack runs for $274.00.  However none of them really seem to have better value for parts then the basic ebay kit listed above.

Many people suggested that kids would love to "drive" their robots with a bluetooth remote control.  One person told me that the IR remotes were too hard to use since you had to stand in front of the robot to get them to work.  These are all options that we can evolve into once we have the robot platforms picked out.

IDEs, Robots and the Limitations of the Arduino

I should also mention that many of the limitations on getting the robots to work is the noise reduction code in the sensors.  Sensors are inherently noisy and the kids really need to master the art of cleaning up the noise in their code to get the robots to perform.

The Arduino Integrated Development Environment (IDE) has a limited ability for the kids to see the actual signal from the sensors.  Which is sad.  Putting a simple graph option in the serial log should be about two pages of Java code.  The students can print the values of the sensors on the serial port, but their is no way to graph them and see the noise levels.  The lack of plugins for the IDE really make it limited, something the Eclipse team learned many years ago.

My friend, Eric Palmer, is also looking into a better "Universal IDE" called, wait for it...The Universal Embedded Computing IDE or the UECIDE.  This IDE will work with both the basic inexpensive Arduino chips as well as the newer and MUCH more powerful mbed chips. 

Since the Arduino chips really are somewhat limited in terms of memory and speed, many of the advanced robotics labs may need to leave the safe world of Arduino and use one of the much more powerful and low-cost MBED chips.  The ARM chips are 6x faster and have 16x the memory as well as options for built-in bluetooth and WiFi.   Here is a nice article on a comparison.

Eric has also mentioned that he is building a popular "self balancing robot" using the mbed processor.  These robots are great ways to teach robot control and dynamic feedback. They demonstrate how to apply both derivatives and integral that we teach in typical freshman Calculus classes.  This is something that our little Arduino may not able to do. If we want to do self balancing robots we need to take our Arduino training wheels off (literally) and move to a better IDE and a faster board.  However our platforms should still be useable with this software and hardware.

Although this IDE is much more "Eclipse" like, I did get it to run the blink lab in about 5 minutes.  So the setup is not too much more difficult.

Sunday, October 26, 2014

Internet of Things Hackday Trip Report

Last week I attended the Internet-of-Things Hackfest at Minnetronix is the Twin Cities.


Dan McCreary working with a student on the Moving Rainbow Arduino Kit


Here is a quick summary of some of the things I learned last week:
  1. Students LOVED the new Moving Rainbow kits I built.  Although some designs worked better than others.
  2. The spark.io device is wonderful but still lacks some features that would make it easy for students to use in classrooms.
  3. Getting IoT remote devices  to work requires specialized skills (like C parsers) that many of us don't have.
  4. When students get their own ideas for projects they get laser focus
  5. Hackdays are a lot of fun!  They are great ways to meet people and learn from others.

I had two roles at the hackfest.  I helped setup and run the "Kids Room" for the hackfest and I did a (very) little work in the Lumière lighting team.

For the Lumière team I mostly helped a few people to learn how to solder wires the LED strips and use heat shrink to secure the wires.  The other two guys on my team (Daniel Feldman and Alan) had much better programming skills and experience with other platforms like the Raspberry Pi.

I did spend many hours working on building an XForms, REST, XQuery application for the spark.io.  And although I did get it working (video), it still needs a lot of polish to allow the commands to be flexible.  I also did get a few new patterns working:
  1. Larson Scanner (Cylon)
  2. Random colors
  3. Candle flicker (simulate the light flickering pattern of real candles)
  4. Up-down patterns
  5. Swipe (redraw colors one pixel at a time)
Many of these still need to be parameterized with arguments such as color and speed.  However doing a general interface still needs work.

To get the Internet-of-Things (IoT) moving rainbow kits to work we need to send a specific "change color pattern" command to each device.  Since the spark.io interface only accepts a simple string, we need to parse the strings into a pattern of colors and motion.  However none of use knew how to use the strtok_r() function so we struggled a bit.  Daniel did a great job under pressure getting one command parser working on the spark.io, however we will need to refine it a bit more over time.

The ideal long-term solution is to develop a small "Addressable LED Strip Markup Language" as a domain-specific command language and write a parser for it in C that would run on the Spark.io.  This is a non-trivial problem for people that don't write C every day.  However I hope to spend some time thinking about the pattern name, color name, delay period syntax and perhaps coming up with a small BNF grammar.

Moving Rainbow Findings

Now I want to review some of the findings around a low-cost Arduino kits for students.


One of the dongle-style Moving Rainbow designs connected to an FTDI programmer

My primary interest is to understand what can we do to get students involved in STEM and to keep them on the strong math and science tracks in high-school and college.  Each time I engage with students I look for things that get them excited and want them to return for more.

My vision is to help design low costs kits that students could take home and show their friends.  These kits should be inexpensive enough that schools, libraries, and park buildings could have them for checkout, just like any library book.  Imagine a bookshelf at the library that contained 100 different electronic kits, each with specific learning goals in mind.

My experience has shown me that with some mentoring and some good electronics kits that many students can use internet resources to do a lot of work on their own.  However some students need a bit more encouragement then others.

This was one of the first times I had my newest collection of my "Moving Rainbow" kits on display.  When the students came into the room I encouraged them to pick up each kit and play with them.  Each of them has slightly different designs, packaging and switches.  When they flick the power switch on the 12-segment RGB LED lights came on and some of the kits had nobs and sensors that change the LED patterns displayed.  I tried to pay special attention to each of the students and they came into the room.  I noted what kits they picked up and what features they were interested in.

One of the things I learned is that there was little interest in the small-versions of the kits.  These were the ones that did not have room for the 12 pixels within the box.  These had a "dongle" design where the LED strip was sticking outside the boxes.  These designs pretty much failed, and looking back I can see why.  My reason to try them was that the smaller boxes were less expensive.  The larger boxes cost $5 or $6.  However, one of the students wanted to purchase the "dongle" designs.

The other thing to remember is that the logistics of getting the Arduino drivers working on both Windows and Apple systems is very time consuming.  Much of the first hour was getting the Arduino drivers working and the FTDI drivers (required for the mini-pros)  installed.

Several of the students did a "color wheel" lab where they had to mix the red, green and blue values together to make purple, yellow and orange colors appear on the LED strip.  The look on their faces when the strip turned either the right or wrong colors was priceless.  These were fantastic teaching moments!  On my TODO list is a "cheat sheet" for the color functions with pictures of color wheels.  This guide would start with functions to set a pixel to a specific RGB value and draw colors within for loops.

The moving color labs was a bit harder.  I need to continue to find good simple examples of lights moving up and down and get them pre-installed in the Arduino Examples area.  This is a great way to teach for loops and if/then/else logic.  As my sister and brother have told me, teacher prep makes a huge difference in students understand and lowering frustration levels.

One incident was that the students really loved the idea of creating a small wearable designs.  I had my "Altoids" example necklace as a demo to show them.  A few of them wanted to make Halloween costumes out of them.  And they were VERY motivated once they realized they could create their own costumes out of them.  At our final presentation to the entire group, two of them came to the front of the group to show their creations.  One of the students specifically thanked me in front of the entire group.  I wish all the students were this polite!

One idea is to have a "Arduino wearables" hackday around the same time next year.  This would require a lot of planning and some volunteer work by people with sewing machines.

What this taught me was that once students get their own idea of a creation it is like a fire-bolt of energy gets lit in them.  Their distractions disappear and they become laser focused on their task.  They seek whatever resources they can get to reach their goals.  Helping each student find their project is what these hackfests is all about!

I also met a few other people that might help us lower the costs of the packaging and find lower costs of the Moving Rainbow kits.  I still have more work to do to learn about placing larger quality orders for components and packaging on volume discount sites such as alibaba.com.

I want to reach out and thank everyone that made the meetup possible.  The sponsors and the people at Minnetronix should get a special thanks!



Monday, October 13, 2014

Motors for Arduino Labs

This week I went through many of the SparkFun Inventor's Kits that we use in our CoderDojo Arduino labs and realized that the kids have been loving our little DC mini-motors to death.  As you can see in Figure 1 over half of the wires had broken off!

Figure 1: DC Motors with most connector broken off


The ones supplied in the SparkFun Inventors kits have small delicate wires on them that break off easily. Since the motor circuit lab is a popular lab we need reliable DC motors!

It was pretty easy to solder the wires back on.  However, I wanted to make sure they didn't get pulled off again.  So I have replaced the old thin wires with some thinker 22 gauge stranded wire. I then use heat shrink and some cable ties to bind the wires to the case of the motors. You can see the heat shrink just before I applied the heat in Figure 2.

Figure 2: DC motors with heat shrink

Next I used some small cable ties (see Figure 3) to firmly secure the wires to the motors.  So if the kids pick up the motor by the wires it will not strain the solder joints.


Figure 3: DC with cable tie


Viewing Motor Direction (Optional)


I also realized that some of the motors don't have gears on them so it is difficult to tell what direction they are spinning.  In our labs (not in the Sparkfun Inventors Kit) we need to show both clockwise and counterclockwise directions in the labs. The direction of rotation is used in the motor labs that use the motor driver H-bridge chips like the popular L293D. So I have added a small drop of hot-glue on the end of the motor spindle and then added a Golden Spiral sticker to help the students see the direction of rotation. I also put a ring of hot-blue around the base and then used a felt pad so that the motor could stand upright.

Figure 4: Motor on base with Golden Spiral label on motor spindle.  The wires are multi-stranded 22 gauge but have a 1/2 inch solid gage at each end to work with the breadboard.



Figure 5 has the artwork from the Golden Spiral Label. This is one of may favorite designs.
Figure 5: Golden Spiral Label to indicate rotation


As an aside, some of you may know that spirals are painted on jet engines like in Figure 6 to indicate that the engine is rotating and give the viewer some indication of direction.

Figure 6: Golden Spiral Graphic painted on jet engine to show motion

A discussion of the golden spiral is also a change to discuss the beauty of mathematics. A PDF file that can be used print out the designs on sheet of labels is here. Let me know if you want the source PPT file. I am not artist and others might have a better spiral design. An SVG image might be a bit smoother.

So some of you might be thinking that this is a lot of detail for a just one lab! You would be right! However, I think that the care and attention to detail we give each of these labs help the kids get quickly to the next level. This principal helps us apply the theory of Constructivism to our labs. Once kids know they can change motor direction directly from within their Arduino code they can then make the logical next step to seeing how they can turn a robot car direction by having one wheel motor go backwards. Each of these are small logical learning steps we need to keep kids coming back to our STEM classes.

In a busy noisy class with 20 students jumping around we don't want to have to search through 20 kits looking for a motor that has the wires attached. This takes time away from our learning objectives.

With a good solid supply of motors that will last through a year of labs like in Figure 7 you will be ready to help all your students get to the really fun parts - building their own Arduino powered robot!

Figure 7: Motors ready for the kids!

We will hear more about this and the learning steps to building robots in future blog posts.

References


Thursday, October 09, 2014

Spark.io Core Evaluation

Last week at the Minnesota Arduino meeting I met the team from Spark.io.  They make a low cost (under $15 in quantity) device that you can program like an Arduino, but it has a full WiFi stack running on it and works with an optional cloud-based service.  They gave me (free) a device to evaluate.  So here is a first look.


The device came in a small box with a half breadboard and a USB connector.  You plug it in and it starts flashing a color that indicates it is "listening for a WiFi login".  If I had a smartphone it would have been easy to give it my WiFi ssid and password, however I did figure out how to give it the credentials using a Windows serial port (putty).  The serial port is not a full unix-like shell, but it got the job done.


Once it got on the local WiFi I went to the spark.io web site and "claimed" my device by putting in its device ID (which I also got via the Windows serial interface).  I could then use a web-based IDE to download the standard Arduino "Blink" app.  I also found the NeoPixel library and ported over some of my Moving Rainbow demos. Here is the spark core running one of the rainbow patterns:





It is interesting to note that you can't use many Arduino Libraries alone.  Someone needs to port them.  However I think that most of the common libraries that students would need are there.


What I really liked about the device is that the cloud interface give each device a REST interface.  For example if I put the following into my browser (or use the UNIX curl), I can get my device status:




We can get the status of our device in a JSON format:


[
 {
   "id": "1234...",
   "name": "dan-test-1",
   "last_app": null,
   "last_heard": "2014-10-10T04:19:08.528Z",
   "connected": true
 }
]


If you have a smartphone there is a nice app you can use to setup the spark core (wifi and password) and do some basic programming.  However I also want to make sure that I can figure out how to make the spark core work in a teaching setting similar to the CoderDojo meetings.  This is where kids come in with Windows PCs and hook up the Arduino to the USB and fire up the Arduino IDE.  I also think that schools may purchase Chromebooks that they also want to use in the labs.  So I still have a bit more learning to do to see if I can create a full Internet-of-things lab that kids with a low-cost Windows PC can really use.


I like the fact that the spark.io system seems pretty open and all the code is on github.  The ARM processor also is a LOT more powerful than the Arduino controller.  The cost is also very low when you consider that a WiFi Arduino shield alone is $90 which is 3x more expensive than the entire device.  The spark.io Internet of Things "starter kit" is currently $39.00.  They have another version with many sensors for $99.00 which seems like a good deal.


My initial impressions are that if you have a Linux or Mac and can run node.js from the command line that you should have very few problems.  That is clearly their target developer audience. However many kids don't have access to a smartphone and a $1,000 Mac.  However, having a low cost Windows computer and using an simple Eclipse-like IDE is still something I hope we can have in the future.


I have started to create a simple web-front end and have some basic unit tests running.  I am using Bootstrap 3 and eXist (my favorite NoSQL database) with a few simple XQuery functions to build the URLs. My initial tests allow you to store the device ID and Access codes in a config file and I have a set of functions that convert the JSON responses into pretty HTML pages.  Next I want to be able to change the colors and light patterns using a web form.  I hope that I have something to demo for the Luminarie team at the hackfest a week from Saturday!

Overall rating 5 out of 5 stars!

Saturday, August 30, 2014

New Arduino Moving Rainbow Lab for Kids for Under $10

After participating as a mentor in the local Twin Cities CoderDojo  in Minneapolis at the University of Minnesota I found that there are two things that kids love: motion and color.  My prior experience with kids is that they love to take projects home with them to show their friends.  So here is the question - could we design an Arduino kit that kids could come to Maker events and actually take a working project home that has both color and motion?

At our local CoderDojo we wisely use the SparkFun Inventor's Kit.  If you have not seen this I encourage you to take a look.  Lots of nice components and a great guide (PDF). Many people undervalue the amount of work going into the guides for these kits.  Good writing is hard to find, are rarely included in many lower-cost projects.  However, since the cost of full Inventors Kit is still around $100 (without shipping) it is beyond the budget of most Maker events to let kids take these home.  So here is my answer: lets find a way to use the low-cost Arduino Compatible Pro Mini combined with a short strip of addressable LEDs in a kit that we could sell for around $10.

Here is a picture of my initial design:


Arduino Moving Rainbow Kit

I think this kit is perfect for kids since we have the wonderful colors of these amazing individually addressable LEDs as well as the ability to create "motion" by programs such as running lights.  The total cost of this should be around $10, depending on the options we use.

Now lets go through the components to verify that we can do this in the $10 price range.

Here is a place on eBay that sells the Pro Mini Arduino Compatible for around $2.00. Although I suspect that we could get them in quantity for less.  Note that these do not have a USB port on them.  More about that later.

Here is an addressable LED strip with 60 LEDs that we can find on eBay for around $17.00. You can cut these up into 6 strips of 10 LEDs each which gives you a price per kit of around $2.80. What is wonderful about these LED strips is that they only take two power pins and a single data pin to hook up!  I use the standard WS2811B which you can program with many libraries.  I have some sample code on github that uses the Adafruit NeoPixel libraries, however I have found other libraries work well also.  Keeping wiring simple is important for kids that don't yet have the fine motor skills (and patience) to wire up complex projects.

Now the last two components are the mini soderless breadboad (around $.70 in quantity), a switch and a power source.  In the kit in the photo above I am using a 3.7volt Lithium Poly battery taken from an old RC helicopter that no longer works. We could also use an external 3 AA or AAA battery holder for around $1.

I also added a power switch and a clear plastic polystyrene box I purchased at the Container Store, although a small plastic leftover box will also work well.

Arduino Pro Mini Rainbow Kit Parts List
  1. Arduino Compatible Pro Mini $2.00
  2. 10 element LED strip $2.70
  3. Solderless mini breadboard $0.80
  4. Battery holder $0.70
  5. Plastic Polystyrene Box $2.50
  6. Wire $.50
  7. On-off switch $1.00
Total: $11.40 (with 3 AA batteries)  I have a detailed parts list here

I also suspect that if we purchase these in quantities of 20 or more the prices would come down a bit and we could get in under $10.

The biggest challenge is that each station must be equipped with a laptop with a FTDI basic breakout programmer with the Arduino software pre-loaded. The Arduino Uno (which runs about $25) does not need the FTDI programmer, but would be 10x more expensive for each Rainbow kit.  Getting these setup and configured is not easy.

I should also acknowledge that when we purchase items off of eBay or other sources, we are not using the official mini supplied by the fine people at Arduino.  However at over $20 these devices would not fit into our under $10 budget.  I still think that for other projects we should encourage people to use original Arduino hardware since the costs go to promote further open Arduino projects and quality educational materials.

Let me know your thoughts on this.  Getting high-quality clear plastic boxes that kids could take home, throw in their back and show their friends is still something I am working on.  I found one source here that I might try but any suggestions you have would be great.  I know other projects use simple baggies for their parts but these seem hard to use and show to others.

There are also other options is to provide a "night light" mode that you could plug into a wall outlet and not have to use batteries.  I have seen 5v USB "wall warts" for under $1.

Although I know that this is a very small project to get kids started in Arduino, I think it might be the platform that other projects could be added.  Creating a clock (12 LEDs required), putting sensors on to display temperature or adding accelerometers and magnetometers might be the next step for these kids.  The key is that they could take them home and show their friends and that keeps them interested and motivate to continue their work.

I also want to thank Gerd Knops for helping me get started on using the LED strips.  He is a brilliant engineer, a great friend and wonderful mentor to me.  I hope that what I learn from Gerd I can use to get more kids interested in science, technology, engineering and math.  We don't really have much time left to do this.

Wednesday, October 02, 2013

Agile Transformation in the Post NoSQL Era

Over the past four years we have seen the NoSQL movement grow from a small "Meetup" in the Bay Area to a technology that is touching all corners of the database world.  Each year NoSQL software becomes more capable, lower cost and easier to use. Document stores, in particular, make the process of doing object-relational mapping unnecessary allowing anyone with god metadata (JSON and XML) to simply drag-and-drop their files into a centralized corporate data store.  There are still a few challenges left.  For example using statistical analysis of inserts and record counts to optimized indexes and putting good query languages (like JSONiq) on top of these data stores.  But in large, these features are just polish on systems that are optimized to scale and be highly-available.  The hard work seems like it has been done and we are now in the stage of refinement, not revolution.

We documented the emergence of the NoSQL database patterns in our book, Making Sense of NoSQL, which now available through Manning Publications.  If you read this book you know that NoSQL systems have a diverse set of architectural patterns and different patterns apply to different problems.  Selecting the right database architecture is a complex process of carefully understanding the subtleties of requirements and weighing the alternatives.  Yet they do work well and once they are setup and configured they make data persistence a straightforward process.

So whats next?

Now that saving data has shifted from a project in its own to a smaller part of the application developer project we see the skills needed to build applications starting to shift.  The need to model your data with ER modeling tools is getting less.  The need to write complex joins with SQL is no longer needed.  The next major skill set we would like to address is the the movement to agile transformation.  How do you get you data out of your database and how do you transform it into the many formats that your application needs?

We think that the answer to this question is clear.  Organizations need to be better at transforming the data in their database to other forms.  This is the shift in skill sets from persistence centric to transformation centric.  And it is not just the software developers that need to be able to transform data.  Everyone on your team including non-programmers can play a role.  They all need skills to quickly transform data from one form to another.

We call this shift the movement to toward Agile Transformation.  We hope to document how organizations are waking up to this new movement and understand the tools and processes they are adopting to empower everyone on their team to quickly transform data.

This process of data extraction and transformation used to be a two-step process.  SQL developers might create a series of tabular reports.  These reports were then converted into the medium needed, HTML, XML, JSON, or even CSV files and other structures needed by other tools.  Now the extract and transformation process can be done with a single step.  Query results are no longer restricted to tabular formats.

Strategies for Agile Transformation

Over the next few months (or perhaps years), we hope to document many of the ways that organizations are attacking the agility challenges.  Here are just a few strategies to get us started.

Single Source Canonical Data Models
If you are in the content management business you know that the concept of single-source publishing is central to your productivity.  Using a single format to store content gets around the many-to-many transform problems that can drag down a teams productivity.  We see the same principals also applying to web applications in general.  Getting many data sources into a single format and then transforming this single format into many forms is the key to organizational productivity.  We call these models "Canonical" since they are the standards that organization can build publish/subscribe web services around.

Flexible Query Languages
If you have every worked with tools like XQuery and JSONiq you know that they are the most flexible query languages around.  These languages have benefited from years of work combining the best features of SQL, XSLT, XPath and dozens of other advanced query languages into a grammar that is designed to transform a variety of use cases.

Reusable Transformation Libraries
One of the first strategies that companies find is that many transformations are similar and can benefit from reusable code.  Languages like SQL do offer a wide variety of non-portable stored procedures.  Yet most of these languages limit your ability to build reusable transformation functions and modules.  Modern languages need to be close enough to your data to understand how queries use indexes but abstract enough to be reused in new applications.

Using Great Tools: IDEs and Report Writers
SQL GUI Report Writers were one of the first tools that tool the complexity out of transforming tabular data.  And we need more tools like these to make NoSQL reporting accessible to non-programmers.  Some NoSQL products like HBASE already have SQL-like query tools.  From our other blog posts you may know that we are big fans of the the oXygen IDE for managing JSON and XML data.  oXygen makes the process of learning how to write XPath expressions easy for even the non-programmer.  Even if their data is complex.  These tools are complex in themselves and require hands-on training if non-programmers are going to get the most out of them.

Simplicity for Non-Programmers
One of the core foundations of agility is not have all your transformation be controlled by a small group of overworked developers.  We learned that simple tools like GUI-based report writers or simple XPath templates can empower a non-programmer, with a bit of training, to build and maintain their own data transformations.  Not needing to understand database joins is a big step in empowerment.  Getting a good foundation library is another great step.  Setting up small but easy to use templates is another good strategy.  Building a search system to find the right library and templates also helps empower new staff and lowers the training burden on existing staff.  In general, we feel that if a user "knows their data" that they should be given the tools to transform their data.

So what is the best practices to build an organization that has agile transformation competency? We would love to know your ideas.  Please send us email or tweet us at @dmccreary on Twitter.