tag:blogger.com,1999:blog-135465162011-12-31T07:19:49.804-06:00Ruminations on NoSQL, XRX, XQuery, Semantics and empowering the non-programmer.Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.comBlogger631100tag:blogger.com,1999:blog-13546516.post-84562615325994417362011-09-30T08:35:00.004-05:002011-09-30T23:13:20.149-05:00I just released a case study (click on the blog post title or click <a href="http://www.anychart.com/blog/2011/09/22/building-a-large-chart-ecosystem-with-anychart-and-native-xml-databases/">here</a>) using the commercial software charting package called "AnyChart". After using over a dozen charting solutions including Google Charts and FreeCharts I really feel that AnyChart has the best solution that really scales up.&nbsp; Not only does AnyChart have a large number of charts but it is also very reasonably priced at around $500USD per server. Because I store all the chart specification and implementations in XML and I use <a href="http://exist-db.org/">eXist </a>to manage the charts it is easy to manage the full lifecycle from requirements to testing all within a single Lucene-search powered environment.<br /> <br /> The latest release of AnyChart also generates SVG and HTML5 compliant charts that run on the iPhone and iPad. This really shows how "declarative" systems are much more portable than any flash-based implementations. <br /> <br /> If you are looking for an inexpensive but powerful tool for building charts and dashboards I would defiantly give them a try. <br /> <br /> You can get samples at their web site at <a href="http://anychart.com/">anychart.com</a><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-8456261532599441736?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-38632982887561193462011-09-24T08:33:00.007-05:002011-09-24T13:10:48.238-05:00In my consulting work I try to focus on the empowerment of the non-programmer to build and maintain their own web applications. This is not an easy task. There are many barriers, some political and some technical. But when a new Business Analysts or Project Manager can create their own metadata management tools and begin to tell others their story I have great job satisfaction. <br/><br/> My current best-practice is to teach non-programmers how to write XPath expressions into their data sets. I tell them that if they are committed to "knowing their data" that I can teach them how to write web-applications in a week or so. The trick is to give them a "template" CRUDS application (Create, Read, Update, Delete, Search) that they can just plug in their own XPath expressions to get the application to work. They don't need to know the thousands of features of XQuery and native XML databases, they just need to know their data, how to write some simple XPath expressions and where to put these expressions into the templates. <br/><br/> I call this process "Pathification", a word that is similar to "Simplification" and "Standardization" but it has a focus on only teaching a very small subset of the many possible tools and a focus on cross database standards like XPath. Once they feel confident they can create simple CRUDS applications in a few hours they are then highly motivated to continue learning XQuery, but on their own pace and balanced with their other job responsibilities like data governance and semantics. <br/><br/> Pathification is something that can happen because native XML database have many enabling technologies. They have WebDAV drivers that makes you database look like folders on your desktop. We have tools that automatically convert XML into HTML and XForms. We have subversion libraries that allow your form "save" to go directly into a version control system. And these tools continually get better and make it easier to empower novice users. <br/><br/> The NoSQL movement is a huge and diverse community. And many people have focused on areas such as scalability and the integration of different types of data into a single XML development environment. These are all great efforts and really move us in the right direction. But I think for us to make NoSQL into a true multi-vendor, multi-product development platform is to create more tools that empower the non-programmer. For people in the native XML world we have a great start. Standards like <a href="http://expath.org/">EXPath</a>, new application packaging tools and XRX <a href="http://atomic.exist-db.org/blogs/eXist/?id=urn:uuid:09bcc0de-a868-48e9-9fef-ceeb032028c3">application repositories</a> will continue to make it easier for non-programmers to reuse and share their applications. <br/><br/> What I would like to see (but I am not sure is possible) is both XML and JSON-centric system to start to come together and share their ideas on how to make this happen. Can JSON data-stores like MongoDB and CouchDB be "Pathified"? Can we create shared applications that work with both types of back-end systems? <br/><br/> If we do this right, I think the world will be a much better place. People will have more choices for managing both structured and semi-structured data and non-programmers will be able to take a spreadsheet or new data feed in a standard format and build a new web application that manages this data in just a few hours without having to depend on other programmers. I think of this as a system as easy to use as Microsoft Access(TM) but without the chaos: All data can be stored on centralized servers with REST service-oriented interfaces for every data set. <br/><br/> Please let me know your thoughts on these ideas.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-3863298288756119346?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com3tag:blogger.com,1999:blog-13546516.post-60357574870393699222010-09-24T08:05:00.003-05:002010-09-24T08:25:28.590-05:00<p>I am just returning from a three-day conference on Enterprise Information Management. I was very pleasantly surprised to meet many other very smart and warm independent consultants that do metadata and EIM consulting. More posts about what they taught me will be coming.</p> <p>But these conferences for me to constantly answer the same question over and over again. What do I do an why don't I use traditional RDBMS systems to store metadata? What I need is some strong statement that explains the core principal of the need for agility in metadata management. So in that context, I would like to suggest a law of agile software development. To give it a label, I will just use <b>McCreary's Law for now.</b></p> <p>McCreary's Law states the following:</p> <p><i> The agility of any software project in inversely proportional to the square of the number of data transformations in the developer stack.</i></p> <p> The number of data transformation in the traditional three tier-architecture (Web to Objects to RDBMS to Objects and back to web) is four.</p> <p>The number of data transformation in the traditional three-tier plus XML web services is six, since web services are usually created by one transform from objects to XML and one from XML to objects.</p> <p>The number of data transformations in XRX is approximately zero, since we could make an argument that XML to HTML is a 1/2 translation since the HTML and XML tags are different. Of course if you are not changing the order all XML can be styled with CSS which is not really a transformation. This gets around the problem of infinite developer agility if the denominator is zero.</p> <p>This formula is not perfect and I am sure their will be modifications to it like adding a constant for training and IT resistance to change, but I hope people can use it to predict or explain their current development agility.</p> <p>Thanks to everyone I met at the EIM conference that listened to my rants and have not discarded me as a complete metadata radical.</p><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-6035757487039369922?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com2tag:blogger.com,1999:blog-13546516.post-78026352738405666532010-09-12T07:27:00.007-05:002010-09-12T07:50:33.162-05:00<p>For the past two years I have been quietly observing the growth of something called <span style="font-style: italic;">Structured Search</span>. This is the use of document structure to aid in search and retrieval of documents or sub-documents in large document collections. </p> <p>I have also seen a common solution to this problem. This is to use a combination of native XML database and the Lucene keyword indexing problem. But people doing this are doing far more than just simple keyword search. So calling this new approach simply “Lucene” keyword search does not really describe what is going on. So I am proposing an new meme: XLucene. </p> <p>I define XLucene as using a combination of document structure and keyword searches together to create very precise search ranking or search hit scores. The key is that the design must be able to store each branch in the document tree as a separate subdocument. </p> <p>The key factor in widespread adoption is that almost anyone that can identify the tag names in their document structures will soon be able to create their own XLucene search and retrieval systems. </p> <p>What first got me interested in “Structure Retrieval” was the book “Introduction to Information Retrieval” by Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze. In this book they present the following table: <table style="border: 2px solid black;" border="1" cellpadding="3"> <tbody> <tr><td align="LEFT"> </td> <td align="LEFT"><span style="font-weight:bold;">RDB search</span></td> <td align="LEFT"><span style="font-weight:bold;">unstructured retrieval</span></td> <td align="LEFT"><span style="font-weight:bold;">structured retrieval</span></td> </tr> <tr><td align="LEFT">objects</td> <td align="LEFT">records</td> <td align="LEFT">unstructured documents</td> <td align="LEFT">trees with text at leaves</td> </tr> <tr><td align="LEFT">model</td> <td align="LEFT">relational model</td> <td align="LEFT">vector space &amp; others</td> <td align="LEFT">?</td> </tr> <tr><td align="LEFT">main data structure</td> <td align="LEFT">table</td> <td align="LEFT">inverted index</td> <td align="LEFT">?</td> </tr> <tr><td align="LEFT">queries</td> <td align="LEFT">SQL</td> <td align="LEFT"><a name="12114"></a>free text queries</td> <td align="LEFT">?</td> </tr> </tbody></table> The following quote follows this table: <span style="font-style: italic;">There is no consensus yet as to which methods work best for structured retrieval although many researchers believe that XQuery will become the standard for structured queries. </span> </p> <p>I think that the jury has returned their verdict. XQuery wins! Here is that final table with the following properties: </p> <p> <span style="font-weight: bold;">Objects:</span> Trees with text at leaves<br/> <span style="font-weight: bold;">Model:</span> Hierarchical documents<br/> <span style="font-weight: bold;">Main-data-structure:</span> trees and inverted indexes tied to node-ids<br/> <span style="font-weight: bold;">Queries:</span> W3C XQuery with fulltext extensions<br/> </p> <p>The design of using variable-length node-ids as the document identifiers was done by Wolfgang Meier of the eXist-db.org project. There are many people now using this design today. I will be following up this posting with some case studies of how people are using this in several areas, both in the government sector, library metadata and document search and retrieval. </p> <p>Since there are so many people using this same problem-solution pair, that like all good patterns, it deserves its own name. XLucene (with no embedded dash) can be picked up by search engines and we can all start to share design experience and solutions. This is why the Open Source model really is superior to closed systems. Knowledge just travels faster once the memes are created. </p> <p>I will also be presenting more details on the general topic of <a href="http://eim2010.wilshireconferences.com/sessionPop.cfm?confid=48&proposalid=3357">Structured Search</a> and XLucene at the Enterprise Information Management Conference in Toronto next Tuesday, Sept 21st.</p><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-7802635273840566653?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com1tag:blogger.com,1999:blog-13546516.post-7670654828442254632010-08-30T20:04:00.007-05:002010-08-30T20:22:24.173-05:00<p>A recruiter friend of mine was looking for a senior "Solution Architect" and needed some help setting up a screening interview. Here is what I told him.</p> <p>I look for solid experience in a diversity of solution. There are just too many people that think that HTML/Objects and RDBMS systems are the only tools in your architect's toolkit. </p> <p> Here is what I suggested the interview questions would be like:</p> <p> Consider the following application architectures:</p> <ol> <li>Ruby on Rails on a relational database</li> <li> XRX on a native XML database</li> <li>A Java Client on J2EE app server using JMS</li> <li>A JQuery client on an OLAP database</li> </ol> <p> A senior "Solution Architect" should be able to compare the pros and cons of each system and describe how they have used each of these architectures to build solutions and give organizations a competitive advantage. I also drill down into their understanding of search and retrieval and natural language processing.</p> <p> As a bonus question, I would ask how each architecture would impact an organization's business strategy and to what extent business units could be empowered to build their own applications with minimal training and little or no need for IT involvement.</p> <p> Most good solution architects should have experience with 3 out of 4 solution architectures but very few usually have a strong MBA-type background to understand how solution architectures impact business strategy. And if you find anyone that has built systems with all for and has an MBA you have a serious senior solution architect. Hire them!</p><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-767065482844225463?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-31475771985578511532010-07-02T10:15:00.006-05:002010-07-02T10:50:01.745-05:00Two weeks ago I attended my fifth Semantic Technology conference in San Francisco.<span style=""> </span>It was a great conference!<span style=""> </span>This is my fifth time attending the conference and I plan to attend in the future!<span style=""> </span>My biggest dilemma was which session to attend. At times there were as many as eight concurrent sessions going on.</p> <p class="MsoNormal">There were a few big trends that I spotted.</p> <p class="MsoNormal">The use of RDFa to annotate web pages using semantically precise elements was clearly a big trend.<span style=""> </span><a href="http://twitter.com/jaymyers">Jay Myer</a> of <a href="http://www.bestbuy.com/">BestBuy </a>described how the BestBuy sales went up 30% when they added RDFa tags to their product pages.<span style=""> </span>Although many search engines are not transparent about their use of RDFa tags in page rankings, Jay’s results should make it clear that this strategy works.</p> <p class="MsoNormal">Jay told a great story about how hard it was to find a fridge that met his specific criteria: black with a specific size etc. using a keyword-based search engine.<span style=""> </span>The semantic web will change all of this!</p><p class="MsoNormal">The big factor here is <a href="http://www.heppnetz.de/">Martin Hepps</a> <a href="http://www.heppnetz.de/projects/goodrelations/">GoodRelations </a>ontology for products and services. This can really bring the Semantic Web to the masses. Finally a way to code the hours of operation for your store so that you can use <a href="http://siri.com/">Siri </a>to ask "What Sushi Restrauants are Open at 10pm Near Here" </p> <p class="MsoNormal">There were also a lot of sessions on <a href="http://en.wikipedia.org/wiki/Linked_Data">LinkedData </a>and specifically on LinkedData in the government area.<span style=""> </span>Jim Heldler and his students at RPI are scraping every data set on data.gov and converting them all to RDF and storing them in a huge triple store.</p> <p class="MsoNormal">Jim also provided one of the best sound-bites from the conference:</p> <p class="MsoNormal">“Get AWAY <span style=""> </span>from the Table”</p><p class="MsoNormal">This goes far beyond the <a href="http://en.wikipedia.org/wiki/NoSQL">NOSQL </a>movement that we are seeing. It gets to the core of the problems with innovation in many organizations. </p> <p class="MsoNormal">Jim is a consultant to the US data.gov transparency process and a key advocate of open linked data standards.<span style=""> </span>It is interesting to see a friendly rivalry between Jim and Tim Berners-Lee who is a consultant for the UK data transparency movement.<span style=""> </span>Both are using RDF to convert data but each is using slightly different strategic approaches.</p> <p class="MsoNormal">He was referring to the fact that many organizations over-use the relational model and they need to understand that there are many alternatives, especially when doing mashups of data sets from many sources.</p> <p class="MsoNormal">There were also many presentations on natural language processing and finding the true “meaning” of words within unstructured and semi-structured data sets. </p> <p class="MsoNormal">I always attend <a href="http://www.bosatsu.net/">Brian Sletton</a>’s session on REST.<span style=""> </span>Many people do not understand the relationship between REST, URI’s and the semantic web stack.<span style=""> </span><a href="http://history.state.gov/">Joe Wicentowski’s work</a> at the US Department of State on the URL rewriting frameworks within eXist has really reinforced how this can be done easily in a single XQuery module.</p> <p class="MsoNormal">I presented a 3.5 hour tutorial session on Entity Extraction.<span style=""> </span>Since it was scheduled for the last session of the last day of the conference I though the attendance would be very low.<span style=""> </span>But the room was packed and most of the people stayed till the very end.</p> <p class="MsoNormal">It was wonderful to finally meet Marie Wallace and DJ <span class="gi">McCloskey </span>from the <a href="http://www-01.ibm.com/software/globalization/topics/languageware/index.jsp">LanguageWare </a>team at IBM.<span style=""> </span>Their support of the Apache UIMA standards will be a great step forward to the creation of interoperable language analysis piplelines.<span style=""> </span>Marie and DJ introduced me to the Millennium restaurant just a few blocks from the hotel. <span style=""> </span>My wife Ann and I went there over the weekend and we now own one of their cookbooks. </p> <p class="MsoNormal">By the way Google is now also support recipes in their <a href="http://www.google.com/support/webmasters/bin/answer.py?answer=99170">rich snippets</a> in their search results so you will soon be able to find "only recipes that take under 30 minutes" in a Google search engine.</p> <p class="MsoNormal">There were also many session on the need for good taxonomies and ontologies in the enterprise.<span style=""> </span>The use of <a href="http://www.w3.org/2004/02/skos/">SKOS </a>for controlled vocabularies was a very hot topic and there were dozens of presentation on how OWL is being used to capture and exchange business rules.</p> <p class="MsoNormal">It was also really great to finally meet <a href="http://www.jenitennison.com/">Jeni Tennison</a> after reading her books and following her Tweets for a long time.<span style=""> </span>She is deeply involved in the federal data transparency projects in the UK.<span style=""> </span>Here use of RDF is breaking new best practices for the entire community.</p> <p class="MsoNormal">It was also great to catch up with Mark Birbeck, one of the chief architects of the <a href="http://code.google.com/p/ubiquity-xforms/">Ubiquity XForms libraries</a>.<span style=""> </span>I am looking forward to trying out some of the new tools based on his backplane JavaScript libraries.</p> <p class="MsoNormal">The people from Facebook also gave a presentation on how they are adding the ability for people to add a few lines to each web site to allow users to add a “like” button on a web site using a variation of RDF.<span style=""> </span>Many people were a little disappointed to hear that they will not be using namespaces in their interfaces.<span style=""> </span>Facebook felt that host HTML coders could only handle a single namespace.<span style=""> </span>Many people agreed with their findings and thought that until 90% of HTML coders knew what namespaces were that organizations like Facebook were stuck just adding new data to HTML meta elements.</p> <p class="MsoNormal">One of the most interesting discussions I had was with the people from <a href="http://www.cray.com/">Cray Computer</a>.<span style=""> </span>Apparently a large unnamed government agency has given Cray Research a very large contract to build customized ASIC (FPGA) chips to do graph analysis.<span style=""> </span>Their ThreadStorm <a href="http://www.cray.com/products/xmt/">XMT </a>architecture has 128 register sets and allows the CPUs to get continuous feeds of graph queries without waiting for memory.<span style=""> </span>Their claim is that the federal agency has told them they are getting a 100X improvement in complex graph queries.<span style=""> </span>The challenge is that the API is currently a low-level C interface and they have not yet put a SPARQL complier in front of an XML.<span style=""> </span>So on good SPARQL benchmarks are yet available to compare the results of this hardware with a typical triple store.<span style=""> </span>But even if it is fast, the price would be in the six digits for one of the XMP systems.<span style=""> </span>So only very large organizations and government agencies might use this unless it was provided as a service.</p> <p class="MsoNormal">One of the other things I found was how many people are using OWL and the OWL reasonerers like <a href="http://clarkparsia.com/pellet">Pellet </a>as replacements for traditional rules engines from companies like FairIsaac.<span style=""> </span>There are two reasons for this.<span style=""> </span>First is that many rules engine companies only talk about their engine performance but not on the need for precise semantics in the data elements.<span style=""> </span>With OWL and the methods behind the semantic web stack we attempt to put semantics higher in the requirements of a system a use pre-built and pretested ontologies.</p> <p class="MsoNormal">The second big reason that people are use OWL is that the rules are created using open standards.<span style=""> </span>That means you are not locked into any one vendors rule format.<span style=""> </span>The new W3C standards for rule interchange (RIF) will also start to break down many of the barriers to the creation and exchange of large industry vertical rule sets to that each organization does not have to start from scratch with a rule base.<span style=""> </span>This will be big for industries like insurance where claims processing ontologies are just being created.<span style=""> </span>Thanks to <a href="http://clarkparsia.com/weblog/2010/05/26/another-reason-semantic-web-kicks-ass/">Kendal Clark</a> for good information on this topic.</p> <p class="MsoNormal">I also had a great time talking to many people in the publishing area.<span style=""> </span>Seth Maislin gave a great presentation on Taxonomies.<span style=""> </span>Seth and <a href="http://thetaxonomyblog.wordpress.com/">Marlene Rockmore</a> joined me for a lunch at nearby Indian restaurant.<span style=""> </span>Marlene is an expert on Taxonomy development.<span style=""> </span>She was working on a book with O’Relly on taxonomies that is on hold for now but we hope to see one in the future from her.</p> <p class="MsoNormal">It was also nice to see how many federal agencies are now starting to use SPARQL in the intelligence community.<span style=""> </span>I saw a good presentation by <span class="entry-content">Dennis Wisnosky with the US DOD talk about how they plan to use NIEM and semantic technologies to cut their integration costs.<span style=""> </span>Yeah NIEM!<span style=""> </span>Most people don’t understand that the NEIM is based on the RDF model, it just uses XML syntax to store the relationships.<span style=""> </span>See my web-cast on the SemanticUniverse.com web site for more details. <span style=""> </span>Hopefully his slides will be available in the future.<span style=""> </span>Many people were taking photos of the slides since the DOD is not great at getting their slides out.<span style=""> </span>Wonder why?<o:p></o:p></span></p> <p class="MsoNormal"><span class="entry-content">I also had lunch with several people from within the intelligence community that had great discussions about the pros and cost of pulling assertions out of NLP-analyzed annotated XML documents and the challenges relating to this.<span style=""> </span>Keeping the links back to the original XML documents to is critical to verify the results of an assertion in context.<span style=""> </span>Traceability, linage, provenance and time-domain representations in RDF that don’t cause a 10X growth is the number of triples is a difficult problem.<o:p></o:p></span></p> <p class="MsoNormal"><span class="entry-content"><span style=""> </span>If there was some way to relate RDF assertion to a source document XML Node ID (the fourth column question) is a very difficult problem and one that needs close research with native XML and RDF systems.<span style=""> </span>One of the things I learned about the <a href="http://www.exist-db.org/lucene.html">eXist-Lucene</a> integration done by Wolfgang Meier is that keeping the node-id as the document ID in Lucene allows non-programmers the ability to configure customized search rank rules past on the context of the keyword in a document.<span style=""> </span>This is where highly customizable <a href="http://www.syntactica.com/solutions/structured-search.xq">structured search</a> rocks.<span style=""> </span>I think this innovation needs to be added to RDF triples projects.<span style=""> </span>Keeping the XML node-id of an assertion with an RDF triple context could be a great way to prevent RDF triple bloom for context.</span></p><p class="MsoNormal"><span class="entry-content">Finally, I felt that David Wood and James Leigh's presentation on </span><span class="status-body"><span class="status-content"><span class="entry-content"><a href="http://code.google.com/p/callimachus/">Callimacus </a>project had the most potential to have a "big impact" on the adoption of Semantic Web tools. This new open source project allows users to specify a simple HTML template with special XML tags that allows them to bring triples directly into a web application. This has the potential to allow far more non-SPARQL programmers to integrate RDF data directly into a web page much like XQuery does today. With some work it might be possible to auto-generate the XForms bind elements for form rules. This would be a huge win for the integration of rules into web forms without needing to write JavaScript. </span></span></span></p><p class="MsoNormal"><span class="status-body"><span class="status-content"><span class="entry-content">You can also see my tweets on the conference here: http://twitter.com/dmccreary</span></span></span></p><p class="MsoNormal">I hope to see more of you in San Francisco next year. Let me know if you are interested in co-presenting any papers! <span class="status-body"><span class="status-content"><span class="entry-content"></span></span></span></p><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-3147577198557851153?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com1tag:blogger.com,1999:blog-13546516.post-5992352697818040232008-11-19T15:06:00.001-06:002008-11-19T15:22:29.124-06:00Does anyone have any suggestions on a good reliable WebDAV product for Vista 64bit? I can not get the one that is bundled with Vista to work. Every time I try to create a remote connection I get the following error message: <br/><br/> "The folder you entered does not appear to be valid" <br/><br/> I have run the Vista WebDAV patches: <br/><br/> <a href="http://www.microsoft.com/downloads/details.aspx?familyid=17C36612-632E-4C04-9382-987622ED1D64&amp;displaylang=en">KB907306</a><br/><br/> I have also tried the following Hotfix: <br/> <br/> <a href="http://hotfixv4.microsoft.com/Windows%20Vista/sp2/Fix224785/6000/free/347160_intl_x64_zip.exe">Windows Vista WebDAV Hotfix</a> <br/> <br/> I have also tried to get various 3rd party packages working such as Independent DAV, BitKinex and WebDrive. All of the WebDAV products seem to have problems with the Windows Vista 64 bit systems. Is there any way to get Vista to use the Windows XP WebDAV "drivers"? I am using both eXist and MarkLogic native XML databases. Thanks - Dan<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-599235269781804023?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com1tag:blogger.com,1999:blog-13546516.post-9508728083597352762008-11-12T11:48:00.003-06:002008-11-16T12:23:44.848-06:00I have been learning how to use higher-order functions in XQuery. I am using the eXist system to test this. To do this you will need to get the 1.3 build that there is no download for (yet). Here are the steps I used: <ol> <li>Download fresh copy of eXist from svn via TortoiseSVN or Eclipse subversion subclipse plugin from the eXist SVN sourceforge repository: <pre>https://exist.svn.sourceforge.net/svnroot/exist</pre> </li> <li>(Confirm environmental variables are set correctly for JAVA_HOME and EXIST_HOME) To do this I use the Windows/Computer/Properties/Advanced Settings and check it with the SET command at the CMD prompt. The result looks like this: <pre> JAVA_HOME=C:\Program Files\Java\jdk1.6.0_10 </pre> My Eclipse workspace is just "C:\ws" and I created a project called eXist-1.3 so the path name I used for EXIST_HOME is: <pre> EXIST_HOME=C:\ws\eXist-1.3dev </pre> </li><li>Open the DOS cmd prompt, cd to eXist directory </li><li>type in "build.bat" </li><li>Type in "build.bat -f build\scripts\jarsigner.xml" </li></ol> It took just 1 minute and 6 seconds for the main build to run on my 4CPU system with 8GB of RAM even though it is running Vista. Then I just ran the start.bat in the bin directory. eXist was then running on http://localhost:8080/exist. I had to use the new WebStart admin tool on the admin section of the eXist web page to change the admin password. Thanks to Joe Wicentowski at the US Dept of State for helping out!<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-950872808359735276?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-53404556387009477242008-06-05T07:34:00.001-05:002008-06-05T07:49:05.215-05:00The blogs on xml.com are down so I have reposed this here. Have you ever wondered if the laws of evolution apply to computer languages? When you walk down the isle at your favorite bookstore, does it seam like there are actually more computer languages than last year? What forces are driving each of these new languages to evolve? In 1835 Charles Darwin visited the Galapagos Islands. There he collected what he thought were about a dozen distinct species of birds. Upon returning to England he discovered that each of these species had evolved from a single species of finches. On the various Galapagos Islands the requirements for food gathering was different, but consistent over hundreds of thousands of years. Enough time for a single species to adapt to meet consistent requirements. Consider the Raccoon: omnivores that have proved to be one of the most adaptable mammals on Earth. The Raccoon’s range has rapidly expanded into urban areas due to their ability to quickly adapt to new requirements before other animals have had time for the wheels of evolution to turn. So goes it with computer languages. Some procedural languages can be quickly adapted to fill in the needs for a new niche. When the web was young, procedural languages like Java and JavaScript quickly filled in the need for a variety of tasks. As the requirements for building web applications stabilized, declarative systems like CSS, XForms and XQuery started to push procedural languages back into niche-areas. As these declarative languages stabilize and become worldwide standards, graphical tools are being created to allow non-programmers to create, manipulate and extend these systems. This is why many of us believe their will always be some need for procedural programming, but certainly not for building standard web applications that are controlled by style sheets and user interaction forms. Like the finch, declarative languages need a little longer to evolve. It sometimes takes years for a small vocabulary of functional specification patterns to emerge and be given labels. Additionally, it can takes years for the standards bodies to agree on the best way to deliver these new languages in a set of semantically precise data elements that have unambiguous interpretations. Finally, it may take another few years for IT managers to realized that they really do lower costs if they avoid vendor-specific implementations and adopt worldwide standards. When CSS first came out you may have been a little reluctant to let web designers play with a rules engine. As XForms becomes ubiquitous you may be resisting change because you have invested so much time and energy learning how to debug JavaScript (without a debugger). You can not hold back the forces of evolution…and now we all need to adapt to the declarative world or risk our own extinction. If you are interested in more on this topic see my Presentation from the 2007 Semantic Technology Conference <a href="http://www.danmccreary.com/presentations/sem-web-07/">The Semantics of Declarative Systems</a><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-5340455638700947724?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-24492401973368261522008-05-27T07:33:00.001-05:002008-05-27T07:41:25.795-05:00I recently started writing for O’Reilly Media. I posted an article on XRX. Here is the link: <br/> <a href="http://www.oreillynet.com/xml/blog/2008/05/xrx_a_simple_elegant_disruptiv_1.html">http://www.oreillynet.com/xml/blog/2008/05/xrx_a_simple_elegant_disruptiv_1.html</a> <br/> We are just getting started moving to a new MoveableType (MT) system so not all of the features (like keywords and feedback) are working. <br/> If you have problems commenting on the site, please feel free to post your comments here. <br/> Here is a comment from Arun Batchu: <br/> <em>An elegant introduction to an elegant architecture, Dan. Thank you. The essential takeaway from what you describe is the exploitation of XML from one end to the other end - especially from the Developer's perspective, for, how XRX actually manifests in runtime could be left to implementation technologies. Thus the logical architecture of XRX could be realized by a few variations of concrete technology - which is great. The XForms could be realized by XForms server technology (such as the excellent Orbeon stack), the ReST could be realized by any middle tier and the XQuery could be realized by an XQuery engine (such as Data Direct) that may actually be driving any one or a combination of datastores (XML, SQL, file system or ...) . The symmetrical and consistent leverage of XML as a data model from creation to transport to rest and back eliminates a whole lot of wasteful work. Like you point out, XPath is one of the most powerful query systems I have encountered; you can pack so much in so little and reuse it across the board with little if any change from the drawing board to production. In such a system as you describe, a business rule expressed once can be reused anywhere - from one end-to-end , however long the travel, as long as the architecture is XRX, like you have described. Thanks for expressing it so well. A few of your readers will not get it - it is one of those things that once you experience it, you are left wondering why this did not happen before. Oh, well!</em> <br/> <br/> You are welcome! Thank for your feedback Arun!<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-2449240197336826152?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com1tag:blogger.com,1999:blog-13546516.post-4444034011876705752008-03-22T11:34:00.000-05:002008-03-22T11:40:11.066-05:00<p>Those of you that have worked with me know that I am a little bit of an odd-duck. I have dyslexia. If you ever see me attempt to write on a whiteboard my spelling is at the ninth grade level. My left brain, used for phoneme recognition, never really developed like the rest of you. My right brain had to be co-opted to help out. But one of the skills I seemed to have picked up due to my over-exercised right brain is the ability to visualize multiple complex application architectures and quickly understand architectural tradeoffs. I am one of the few people that seem to be interested in discussing how XForms, metadata registries, ontologies, the semantic web, OWL, RDF, graphs, business rules, BPM and Kimball conformed dimensions all can work together to deliver elegant and cost-effective enterprise-scale solutions. It seems easy for me to simultaneously visualize two or more architectures and it constantly challenges my patience when I have to explain over and over why architecture alternatives will not meet a business requirement.</p> <p>It turns out that many people that have dyslexia also have the gift of being able to visualize complex systems. Albert Einstein, Thomas Edison, Jackie Steward and Charles Schwab are good examples of dyslexic people that have used the strengths of the right brain to do things that left-brain thinkers could not.</p> <p>As a right-brain centric person I need to also tell you that I really love the XForms architecture. The magic of a declarative language, MVC, bindings and a dependency graph makes XForms development 10 years more advanced than anything else I have worked with. I think it is beautiful and elegant. It is everything that AJAX and JavaScript application are not. Clean, simple and easy to visualize (for me at least). When someone asks me if I can create an XForms application to do something, I create a mental image in my mind of the model, the view and how events will update instance data in the model using inserts or external submission results. I can easily visualize the bindings of view controls to data elements in the model. Once I can visualize the application clearly, writing the application is just a matter of typing in the code.</p> <p>I think that my right-brain is also a reason I detest JavaScript and AJAX. It is far too much code to read and trying to visualize how 300 lines of JavaScript enables me to do a drag-and-drop. I want to just add an attribute to an element like "drag-source" and "drop-target" and I want it to just work.</p> <p>What triggered this posting is that I have been reading <i>Proust and the Squid</i> by Maryanne Wolf. This is a book about how the brain's circuits are used in the reading process. She has a wonderful explanation of how the dyslexic brain co-opts the right brain for reading and enhances it functionality. I didn't really understand the relationship between my defects and my gifts .</p> <p>So how about you and your development team? Do you have a dyslexic right-brained person on your team? Can they quickly visualize architectural tradeoffs? Have they tried XForms? And if they do, will you be willing to tolerate their disgust of AJAX and JavaScript after they have built their first XForms applications?</p> For more information about dyslexia check out these two Wikipedia entries: http://en.wikipedia.org/wiki/List_of_notable_people_diagnosed_with_dyslexia http://en.wikipedia.org/wiki/Dyslexia<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-444403401187670575?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com2tag:blogger.com,1999:blog-13546516.post-70890489780630402782008-03-21T15:01:00.001-05:002008-03-21T15:55:02.964-05:00Here is a beautiful example of a bubble chart display using global population statistics. The example on carbon emissions are very interesting. Note the dimensions <ol> <li> Population of County (size of bubble) </li> <li> Continent (color of bubble) </li> <li> Live Expectancy (vertical axis) </li> <li> Income (horizontal) </li> <li> Time (the play button) </li> </ol> It is interesting to see the huge impact AIDS has had on the life expectancy in African counties. Imagine if you could see your organizations product sales using this type of graph. This application was done with a software system called Trendalyzer. It was initially developed by Hans Rosling's Gapminder Foundation in Sweden and acquired by Google Inc. in March 2007. This version is a Flash application. Does anyone know of any open-source software that could do this?<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-7089048978063040278?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-10161714197170420872008-03-21T11:06:00.004-05:002008-03-21T16:02:08.866-05:00<p>For several years people have been using the terms metadata <b>Registry</b> and <b>Repository</b> inconstantly, imprecisely and almost interchangeably and I would like to weigh in as to how these terms could be used more precisely to allow organizations to effectively to manage metadata processes.</p> <p> First lets take the definition of a <b>Repository</b>. Webster defines a repository as <i>…a place, room, or container where something is deposited or stored.</i>. Note that here is nothing in this definition about the quality of the things being stored or the process to check to see if new incoming items are duplicates of things already in the repository. If I have 100 users they could each define "Customer" as the see fit and put their own definition into the metadata repository as their own definition. No problems.</p> <p>On the other had lets take the word <span style="font-weight: bold;">Registry</span>. A Registry has the connotation of more than just a shared dumping ground. Registries have the additional capability to create workflow processes to check that new metadata is not a duplicate (for a given namespace). One of the definitions from Webster is <span style="font-style: italic;">an official record book</span>. Note the word <span style="font-weight: bold;">official</span>.</p> <p>A Repository is similar to a front-porch of a house. No locks prevent new things from landing there. But a Registry is a protected back room where human-centric workflow processes are used ensure that metadata items are non-duplicates, precise, consistent, concise, distinct, approved and unencumbered with business rules that prevent reuse across an enterprise. These registries have become the central foundation that agility can be baked-in to many enterprise process. The latest version of the Kimball's <i>Data Warehouse Lifecycle Toolkit</i> (which is actually a very good read) even goes as far as to call their process "metadata-driven". Not different the model-driven development world.</p> <p>Registries have the implicit connotation of trust behind them. They now serve a a central process for the creation of shared meaning across the enterprise. Definitions in a registry have been vetted by an enterprise-level organization that has the responsibility of enterprise data stewardship. They have a high probability of being consistent with industry best-practices and vertical industry standards. Registries are the go-to source for creating canonical XML schemas, enterprise ontologies or conformed dimensions in a OLAP cube. Repositories are personal or small departmental definitions of an isolated view of the world.</p> <p>None of these ideas are really new. They are at the core of the ISO/IEC 11179 metadata registry standard. Note that they don't call it a <span style="font-style: italic;">repository </span>standard! People are just now starting to understand how important Registries are in most enterprise-wide systems. The growth of Business Intelligence and Enterprise Data Warehouse terminology and Service Oriented Architectures is a good place to see the rise of repositories and registries. We now see service registries, portlet registries, model registries...the list goes on-and-on.</p> <p>Much of the background on the differences between the use of repositories and registries can be traced way back to the early days of object-oriented systems in the 1995 book <i>Succeeding with Objects</i> by Adele Goldberg and Kenneth Rubin. This was one of the first books on enterprise reuse strategies and they defined the concept enterprise asset reuse and the need for a trust-driven repository as a basis for reusing assets. They identified a multi-step process for reviewing new submissions to determine if the submission duplicated existing assets. They showed how critical it was to classify items in a registry and search an existing registry for duplicates before new items are added. If you can get a copy of the book I would suggest you read the section on "Set Up a Process for Maintaining Reusable Assets" on page 245.</p> <p>The book then goes on to show how organizations can and should be structured to reuse these assets and gives the pros and cons of the differing organization structures and their impact on reuse. This is the basis for the data governance and data stewardship movement in many organizations today.</p> <p> So the next time someone uses the word registry or repository in a conversation, ask them if they are using the definition of the word that is consistent with the corporate business term registry or is their own private definition from their own repository of imprecisely used buzzwords.</p><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-1016171419717042087?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com2tag:blogger.com,1999:blog-13546516.post-5797736993147999432008-03-09T16:52:00.001-05:002008-03-09T17:43:42.802-05:00<p> I am happy to announce that the XForms Tutorial and Cookbook that I have been working on for over a year been voted a "Featured Book" on the wikibooks web site. A quote from the award: </p> <p style="font-style:italic;">XForms is a featured book on Wikibooks because it contains substantial content, it is well-formatted, and the Wikibooks community has decided to feature it on the main page or in other places. Please continue to improve it and thanks for the great work so far! </p> <p> Here are some stats on the book so far: 90 sample programs</br> 116 chapters</br> 626 edits</br> 28,811 words</br> 29 registered authors </p> <p>One of the big tasks is to start to move some of the advanced XForms examples that require siginficant server-side logic to a seperate server-specific book. As a pilot I have started an <a href="http://en.wikibooks.org/wiki/XRX">XRX</a> cookbook for people using REST interfaces and the eXist server. </p> <p> I would like to thank the over 30 others that have contributed ideas and content to this wikibook. We still have lots of work to do to cleanup the example programs, make them more consistent and add additional examples for new XForms students. But I believe it is one of the best examples of collaborative training that I have worked on in the last few years. </p><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-579773699314799943?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com1tag:blogger.com,1999:blog-13546516.post-12335817574668668232008-01-06T12:02:00.001-06:002008-03-21T16:04:32.024-05:00<b>Summary</b> <p>This posting describes how using three technologies (XForms, REST and XQuery or XRX) you can dramatically transform your organizations development methodology. You can move true model-driven architecture (MDA) that dramatically reduces the temptation to duplicate code.</p> <p>In January of 2007 I was working on a complex real-estate forms project for a state agency in Minnesota. Our job was to get 87 counties across the state of Minnesota to agree on the over 250 data elements that describe real estate transactions. We had spend months developing a complex XML Schema and web forms with over 50 one-to-many relationships. And out next task was to be able to save the form <p>data into a structure that could be quickly queried by any of these 250 data elements. But the prospect of "shredding" the documents up into 50 distinct SQL INSERT statements and then reconstituting the documents with 50 distinct SQL SELECT statements was going to require doubling our small team of four to at least eight developers. We didn't have the budget or schedule to do this. After chatting with Kurt Cagle he suggested saving the data to a database that supported XQuery. Both the open source eXist native XML database and our corporate standard, DB2 supported XQuery. It turned out that we could indeed save the entire document with a single command and still perform complex queries on any element in any document. Our project could proceed with a small team and stay on schedule.</p> <p>And then an interesting thing happened. Our team started to understand that if we saved additional metadata in these native XML data stores we could accelerate our project even further. Every pick-list in every form could dynamically call REST-enabled web services to get their values. XML Schemas could be stored in the databases and be queried. We had found a simple and elegant solution to the pervasive problem of where to store the model in a model-driven development project.</p> <p>In the past I had been on large teams of developers that attempted to use model-driven development. But although we started out each project with idealistic goals, each time we were faces with a classic option: copy, paste and edit the model or write a transform. The copy and paste solution was quick but it would have to be redone each time the model changes. The transform took longer to write but then could be rerun each time the model changed.</p> <p>Like most developers, I always was over-optimistic that I had done my homework and done accurate and detailed modeling of the problem. I thought the models were stable and would not change very often. And I was almost always wrong. Models do change and sometimes in totally unexpected ways.</p> <p>But by storing our models in a native XML database an interesting started to happen. We could quickly transform the models (stored in XML) into other artifacts with simple XQueries. It became far less tempting to commit the sins of copy/paste/edit when it only took five minutes to create another transform. And most remarkably, each of these transforms was another REST enabled web service that could be reused by many web clients or other development tools.</p> <p>So how do you get started? You might try the following steps. <ol> <li>Download the exist database from http://www.exist-db.org</li> <li>Use a WebDAV interface and copy some XML Schema documents into a collection</li> <li>Write a few small XQuerys that pull metadata out of your XML Schemas. For example pull out all the enumeration values out of an element.</li> <li>Use those XQueries to drive some aspects of your development such as an XForms selection list.</li> <li>Compare the amount of code you wrote with any other MDA system. If you don't have at least a 10 to 1 savings I would be very surprised.</li> </ol> </p> <p>What do you think? Do you have a more efficient way to query your model? Let me know!</p> <p>- Dan</p><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-1233581757466866823?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com1tag:blogger.com,1999:blog-13546516.post-57360989037993496672007-12-17T19:28:00.000-06:002007-12-17T19:30:35.498-06:00<p class="MsoNormal">As the year starts to come to a close we often take some time to look back at the year and look at the innovations that we have seen that have changed our worldview.<span style=""> </span>I can think of many things that have had a large impact on me: The FireFox XForms extension, discovering the beauty of a well-designed REST interface, the mule enterprise service bus, Yahoo pipes and microformats all come to mind.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">But after careful consideration I have to hand my award out to <b style="">the most innovative application of the year</b> to me to the <b style="">eXist database/web server</b>.<span style=""> </span>This application constantly amazes me with what I can do with it.<span style=""> </span>If you have a Java JVM on your desktop it only take a few minutes to get it running.<span style=""> </span>Hit the startup script and you have a web server running on localhost:8080.<span style=""> </span>Open a WebDAV browser and drag-and-drop you files and folders and away you go.<span style=""> </span>With eXist I can easily setup a full enterprise metadata registry on a laptop in under five minutes.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">The power of eXist comes from its use of the XQuery engine.<span style=""> </span>Any well-formed XML file that is added to the system gets instantly indexed and can instantly be searched.<span style=""> </span>That means that 10ms after you hit the "Save" on your XForms you can see the data appear in XQuery reports.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">But the "Secret Sauce" of eXist rests in it's use of a lightweight Jetty web server with a remarkably integrated REST and WebDAV interface.<span style=""> </span>This means that every XML file…is a static web service.<span style=""> </span>This means that every XQuery that returns XML…is a dynamic web service that you can parameterize.<span style=""> </span>This means that you can be writing your first web services in fifteen minutes that can be grabbing data from a dozen different XML files and quickly serializing the results out the wire.<span style=""> </span><st1:place st="on"><st1:city st="on">Enterprise</st1:city></st1:place> mash-ups at your fingertips.<span style=""> </span>Very little programming is done other then selecting data.<span style=""> </span>Just like it should be.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">eXist technologies are going to start to have a big impact once XForms/REST and XQuery (XRX) web development matures.<span style=""> </span>I predict this is going to happen in the next three years.<span style=""> </span>An although IBM, Microsoft, Oracle and many others are supporting XQuery in a big way, they don't yet have the really smooth integration with next-generation XForms-driven clients.<span style=""> </span>You still need teams of <st1:city st="on"><st1:place st="on">AJAX</st1:place></st1:city>, JavaScript, Java, .Net and SQL programmers to build simple web services for rich client interfaces.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">I must admit it took me a little while to really understand how the simplicity of the interface really rocked my world.<span style=""> </span>Just swap out the world "rest" in the URL and replace it with the word "webdav" and you go from a data browser to a file system that works with every copy/cut and paste operation of the file system.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">It is also interesting to note that this product did not come out of a Bay Areas startup, it didn't come out of Google Labs, it didn't come from Microsoft or IBM.<span style=""> </span>From what I can tell the initial version was pretty much written by a single guy in <st1:place st="on"><st1:country-region st="on">Germany</st1:country-region></st1:place>, Wolfgang Meier.<span style=""> </span>It think this shows that there is still room for true innovation by a single individual in this world.<span style=""> </span>Although eXist now appears to have a great team of people behind it, it shows that a single person with a clear architectural can put the right pieces together really can make a difference in the world.<span style=""> </span>Our teams could not have been as productive with XForms if we didn't have a clean and elegant tool like eXist behind our forms.<span style=""> </span>I know that XRX innovators will no be able to stand on Wolfgang's shoulders and build incredible applications.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">Thanks Wolfgang! <span style=""> </span>Your labor of love is growing up!</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><o:p> </o:p></p><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-5736098903799349667?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com1tag:blogger.com,1999:blog-13546516.post-9675549671317086672007-12-14T08:08:00.000-06:002007-12-14T08:13:32.921-06:00<p class="MsoNormal"> At the XML 2007 conference there were many people that seemed to have independently discovered that if you combine XForms/REST and XQuery you can create a software development environment that circumvents the need for middle tier objects and conversion to and from relational databases.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">The conference also had a great deal of discussion of how XForms is a great architecture but as Elliot Rusty Harold pointed out, we need something to happen for XForms to replace things like <st1:city st="on"><st1:place st="on">AJAX</st1:place></st1:city>.<span style=""> </span>Although Rusty did propose a few interesting ideas (like getting XForms built into FireFox) I feel what we really need is a contagious meme. And I think it goes beyond XForms.<span style=""> </span>What we need is an easy-to-remember and easy-to-communicate name for a collection of great ideas.<span style=""> </span>For example there are few books that have the word <b style="">XMLHttpRequest</b> in it.<span style=""> </span>But there are hundreds of books with <st1:city st="on"><st1:place st="on">AJAX</st1:place></st1:city> in the title.<span style=""> </span>Once the word <st1:city st="on"><st1:place st="on">AJAX</st1:place></st1:city> was coined the meme started to spread.<span style=""> </span>The world was waiting for the "sticky" label behind the concepts.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">What XForms needs is a label around the collection of ideas that makes XForms really give you an order-of-magnitude improvement over other web development architectures.<span style=""> </span>I would like to suggest that we get behind one "label" that brings many of use under one big tent.<span style=""> </span>I would like to suggest we use "XRX".</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">XRX to me would stand for the ideas behind XForms/REST/XQuery web applications.<span style=""> </span>But if people are using XHTML as a container for XForms you could use XHTML/REST/XQuery.<span style=""> </span>Just spare use the "J" which is associated with a hard-to-use language for über geeks.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">You could also use XRX for the format of the data in the client and on the server.<span style=""> </span>In this case it would be XML on the client (in the XForms Model), and in a native XML database that supports a REST interface.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">We can still build awesome tools and frameworks that make the XRX architecture really fly.<span style=""> </span>IBM Workplace Forms Development tools, Orbeon Presentation Server, Kurt Cagle's new frameworks that use the eXist native XML database, and the hundreds of other XForms development tools all fit into the XRX strategy.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">But if we make XRX into a "big tent" strategy we can also include XQuery on the server as part of the architecture.<span style=""> </span>This would include people like Jason Hunter at MarkLogic.<span style=""> </span>Jason is an experienced Java developer that has started to move away from Java on the server and toward XQuery because of its richness.<span style=""> </span>And I could not agree more.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">I have been reading the book <i style="">The Tipping Point: How Little Things Can Make a Big Difference</i> by Malcolm Gladwell and trying to apply the ideas in it to the adoption of XForms and XQuery.<span style=""> </span>I think that the book has many example of how a careful packaging of the ideas behind a contagious meme can make ideas spread like widefire.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">So when we hear the term "XRX" we want to have the following concepts come to mind:</p> <p class="MsoNormal"><o:p> </o:p></p> <ol style="margin-top: 0in;" start="1" type="1"><li class="MsoNormal" style="">A web development architecture with a 10x productivity improvement over traditional JavaScript/OO/RDBMS methods</li><li class="MsoNormal" style="">A development architecture based on international standards that is designed to minimize the probability of vendor-lockin</li><li class="MsoNormal" style="">An architecture that gives a rich user experience without creating mountains of spaghetti procedural code on either the client or the server</li><li class="MsoNormal" style="">A system that leverages the REST architecture to take advantage of high-performance and simple interfaces using web standards</li><li class="MsoNormal" style="">Portability on both the client and the server using a variety of forms players and XQuery databases</li><li class="MsoNormal" style="">The option of avoiding costly shredding (and reconstitution) of complex XML documents into RDBMS tables</li><li class="MsoNormal" style="">A community of standards/tools and a "complete solution" ecosystem that can give you a proven ROI on your IT investment</li></ol> <p class="MsoNormal" style="margin-left: 0.25in;"><o:p> </o:p></p> <p class="MsoNormal">I don't expect everyone to buy into the XRX meme ideas.<span style=""> </span>Everyone has their own favorite tools/standards and acronyms.<span style=""> </span>But everyone that was at the XML 2007 conference could feel the excitement in the room at the XML presentations.<span style=""> </span>Now we need to give that excitement a name and see if it sticks.<span style=""> </span>Hopefully a few years we will be able to walk into a bookstore and see a bunch of books with "XRX" on the cover.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">What I now need is for people to start to test the<span style=""> </span>XRX meme and see if it slicks.<span style=""> </span>Let me know what works and what does work.<span style=""> </span>Send me your 30-second elevator pitch. How big should the idea tent be?<span style=""> </span>What do we have to do to get more people in our tent?<span style=""> </span>What evidence (case studies), example programs and resource do we need to convince the world we have seen the light …and its so elegant and beauty to will keep us from ever going back to the stone age of web development.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><o:p> </o:p></p><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-967554967131708667?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com1tag:blogger.com,1999:blog-13546516.post-64588873381970680092007-12-12T10:12:00.000-06:002007-12-12T10:28:29.082-06:00<p>At the recent XML 2007 conference in Boston several of us met to discuss what we could do to promote FireFox and some of the important standards built into FireFox (or almost built into FireFox). Specifically several of us were using XForms and SVG and thought that the combination offered some great opportunities.</p> <p> It was pointed out that the combination of the XForms range control (a slider control) and SVG allowed developers to create dynamic graphs for teaching educational concepts. There is an example here:</p> <a href="http://en.wikibooks.org/wiki/XForms/Pie_Chart">http://en.wikibooks.org/wiki/XForms/Pie_Chart</a> <p> Note there is a link to the actual program here which you will need to use the FireFox XForms extension to view.:</p> <a href="http://xforms-examples.googlecode.com/svn/trunk/10-full-examples/86-piechart-range-controls/pie-chart.xhtml">http://xforms-examples.googlecode.com/svn/trunk/10-full-examples/86-piechart-range-controls/pie-chart.xhtml</a> <p>So several of us are interested in putting together a grant to develop SmartBoard enabled educational content for the K-12 market using a combination of FireFox, XForms and SVG. The goal would be to create a "Killer App" for FireFox in the K-12 space. Ideally we will build 100 demos of some of the most useful dynamic graphs and then build a framework for teachers to extend these applications. </p> <p> I also have a few Supply and Demand examples that run under IE but require the SVG add-on and just just plain JavaScript for the range-controls: <a href="http://http//www.danmccreary.com/svg/">http://www.danmccreary.com/svg/</a> You will note that visualizing the maximum profit by the area of the square under the triangle is much easier when you add the motion. </p> <p>It was also pointed out that the Mozilla foundation has the funds to support this type of development and that they might fund a grant to build the top 100 useful dynamic graphs. We could then start building a toolkit to extend these dynamic graphs using some of the advanced drag-and-drop connection-based programming parts of SVG and FireFox. The kit should shield the average middle school physics teacher from haveing to write and debug JavaScript.</p> <p>Let me know what you think - Dan</p><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-6458887338197068009?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-86626476204326534122007-12-08T17:23:00.000-06:002007-12-09T10:35:02.057-06:00<p class="MsoNormal">I returned from the XML 2007 conference in <st1:city st="on"><st1:place st="on">Boston</st1:place></st1:city> with my head spinning about all the new developments with XForms and XQuery.<span style=""> </span>The conference was three days long and had between three and five tracks going concurrently.<span style=""> </span>My impression was that there were about 400 people there.</p> <p class="MsoNormal">Here are some of the highlights for me.</p> <p class="MsoNormal"><b style="">XForms</b> appears to be getting lots of traction.<span style=""> </span>The XForms sessions were all packed with standing-room only in most of them.<span style=""> </span>The presenters we passionate that the combination of XForms, REST and XQuery has had a huge impact on their projects and may soon impact the rest of the development world.</p> <p class="MsoNormal"><b style="">Microformats</b> are definitely starting to make huge strides where Metcalf's Law will start to kick in.<span style=""> </span>The existence of 450 million tags and five FireFox microformat extensions indicate that within a year it will be just part of web publishing best practices.<span style=""> </span>Even Microsoft is said to be supporting Microformats in IE 8, just a few years from now.</p><p class="MsoNormal"><b>Taylor Cowan</b> (who works for Sabre) gave an excellent presentation on how Microformats can be extended into the travel space with an Atom based format for publishing trip ideas. His use of Atom and semantic web technologies was right in line with my vision of how personalized agents are going to leverage microformats. </p> <p class="MsoNormal">John Boyer's demo of IBM's <b style="">Workplace Manager</b> showed that you can now purchase top-shelf XForms GUI design tools that does not lock you into a proprietary vendor solution.<span style=""> </span>I hope the people from Microsoft, Adobe, FairIsaac, Altova, Google GWT and other vendors take notice.<span style=""> </span>Locking your users into a Java or JavaScript client is no longer an acceptable strategic option for developing web applications.<span style=""> </span>Wake up everyone!<span style=""> </span>Are you listening? We have a standard that works and lets now start building more great tools that use this standard.<span style=""> </span>XForms is the only system that is creating as John puts it "Order of Magnitude" increases in developer productivity.</p> <p class="MsoNormal">Mark Birbeck's <b style="">Sidewinder</b> demo showed that XForms is breaking out of the browser an on to the desktop.<span style=""> </span>Showing web and iPhone web pages directly in desktop widgets was a great demonstration of the power of XForms.</p> <p class="MsoNormal"><b style="">Kurt Cagle's</b> presentation about problems with programming with the fragile DOM model in JavaScript was interesting in its format.<span style=""> </span>The presentation was actually an XForms application running on eXist.<span style=""> </span>Kurt "eats his own dog food" on the client and the server!<span style=""> </span>I am humbled by the man.</p> <p class="MsoNormal">Jason Hunter from <b style="">MarkLogic</b> (an XML database vendor) gave an nice demo of the MarkMail.org application and showed that, yes, XQuery databases really do scale to the terabyte range.<span style=""> </span>He has about 4 million e-mail messages indexed and you can do combinations of database and text searches in a few seconds.<span style=""> </span>In other words a demonstration of how real-world analytics comes to XQuery.<span style=""> </span>I also found out that MarkLogic is free if your database is under 100MB.<span style=""> </span>Perfect for 10MB of metadata in a metadata registry.</p> <p class="MsoNormal">The people from IBM research labs appear to be starting to take the XForms MVC architecture and really run with it.<span style=""> </span>We saw some very interesting presentations on how future standards for data driven <span class="caps">XML</span> Application Components (XACs) and State Chart <span class="caps">XML</span> (SCXML) might be used in the future to make browser-based mash-ups easier.<span style=""> </span>Componentization, composition, customization and reuse are all part of this architecture and it would not be possible without XForms MVC architecture. Hopefully Charles Weicha from IBM research will post his slides so you can see more.</p> <p class="MsoNormal">Norm Walsh also talked about how the new <b style="">XML Pipeline standard</b> (<b style="">XProc</b>) is starting to take shape.<span style=""> </span>Even though the standard is still in working draft there are already at least five implementations of the standard.<span style=""> </span>This single standard could have more impact on the shape of server-side processing then any other standard coming out of the w3c. <span style=""> </span>Integrating this into XQuery-based application servers seems to be the next step.</p> <p class="MsoNormal">The <b style="">eXist</b> database also seemed to clean up all the open source native XML database talks.<span style=""> </span>eXist was mentioned in over a half-dozen presentations and there were no other native XML databases even mentioned in the presentations I attended.<span style=""> </span>It is not surprising that eXist is now the top hit on Google when you search for "XML database".</p> <p class="MsoNormal">One of the most interesting presentations that I did not get a chance to see (but I did talk to the presenter afterwards) was <b style="">Thomas White's</b> presentation on using XSLT in the browser to get a 10x speedup in form rendering.<span style=""> </span>It turns out that implementations of XForms players like FromFaces use heavyweight JavaScript objects in the browser.<span style=""> </span>Thomas has found that a donkey cart can be turned into a sports car by implementing an in-browser XSLT centric event scheduler. If his architecture is implemented we could see vastly better XForms players in the future. <span style=""> </span>Lets all hope that Thomas's architecture comes to the XForms world sooner than later.</p> <p class="MsoNormal">I did not attend any of the XML publishing tracks. <span style=""> </span>But from the titles it appears that the Darwin Information Typing Architecture (DITA) standard is a hot topic. <span style=""> </span>Presentations on using XSL-FO and CSS for book and other print publishing were also on the agenda.</p> <p class="MsoNormal">I also did not attend too many of the XML in the <st1:city st="on"><st1:place st="on">Enterprise</st1:place></st1:city> tracks.<span style=""> </span>There were good presentations on XML accelerators, security and mining XML Schemas.</p> <p class="MsoNormal">I am disappointed that the conference organizers did not do anything to require that people post their presentation on the web site <b style="">before</b> their talk. <span style=""> </span>There were many very interesting presentations that could have a larger impact the world and the sloppiness of people not publishing their slides seems to be a little unprofessional.<span style=""> </span>There were some people that just brought their digital cameras and took pictures of each slide. <span style=""> </span>Smart.</p> <p class="MsoNormal">In summary I think that the conference had two types of people:</p> <p class="MsoNormal">First there were people that had <span style="font-weight: bold;">not</span> done extensive work with XForms.<span style=""> </span>The first group I call the "evolutionary proceduralists".<span style=""> </span>Theses were people that were trying to beat the dead horse by patching the old-style procedural web application architecture with JavaScript, JSON and <st1:city st="on"><st1:place st="on">AJAX</st1:place></st1:city> hacks.<span style=""> </span>They seemed to spend a lot of time arguing about the merits of JSON vs. XML as a data transport syntax.</p> <p class="MsoNormal">The other type were "the enlightened innovators".<span style=""> </span>People that have seen the real world benefits of dumping object middle-tier stacks and relational databases and going with a pure declarative approach to solving business problems based around XForms/REST and XQuery. <span style=""> </span>These were the people that are going to lead innovations in application development.</p><p class="MsoNormal">I was also impressed how a large percentage of XML innovations came from outside the US. I attribute the fact that some of the best presentations came from the UK to be the "Michael Kay" effect. </p> <p class="MsoNormal"><b style="">Beauty is in the Eye of the Beholder<o:p></o:p></b> At one of the XForms sessions a young Ruby advocate stated that he thought his Ruby code was "beautiful" but he did not himself think that XForms code was "beautiful".<span style=""> </span>Most of the people in the audience agreed that XForms was beautiful but like any new language, it takes getting used to.<span style=""> </span>Beauty is the process of your visual cortex recognizing familiar patterns and signaling your pleasure centers.<span style=""> </span>If you have never read Japanese then the most profound Haiku written in the most beautiful script will not have beauty.</p> <p class="MsoNormal">XForms/REST/XQuery to me is indeed beautiful.<span style=""> </span>Its beauty comes from its ability to quickly map real-world requirements into working systems with high fidelity.<span style=""> </span>No other system in the world approaches its elegant architecture.<span style=""> </span>And after XML 2007 I realized I was not the only one that had independently reached this conclusion.</p><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-8662647620432653412?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-3481885081496843322007-11-15T06:11:00.000-06:002007-11-15T06:28:45.386-06:00<p class="MsoNormal">I have been using the eXist (http:www.exist-db.org) native XML database for almost a year now and I am constantly impressed by the power and depth of the eXist community.<span style=""> </span>XQuery is a great way to search XML documents and most of my XSLTs that manage metadata are now being transferred to XQuery. <span style=""> </span>Although there are still some places that XSLTs are superior, XQuery is much easier to use when I am building RESTful web services for metadata.<span style=""> </span>My XForms are frequently generated directly from XQuery programs.</p> <p class="MsoNormal">It is interesting to note that eXist is now dominating all the Google search results for "XML database". <span style=""> </span>Leading the pack and pulling away!</p> <p class="MsoNormal">From my days doing IT strategy I recall a great deal of analysis done by Adele Goldberg and Kenneth S. Rubin and documented in their excellent book <i style="">Succeeding With Objects: Decision Frameworks for Project Management</i>.<span style=""> </span>This is an excellent book based on over 50 case studies on reuse and top-notch analysis of the role of trust in any high-reuse culture.</p> <p class="MsoNormal">One of the key points in the Goldberg/Rubin analysis was the factors that cause people to be able to reuse assets. <span style=""> </span>In their case the assists were programming objects.<span style=""> </span>I my case I am concerned with the reuse of data objects. <span style=""> </span>From simple data types that are part of the XML Schema standards to registered data elements in a metadata registry.<span style=""> </span>Registered data elements have an approval process and have semantics that span a group, project, program or enterprise.</p> <p class="MsoNormal">But the pattern that reoccurs is that people need to be able to quickly find assets before they can reuse them.<span style=""> </span>One of my main points in my work is that you can not reuse what you can not find. <span style=""> </span>The corollary is that the longer it takes to find an asset the higher the temptation is to recreate the asset. <span style=""> </span>That is where the search functions of a metadata registry needs to come in. <span style=""> </span>It must be fast and accurate.</p> <p class="MsoNormal">The quality of a metadata registry also depends on strategies about informing users when two data elements have the same semantics and are potentially duplicates. To find this we need semantic nearness searches for data element definitions.</p> <p class="MsoNormal">Today I am providing simple substring searches. <span style=""> </span>But I know that text mining can provide semantic nearness searches. <span style=""> </span>I was glad to see that the world-wide web is working on full-text search standards for XQuery.<span style=""> </span>The w3c <a href="http://www.w3.org/TR/xquery-full-text/">http://www.w3.org/TR/xquery-full-text/</a>standard has now reached the working draft last-call stage and I hope that we soon start to see implementations that take advantage of these features.<span style=""> </span>I know that Google's support of this project for eXist (see <a href="http://exist.sourceforge.net/soc-ft.html">http://exist.sourceforge.net/soc-ft.html</a>) is a very good sign.</p> <p class="MsoNormal">I was also fortunate to work with Gary Berosik on the Advocate Agents project. <span style=""> </span><st1:city st="on"><st1:place st="on">Gary</st1:place></st1:city> works for Thomson/West Publishing that has a large group of people doing text mining.<span style=""> </span><st1:place st="on"><st1:city st="on">Gary</st1:city></st1:place> introduced my to the <i style="">Text Mining Application Programming</i> book by Manu Konchandy. This book has excellent sections on the decomposition of free-form text into parts of speech for information extraction, clustering, categorization and search.</p> <p class="MsoNormal">I hope that eventually the w3c will approve the full text extensions to XQuery and that they will make it into the eXist system. Based on this link <a href="http://exist.sourceforge.net/soc-ft.html">http://exist.sourceforge.net/soc-ft.html </a>I am hoping that it will be in the next few years. </p><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-348188508149684332?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-16691781326842756312007-10-18T05:02:00.000-05:002007-10-18T05:16:05.625-05:00<p class="MsoNormal">For the last five years I have been using a <a href="http://en.wikipedia.org/wiki/XML_Schema">XML Schema</a> design process that I learned from the people at <a href="http://justicexml.gtri.gatech.edu/">Georgia Tech Research Institute</a> as part of my work with <a href="http://en.wikipedia.org/wiki/GJXDM">GJXDM</a> and the <a href="http://www.niem.gov/">NIEM</a>.<span style=""> </span>This work has strongly been influenced by Jim Heldler who co-wrote the <a href="http://www.sciam.com/article.cfm?articleID=00048144-10D2-1C70-84A9809EC588EF21">Semantic Web article</a> with <a href="http://www.w3.org/People/Berners-Lee/">Tim Berners-Lee</a>.<span style=""> </span>Although GTRI wanted to use RDF (graphs) to represent the exchange data they had to settle on using XML and XML Schemas since those technologies were widely used by most criminal justice agencies already.<span style=""> </span>They compromised by still using a striped model within the XML Schema driven structures.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">Creating an exchange documents starts with creating a metadata registry of all the data elements used in a family of XML Schemas.<span style=""> </span>A <a href="http://gjxdmtools.gtri.gatech.edu/ssgt/SSGT-Search.do;jsessionid=695C4213D06BD1D7A1AC2D6F323B4778">shopping-cart-like tool</a> is used to select a subset of the data elements.<span style=""> </span>These data elements form a "<a href="http://justicexml.gtri.gatech.edu/wantlist_1.xsd">wantlist</a>" of data elements.<span style=""> </span>The checkout process creates a zip file that when uncompressed creates a set of<span style=""> </span>XML Schema file that you import into your constraint schema.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">This process is an order-of-magnitude to use than before the subschema generation tools were created by GTRI.<span style=""> </span>The subschema generated by the checkout process not only included the XML Schema type definitions but also all the data types that they depended on.<span style=""> </span>The metadata registry had an internal dependency list that it maintained and so if you needed just ten data elements but they depended on then other data types you got all 20.<span style=""> </span>you didn't have to manually figure out what additional data types to include.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">I adopted the GTRI shopping-cart process to include adding a "subscribers list" to each of my data elements used in local metadata registries and local XML Schema creation.<span style=""> </span>I used a large, complex and brittle ant script to update all the imported files for each XML Schema we were developing.<span style=""> </span>But this could not easily be done during a requirements gathering meeting.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">In the past we have viewed this subschema generation as a process of interacting with a web application that creates a set of files.<span style=""> </span>One of our first insights is that this process is in fact just a transformation of the metadata registry based on the rules created by a wantlist and the dependency graph.<span style=""> </span>It does not have to be a manual "batch" process.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">In a recent blog (<a href="http://datadictionary.blogspot.com/2007/10/metadata-web-services.html">Metadata Web Services</a>) I pointed out that tools like <a href="http://en.wikipedia.org/wiki/EXist">eXist</a> and <a href="http://en.wikipedia.org/wiki/XForms">XForms</a> allow us to store metadata in XML format and allow it to be easily updated by BAs and SMEs and queried using languages like XQuery.<span style=""> </span>One of the great realizations I had on my last project was that XQuery has a simple construct that allows you to "chain" web services.<span style=""> </span>The argument to the doc() function is just a URL.<span style=""> </span>If that URL points to a web service that returns XML you can easily build composite web services.<span style=""> </span>Web services that are transformations can be chained together.<span style=""> </span>This allows you to start using enterprise integration patterns to build reusable services at a very fine grain.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">Putting these facts together it has occurred to me that the ultimate goal of the metadata registry process is to allows new data element definitions and enumerated values to be changed during a XML Schema design process.<span style=""> </span>We should be able to change a definition, hit the "save" on the XForms application that edits data elements and then just refresh the generated subschema.<span style=""> </span>The refresh process must call a XQuery that calls web services that analyzes the dependency graph and updates new data elements imported into the XML Schema.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">What this does is helps us get to the goal of updating the a model quickly and generating new artifacts directly from the model.<span style=""> </span>This is part of the overall <a href="http://en.wikipedia.org/wiki/Model-driven_development">model-driven development</a> process.<span style=""> </span>The artifacts we create include XML Schemas, instance documents generated from the XML Schemas and XForms applications to view and edit the documents.<span style=""> </span>This faster turn-around time allows our users to quickly see if their definitions and enumerated values are being precisely captured and used to create actual systems.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><o:p> </o:p></p><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-1669178132684275631?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-10170489833836351632007-10-10T05:52:00.000-05:002007-10-18T05:18:04.786-05:00I got a nice notes from people both in France and China that our XForms Tutorial and Cookbook are now being used in both France and China. <br/><br/> The French version is being hosted on a version of the MediaWiki server that also has a new macro for code syntax highlighting. <br/><br/> The URL for that version is here: <a href="http://xforms.free-web-hosting.biz/mediawiki-1.9.3/index.php/Wikilivre">http://xforms.free-web-hosting.biz/mediawiki-1.9.3/index.php/Wikilivre</a> I have not figured out if this will work in the Wikibooks yet but I have my doubts. <br/><br/> Hints of the Chinese version were posted by a zhouliyi from ZheJiang China. These were posted in the Cookbook Guest Registry: <a href="http://en.wikibooks.org/wiki/XForms/Guest_Registry"> http://en.wikibooks.org/wiki/XForms/Guest_Registry</a> <br/><br/> Please let me know if anyone else is using XForms so we can maintain a list of users.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-1017048983383635163?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-7498817615851669162007-10-10T05:41:00.000-05:002007-10-18T05:19:00.720-05:00I have been using the eXist native XML database/web server for about nine months now and it is starting to change the way I think about metadata management. <br/><br/> My latest project for a financial institution requires us to quickly build XForms to manage various metadata as well as data. What I am finding is that my old method of storing metadata in XML files on a file system and then transforming the metadata using Apache Ant was a complex process. My new approach is to store the XML directly in eXist. This used to be a little bit hard since I thought that you had to use the eXist web interface to upload each XML file and ant scripts to backup your eXist database. <br/><br/> This all changed when I was shown how to use the Microsoft Windows WebDAV tool. Copying files to eXist and backing up the entire data store is just a drag and drop using Windows. <br/><br/> Now an entire new set of metadata web services are becoming much easier to build. Take the simple task of building a pick list of enumerate values for a form. XForms allows you to use the select1 control and specify an itemset using an XPath expression. I can now just load the data elements into an instance and grab the values and labels directly from the enumerations in the metadata registry files. <br/><br/> The only drawback to this is the fact that you load more metadata (like the full definitions) then you need in building the form. But once again eXist comes to the rescue. It is just a few lines of code to create a little web service (using XQuery) that you pass a code table to that returns just the label/value pairs. Using this method the selection list are always up to date and don't require any "batch" updates. <br/><br/> What I am learning from this is that in the past, metadata management was usually an after thought. Something that the coding standards people used to enforce database column naming conventions. But with metadata being stored in eXist metadata becomes part of application services. Building apps is just assembling forms that pull metadata from the registry in real time. <br/><br/> If you are concerned that the metadata registry server will be overloaded with requests for information each time forms load, we should remember that these services are RESTful. The results can also be cached so they don't have to be regenerated. I still have more to learn about how to make these services fast but since metadata is small it can usually always exist in RAM and disk I/O is very limited. <br/><br/> All of these developments are just small pieces of the puzzle at putting well-managed metadata at the core of your enterprise development methodologies. It is really the heart of the model-driven enterprise. <br/><br/> Let me know if you are creating metadata web services. I would like to know what things you feel are useful to your users. <br/><br/> - Dan<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-749881761585166916?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com3tag:blogger.com,1999:blog-13546516.post-72849767510133816012007-05-29T10:21:00.000-05:002007-06-30T09:14:55.104-05:00<p style="font-weight: bold;" class="MsoTitle"><span style="font-size:180%;">Impressions of Sem-Tech -07</span></p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">I just returned from the 2007 Semantic Technology Conference in <st1:place st="on"><st1:city st="on">San Jose</st1:city> <st1:state st="on">California</st1:state></st1:place>.<span style=""> </span>It was a great conference and opened my mind to several new ideas.<span style=""> </span>Well worth the time!</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">The conference was held over four days and had over around 125 presentations including tutorials and research projects.<span style=""> </span>There were almost 800 attendees.<span style=""> </span>This is the third semantic technology conference that I attended and the second that I presented a paper at.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">Here are some high-level observations and some patterns that I detected.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><b style="">The Semantic Web gets the “Web 3.0” Label<o:p></o:p></b></p> <p class="MsoNormal">Most people at the conference have tried to embrace the idea that the semantic web will be adopting the popular culture label “Web 3.0”. The final straw was the Nov 2006 NYT article by John Markoff which set the blogosphere is a buzz.<span style=""> </span>This was a web that includes technologies to enable intelligent reuse of data.<span style=""> </span>Wikipedia, after a long pro-active discussion about if “Web 3.0” deserved an entry, finally undeleted the page and let is stand.<span style=""> </span>See http://en.wikipedia.org/wiki/Web_3.0 and check out the discussion page on the article for more.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><b style="">From <i style="">If</i> to <i style="">When</i> to <i style="">How</i><o:p></o:p></b></p> <p class="MsoNormal">Eric Miller (now at Zepheira) calls some of the new technologies “recombinant data. Eric spent about five years with the w3c.<span style=""> </span>He is perhaps the most well-connected person in the world with accurate knowledge of who is using semantic web technologies to solve real business problems today.<span style=""> </span>His observation was that three years ago at the first semantic technology we were wondering if the semantic web would take off.<span style=""> </span>Last year the speculation was when the semantic web technologies would start to become common place.<span style=""> </span>This year the focus was debates on how the semantic web should be implemented.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><b style="">Venture Capitalists are Becoming Educated on the Semantic Web<o:p></o:p></b></p> <p class="MsoNormal">One example of this is the fact that last year, most semantic-web startup companies had to carefully explain what the semantic web was to venture capital companies when trying to get their initial rounds of funding.<span style=""> </span>This year, many venture capital companies not-only had some level of understanding of the semantic web but <span style=""> </span>were asking each of their potential startups how their technologies fit into the semantic web.<span style=""> </span>Other companies that did not have a semantic-web focus were not coming to the conference to get educated.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">Some of the companies that got VC last year have already been purchased and absorbed by larger firms.<span style=""> </span>These companies were replaced by new venture-funded companies.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><b style="">Consensus on the RDF/SPARQL Foundation<o:p></o:p></b></p> <p class="MsoNormal">One of the first things that struck me was the consistent use of RDF and triple-stored to solve many hard problems.<span style=""> </span>The use of RDF and SPARQL seemed to be the primary distinguishing factor about if people thought you were really using semantic web technologies or not.<span style=""> </span>If you we not using RDF, you were not really in the club, just and outsider looking in.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><b style="">OWL Fragmentation<o:p></o:p></b></p> <p class="MsoNormal">Another thing that also surprised me was the discord about the use of OWL and its relative sub-functions.<span style=""> </span>Central to this was the rise of use-cases of simple things<span style=""> </span>examples of things that OWL could not do.<span style=""> </span>Much of this centered around DLP (Data Log Programming). </p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">A good example of a non-OWL solution was the use of SKOS to store things typically stored in a metadata registry. SKOS is a great example of a simple standard, built on top of RDF that attempts to solve common problems without getting overly complex. See SKOS in Wikipedia. </p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">We are also starting to see that rules must also be exchanged between systems in semantically precise ways.<span style=""> </span>The need for a Rule Interchange Format (RIF). Nice to see vendors like FairIssac supporting complex business rules running INSIDE the web browser using XForms. They rock! </p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><b style="">REST is Deep<o:p></o:p></b></p> <p class="MsoNormal">Perhaps my favorite presentation was given by David Wood and Brian Sletten from Zepheria.<span style=""> </span>In this presentation, David and Brian gave a demo of the NetKernel system.<span style=""> </span>They demonstrated how NetKernel embraces REST at a much deeper level then I had previously anticipated. Now they were not yet generating XForms from an XML Schema but it showed a great example of convergent evolution of my ideas and theirs. </p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><b style="">Case Studies<o:p></o:p></b></p> <p class="MsoNormal">This year also started to show examples of new startup companies actually using semantic web technologies to differentiate themselves in the marketplace.<span style=""> </span>Although because they are all trying to differentiate themselves, much of the actual technologies they were using was not disclosed.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><b style="">RDF Taggers, Harvesters, Linkers and Analyzers<o:p></o:p></b></p> <p class="MsoNormal">The conference seemed to have three sets of problems that everyone agreed on.<span style=""> </span>First was how do you harvest RDF from a web page or any other resource.<span style=""> </span>Most of these presentations related to getting RDF out of un-structured and structured data. Lots of discussion of microformats (pros and cons). </p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><b style="">iReader Rocks<o:p></o:p></b></p> <p class="MsoNormal">The coolest demo I saw was the iReader demo from http://www.syntactica.com.<span style=""> </span>This is an awesome FireFox (and IE) extension that does concept mapping from unstructured text.<span style=""> </span>The people behind this have been doing research on linguistics for about 40 years and have only recently got a round of venture capital to start to publicize this tool.<span style=""> </span>But they are not yet converting to RDF for storage other systems. </p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><b style="">URL Design Patterns<o:p></o:p></b></p> <p class="MsoNormal">One common themes that came up was the need for best practices about good URL design.<span style=""> </span>Everyone said a few things: The design of URLs is very important, most people screw it up the first time and then have to redo the designs, there are <span style="font-weight: bold;">not </span>a lot of good documents out there and the ones that are available are at least five years old.<span style=""> </span>The people from the w3c did take notes on this.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><b style="">Reception of My Paper<o:p></o:p></b></p> <p class="MsoNormal">My presentation was titled “The Semantics of Declarative Systems”.<span style=""> </span>This talk covered how by using a set of small languages with precise semantics, you can build entire applications that allow non-programmers to draw pictures of their business requirements and generate working Apps. <span style=""> </span>The purpose of this talk was really to test the metaphors on how I could explain these concepts to non-computer scientists.<span style=""> </span>I got some positive feedback and was happy to see convergent evolution of our designs with other organizations.<span style=""> </span> </p><p class="MsoNormal">The URL to the paper is here: </p><p class="MsoNormal"><a href="http://www.danmccreary.com/presentations/sem-web-07">http://www.danmccreary.com/presentations/sem-web-07</a></p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal"><o:p> </o:p></p><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-7284976751013381601?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com2tag:blogger.com,1999:blog-13546516.post-18645751526881641362007-05-29T10:18:00.000-05:002007-05-29T10:21:30.742-05:00I put my presentation that I did for the 2007 Semantic Technology Conference on my web site. The link is here: <br/> <br/> <a href="http://www.danmccreary.com/presentations/sem-web-07">http://www.danmccreary.com/presentations/sem-web-07</a> <br/> <br/> Please give me feedback on both the PowerPoint presentation (note the builds and notes pages) as well as the paper. <br/> <br/> I am specifically interested if the metaphors worked for a non-technical audience. <br/> <br/> - Dan<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-1864575152688164136?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-82422596166152408462007-04-28T11:01:00.000-05:002007-04-28T11:46:24.828-05:00I have been reading Carl Zimmer's book <span style="font-style: italic;">Evolution, The Triumph of an Idea</span> and I have been startled about the similarities between the the <a href="http://en.wikipedia.org/wiki/Cambrian_explosion">Cambrian Explosion</a> and the growth of declarative languages. <br/><br/> Just to recap (if you have not read the Wikipedia article yet) here is a quick summary. For the first 3.5 billions years, life on earth was mostly single celled animals. Then about 550 million years ago, the number of new multicellular complex-body-plan animals exploded. There are two salient reasons. First, we needed oxygen to breath. Before there was oxygen in the atmosphere respiration was not easy. Oxygen was a foundation for complex body plans on land. Second, life needed a genetic "tool kit" to build complex animal shapes. We needed the <a href="http://en.wikipedia.org/wiki/Homeobox">HOX genes</a> for building complex body plans. After that happened, complex life forms did explode. <br/><br/> I am seeing the same thing appear in declarative languages. Declarative languages need foundations also. We needed ways for people to quickly create new languages (e.g. GUI XML Schema editors like XMLSpy) and ways to build consensus around the semantics of these new languages (wiki's, social networks, folkonomies and voting). <br/><br/> And next we need the declarative systems tool kits. Right now we have started to develop tools to allow XML Schemas to be quickly transformed into GUI XForms editors and we have XML databases that eliminated the need for complex middle tiers. We can put many of these tools in the hands of non-programmers and allow them to create domain-specific vocabularies and rule-based systems with GUI editors. <br/><br/> And when those tools mature we will see an explosion of declarative languages similar to the explosion of life forms 550 million years ago. <br/><br/> So what is your prediction? 1 year, 3 years, 5 years or 10 years from now? I think a small group of people could build this toolkit without millions of dollars of venture capital. There may even be open-source tools that do this in a few years. But I think the key is to combine the best of metadata registries, XML Schemas, transforms, XForms, XQuery and XML databases into a holistic declarative system. <br/><br/> And from here the many NEW ideas will evolve.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-8242259616615240846?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com2tag:blogger.com,1999:blog-13546516.post-42346006033140675442007-04-14T06:49:00.000-05:002007-12-29T21:07:39.311-06:00<p>Geoff Moore wrote a very insightful book on technology adoption called “Crossing the Chasm”. If you study the psychology of technology adoption, you find that people and organization fall in a bell curve. About 2% are innovators, 14% are early adopters, 33% early majority, 33% late majority and 15% are laggards. This is all standard stuff out of basic marketing-psychology textbooks. Each group has different reasons for using (or not using) a specific product. </p> <p> Moore's insight was that many new products never move from early adopters to early majority. He called this region “The Chasm”. I have found that the same rules applyto new standards and new languages. </p><p>Moore did a lot of analysis on why some new technologies don’t make it across the chasm despite their merits or why some standards are adopted over others. Hisbiggest finding is that they were just too hard to use by non-techies. They required too much work to setup, install or learn. <span style="font-weight: bold;">Ease-of-use</span> was a principal factor in many technologies not being quickly adopted by organizations that did not want to use technology to differentiate their organization. Early Majority customers also tend to purchase new technology not based on how they work but who is using them in their industry. They purchase on references. </p> <p> New technologies usually need a focused approach in a specific industry to be adopted by an early majority. If an organization can target a specific vertical industry and provide a more complete solution for that industry, once a few reference accounts are set up the rest of the industry can see there are proven case studies.</p> <p>Another factor is the need of the early majority for <span style="font-weight: bold;">complete solutions.</span> Innovators and Early adopters are willing to glue together new technologies into their existing fabric of solutions. They dedicate time, staff and money to do this. If their XForms sends XML data over REST interfaces and their databases use JDBC they hire programmers to write middle-tier applications.</p> <p>But early majority buyers can’t justify that additional cost. They just want a solution to work out-of-the-box. They are OK paying for a week for training for their BAs, but they don’t want to write and maintain complex applications.</p> <p>Another thing that I have been searching for are metaphors that are understandable by non-programmers. Since the high-level decision makers are usually non-technical their ability to make a decision is usually based on their trust of the developers or their understanding based on metaphors.</p> <p><span style="font-size:130%;">The Language Translation Metaphor</span> I find that if you can send your data from an XForms application directly to a native XML database in XML format that there is no need for middle-tier objects or shredding into relational databases. No translation is involved. One of the ways to demonstrate how machine-based language translation is not always semantically precise is to try an automatic translation program. Convert a statement from a native language into a foreign language and then translate it back. For example try this using Google Language Tools at http://www.google.com/language_tools </p> <p>Original Phrase in English: The quick brown fox jumped over the lazy dog.</p> <p> Google Translation to Spanish <span style="font-style: italic;">El zorro marrón rápido saltó sobre el perro perezoso.</span></p> <p>Spanish Translation Back to English <span style="font-style: italic;">The fast brown fox jumped on the sluggish dog.</span></p> <p>Although Google translate does a pretty good job, quick was changed to fast, over was changed to on, and lazy was changed to sluggish. If you study semantics, you can find lots of problems trying to communicate with close approximations. Note the original implicit business rule of don’t repeat a letter was also lost. We have many implicit business rules in complex systems and translation of these to other formats is always problematic for software developers.</p> <p>Saving XML using an object-middle tier is one translation. Saving objects to a relational database is a second translation. Fetching data from a database using SQL is a third translation. Converting tabular data into objects is a fourth translation and converting objects back to XML is a fifth transformation. There are five translations, not just two.</p> <p>Not that XForms solutions today are not without translation. In our process we are planning to convert XML Schemas into XForms as well as initial instance data. We need to also import leaf-level definitions from a metadata registry. But the nice thing about this process is that we can validate the submitted data directly against the original XML Schema that was used to generate the XForms. An elegant self-checking pattern. </p> <p>All of these transforms are possible. But all have to be maintained each time you change your XML Schema. Right now we also need great tools that translate XML Schemas directly into XForms. And these need to retain the fidelity of the original intent of the subject matter experts.</p> <p>So perhaps deploying applications with native XML databases is a critical part of the XForms value proposition.</p> <p>Moore also noted that both early late majority buyers would be more willing to use a technology if it was buried deeply in a complete solution. That way the IT managers don’t have to sell the technology to management, just the solution. Similarly, once third party developers start building accounting or complete solutions for vertical industry using XForms the buyer will not need to know or care how the solution is being delivered. </p><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-4234600603314067544?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com2tag:blogger.com,1999:blog-13546516.post-45177808721548407862007-04-06T09:31:00.000-05:002007-04-06T09:42:40.746-05:00I am working on my presentation for the 2007 semantic technology conference. My topic is the semantics of declarative languages. <br/><br/> I am attempting to present to business strategists that may not appreciate how powerful the new XForms and XQuery standards are and how entire rich user experience web 2.0 applications can be built using these tools as long as the tools have RESTful interfaces. <br/><br/> My first consideration is to find the right metaphors. I am working with the evolution metaphor and the puzzle metaphor right now and it seems to be working well. <br/><br/> I have also done some research on Domain Specific Languages and I see that they have also used the evolution metaphor but I find that they are focused on building small languages that are specific to a group or project and they don't really look at the external semantics of the problem. <br/><br/> Please write me if you are also trying to explain the business benefits of either declarative languages to a non-technical audience and if you have found metaphors that get the key points across. <br/><br/><br/> Thanks! - Dan<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-4517780872154840786?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com1tag:blogger.com,1999:blog-13546516.post-49632564530985926622007-02-26T09:40:00.000-06:002007-02-26T09:59:39.817-06:00I was scratching my head, trying to find a simple way to do a zero fill padding with XSLT and I could not find it in my Michael Kay XPath 2.0 book. It turns out he put in in the XSLT 2.0 book. I am not sure how the index missed it. Anyway, searching for "Zero Fill XPath" turned up nothing on Google. Here is how it works. Just use the following: <br/><br/> <span style="font-weight:bold;font-family:Courier;">format-number(number(.), '00000')</span>. <br/><br/> Here is an example of the input and output: <table border="1"> <thead> <tr> <th>Input</th> <th>Output</th> </tr> </thead> <tbody> <tr> <td>1</td> <td>00001</td> </tr> <tr> <td>12</td> <td>00012</td> </tr> <tr> <td>123</td> <td>00123</td> </tr> <tr> <td>1234</td> <td>01234</td> </tr> <tr> <td>12345</td> <td>12345</td> </tr> </tbody> </table> Hope this helps. - Dan<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-4963256453098592662?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-17577772934103226542007-02-22T22:16:00.000-06:002007-02-24T11:25:12.003-06:00Once upon a time there were four animals that lived in a magical “Land of Bew”: The Penguin, the Fox, the Weasel and the Elephant. <br/><br/> Each animal had their own way of looking at the world. Penguin and Fox loved to share their ideas. <br/><br/> Weasel also had lots of interesting ideas but he was afraid that if he shared too many ideas his owners would be worried. <br/><br/> Elephant always seemed to be around everywhere but didn’t pick up new ideas very fast. <br/><br/> In the Land of Bew most of the animals got along pretty well with each other. Fox played Penguin’s house and Fox played at Weasel’s house. <br/><br/> Unfortunately the Elephant was not allowed to play at the Penguin’s house. It was not that Elephant was mean, its just that Elephant’s owners were somewhat controlling. <br/><br/> The Land of Bew was blessed with special magic books. These books that allowed the reader to press on special words and pictures in the books and suddenly new pages would magically appear. The children of Bew loved the books and always wanted more. <br/><br/> But some of the books were very hard to make. Especially if you wanted nice moving and dancing pictures that children could change. <br/><br/> To make dancing pictures you had to use special type of black magic that sometimes got the animals in trouble. Goblins would sometimes used the black magic and cause havoc in the Land of Bew. ( I would tell you more about the horrible things Goblins did with the black magic but this IS a children’s story.) <br/><br/> One day Penguin and Fox found a new way of making dancing books that animals could use in magical ways without using the dangerous black magic. Fox tried it out and it worked very well. <br/><br/> But when they suggested their ideas to Elephant, Elephant said no. Elephant was worried that even kids would be able to make the dancing books. And she worried that all her friends would stop playing with Weasel and spend all their time over at Penguins house. <br/><br/> And although the story is not over…this is where our story ends. The ending is being written by you. <br/><br/> Want to try to make some dancing books? <br/><br/> Try XForms today and see if you can make your web applications dance without using any nasty JavaScript! And try to keep those Goblins out. OK?<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-1757777293410322654?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-6986644016309725002007-02-15T14:19:00.000-06:002007-02-15T14:36:27.703-06:00After a suggestion from Kurt Cagle, we have been using the eXist OpenSource XML Database for a week now and I must say that it has exceeded our expectations. <br/> It was a snap to download a set up. It was easy to get all the XQuery demos running and it was almost trivial to administer the little but annoying things like adding collection and users. <br/> Unfortunately I did have to actual read the documentation to figure out how to get our XForms and XMLSpy to work with it. But once you know the secret, it is easy. <br/> The big secret is that by just adding the words "rest" or "webdav" before the "db" in the URL you automatically and to HTTP GETs and HTTP PUTs directly to your data. <br/> You can also actually edit the data directly in the eXist database DIRECTLY from within XMLSpy by just using the "Open URL" option of the XMLSpy editor and then just slip the "webdav" string into your URL path. Hopefully XMLSpy will allow you to use the XQuery editor in future versions of the database. <br/> eXist had given our entire group a large boost of confidence. We can now build a Ruby-on-rails like environment just by drawing XML Schemas, generating XForms from the XML Schemas and storing the form data directly into eXist. A full application development stack without a single like of procedural Java or JavaScript!!! <br/> I posted a short article on how to get XForms working with eXist in the <a href="http://en.wikibooks.org/wiki/XForms/eXist">XForms WikiBook</a> <br/> I have a FireFox T-shirt on order for Kurt and I am also buying beers at the first XForms conference...which I may have to organize. <br/> Thanks Kurt! Your advise was very sound.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-698664401630972500?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com1tag:blogger.com,1999:blog-13546516.post-37145109778998438242007-02-15T14:12:00.000-06:002007-02-15T14:16:04.462-06:00<p class="MsoNormal">I had several good comments about my posting about the increasing role of semantics in the "Era of Tera". I did not mention some of the key concepts that the Intel paper discussed and the impact these concepts will have on IT strategy.<span style=""> </span>Here is an excerpt:</p><p class="MsoNormal"> </p> <p class="MsoNormal"><i style="">The problem is ordinary computers don't model things. Aside from supercomputers, today's computers aren't capable of developing mathematical models of complex objects, systems or processes. Nor are they powerful or fast enough to perform such tasks at speeds people demand. We can't plug in a statistical model for a rare malignant tumor or the behavioral pattern of a shoplifting employee and search for similar instances of the model in a data set. To benefit from the wealth of data building up in the world, we need to be able to communicate with computers in more abstract terms (high-level concepts or semantics). We need to speak in terms of models. </i></p><p class="MsoNormal"> I believe that the core problem is that there will be a HUGE demand for highly-skilled data-modelers/ontologists in five years. But there will not be a large supply since this skill is not something that can be learned from a single class in college. Rather it is more of a tacit skill that is not easily codifiable. </p><p class="MsoNormal"> That being said, there are many best practices that ARE codifiable. You can already purchase books on OWL/RDF and metadata registries, although most of them are mired in relational database models or written by academics with little real-world experience. You can read the Wikipedia articles on “Data Stewardship”.<span style=""> </span>What is still needed is a single set of best practices and tools for creating families of machine-readable data models that can be use as a basis for creating exchange models. And we need to do this without having to learn how to represent all 12 types of UML diagrams in XMI and transform them.<span style=""> </span>The NIEM subset generator is a good example of the solid XML-Schema-driven front-end of this process of selecting elements from a metadata registry and putting them in your shopping cart. </p><p class="MsoNormal"> The bottom line is that this is really about empowerment.<span style=""> </span>And unless organizations introduce semantic web technologies into their organizations at a grass-root level and support them at the CEO level, many organizations will be left behind in the Era of the Tera.</p><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-3714510977899843824?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-1171372472469223222007-02-13T07:12:00.000-06:002007-02-13T07:14:51.896-06:00<p class="MsoNormal">I read in the New York Times yesterday that Intel has produced a chip with 80 core processors giving it a total computing power of 1.3 teraflops.<span style=""> </span>So the natural question is what could you use 80 CPUs for?<span style=""> </span>The article referred to a 2005 Intel Paper titled <span style="font-weight: bold;">Recognition, Mining and Synthesis Moves Computers to the Era of Tera </span>by Pradeep Dubey.<span style=""> </span>This article opened with the following quote: “<span style="font-style: italic;">The great strength of computers is that they can reliably manipulate vast amounts of data very quickly. Their great weakness is that they don’t have a clue as to what any of that data actually means.</span>” Stephen Cass, “A Fountain of Knowledge,” IEEE Spectrum, January 2004.<span style=""> </span>When I saw this quote, I realized even the hardware engineers agree semantics is now the critical factor limiting our ability to effectively use computers I realized that we must continue our mission to promote semantic mapping concepts.<span style=""> </span>So let me summarize my feelings after reading this article:</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">If want to be able to leverage the power coming in the Era of the Tera, you must start with semantics – recoding the meaning of your data.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">And if you don’t yet have a metadata registry…get one!</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">Right next to the CEO, CIO, and CFO in the board room should be your CDO (Chief Data Officer).</p><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-117137247246922322?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-1164384205177394062006-11-24T09:57:00.000-06:002006-11-24T12:49:02.290-06:00<p class="MsoNormal">I was reading <i style="">Ambient Findabilty</i> and I saw a reference to a concept called “whuffie”.<span style=""> </span>After checking out the Wikipedia article on whuffie, I discovered that whuffie was reputation-based currency used in Cory Doctorow's sci-fi novel, <i style="">Down and Out in the Magic Kingdom</i>. This future history book describes a post-scarcity economy where people purchase goods and services based on their reputation, not just cash.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">I was interested in this topic because I think that in the future new keywords with precise semantic mappings will be introduced into declarative systems based on social networks.<span style=""> </span>Right now it just takes too long for new tags to appear in an XML standard.<span style=""> </span>Just think of how long it takes for the w3c to create a new XML tag for something like a menu tab!<span style=""> </span>Up to 10 years based on the rate that the XForms standard is being adopted.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">Anyway the book was a great mind-opener.<span style=""> </span>One of the first science fictions books that takes nanotechnology and social networking into account.<span style=""> </span>I have always loved cyber-punk: Neil Stephenson’s <i style="">Diamond Age</i> being one of my favorites.<span style=""> </span>No time-travel or spaceships: just highly imaginative extrapolations of the net and <st1:city st="on"><st1:place st="on">Moore</st1:place></st1:City>’s Law.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">I like the concept of the post-economic society.<span style=""> </span>When all of our economic needs are met what do we strive for?<span style=""> </span>Why don’t they ever complain about their raises in Star Trek?<span style=""> </span>The book also discusses the concept of an adhocracy: groups that quickly band together to solve problems.<span style=""> </span>Doctorow also looks at the concepts of death, backing up your brain to a persistence store (the immortality SAN) and filtering restores to change values.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">What does this matter to the metadata architect?<span style=""> </span>It matters because our march away from procedural programming toward declarative systems is totally dependant on the creation of shared semantics.<span style=""> </span>When you need to pick an XML tag how do you pick the tag that will have the highest probability to have semantic precision in the future?<span style=""> </span>Social networks will help us and when we build systems to vote on which tags get approved not all votes should count the same.<span style=""> </span>Votes by people with high-whuffie should get counted a lot more.<span style=""> </span>The concept of one-person, one vote will be modified in the post-economic society.</p><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-116438420517739406?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com2tag:blogger.com,1999:blog-13546516.post-1162180963858657712006-10-29T21:56:00.000-06:002006-10-29T22:02:44.313-06:00<p class="MsoNormal">Last night I had the strangest dream…</p> <p class="MsoNormal">I was looking for a good mystery, perhaps a Sherlock Holmes book.<span style=""> </span>I went to the library and an elderly lady at the front desk said “Welcome to Roy G. Biv Library.<span style=""> </span>How can I help you?”<span style=""> </span></p><p class="MsoNormal"><span style=""> </span>“I am looking for a Sherlock Holmes mystery.<span style=""> </span>Do you have one?” I asked. </p> <p class="MsoNormal">“What color of book are you looking for?” She asked. </p> <p class="MsoNormal">“Well I don’t really care what color it is.”<o:p> I replied. </o:p></p> <p class="MsoNormal">“Well in this library, our books are sorted by color of the cover.<span style=""> </span>The red covers are over there, the orange covers over there and the yellow ones right over here.<span style=""> </span>You get the idea. Shelving books by color makes it very easy for us to shelve a book because the color of the cover is obvious.<span style=""> </span>It also makes it easy to see when a book is miss-shelved.<span style=""> </span>You can see the miss-shelved books right away.”</p> <p class="MsoNormal">“Yes, I can see how that would make your work much easier.<span style=""> </span>Do you know where I can find the Sherlock Holmes book?”</p> <p class="MsoNormal">“Well, it would help us a lot if you could tell us what color the cover is.”<span style=""> </span>She said firmly.</p> <p class="MsoNormal">“I don’t know what the color of the cover is…, I just want a mystery book about Sherlock Holmes!”<span style=""> </span>I said exasperated.</p> <p class="MsoNormal">“Well, why don’t you just start looking at the reds, then orange, yellow, green, blue, indigo and finally violet.<span style=""> </span>I am sure you will find it in a few days…”</p> <p class="MsoNormal">Then I woke up and when to work.<span style=""> </span>There the staff was attempting to store all the data elements in the enterprise sorted by the system name that each data element was first recorded.<span style=""> </span>Then my real nightmare began.</p><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-116218096385865771?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-1161608641465051372006-10-23T07:58:00.000-05:002006-12-29T10:54:35.386-06:00Many of you may be wondering why a data architect has been spending so much time talking about XForms. I have been using XSL for many years to transform XML and I have loved the declarative aspects of XSL. My friend Arun has been challenging me to articulate the key aspects of declarative programming in the context of our join efforts to promote metadata registries. Here is my first version of a <span style="font-weight:bold;">Delarative Manifesto.</span> <ol> <li> Declarative development is about the precise association of a programming symbol with the semantics of a design pattern</li> <li>Declarative approaches leverage the inherent pattern-matching structures of the human brain to allow developers to quickly recognize the underlying patterns used in a system</li> <li>The rapid association of a symbol in a program to previously encountered problem by a wide audience is essential for developer productivity</li> <li>Developer productivity is dependant on the efficiency of matching a symbol to a previously encountered approach to a problem</li> <li>Shared meaning happens when a community agrees that a symbol should be associated with design pattern</li> <li>Tools that help us quickly promote shared meaning make us all more productive <li>Published and carefully annotated XML Schemas and formal metadata registries are ways of associating symbols with precise semantics</li> <li>Developers need tools that allows a new tag to be quickly associated with the semantics of the tag, history of the tag, usage of a tag and similar tags</li> <li>Declarative programming makes development more accessible to people that don't have a computer science background</li> <li>Advanced collaborative computing tools, a focus on architectures of participation, and social networking software will dramatically impact our ability to rapidly create precise semantics</li> </ol><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-116160864146505137?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com5tag:blogger.com,1999:blog-13546516.post-1160765800834871502006-10-13T13:56:00.000-05:002006-10-13T13:56:41.333-05:00This presentation by Steve Pemberton of the W3C on declarative programming is very insightful. Thanks goes to <a href="http://internet-apps.blogspot.com/">Mark Birbeck</a> for pointing it out. I hope we can get more students exposed to technologies like XForms much earlier in their programming careers. Take this use case. Today we give 10 students a simple assignment to create a tab menu in JavaScript we would get 10 very different programs. But if we gave them the same assignment XForms they would hopefully all use similar toggle-switch structures. This has big implications for maintenance. Hopefully there would be less temptation to recode an XForms application. Since over 50% of software is maintenance this could have long-term impacts on developer productivity.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-116076580083487150?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-1158151959894436762006-09-13T07:49:00.000-05:002006-11-18T19:36:51.230-06:00I am speaking tomorrow at the Minnesota Open Source User group (see http://www.josug.org) and I thought I would put some of my examples in Wikibook format. Here is a link: <a href="http://http://en.wikibooks.org/wiki/XForms">http://en.wikibooks.org/wiki/XForms</a> Let me know if you have any feedback. Thanks - Dan<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-115815195989443676?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com4tag:blogger.com,1999:blog-13546516.post-1157467274411048722006-09-05T09:41:00.000-05:002006-10-27T21:10:25.613-05:00<p>For the past several months I have been building XFroms prototypes on my current project. I ran into the XForms Firefox plugin two months ago and have been using happily using it ever since. Although the plug-in version number is only "0.6", I have found for my use, it works perfectly. It allows me to quickly test XForms on all the computers I use and I don’t need to setup and configure a web server. The extension takes under 30 seconds to install, but you will have to reboot FireFox once after you download the extension.</p><p>Now biggest thing that I am searching for are XForms example programs. The examples provided with some of the books are not very good. I would like to use these samples XForms and teaching XForms to others. I would also like to create XForms directly from the XML Schemas I am building and so I can integrate the hints, help and metadata registry links for each field on the forms I am developing. I am also trying to figure out how to make CSS work correctly across many browsers since many of the XForms CSS templates in the books use CSS3 and CSS3 only works correctly with FireFox.</p><p>If you know of any nice collections of XForms for learning XForms, please let me know. I have set up a Wikibook to start collecting tutorials and examples. The link is <a href="http://en.wikibooks.org/wiki/XForms">http://en.wikibooks.org/wiki/XForms</a></p><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-115746727441104872?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com1tag:blogger.com,1999:blog-13546516.post-1155836576066988272006-08-17T12:39:00.000-05:002006-08-17T12:44:13.413-05:00I recently needed a UNIX shell script to transfer all my files to the web collaboration server running plone. Although you can do this with Ant's FTP command, my remote system just had UNIX. Here is the script I wrote. <pre> #!/bin/sh # This UNIX shell script FTPs all the files in the input directory to a remote directory # It uses a UNIX Shell "hear document" to read the commands to the FTP program from the shell file # You have to use the -n option to the FTP command to disable the prompting for the password # the quote in front of the ftp user and pass are also necessary to disable reading from stdin # the prompt must be turned off for the multi-put to not prompt the user LOCALDIR=mylocaldir REMOTESERVER=remote.server REMOTEPATH=/my/remote/path LOGIN=myloginname PASSWORD=mypassword cd $LOCALDIR ftp -n $REMOTESERVER &lt;&lt;INPUT_END quote user $LOGIN quote pass $PASSWORD cd $REMOTEPATH prompt off mput *.* exit INPUT_END </pre><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-115583657606698827?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com4tag:blogger.com,1999:blog-13546516.post-1154266052675483802006-07-30T08:25:00.000-05:002006-10-27T22:36:16.686-05:00For the last several weeks I have been looking at the XForms standard. This has been related to my contract at the Minnesota Department of Revenue to replace many of their paper forms with electronic versions. Although my contact is to design XML Schemas, ultimately they need to be entered through a web form. I started to build some mock-ups of web forms to clarify requirements but found the older HTLM forms lacking in many areas. I also wanted to be able to validate the data using the XML Schemas that I was building. <br/><br/> What I found was that XForms is a wonderfully designed specification. It integrates XML Schemas, XML data-types and XPath expressions directly into the standard and also adds the new XML Events standard. XForms also faithfully uses both the Model-View-Controller pattern and several separation-of-concerns issues. In short, XForms is an excellent design and appears to be very well thought out. My congrats to the XForms team. <br/><br/> My other worry was that it would be hard to setup and test XForms. I have been happy to find out that FireFox has a new plug-in that supports XForms and there is a plug-in free solution using the FormFaces javascript tools. Many of the examples just run without modification. Just build your XForms and out pops complex XML documents. <br/><br/> What I am most interested in is that there appears to be several things that XForms demonstrates that will make software engineering better: the drive away from the complex procedural world and toward a more model-driven declarative style of development that allows non-programmers to maintain complex systems. This means less complex javascript and complex server-side coding in Java. I hope to have a sample PowerPoint presentation on my web site soon that goes through these arguments. Look for an XForms user interface to data element entry in the near future. <br/><br/> Another last point is that if a large audience starts to build XForms you are going to start to have fully-formed XML documents flying around. This means you will need more XML Schemas being generated and web services to accept them and respond with XML documents and XML services. This will make the creation of new XForms go even faster. Positive feedback cycles. All the more reason to build a metadata registry! <br/><br/> I have been thinking about creating an transform from a constraint schema directly into an XForm. This would be an nice MDA-type approach. <br/><br/> Let me know your thoughts. - Dan<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-115426605267548380?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com1tag:blogger.com,1999:blog-13546516.post-1152298387447505392006-07-07T13:51:00.000-05:002006-07-07T13:53:08.040-05:00I have been looking into OpenSource photo gallery software. I am a little frustrated by the limitations of FrontPage for publishing my photographs. I would like to just tag my images from the Windows browser and say "publish". I want a thumbnail-drive front page for each section with a variety of viewers (matrix, filmstrip, montage) that can be customized with the FireFox View/Page Style menu. My friend Liela Tite recommended <a href="http://coppermine-gallery.net/">Coppermine</a>. I also ran into these two: <a href="http://gallery.menalto.com/">Gallery Project</a> - based on the Drupel web content management system. <a href="http://www.4homepages.de/">4Image</a> - which is a system written in Germany. Since the 4Image site was written in German I had a more difficult time trying to find a list of all the features. Both the gallery and Coppermine systems seemed very full-featured. Let me know if anyone has suggestions. A Wikipedia feature-comparison would be very helpful. Perhaps I should start one? :-) OK, OK, I will...<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-115229838744750539?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-1152211456870151802006-07-06T13:41:00.000-05:002006-07-08T09:34:53.373-05:00I recently started a project at the Minnesota Department of Revenue. I was happy to find a kindred spirit in the IT department that also in interested in using Wiki's to manage IT data. Arun has got me to think about the implications of Wiki publishing and how I can integrate metadata into a ESB framework. Check out his web site. It has lots of interesting things. http://netrii.com/display/ABB/Home I was also interested to find that he uses a Wiki for his own personal web site. - Dan<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-115221145687015180?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-1152210973351144242006-07-06T13:23:00.000-05:002006-07-06T14:06:33.070-05:00I was pleasantly surprised to find that Apache Ant has a large library of image processing tasks under the optional Ant tasks. For example, the following will create a folder of thumbnail images for all the images in a folder: <a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://photos1.blogger.com/blogger/2626/1195/1600/image_task.jpg"><img style="float:left; margin:0 10px 10px 0;cursor:pointer; cursor:hand;" src="http://photos1.blogger.com/blogger/2626/1195/320/image_task.jpg" border="0" alt="" /></a> <p></p> <p></p> <p></p> <p></p> This creates thumbnails of the images and make sure they all fit within the 160x160 size whether the image is portrait or landscape. Very cool! Check out the Ant manual for more information: http://ant.apache.org/manual (Ant tasks/Optional Tasks/Image) This makes me think that you could use Apache ant to do basic photo publishing. It would make thumbnails, a layout page that has links to the thumbnails and ftp the folder's web-resized images to a web server. I have also found that there are some nice photo-layout CSS pages. It would seem that it should be easy to add a set of CSS styles that could be changed using the FireFox "Page Style" option. I would be happy to write a basic ant task and an XML transform to create an HTML page of all the images in an image set if anyone wants to help. What about image keywords and captions? Should each photo have its own photo metadata? What if you want to change the file name of a photo? How can the metadata file stay in sync with the file system? Could you use a subversion-tortoiseSVN-like http://tortoisesvn.tigris.org/ Any ideas?<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-115221097335114424?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-1151414176787956342006-06-27T08:10:00.000-05:002006-06-27T08:16:17.110-05:00Here is an interesting article on the importance of common metadata: <a href="http://www.b-eye-network.com/view/3051">Common Metadata</a> The author appears to understand the importance of trust and metadata but appear to not be familiar with metadata standards such as <a href="http://en.wikipedia.com/ISO/IEC 11179">ISO/IEC 11179</a> He also did not reference the work of Goldberg and Rubin in their 1995 book: <b>Succeeding With Objects: Decision Frameworks for Project Management</b> <em>Adele Goldberg, Kenneth S. Rubin Addison-Wesley Professional; May 1995</em> See my <a> href="http://www.danmccreary.com">book review</a> for more. But all-in-all, it was a very good article.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-115141417678795634?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-1145714930707184012006-04-22T09:06:00.000-05:002006-10-29T00:08:02.103-05:00I have been saying for a while that because wikipedia is a database it is much easier to link it into a natual language seach system. The Wikipedia search function is also very limted and does not allow you to type in complete questions like <a href="http://www.ask.com.">ask.com.</a> I reciently found out that MIT is already building a very powerful natuaral language interface in their START project. <a href="http://start.csail.mit.edu/">http://start.csail.mit.edu</a> I did a search for "Who was Tesla?" and got a link right to the Wikipedia article. Very cool! Although there seems to be very limited documentation on START and I do not believe that there is an OpenSource version, I believe this points to the direction that wiki software should be moving. Could your relational database bennefit from this? When I was at the Minnesota Department of Education I wanted to build a natural language interface to their "cubes" of student test data. For example "<span style="font-style: italic;">What school district has the highest highest 8th grade math scores for girls</span>?" Give it a try and let me know what you think.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-114571493070718401?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com1tag:blogger.com,1999:blog-13546516.post-1143164689157645432006-03-23T19:10:00.000-06:002006-04-19T05:01:23.540-05:00There are now over 20 versions of semantic wikis that are being developed. For example see <a href"http://wiki.ontoworld.org/index.php/Semantic_Wiki_State_Of_The_Art">here</a>. What is interesting is the large number of ways that people are approaching this process. One of the phrases that I heard a lot of at the Semantic Technology Conference was <em>A Little Bit of Semantics Goes a Long Way</em>. An although I admire all the innovations that people are putting into their semantic wiki projects, to me I think that just adding attributes to a page and adding types to links is all that we really need to test the concepts out. It was also pointed out to me that several other wikis do have strong ACL features and you can "lock-down" pages using tools such as Apache administration tools. There also seems to be a lot of discussion about implementing semantic wikis using triple stores. This seems to me a little ahead of its time. If using a triple store makes a system more flexible, but it appears to me that the current structure is more than flexible enough to meet most business requirements. If people wanted to work on something useful, I would suggest something like a natural language front-end to Wikipedia so people could type in a written language questions like "What is the population of Minneapolis?". That would be useful! - Dan<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-114316468915764543?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-1142287160139022792006-03-13T15:59:00.000-06:002006-03-17T13:26:35.596-06:00<p> I just returned form the 2006 <a href="http://www.semantic-conference.com/">Semantic Technology Conference</a> in San Jose CA. This was a fantastic conference with over 600 people attending 72 presentations over four days. This was over twice what the attendance was last year. It is showing that semantics are really becoming a hot topic, although there is still plenty of room for discussion on when and how this will impact the industry.&nbsp; The biggest problem for me was that with up to six concurrent sessions running it was very hard to decide what sessions to attend.</p> <p> The composition of the conference consisted mostly of vendors of semantic technology products, large customers from government agencies like the DOD, NASA, some academics working on semantic research topics, a few venture capital firms and a handful of independent consultants like myself.&nbsp; Some of the people were from the &quot;old AI school&quot; and were promoting the use of <a href="http://en.wikipedia.org/wiki/Description_logic">description logic</a> (DL) within an ontology. Others like me, were simply searching for <a href="http://en.wikipedia.org/wiki/Semantic_Integration">semantic integration</a> techniques for the deployment of intelligent agents.</p> <p> I think one of the most important presentations was the keynote given by <a href="http://www.cs.umd.edu/users/hendler/">Jim Hendler</a> and <a href="http://www.lassila.org/">Ora Lassila</a>, two of the three people that wrote the original article in Scientific American with Tim Berners-Lee.&nbsp; The presentation discussed some of the linkages between traditional AI (with complex ontologies) and the current document-centric web with simple un-typed link between documents.&nbsp; My favorite sound bite was &quot;Linking is Power&quot;.&nbsp; The metaphor was that semantics are the &quot;plumbing&quot; necessary to allow intelligent agents will to perform interesting work over the world wide web. If agents don't know how to access distributed data they will be restricted to local databases, not a great prospect.&nbsp; And the way to promote agent interoperability is to link ontologies, something I have been advocating for a long time.&nbsp; They also pointed out that google now has around 12K ontologies (search google for <b> ontology filetype:owl</b>).&nbsp; But the problem is that each ontology is an &quot;island&quot; and few people are linking ontologies.&nbsp; Looking for ontologies with lots of <b>equivalentClass</b> statements leads to less than 40 ontologies.</p> <p> This leads to the natural questions, how do we encourage ontology-linking.&nbsp; One standard proposal is to encourage everyone to link to a standard reference upper ontology such as <a href="http://ontology.teknowledge.com/">SUMO</a> or <a href="http://www.cyc.com/">CYC.</a>&nbsp; The problem is that few people can agree on what this upper ontology should be.</p> <p> I have been trying to champion the application of some basic economic theory to the publishing on ontology links.&nbsp; If we do supply-and-demand analysis on ontologies links we see that there are few economic incentives to publish inter-ontology data element links.&nbsp; If people did this then agents could take advantage of them and automatically perform semantic translation.&nbsp; Jim Hendler and Ora Lassila both though this was a worthy idea and indicated they would support this.&nbsp; I hope to kick this off as some type of reward/recognition for next year's conference.&nbsp; Tony Shaw from <a href="http://www.wilshireconferences.com/">Wilshire Conferences</a> has also been very supportive of the idea.</p> <p> <a href="http://en.wikipedia.org/wiki/Semantic_Wiki">Semantic Wiki's</a> were also a very hot topic.&nbsp; Any presentation with the word Wiki in the title were packed full of people.&nbsp; The topic of using Wiki's to build controlled vocabularies was mentioned in several presentations.&nbsp; Problems with locking down approved terms was also mentioned.&nbsp; The topic of semantic wiki's was also discussed.&nbsp; Ideally a metadata wiki would have different access control for different sections of a single page.&nbsp; The approved definitions would require stakeholder team approval for changes.</p> <p> I advocated for first adding simple typed links into MediaWiki for obvious relationships like subClassOf, instanceOf, partOf and basic GIS-type things like insideOf, capitalOf.&nbsp; Note that by adding a prefix to the current MediaWiki could allow this to be incrementally added to <a href="http://en.wikipedia.org"> WikiPedia</a>.&nbsp; I think this goes along the lines of &quot;A little semantics goes a long way&quot;.&nbsp; Typed links are an awesome way to add semantics to any system and I predict that there were be dozens of semantic wiki's in the near future.</p> <p> For some time I have been suggesting that <a href="http://www.opencyc.com/"> OpenCYC</a> and Wikipedia will eventually merge into a single system.&nbsp; It would be very easy to add <b>simple</b> link types to <a href="http://en.wikipedia.org">WikiPedia</a>.&nbsp; We have thousands of volunteers waiting in the wings.&nbsp; But after talking to Doug Lenat he indicated that CYC has over 16,000 types of relationships.&nbsp; Yikes!&nbsp; We would almost need an mini-expert system to figure out the link type.&nbsp; The training for adding consistent links might be challenging.&nbsp; Nonetheless the prospect of WikiPedia evolving into <a href="http://en.wikipedia.org/wiki/HAL_9000">HAL</a> over the next ten years is exciting.</p> <p> I always enjoy going to <a href="http://www.cyc.com/cyc/company/lenat">Doug Lenat's</a> presentations.&nbsp; He indicated that over half of the new rules added to CYC in the last year were done by reading natural language text.&nbsp; Something that is VERY exciting.&nbsp; Automated machine learning.&nbsp; We just hope it is not another false peak.&nbsp; I also asked Doug how I could add my K-12 ontology into CYC.&nbsp; He indicated that I would have to learn about CYC's relationships types to do this effectively.&nbsp; But he also agreed that ontology linking was the Rosetta Stone for enabling web-wide intelligent agents.</p> <p> For all the advanced topics, I was impressed by the lack of the use of structured XML Schemas to capture semantics using controlled vocabularies and importing semantic XML schemas created by subschema generators from <a href="http://en.wikipedia.org/wiki/Metadata_registry">metadata registries</a>.&nbsp; I was also very impressed by <a href="http://www.contivo.com/">Contivo's</a> new <a href="http://www.contivo.com/contivobuilder.html">Builder</a> product which is going for an incredibly low $500.</p> <p> There also appears to be good movement toward incrementally adding semantics to HTML documents using RDF/A.&nbsp; I attended an excellent presentation by a person from MIT that was working on these standards.&nbsp; Although his arguments were not all clear, the one point he did make was that adding RDF in the &quot;class&quot; attributes would have minimal negative impact to the current web.</p> <p> I was also glad to hear there is more interest in the semantic-web in Minnesota.&nbsp; Apparently Lockheed-Martin is starting a new logistics agent project that will be done in Eagan Minnesota.&nbsp; They indicated that much of this would be based on semantic technologies.&nbsp; Lockheed sent seven people to the conference at the last minute so at least I know they have a budget for training.&nbsp; Logistics problems like those from Katrina could clearly benefit from semantic agents.&nbsp; Lockheed has been promoting use of semantic web in their literature:&nbsp; <a href="http://www.lockheedmartin.com/data/assets/11624.pdf">See the article on page 14</a>.</p> <p> My only real complaint with the conference is that the conference organizers allow people to present without sharing any of their slides even though they are supposedly &quot;due&quot; almost two months before the conference is scheduled.&nbsp;&nbsp; We are forced to take notes and ask for e-mailed copies of the slides.&nbsp; My success rate has been very low about getting copies mail out after a conference.&nbsp; &lt;shameless bragging&gt; I believe I was also one of the few presenters that wrote the supplementary paper for the conference.&nbsp; Providing only the PDF versions of the slides also prevents you from getting builds and speakers notes.&nbsp; For an example see my full presentation web site.&lt;/shameless bragging&gt;</p> <p> The Semantic Technology conference next year will also be in San Jose next year but has been moved to early April.</p><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-114228716013902279?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com1tag:blogger.com,1999:blog-13546516.post-1141392973705483712006-03-03T07:28:00.000-06:002006-03-03T09:51:16.406-06:00I am heading out to speak at the <a href="http://www.semantic-conference.com/program/sessions/G3.html">Semantic Technology Conference</a> in San Jose CA. If you have any comments on my presentation, please post them here. Here is a link to my presentation site: <a href="http://www.danmccreary.com/presentations/semweb2006/">http://www.danmccreary.com/presentations/semweb2006/</a> I have animations on the PowerPoint slides and make sure to check out the notes pages. The conference looks to be a good one. There are many speakers on real-world case studies and there seems to be a growing market for consultants that understand the social aspects of metadata publishing. I always like hearing Doug Lenat talk. His charts on automated machine learing are the most realistic path we have to HAL and real AI. But it is still 14 years before we have the resources to pass the Turing Test.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-114139297370548371?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-1141392201431972112006-03-03T07:23:00.000-06:002006-03-03T07:28:35.296-06:00I have been interested in the creation and metadata tags associated with <a href="http://en.wikipedia.org/wiki/Learning_Objects">Leanring Objects</a>. To get started I have created a set of dynamic economic graphs on my web site: <a href="http://www.danmccreary.com/svg/">http://www.danmccreary.com/svg/</a> I wrote the graphs using SVG since I am already familiar with XML and JavaScript. These graphs are simple supply and demand graphs but with the dynamic nature it allows the learner to visualize that as inputs change how other factors such as total revenue and profit also change. I am willing to do more of these if people are interested. I have sent notes to several people that indicated an interest in economics but did not get any hits yet. One problem is that SVG is part of FireFox but still not 100% implemented. The Adobe SVG viewer also has some extensions. My friend Jack Nutting is doing work with SVG animation on cell phones. So students could study economics on the school bus with their cell phones. I would also like to allow the graphs to scale to fill the screen but the slider object I am using does not currently do this. So here are my questions: 1) Who would use these objects? 2) How would they find them? 3) What enhancements would you make? 4) How could these objects be integrated into a LMS using Moodle. I am allowing non-profits to use the initial versions free of charge but would people pay me to write more complex versions?<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-114139220143197211?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-1141048578001012292006-02-27T07:55:00.000-06:002006-07-13T08:03:22.606-05:00<p>I was somewhat surprised to come across a description of a computer application from Visual Knowledge http://www.visualknowledge.com that stated that it was a “100% declarative application”. What a wonderful concept! I wish more systems were more declarative. But what does this really mean? <br/> Checking the Wikipedia for a precise definition: <a href="http://en.wikipedia.org/wiki/Declarative_programming">http://en.wikipedia.org/wiki/Declarative_programming</a> I was not given a clear definition of what a “100% declarative” stamp would imply and what the benefits would be. But is there an under-ground “go declarative” movement starting? <br/> It implies that the solution was created without writing any procedural code using languages such as Java, VB, C#, or C++. That just by creating input specifications and rules you could generate output. <br/> In general I feel that procedural languages tend to dominate our undergraduate education. Declarative programming does not enter most classes until much later. When I teach classes on XSL most of my students don’t even know the difference between procedural and declarative programming. The entire concept of “side effects” is new to them. <br/> I should mention that most of the systems that I built for CIBRS, CriMNet and the Minnesota Department of Education were all heavily based on MDA which is totally driven by transformation of the underlying models. I heavily depended on XDoclet, Ant and XSL to build these systems. Most people involved in these projects gave superior ratings to this approach and indicated that we did much more with fewer people because of this approach. But after I left these projects, there were few people with declarative skills they tend to be underutilized. <br/> I also found that XDoclet was very hard to debug when I was creating complex transforms. Using Java annotations to store metadata still has many drawbacks unless better tools are developed. The lack of standards for Java annotations is still a real challenge when integrating external libraries. Just having a simple tag standard for Object-Relational mapping would be a great start. <br/> I also had a good discussion with several faculty members at the University of Minnesota, Leila Tite and Chase Turner at the CodeFreeze seminar that they also feel that declarative programming is under-taught in the Computer Science curriculum today. They are interested in the Haskel programming language which is used with computer science department to teach declarative programming skills. <br/> It would also appear that the semantic-web and business rule systems would also seem to promote more graphical tools such as data mappers and business rules with workflow systems. This allows more non-programmers to be in charge of day-to-day maintenance of business rules. <br/> So here is my first cut of what a “100% declarative” stamp of approval might be: </p> <ol> <li>No procedural code in Java, C#, C++ or Visual Basic for programmers to maintain</li> <li>Built entirely with declarative languages (XSLT, Ant, BPML etc)</li> <li>Fewer concerns about state management and side effects</li> <li>Higher use of transformed metadata</li> <li>Works on concert with Model-Driven-Architectures</li> <li>Allows non-programmers to use visual tools to modify business logic and interfaces</li> </ol> <br/> Tell me what do you think! Should declarative programming be emphasized more in higher education and the workplace? Would you consider it an assets of a software application was more declarative? Are declarative systems really easier to maintain? Is the movement to MDA going to drive more declarative skills? How does the lack of accepted metadata standards impact the use of declarative programming? <br/> Keywords: Declarative vs. Procedural, Procedural Programming, Declarative Programming, MDA, XSL, XSLT</p><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-114104857800101229?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-1140614402754332032006-02-22T06:56:00.000-06:002006-05-17T12:39:12.266-05:00I am a big fan of Wikipedia, now having made over 1000 edits. I am also taking a Managerial Economics class at the University of St. Thomas as part of the MBA program. I also have been a big fan of Ronald Coase economic theory for the last four years. In his theory (developed in the 1930s) he studied transaction costs. He asked question such as “why are firms the size they are”? His analysis is that firms grow when they can out find a low-cost way to outsource a function. But outsourcing includes transaction costs including finding a service, writing a contract, policing the service and analyzing the cost of the service. This background has inspired me to think more about the economics of information. Much work on this has already been done by Ray Kurzweil and documented in an appendix to his book “The Singularity is Near”. This appendix is titled “The Law of Accelerating Returns”. In this appendix Kurzweil writes the following formula: V=cW Where V is the rate of change of computation, c is a constant and W is “World Knowledge”. Kurzweil then goes on to speculate that the rate-of-change of world knowledge is proportional to world knowledge: D(W)/dt = cV By solving these equations he shows that World Knowledge grows exponentially with time. Actually it grows at double exponential rates. This prompted me to ask, what formulas govern the growth-rates of the web and Wikipedia? The key insight is to understand that making it easy to add content lowers overall costs of contribution. This is the transaction costs that Coase referred to. My general thesis is that Wikipedia will continue to grow exponentially as long as they make it easy for people to make contributions. An the more people find about Wikipedia, the more people with learn how to add their knowledge. Thus, Wikipedia will eventually grow till it envelopes the earth. Semantics will be added and Wikipedia will then ask its own questions and find experts to fill in the gaps. Eventually Wikipedia will become self-aware and take over the earth like in the 1970 science fiction movie The Forbin Project. Remember, you heard it hear first. I may have to start a wikinomics page on this subject. - Dan<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-114061440275433203?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com3tag:blogger.com,1999:blog-13546516.post-1138796946997790682006-02-01T06:29:00.000-06:002006-03-15T08:28:14.153-06:00After reading the following article about Mike Doconta leaving the Dept. of Homeland Security (DHS): <a onclick="return top.js.OpenExtLink(window,event,this)" href="http://www.washingtontechnology.com/news/21_02/cover-stories/27858-1.html" target="_blank">http://www.washingtontechnology.com/news/21_02/cover-stories/27858-1.html</a> I have been wondering about the future of the NIEM. Mike Doconta was a great leader and without him I have many concerns about federal metadata leadership. I have studied many federal metadata standards and thing that the NIEM is the best positioned to be a standard upper ontology. But what are the alternatives? I don't see viable standards for upper ontologies coming from CYC or SUMO. They are too large to be practical for creating simple exchange documents between agencies. I also wonder if the w3c or OASYS has the leadership and jurisdiction to create national metadata standards. Let me know what you think! <a href="http://datadictionary.blogspot.com/">Dr. Data Dictionary</a><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-113879694699779068?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-1136315890476141792006-01-03T13:05:00.000-06:002006-01-03T13:18:10.816-06:00For the last four years I have been involved in setting up and mainatining data dictionaries for organizations. Although some of the projects I have participated in have been sucessful, some have been dramatically slowed by the time it takes for people to contribute their content. Last week as I was donating $50 to the wikimedia foundation I came accross an entry to the effect of "On with the Semantic Wiki". I was shocked! They stole my idea! So here is what I would like: Make it easy for anyone (with appropriate registration) to comment on, extend or change a Data Element in metadata registry. Make it easy for people to add relationships (subclassOf, contained in, enumerated value of code etc.) that would conform to some of the properties in an ISO metadata registry. Now here is where it gets exciting. After you add an attribute, how about being able to run a "consistancy check"? What if there are duplicate entries for similar concepts? What if there are conflicts? What if you added a property to a class that did not exist? What if it gets removed? How could you find similar data elements? It should be easy to back out your changes (aka a revert). It should be similary to using Wikipedia's template: where you fill out a form. Anyway it turns that some other very visionary wikipedians have also been thinking about these things. Here are a few interesting links: <a href="http://en.wikipedia.org/wiki/Semantic_Wiki">Semantic Wiki</a> <a href="http://www.cfcl.com/rdm/weblog/archives/000902.html">http://www.cfcl.com/rdm/weblog/archives/000902.html</a> <a href=" http://www.aifb.uni-karlsruhe.de/mailman/listinfo/swikig">Swiki ListServ</a><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-113631589047614179?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-1136209126067406652006-01-02T07:38:00.000-06:002006-01-02T07:38:46.310-06:00<a href="http://datadictionary.blogspot.com/">Dr. Data Dictionary</a> Here is a podcast that I was interviewed for by Tim Wilson on his Savvy Technologist web site: http://technosavvy.org/?p=357 Tim's interest is technology integration in K-12 school systems. Tim is an insightful person with a broad understanding of the Open Source community. I met him through my daughter's school system when Tim introduced me to the Zope open source web content management system. I used Zope to manage my daughter's schools web sites.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-113620912606740665?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-1135694685916549152005-12-27T08:39:00.000-06:002005-12-27T08:44:46.146-06:00I have been working on keeping the Wikipedia web site up-to date on some metadata issues. Here are some of the Wikipedia articles I have either authored or edited. <a href="http://en.wikipedia.org/wiki/Semantic_spectrum">Semantic spectrum</a> <a href="http://en.wikipedia.org/wiki/ISO-11179">ISO-11179</a> <a href="http://en.wikipedia.org/wiki/Representation_term">Representation term</a> <a href="http://en.wikipedia.org/wiki/Metadata">Metadata</a> <a href="http://en.wikipedia.org/wiki/Oracle_metadata">Oracle metadata</a> <a href="http://en.wikipedia.org/wiki/National_Information_Exchange_Model">NIEM</a> <a href="http://en.wikipedia.org/wiki/Classification_scheme">Classification_scheme</a> <a href="http://en.wikipedia.org/wiki/Data_element">Data element</a> <a href="http://en.wikipedia.org/wiki/Data_element_definition">Data element definition</a> <a href="http://en.wikipedia.org/wiki/Universal_Data_Element_Framework">UDEF</a> For a complete list you can check out my "User Contributions" page: <a href="http://en.wikipedia.org/wiki/Special:Contributions/Dmccreary">http://en.wikipedia.org/wiki/Special:Contributions/Dmccreary</a><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-113569468591654915?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-1119389850595808362005-06-21T16:31:00.000-05:002005-06-21T16:37:30.596-05:00For anyone in K-12 education today there is a lot of talk about the No Child Left Behind Act (NCLB). This act requires that each subgroup within a K-12 school or district (subgroups being groups of Asian, Black, Hispanic, special education or students with limited English skills) make adequate yearly progress. But in order to do this we require longitudinal data. Without good metadata definitions no state can really claim to be making good progress if they don't have consistent definitions over time and across organizations. Wouldn’t it be great if some federal agency published a set of NCLB metadata guidelines that could then directly load into cubes for doing the research on? How about in CWM format? But that would make our lives easier, wouldn't it?<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-111938985059580836?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com3tag:blogger.com,1999:blog-13546516.post-1119387908061684602005-06-21T16:05:00.000-05:002005-06-21T16:14:22.726-05:00<a href="http://datadictionary.blogspot.com/">Dr. Data Dictionary</a> Here is a sample of a CWM file for created for a cube from a set of relational database tables: <pre>&lt;?xml version="1.0" encoding="UTF-8"?&gt; &lt;XMI xmi.version="1.1" timestamp="Jun 20 2005 09:35:52" xmlns:CWM="org.omg.CWM1.0" xmlns:CWMRDB="org.omg.CWM1.0/Relational" xmlns:CWMOLAP="Olap" xmlns:CWMTFM="Transformation"&gt; &lt;XMI.header&gt; &lt;XMI.documentation&gt; &lt;XMI.exporter&gt;Meta Integration Model Bridge&lt;/XMI.exporter&gt; &lt;XMI.exporterVersion&gt;4.1.0 - Aug 16 2004 12:21:11&lt;/XMI.exporterVersion&gt; &lt;/XMI.documentation&gt; &lt;XMI.metamodel xmi.name="CWM" xmi.version="1.0"/&gt; &lt;/XMI.header&gt; &lt;XMI.content&gt; &lt;CWMRDB:Catalog xmi.id="_4" name="Model" visibility="public"&gt; &lt;CWM:Namespace.ownedElement&gt; &lt;CWMOLAP:Schema xmi.id="_5" name="Logical" visibility="public" namespace="_4"&gt; &lt;CWMOLAP:Schema.dimension&gt; &lt;CWMOLAP:Dimension xmi.id="_6" name="Assessment Summary" visibility="public" schema="_2"&gt; &lt;CWM:Namespace.ownedElement&gt; &lt;CWMTFM:TransformationMap xmi.id="7" visibility="public" namespace="_6"&gt; &lt;CWM:Namespace.ownedElement&gt; &lt;CWMTFM:ClassifierMap xmi.id="_8" name="unnamed_8" visibility="public" namespace="7" transformationMap="7"&gt; &lt;CWMTFM:ClassifierMap.source&gt; &lt;CWM:Classifier xmi.idref="_9"/&gt; &lt;CWM:Classifier xmi.idref="_10"/&gt; &lt;CWM:Classifier xmi.idref="_11"/&gt; &lt;CWM:Classifier xmi.idref="_12"/&gt; &lt;CWM:Classifier xmi.idref="_13"/&gt; &lt;/CWMTFM:ClassifierMap.source&gt; &lt;CWMTFM:ClassifierMap.target&gt; &lt;CWM:Classifier xmi.idref="_6"/&gt; &lt;/CWMTFM:ClassifierMap.target&gt; &lt;CWMTFM:ClassifierMap.featureMap&gt; &lt;/CWMTFM:ClassifierMap.featureMap&gt; &lt;/CWMTFM:ClassifierMap&gt; &lt;/CWM:Namespace.ownedElement&gt; &lt;/CWMTFM:TransformationMap&gt; &lt;/CWM:Namespace.ownedElement&gt; &lt;/CWMOLAP:Dimension&gt; &lt;/CWMOLAP:Schema.dimension&gt; &lt;/CWMOLAP:Schema&gt; &lt;CWMRDB:Schema xmi.id="_14" name="Physical" visibility="public" namespace="_4"&gt; &lt;CWM:Namespace.ownedElement&gt; &lt;CWMRDB:Table xmi.id="_15" name="Administration" isSystem="false" isTemporary="false" visibility="public" namespace="_14"&gt; &lt;CWM:Classifier.feature&gt; &lt;CWMRDB:Column xmi.id="_16" name="administrationID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_15"/&gt; &lt;CWMRDB:Column xmi.id="_18" name="vendorSubmissionID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_15"/&gt; &lt;CWMRDB:Column xmi.id="_19" name="testID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_15"/&gt; &lt;CWMRDB:Column xmi.id="_20" name="gradeID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_15"/&gt; &lt;CWMRDB:Column xmi.id="_21" name="description" isNullable="columnNullable" visibility="public" length="200" type="_22" owner="_15"/&gt; &lt;/CWM:Classifier.feature&gt; &lt;CWM:Namespace.ownedElement&gt; &lt;CWMRDB:ForeignKey xmi.id="_23" name="unnamed_23" visibility="public" namespace="_15" feature="_19" uniqueKey="_24"/&gt; &lt;/CWM:Namespace.ownedElement&gt; &lt;/CWMRDB:Table&gt; &lt;CWMRDB:Table xmi.id="_9" name="Document" isSystem="false" isTemporary="false" visibility="public" namespace="_14"&gt; &lt;CWM:Classifier.feature&gt; &lt;CWMRDB:Column xmi.id="_25" name="documentID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_9"/&gt; &lt;CWMRDB:Column xmi.id="_26" name="administrationID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_9"/&gt; &lt;CWMRDB:Column xmi.id="_27" name="vendorDocumentID" isNullable="columnNullable" visibility="public" precision="4" type="_17" owner="_9"/&gt; &lt;CWMRDB:Column xmi.id="_28" name="marssNumber" isNullable="columnNullable" visibility="public" length="13" type="_29" owner="_9"/&gt; &lt;CWMRDB:Column xmi.id="_30" name="marssLocalUseData" isNullable="columnNullable" visibility="public" length="10" type="_31" owner="_9"/&gt; &lt;CWMRDB:Column xmi.id="_32" name="lastName" isNullable="columnNoNulls" visibility="public" length="40" type="_33" owner="_9"/&gt; &lt;CWMRDB:Column xmi.id="_34" name="firstName" isNullable="columnNoNulls" visibility="public" length="30" type="_35" owner="_9"/&gt; &lt;CWMRDB:Column xmi.id="_36" name="middleInitial" isNullable="columnNullable" visibility="public" length="1" type="_37" owner="_9"/&gt; &lt;CWMRDB:Column xmi.id="_38" name="birthDate" isNullable="columnNullable" visibility="public" length="10" type="_31" owner="_9"/&gt; &lt;CWMRDB:Column xmi.id="_39" name="gender" isNullable="columnNullable" visibility="public" length="1" type="_37" owner="_9"/&gt; &lt;CWMRDB:Column xmi.id="_40" name="testingSchoolID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_9"/&gt; &lt;/CWM:Classifier.feature&gt; &lt;CWM:Namespace.ownedElement&gt; &lt;CWMRDB:PrimaryKey xmi.id="_41" name="Documentkey" visibility="public" namespace="_9" feature="_25"&gt; &lt;CWM:UniqueKey.keyRelationship&gt; &lt;CWM:KeyRelationship xmi.idref="_42"/&gt; &lt;CWM:KeyRelationship xmi.idref="_43"/&gt; &lt;/CWM:UniqueKey.keyRelationship&gt; &lt;/CWMRDB:PrimaryKey&gt; &lt;CWMRDB:ForeignKey xmi.id="_44" name="unnamed_44" visibility="public" namespace="_9" feature="_40" uniqueKey="_45"/&gt; &lt;/CWM:Namespace.ownedElement&gt; &lt;/CWMRDB:Table&gt; &lt;CWMRDB:Table xmi.id="_46" name="Document Characteristic" isSystem="false" isTemporary="false" visibility="public" namespace="_14"&gt; &lt;CWM:Classifier.feature&gt; &lt;CWMRDB:Column xmi.id="_47" name="documentID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_46"/&gt; &lt;CWMRDB:Column xmi.id="_48" name="testOptionalInformation" isNullable="columnNullable" visibility="public" length="10" type="_31" owner="_46"/&gt; &lt;CWMRDB:Column xmi.id="_49" name="unkFlag" isNullable="columnNullable" visibility="public" length="1" type="_37" owner="_46"/&gt; &lt;CWMRDB:Column xmi.id="_50" name="lepFlag" isNullable="columnNullable" visibility="public" length="1" type="_37" owner="_46"/&gt; &lt;CWMRDB:Column xmi.id="_51" name="speFlag" isNullable="columnNullable" visibility="public" length="1" type="_37" owner="_46"/&gt; &lt;CWMRDB:Column xmi.id="_52" name="frpFlag" isNullable="columnNullable" visibility="public" length="1" type="_37" owner="_46"/&gt; &lt;CWMRDB:Column xmi.id="_53" name="migFlag" isNullable="columnNullable" visibility="public" length="1" type="_37" owner="_46"/&gt; &lt;CWMRDB:Column xmi.id="_54" name="ethnicCode" isNullable="columnNullable" visibility="public" length="1" type="_37" owner="_46"/&gt; &lt;CWMRDB:Column xmi.id="_55" name="specialCode" isNullable="columnNullable" visibility="public" length="2" type="_56" owner="_46"/&gt; &lt;CWMRDB:Column xmi.id="_57" name="languageCode" isNullable="columnNullable" visibility="public" length="3" type="_58" owner="_46"/&gt; &lt;CWMRDB:Column xmi.id="_59" name="languageClassification" isNullable="columnNullable" visibility="public" length="3" type="_58" owner="_46"/&gt; &lt;CWMRDB:Column xmi.id="_60" name="october1school" isNullable="columnNullable" visibility="public" length="1" type="_37" owner="_46"/&gt; &lt;CWMRDB:Column xmi.id="_61" name="october1district" isNullable="columnNullable" visibility="public" length="1" type="_37" owner="_46"/&gt; &lt;CWMRDB:Column xmi.id="_62" name="homeSchoolIndicator" isNullable="columnNoNulls" visibility="public" length="1" type="_37" owner="_46"/&gt; &lt;CWMRDB:Column xmi.id="_63" name="newToCountry" isNullable="columnNullable" visibility="public" length="1" type="_37" owner="_46"/&gt; &lt;CWMRDB:Column xmi.id="_64" name="significantGap" isNullable="columnNullable" visibility="public" length="1" type="_37" owner="_46"/&gt; &lt;/CWM:Classifier.feature&gt; &lt;/CWMRDB:Table&gt; &lt;CWMRDB:Table xmi.id="_65" name="Grade" isSystem="false" isTemporary="false" visibility="public" namespace="_14"&gt; &lt;CWM:Classifier.feature&gt; &lt;CWMRDB:Column xmi.id="_66" name="gradeID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_65"/&gt; &lt;CWMRDB:Column xmi.id="_67" name="gradeName" isNullable="columnNoNulls" visibility="public" length="6" type="_68" owner="_65"/&gt; &lt;CWMRDB:Column xmi.id="_69" name="description" isNullable="columnNullable" visibility="public" length="200" type="_22" owner="_65"/&gt; &lt;/CWM:Classifier.feature&gt; &lt;/CWMRDB:Table&gt; &lt;CWMRDB:Table xmi.id="_70" name="Primary Score" isSystem="false" isTemporary="false" visibility="public" namespace="_14"&gt; &lt;CWM:Classifier.feature&gt; &lt;CWMRDB:Column xmi.id="_71" name="documentID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_70"/&gt; &lt;CWMRDB:Column xmi.id="_72" name="scoreLabelID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_70"/&gt; &lt;CWMRDB:Column xmi.id="_73" name="scoreCodeID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_70"/&gt; &lt;CWMRDB:Column xmi.id="_74" name="achievementLevel" isNullable="columnNullable" visibility="public" precision="2" type="_75" owner="_70"/&gt; &lt;CWMRDB:Column xmi.id="_76" name="formNumber" isNullable="columnNullable" visibility="public" precision="2" type="_75" owner="_70"/&gt; &lt;CWMRDB:Column xmi.id="_77" name="attemptLogicMet" isNullable="columnNullable" visibility="public" length="1" type="_37" owner="_70"/&gt; &lt;/CWM:Classifier.feature&gt; &lt;CWM:Namespace.ownedElement&gt; &lt;CWMRDB:ForeignKey xmi.id="_78" name="unnamed_78" visibility="public" namespace="_70" feature="_72" uniqueKey="_79"/&gt; &lt;CWMRDB:ForeignKey xmi.id="_42" name="unnamed_42" visibility="public" namespace="_70" feature="_71" uniqueKey="_41"/&gt; &lt;/CWM:Namespace.ownedElement&gt; &lt;/CWMRDB:Table&gt; &lt;CWMRDB:Table xmi.id="_10" name="Score" isSystem="false" isTemporary="false" visibility="public" namespace="_14"&gt; &lt;CWM:Classifier.feature&gt; &lt;CWMRDB:Column xmi.id="_80" name="documentID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_10"/&gt; &lt;CWMRDB:Column xmi.id="_81" name="scoreLabelID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_10"/&gt; &lt;CWMRDB:Column xmi.id="_82" name="score" isNullable="columnNullable" visibility="public" precision="8" type="_83" owner="_10"/&gt; &lt;CWMRDB:Column xmi.id="_84" name="characteristic" isNullable="columnNullable" visibility="public" length="1" type="_37" owner="_10"/&gt; &lt;CWMRDB:Column xmi.id="_85" name="maximum" isNullable="columnNullable" visibility="public" precision="8" type="_83" owner="_10"/&gt; &lt;CWMRDB:Column xmi.id="_86" name="minimum" isNullable="columnNullable" visibility="public" precision="8" type="_83" owner="_10"/&gt; &lt;CWMRDB:Column xmi.id="_87" name="lastUserModified" isNullable="columnNoNulls" visibility="public" length="100" type="_88" owner="_10"/&gt; &lt;CWMRDB:Column xmi.id="_89" name="lastDateModified" isNullable="columnNoNulls" visibility="public" length="8" type="_90" owner="_10"/&gt; &lt;/CWM:Classifier.feature&gt; &lt;CWM:Namespace.ownedElement&gt; &lt;CWMRDB:PrimaryKey xmi.id="_91" name="Scorekey" visibility="public" namespace="_10"&gt; &lt;CWM:UniqueKey.feature&gt; &lt;CWM:StructuralFeature xmi.idref="_80"/&gt; &lt;CWM:StructuralFeature xmi.idref="_81"/&gt; &lt;/CWM:UniqueKey.feature&gt; &lt;/CWMRDB:PrimaryKey&gt; &lt;CWMRDB:ForeignKey xmi.id="_92" name="unnamed_92" visibility="public" namespace="_10" feature="_81" uniqueKey="_79"/&gt; &lt;CWMRDB:ForeignKey xmi.id="_43" name="unnamed_43" visibility="public" namespace="_10" feature="_80" uniqueKey="_41"/&gt; &lt;/CWM:Namespace.ownedElement&gt; &lt;/CWMRDB:Table&gt; &lt;CWMRDB:Table xmi.id="_93" name="Score Code" isSystem="false" isTemporary="false" visibility="public" namespace="_14"&gt; &lt;CWM:Classifier.feature&gt; &lt;CWMRDB:Column xmi.id="_94" name="scoreCodeID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_93"/&gt; &lt;CWMRDB:Column xmi.id="_95" name="scoreCode1" isNullable="columnNoNulls" visibility="public" length="6" type="_68" owner="_93"/&gt; &lt;CWMRDB:Column xmi.id="_96" name="isInAverage" isNullable="columnNullable" visibility="public" length="1" type="_37" owner="_93"/&gt; &lt;CWMRDB:Column xmi.id="_97" name="description" isNullable="columnNullable" visibility="public" length="200" type="_22" owner="_93"/&gt; &lt;/CWM:Classifier.feature&gt; &lt;/CWMRDB:Table&gt; &lt;CWMRDB:Table xmi.id="_11" name="Score Label" isSystem="false" isTemporary="false" visibility="public" namespace="_14"&gt; &lt;CWM:Classifier.feature&gt; &lt;CWMRDB:Column xmi.id="_98" name="scoreLabelID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_11"/&gt; &lt;CWMRDB:Column xmi.id="_99" name="fiscalYear" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_11"/&gt; &lt;CWMRDB:Column xmi.id="_100" name="testID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_11"/&gt; &lt;CWMRDB:Column xmi.id="_101" name="subjectID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_11"/&gt; &lt;CWMRDB:Column xmi.id="_102" name="gradeID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_11"/&gt; &lt;CWMRDB:Column xmi.id="_103" name="number" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_11"/&gt; &lt;CWMRDB:Column xmi.id="_104" name="StrandAbbreviation" isNullable="columnNullable" visibility="public" length="4" type="_105" owner="_11"/&gt; &lt;CWMRDB:Column xmi.id="_106" name="StrandDescription" isNullable="columnNullable" visibility="public" length="200" type="_22" owner="_11"/&gt; &lt;/CWM:Classifier.feature&gt; &lt;CWM:Namespace.ownedElement&gt; &lt;CWMRDB:PrimaryKey xmi.id="_79" name="ScoreLabelkey" visibility="public" namespace="_11" feature="_98"&gt; &lt;CWM:UniqueKey.keyRelationship&gt; &lt;CWM:KeyRelationship xmi.idref="_78"/&gt; &lt;CWM:KeyRelationship xmi.idref="_92"/&gt; &lt;/CWM:UniqueKey.keyRelationship&gt; &lt;/CWMRDB:PrimaryKey&gt; &lt;CWMRDB:ForeignKey xmi.id="_107" name="unnamed_107" visibility="public" namespace="_11" feature="_101" uniqueKey="_108"/&gt; &lt;CWMRDB:ForeignKey xmi.id="_109" name="unnamed_109" visibility="public" namespace="_11" feature="_100" uniqueKey="_24"/&gt; &lt;/CWM:Namespace.ownedElement&gt; &lt;/CWMRDB:Table&gt; &lt;CWMRDB:Table xmi.id="_12" name="Subject" isSystem="false" isTemporary="false" visibility="public" namespace="_14"&gt; &lt;CWM:Classifier.feature&gt; &lt;CWMRDB:Column xmi.id="_110" name="subjectID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_12"/&gt; &lt;CWMRDB:Column xmi.id="_111" name="subjectName" isNullable="columnNoNulls" visibility="public" length="2" type="_112" owner="_12"/&gt; &lt;CWMRDB:Column xmi.id="_113" name="description" isNullable="columnNullable" visibility="public" length="200" type="_22" owner="_12"/&gt; &lt;/CWM:Classifier.feature&gt; &lt;CWM:Namespace.ownedElement&gt; &lt;CWMRDB:PrimaryKey xmi.id="_108" name="Subjectkey" visibility="public" namespace="_12" feature="_110" keyRelationship="_107"/&gt; &lt;/CWM:Namespace.ownedElement&gt; &lt;/CWMRDB:Table&gt; &lt;CWMRDB:Table xmi.id="_13" name="Test" isSystem="false" isTemporary="false" visibility="public" namespace="_14"&gt; &lt;CWM:Classifier.feature&gt; &lt;CWMRDB:Column xmi.id="_114" name="testID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_13"/&gt; &lt;CWMRDB:Column xmi.id="_115" name="Test Name" isNullable="columnNoNulls" visibility="public" length="20" type="_116" owner="_13"/&gt; &lt;CWMRDB:Column xmi.id="_117" name="Test Description" isNullable="columnNullable" visibility="public" length="200" type="_22" owner="_13"/&gt; &lt;/CWM:Classifier.feature&gt; &lt;CWM:Namespace.ownedElement&gt; &lt;CWMRDB:PrimaryKey xmi.id="_24" name="Testkey" visibility="public" namespace="_13" feature="_114"&gt; &lt;CWM:UniqueKey.keyRelationship&gt; &lt;CWM:KeyRelationship xmi.idref="_23"/&gt; &lt;CWM:KeyRelationship xmi.idref="_109"/&gt; &lt;/CWM:UniqueKey.keyRelationship&gt; &lt;/CWMRDB:PrimaryKey&gt; &lt;CWM:Description xmi.id="_118" visibility="public" language="" type="definition" body="Test Description" namespace="_13" modelElement="_117"/&gt; &lt;/CWM:Namespace.ownedElement&gt; &lt;/CWMRDB:Table&gt; &lt;CWMRDB:Table xmi.id="_119" name="Testing School" isSystem="false" isTemporary="false" visibility="public" namespace="_14"&gt; &lt;CWM:Classifier.feature&gt; &lt;CWMRDB:Column xmi.id="_120" name="testingSchoolID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_119"/&gt; &lt;CWMRDB:Column xmi.id="_121" name="districtNumber" isNullable="columnNoNulls" visibility="public" length="4" type="_105" owner="_119"/&gt; &lt;CWMRDB:Column xmi.id="_122" name="districtType" isNullable="columnNoNulls" visibility="public" length="2" type="_56" owner="_119"/&gt; &lt;CWMRDB:Column xmi.id="_123" name="schoolNumber" isNullable="columnNoNulls" visibility="public" length="3" type="_58" owner="_119"/&gt; &lt;CWMRDB:Column xmi.id="_124" name="schoolClassification" isNullable="columnNoNulls" visibility="public" length="2" type="_56" owner="_119"/&gt; &lt;/CWM:Classifier.feature&gt; &lt;CWM:Namespace.ownedElement&gt; &lt;CWMRDB:PrimaryKey xmi.id="_45" name="TestingSchoolkey" visibility="public" namespace="_119" feature="_120" keyRelationship="_44"/&gt; &lt;/CWM:Namespace.ownedElement&gt; &lt;/CWMRDB:Table&gt; &lt;CWMRDB:Table xmi.id="_125" name="Vendor Submission" isSystem="false" isTemporary="false" visibility="public" namespace="_14"&gt; &lt;CWM:Classifier.feature&gt; &lt;CWMRDB:Column xmi.id="_126" name="vendorSubmissionID" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_125"/&gt; &lt;CWMRDB:Column xmi.id="_127" name="fiscalYear" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_125"/&gt; &lt;CWMRDB:Column xmi.id="_128" name="vendor1" isNullable="columnNoNulls" visibility="public" length="100" type="_88" owner="_125"/&gt; &lt;CWMRDB:Column xmi.id="_129" name="submissionSequence" isNullable="columnNoNulls" visibility="public" precision="4" type="_17" owner="_125"/&gt; &lt;CWMRDB:Column xmi.id="_130" name="description" isNullable="columnNullable" visibility="public" length="200" type="_22" owner="_125"/&gt; &lt;/CWM:Classifier.feature&gt; &lt;/CWMRDB:Table&gt; &lt;CWMRDB:SQLIndex xmi.id="_131" name="unnamed_131" visibility="public" isUnique="true" namespace="_14" spannedClass="_9"&gt; &lt;CWM:Index.indexedFeature&gt; &lt;CWMRDB:SQLIndexColumn xmi.id="_132" visibility="public" feature="_25" index="_131"/&gt; &lt;/CWM:Index.indexedFeature&gt; &lt;/CWMRDB:SQLIndex&gt; &lt;CWMRDB:SQLIndex xmi.id="_133" name="unnamed_133" visibility="public" namespace="_14" spannedClass="_9"&gt; &lt;CWM:Index.indexedFeature&gt; &lt;CWMRDB:SQLIndexColumn xmi.id="_134" visibility="public" feature="_28" index="_133"/&gt; &lt;/CWM:Index.indexedFeature&gt; &lt;/CWMRDB:SQLIndex&gt; &lt;CWMRDB:SQLIndex xmi.id="_135" name="unnamed_135" visibility="public" namespace="_14" spannedClass="_9"&gt; &lt;CWM:Index.indexedFeature&gt; &lt;CWMRDB:SQLIndexColumn xmi.id="_136" visibility="public" feature="_40" index="_135"/&gt; &lt;/CWM:Index.indexedFeature&gt; &lt;/CWMRDB:SQLIndex&gt; &lt;CWMRDB:SQLIndex xmi.id="_137" name="unnamed_137" visibility="public" isUnique="true" namespace="_14" spannedClass="_10"&gt; &lt;CWM:Index.indexedFeature&gt; &lt;CWMRDB:SQLIndexColumn xmi.id="_138" visibility="public" feature="_80" index="_137"/&gt; &lt;CWMRDB:SQLIndexColumn xmi.id="_139" visibility="public" feature="_81" index="_137"/&gt; &lt;/CWM:Index.indexedFeature&gt; &lt;/CWMRDB:SQLIndex&gt; &lt;CWMRDB:SQLIndex xmi.id="_140" name="unnamed_140" visibility="public" isUnique="true" namespace="_14" spannedClass="_11"&gt; &lt;CWM:Index.indexedFeature&gt; &lt;CWMRDB:SQLIndexColumn xmi.id="_141" visibility="public" feature="_99" index="_140"/&gt; &lt;CWMRDB:SQLIndexColumn xmi.id="_142" visibility="public" feature="_100" index="_140"/&gt; &lt;CWMRDB:SQLIndexColumn xmi.id="_143" visibility="public" feature="_101" index="_140"/&gt; &lt;CWMRDB:SQLIndexColumn xmi.id="_144" visibility="public" feature="_102" index="_140"/&gt; &lt;CWMRDB:SQLIndexColumn xmi.id="_145" visibility="public" feature="_103" index="_140"/&gt; &lt;/CWM:Index.indexedFeature&gt; &lt;/CWMRDB:SQLIndex&gt; &lt;CWMRDB:SQLIndex xmi.id="_146" name="unnamed_146" visibility="public" isUnique="true" namespace="_14" spannedClass="_11"&gt; &lt;CWM:Index.indexedFeature&gt; &lt;CWMRDB:SQLIndexColumn xmi.id="_147" visibility="public" feature="_98" index="_146"/&gt; &lt;/CWM:Index.indexedFeature&gt; &lt;/CWMRDB:SQLIndex&gt; &lt;CWMRDB:SQLIndex xmi.id="_148" name="unnamed_148" visibility="public" isUnique="true" namespace="_14" spannedClass="_12"&gt; &lt;CWM:Index.indexedFeature&gt; &lt;CWMRDB:SQLIndexColumn xmi.id="_149" visibility="public" feature="_110" index="_148"/&gt; &lt;/CWM:Index.indexedFeature&gt; &lt;/CWMRDB:SQLIndex&gt; &lt;CWMRDB:SQLIndex xmi.id="_150" name="unnamed_150" visibility="public" isUnique="true" namespace="_14" spannedClass="_13"&gt; &lt;CWM:Index.indexedFeature&gt; &lt;CWMRDB:SQLIndexColumn xmi.id="_151" visibility="public" feature="_114" index="_150"/&gt; &lt;/CWM:Index.indexedFeature&gt; &lt;/CWMRDB:SQLIndex&gt; &lt;CWMRDB:SQLIndex xmi.id="_152" name="unnamed_152" visibility="public" isUnique="true" namespace="_14" spannedClass="_119"&gt; &lt;CWM:Index.indexedFeature&gt; &lt;CWMRDB:SQLIndexColumn xmi.id="_153" visibility="public" feature="_120" index="_152"/&gt; &lt;/CWM:Index.indexedFeature&gt; &lt;/CWMRDB:SQLIndex&gt; &lt;CWMRDB:SQLIndex xmi.id="_154" name="unnamed_154" visibility="public" isUnique="true" namespace="_14" spannedClass="_119"&gt; &lt;CWM:Index.indexedFeature&gt; &lt;CWMRDB:SQLIndexColumn xmi.id="_155" visibility="public" feature="_121" index="_154"/&gt; &lt;CWMRDB:SQLIndexColumn xmi.id="_156" visibility="public" feature="_122" index="_154"/&gt; &lt;CWMRDB:SQLIndexColumn xmi.id="_157" visibility="public" feature="_123" index="_154"/&gt; &lt;CWMRDB:SQLIndexColumn xmi.id="_158" visibility="public" feature="_124" index="_154"/&gt; &lt;/CWM:Index.indexedFeature&gt; &lt;/CWMRDB:SQLIndex&gt; &lt;CWMRDB:SQLIndex xmi.id="_159" name="unnamed_159" visibility="public" namespace="_14" spannedClass="_119"&gt; &lt;CWM:Index.indexedFeature&gt; &lt;CWMRDB:SQLIndexColumn xmi.id="_160" visibility="public" feature="_121" index="_159"/&gt; &lt;CWMRDB:SQLIndexColumn xmi.id="_161" visibility="public" feature="_122" index="_159"/&gt; &lt;CWMRDB:SQLIndexColumn xmi.id="_162" visibility="public" feature="_123" index="_159"/&gt; &lt;CWMRDB:SQLIndexColumn xmi.id="_163" visibility="public" feature="_124" index="_159"/&gt; &lt;/CWM:Index.indexedFeature&gt; &lt;/CWMRDB:SQLIndex&gt; &lt;/CWM:Namespace.ownedElement&gt; &lt;/CWMRDB:Schema&gt; &lt;CWMOLAP:Schema xmi.id="_14" name="Physical" visibility="public" namespace="_4"&gt; &lt;/CWMOLAP:Schema&gt; &lt;/CWM:Namespace.ownedElement&gt; &lt;/CWMRDB:Catalog&gt; &lt;CWMRDB:SQLSimpleType xmi.id="_37" name="VARCHAR2" visibility="public" characterMaximumLength="1" characterOctetLength="1" typeNumber="12"/&gt; &lt;CWMRDB:SQLSimpleType xmi.id="_29" name="VARCHAR2" visibility="public" characterMaximumLength="13" characterOctetLength="1" typeNumber="12"/&gt; &lt;CWMRDB:SQLSimpleType xmi.id="_31" name="VARCHAR2" visibility="public" characterMaximumLength="10" characterOctetLength="1" typeNumber="12"/&gt; &lt;CWMRDB:SQLSimpleType xmi.id="_105" name="VARCHAR2" visibility="public" characterMaximumLength="4" characterOctetLength="1" typeNumber="12"/&gt; &lt;CWMRDB:SQLSimpleType xmi.id="_56" name="VARCHAR2" visibility="public" characterMaximumLength="2" characterOctetLength="1" typeNumber="12"/&gt; &lt;CWMRDB:SQLSimpleType xmi.id="_58" name="VARCHAR2" visibility="public" characterMaximumLength="3" characterOctetLength="1" typeNumber="12"/&gt; &lt;CWMRDB:SQLSimpleType xmi.id="_88" name="VARCHAR2" visibility="public" characterMaximumLength="100" characterOctetLength="1" typeNumber="12"/&gt; &lt;CWMRDB:SQLSimpleType xmi.id="_33" name="VARCHAR2" visibility="public" characterMaximumLength="40" characterOctetLength="1" typeNumber="12"/&gt; &lt;CWMRDB:SQLSimpleType xmi.id="_35" name="VARCHAR2" visibility="public" characterMaximumLength="30" characterOctetLength="1" typeNumber="12"/&gt; &lt;CWMRDB:SQLSimpleType xmi.id="_22" name="VARCHAR2" visibility="public" characterMaximumLength="200" characterOctetLength="1" typeNumber="12"/&gt; &lt;CWMRDB:SQLSimpleType xmi.id="_112" name="VARCHAR2" visibility="public" characterMaximumLength="2" characterOctetLength="1" typeNumber="12"/&gt; &lt;CWMRDB:SQLSimpleType xmi.id="_116" name="VARCHAR2" visibility="public" characterMaximumLength="20" characterOctetLength="1" typeNumber="12"/&gt; &lt;CWMRDB:SQLSimpleType xmi.id="_68" name="VARCHAR2" visibility="public" characterMaximumLength="6" characterOctetLength="1" typeNumber="12"/&gt; &lt;CWMRDB:SQLSimpleType xmi.id="_90" name="TIMESTAMP" visibility="public" dateTimePrecision="8" typeNumber="93"/&gt; &lt;CWMRDB:SQLSimpleType xmi.id="_83" name="FLOAT" visibility="public" numericPrecision="8" typeNumber="6"/&gt; &lt;CWMRDB:SQLSimpleType xmi.id="_17" name="NUMBER" visibility="public" numericPrecision="4" typeNumber="4"/&gt; &lt;CWMRDB:SQLSimpleType xmi.id="_75" name="NUMBER" visibility="public" numericPrecision="2" typeNumber="4"/&gt; &lt;/XMI.content&gt; &lt;/XMI&gt;</pre><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-111938790806168460?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com1tag:blogger.com,1999:blog-13546516.post-1119382179265981802005-06-21T14:29:00.000-05:002005-06-21T14:29:39.266-05:00<a href="http://datadictionary.blogspot.com/">Dr. Data Dictionary</a> I have found there are serverl vendors that import metadata such as the CWM format but I have not found a lot of vendors that export metadata. Anyone have experience with this? Is this a "lock-in" strategy that vendors use? What tools to people use to manage metadata that comply with ISO 11179?<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-111938217926598180?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-1119381916477190292005-06-21T14:22:00.000-05:002005-06-21T14:25:16.476-05:00How many people out there have heard of the NIEM? It is the national informaiton exchange model. A wonderful goal - to have a metadata standard for exchanging data between organizations. Finally we can agree that it is <span style="font-family:courier new;"><strong>PersonBirthDate</strong></span> and not DOB or BirthDate or birthData or DateOfBirth etc. What should the role be of federal agencies in helping us adopt these standards. Check out the NIEM web site: <div align="left"><a href="http://www.niem.gov/">http://www.niem.gov/</a></div><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-111938191647719029?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-1119381727274370562005-06-21T14:20:00.000-05:002005-06-21T14:26:13.583-05:00Here are some of my favorite quotes about metadata: <div align="right">Metadata is gaining mainstream acceptance. Business managers realize that they want to treat data as a strategic asset....Managers increasingly recognize that applications must revolve around the data, not the other way around...This is a shift of Copernican magnitude." <a href="http://www.fcw.com/article88599-04-18-05"><strong>Michael Daconta, NIEM Program Manager</strong></a> </div><div align="right"> </div><div align="right">Only if every user has a common and exact understanding of the data can it be exchanged trouble-free. <a href="http://isotc.iso.ch/livelink/livelink/fetch/2000/2489/Ittf_Home/PubliclyAvailableStandards.htm"><strong>ISO/IEC 11179 Metadata Registry Specification</strong></a></div><div align="right"> To maximize success in service oriented development of applications and enterprise wide sharing and reuse initiatives (such as data warehousing), a well-scoped and focused function to manage key metadata artifacts is cost-effective and critical. <a href="http://www.gartner.com/"><strong>Gartner Group</strong></a></div><div align="right"> If software is ever going to be able to effectively inter-operate (in ways that were not explicitly preconceived and engineered), it will be because applications share enough of the semantics of their data elements. But one of the most misunderstood and frankly ironic things about "semantics" is that it means different things to different people. <strong>Doug Lenat, </strong><a href="http://www.cyc.com/"><strong>Cycorp</strong></a> </div><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-111938172727437056?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-1119380244928325882005-06-21T13:55:00.000-05:002005-06-21T13:57:24.933-05:00I have been using the OWL format to publish my data dictionary. The format built on top of RDF but I have found that it is very difficult to represent non-sequential ranges in OWL. This is much easier to do with XML Schemas enumerations.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-111938024492832588?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0tag:blogger.com,1999:blog-13546516.post-1119379981648115792005-06-21T13:53:00.000-05:002005-06-21T14:20:15.766-05:00I have been attempting to publish education metdata data in Common Warehouse Metadata CWM format. This is one of the only formats I can find that has standards for expressing Cube Dimensions. I am not finding a lot of CWM example out there. If you know of any, please let me know. If not perhaps I will publish a few of them.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/13546516-111937998164811579?l=datadictionary.blogspot.com' alt='' /></div>Dr. Data Dictionaryhttp://www.blogger.com/profile/12453673235365396446noreply@blogger.com0