For so many years we had marketplaces for virtually anything that comes in contact with us or takes part in our daily life. From iron ore to software applications, from utilities to chocolate, everything can be traded in one or more well-known and well-established marketplaces. As a particular case, software is starting to get closer and closer to the “classics” in terms of how it gets to consumers. Popular devices (iPhone, Windows Phone, and many others) have nowadays their own marketplaces where you can buy and integrate software as easily and quickly as you can buy anything else in the virtual world.

The natural evolution of the current situation places data itself in the spotlight. While code that handles data is today ubiquitous in terms of availability via marketplaces, the situation with data is rather different. It’s true, there are some particular forms of data that have been extremely successful from this point of view. Just think about MP3s, movies, maps & navigation, or encyclopedias. Yet data, as a generalized type of merchandise, is not widely available. According to what we’ve learned at MIX10 though, this situation is about to change.

When a company as strong in data programmability as Microsoft decides to play a data card, it usually pays off to watch and analyze seriously. It’s not like they haven’t proved in the past the ability to be ice-breakers in this particular field. Well … which is the card you may ask. The final name seems to be OData (Open Data Protocol), a protocol that has been factored out from WCF Data Services (previously known under the names of ADO.NET Data Services and “Astoria”) and morphed into a standard that aims to be both platform-neutral and open.

What is OData after all? The best answer is hidden in two of its fundamental objectives:

• help unlock and free the data itself from the silos of today’s applications;
• enable new levels of data integration and interoperability;

It’s also worth noting that OData doesn’t play alone. Being a true data platform company, Microsoft supports the protocol proposal with a few more than tangible things:

• Windows Azure and SQL Azure are capable of exposing OData. Even more, “Dallas” (Windows Azure marketplace for commercial data feeds) and “Houston” (Silverlight tool that helps us create SQL Azure databases and expose them as OData feeds) provide the foundation for what is envisioned to be an general-purpose, cloud-based data market;
• WCF is native with OData. I dare to say writing and OData service with WCF is close to trivial;
• PowerPivot consumes OData feeds;
• Reporting Services in SQL Server 2008 R2 exposes reports as OData feeds;
• SharePoint 2010 exposes lists as OData feeds;
• The marriage between OData and Silverlight is just natural. Expect this to enable amazing functionalities in all devices that are Silverlight-aware (from desktops to Windows Phones);

Will OData be a catalyst for data commoditization? Not just yet. Its commonly accepted that a product becomes a commodity when customers perceive little or no value difference between brands and/or versions. So far, we’re in the stage when data is just barely shaping up as a product. Interestingly enough, so far data has been much more a byproduct than being a product. Once data becomes a first-class product on the global market, I’m more than convinced a new stage of the digital era will begin to unfold. Right now, that’s the big promise and bet Microsoft does with OData.

Will OData ultimately be successful? It’s probably too early to tell, but the ice has been definitely been broken here and others will follow. On the long run, there’s probably a consensus around the idea that data (in any form or format) will become a global merchandise and eventually, the commoditization of data will be a fact of life. As it did several times in the past, Microsoft is setting the stage and leading the pack. The big difference is that this time, it does it in style via open, standardized, and clear specs combined with deep integration into both on-premises and cloud-based software services.

You know how they say, beauty is in the eye of the beholder… well, I think the same applies when it comes to document interoperability. I had several interesting debates a few days ago with users who, due to the nature of their work, interact frequently with both OOXML and ODF documents. Even more than that, they quite often need to transform those documents from OOXML to ODF and vice versa.

Having written an earlier post about conformance testing, I was quite interested to see the perception these persons have about interoperability. Quite interestingly, we’re not talking here about IT professionals but more about high level information workers. As I expected, the views on interoperability expressed during the discussions were quite different from those I got from people heavily involved in the standardization processes. Some interesting conclusions I draw are:

• Regular people really don’t care about perfect standards-based transformation of documents. As long as they look pretty much the same and no information is lost, that’s just fine.
• Roundtrips occur far less than I would have thought. When I say roundtrip, I refer to the need to transform say an OOXML document into an ODF document and then, at some point in time back to an OOXML document. Much of the ‘academic’ debate in the field of standardization revolves around the roundtrip concept. Well, I have my doubts that this is actually a real concern.
• Formulas in spreadsheets are a real pain point. That’s were transformation between OOXML and ODF fails big time. And honestly, I don’t think it’s OOXML’s fault.

Obviously, these are just some let’s say empirical observations, but still, I guess they are an interesting indicator of the mismatch that exists between interoperability as it is perceived by information workers and interoperability as it is perceived by the standardization folks.

This week I participated in some very interesting discussions during the Document Interoperability Initiative event held in London. One of these discussions was related to conformance testing. Obviously, the world of documents is moving forward and one of the important processes going on is the continuous improvement of standards. But standards by themselves are not enough.

It turns out that when one talks about conformance testing in the world of documents there are at least two issues to talk about. The first one is document conformance testing and it’s the easier one (nota bene, I’m saying easier not easy). It is all about the conformance of a particular document to a published standard. When it comes to OOXML, the standard is ISO/IEC 29500 with its four parts:

• ISO/IEC 29500-1:2008 – Fundamentals and Markup Language Reference
• ISO/IEC 29500-2:2008 – Open Packaging Conventions
• ISO/IEC 29500-3:2008 – Markup Compatibility and Extensibility
• ISO/IEC 29500-4:2008 – Transitional Migration Features

Who’s in charge of the standard? Well… that’s a more complicated question. The International Standards Organization (ISO) and the International Electrotechnical Commission (IEC) are running a collaborative effort named Joint Technical Committee 1 (JTC1) whose members are National Bodies (42 participating members and 44 observers). All in all, there are about 2100 technical experts from around the world that work with JTC1. Among the subcommittees of JTC1, there is one named SC34 – Document Description and Processing Languages which is in charge, among many other things with the ISO/IEC 29500 standard. SC34 has 5 Working Groups:

• WG1 – Information description
• WG2 – Information presentation
• WG3 – Information association
• WG4 – Office Open XML
• WG5 – Document interoperability

As you might notice, despite the fact that the documents and the bodies governing them are quite complex (remember, it’s the easier side of conformance, not the easy one :) ), testing document conformance is pretty much an exact science.

The second aspect of conformance testing, is unfortunately a bit more fuzzy. I’m talking here about application conformance testing. This one is supposed to test whether an application that works with certain types of documents does what it is supposed to do. Because we’re talking here mainly about user interaction, testing for conformance is much more complicated. Think about the simplest example possible: is the application displaying italic fonts in all places where the underlying document states so? Well … seems to be quite a difficult test to be implemented in an automated way, right? That’s why, unfortunately, much of today’s application conformance testing is still performed manually.

Ok, you might ask, what’s all the talk about OOXML and conformance testing anyway? Well, some of it has to do with JTC1/SC34/WG5, or in plain English, with document interoperability. Now that Office 2007 SP2 supports ODF 1.1, all of the sudden the issue of document interoperability got much hotter. And obviously, conformance testing is going to play a major role in this area for years to come.

I am attending today a DII workshop in London which gives me a good opportunity to talk about what I perceive to be a new face for Microsoft. For those of you who are not familiar with the term DII, I’ll start by saying that it stands for Document Interoperability Initiative. DII workshops are currently organized all over the world, and the main goal of such workshops is to talk about the most pressing issues faced today in document interoperability, from two major points of view, business and standardization.

As a consultant I encountered (and actually worked in) a lot of projects dealing with documents on the server side of the the world. Traditionally this has not been an easy ride. Not even a medium one :D I can’t help remembering the days of proprietary and binary Office document formats when you needed high class wizardry to make winword.exe or excel.exe work within a Windows service on the server side. For many many years Microsoft has been struggling with its inability to get passed the “we’re the only ones making the right stuff” mindset spiced with the “you’re not entitled to understand what’s happening behind the curtain” approach.

I was looking today at the agenda of the workshop and I couldn’t help thinking about the fundamental changes going on with Microsoft for the past 10 years. Today we’re talking about stuff like standards-based validation of documents, PowerPoint presentations generated from PHP, converting to and from ODF, and so on. All of this in a context where Office Open XML is an ISO standard. This is a move from Microsoft towards openness and interoperability that even its toughest critics cannot ignore.

If you want to find out more about what Microsoft is currently doing in the area of interoperability you can start by reading their Interoperability Principles.