How Do Bandwagons Fit into a Project Plan?

That thunder behind you is the REST bandwagon comin’ your way. Anne Thomas-Manes at the Burton Group has proclaimed that the future of SOA lies in REST. I’m not saying the Burton group is itself just jumping on a bandwagon – but their declaration is significant enough to convince others to take a leap. The REST dust-up is in such a very early phase and it’s far from clear whether the bandwagon is going to help me or hurt me. David thinks he might jump on, but let someone else do the driving. Nothing wrong with that strategy, but does it feels like history has come around a bit, eh David?

I think the REST bandwagon, the long-time Lords of the Web, and the WS-* camps are heading toward a colossal and rather fun collision. There will be friction between the stewards of HTTP and those who warp it to meet their RESTful needs. The REST camp likes to say HTTP is all you need – and I agree. But don’t be fooled into thinking that the foresight of the authors of RFC-2616 and RFC-2396 took all this into account. The RESTafarians probably won't want to see their momentum slowed by the standards trust. And then someone is going to say, “Hello! Is that an XML document you’re handing me? Care to sign it?” The WS-* folks are already figuring out how to chase that ambulance.

I’ve wasted a lot of cycles worrying that the Lords of the Web are going to tell me I’m perverting the Web’s majesty. For whatit's worth, here is my advice. First, it’s OK to say you are doing REST even if you haven’t read the dissertation. Personally, I think the magic is in the URIs and the links embedded in the payload. But that’s just me. Second, use the HTTP methods however you see fit. I use GET and POST exclusively myself. I use the occasional URN and I don’t think it’s an evil thing. Some people use the HTTP media-type to indicate what the payload contains. But RFC-2616 “discourages” the use of media-type values that are unregistered (section 3.7). Just use media-type if it helps you. When my payloads are XML, I have a namespace (a URN, BTW) to indicate what it is. For HTML, I haven’t really settled on anything yet.

No matter how far ahead of the REST bandwagon you think you are, it *will* overtake you. I’m sure our marketing department will snap up REST as fast as we did “web services”. I blinked one day and the word “SOA” appeared on the collateral. But you don’t need a REST toolkit – use the bits you have now. More importantly, don’t let the toolkit vendors say “just make objects – we’ll hide the REST part for you”. Good musicians learn to improvise by first knowing their scales. If you know the fundamentals of data modeling, URIs, and HTTP you shouldn’t need WADL, WCF, O’Reilly, or the bandwagon to be successful.

Harry & David

Harry and David (not these guys) are blogging comments to each other about REST. I wasn't sure where to leave my own, so I guess it's here...

David wonders whether widely varying interpretations of REST is an interop killer. Now that is an interesting question. It’s arguable that if two systems have completely different notions of state transitions and URI constructs, interop will suffer and you’re back to writing glue – or pitching for standards. On the other hand, there are those who argue that RFC-2616 has everything you need to be “good enough”, which, conveniently for them, makes interoperability an app domain issue. I think the web wonks have a good point. But it’s hard to know if resource-orientation (sorry) is a way forward unless you first let go of interfaces. One thing that scares the crap out of me is that someone will wire URIs and BPEL together and declare “Mission Accomplished”.

In Perfect Land, you can look at a URI and know what it does. URIs aren’t overloaded by packing the query string. Behavior is consistent and payloads for a given URI have the same format no matter whether you GET, POST, or PUT. But the RFCs (rightly) do not attempt perfection – real-world experience shows you rarely POST exactly what you GET. Also, ambiguities in the specs used to be settled by the whims of browsers. But now those specs are being used to connect systems instead of browsers to servers. I’m not ready to look a trading partner in the eye and say “hmm, let’s see how Opera handles an HTTP 415”. I’ll instead go hide in a closet and do just that.

So, even with the flawed cast of characters you see a lot of whining about – HTTP, URI, XML, and even (gasp!) XML Schema – the pieces are there to build good systems that also make great constituents in anyone’s SOA. The specs, with one glaring exception, are easy to digest. My advice is to start by thinking about a URI strategy that people can follow intuitively. Don’t underestimate the value of a good URI set or the design skills it takes to build. I would also ignore worries about pissing off the Old-Hands of the Web by somehow not using the Web exactly as intended. Their crystal balls weren’t clearer than anyone else’s and these guys do put pants on the usual way (not that I’ve personally witnessed it).

And on Harry’s point that REST and CRUD, I agree that Tim was simply advising people not to limit their comprehension of REST around entities accessed via GET and PUT. REST resources are like views that may or may not be underpinned by an entity model. REST state transitions use URIs to label and invoke services which may or may not use an entity/CRUD programming model under the hood. I agree with Tim in that the presence of those URIs as links is what actually defines REST – NOT simply that you’ve labeled data with URIs.

REST Protocols are the Service Layer

Business operations look a lot like protocol state machines. One of the areas I’ve been looking at (for several years) is how to better leverage state machine thinking into data-driven applications design. I read the REST dissertation, but didn’t see how Roy Fielding’s notion of state transitions applied to my projects. A couple of years ago, David and I looked at how server-side state transitions could drive a service layer (and more). We talked about writing up a paper, but I flaked out so David went ahead with a post.

So, I read Tim Ewald’s posts about REST (first post is here) with a lot of interest. Tim’s description of REST is centered on client protocol state transitions and, critically, ensuring responses to GET requests include the URIs (links) that transition between client states. Up until now, I ignored the idea of URIs for client states that are not mapped to system state transitions on the server. I also questioned the wisdom of including links at all in REST-style responses. Here’s why.

A classic blunder in applications development is designing the programming model (and it's API) around a specific client interaction style. An acid test for an API is how well it works for different calling scenarios, especially those unknown at design time. On that second point, only time can tell you if you have a winner. SOA itself was invented largely to counter unsuitable APIs (and fill space on collateral). Service layers usefully loosen the coupling (you don’t need my binaries to call me) and serve as adapters between an application and the potentially numerous and radically different callers.

So, isn’t it a slippery slope to presumptuously inject URIs into messages for operations that might not apply to the client’s intention? In Tim’s example, the server returns a list of itineraries along with URIs that transition the client to one of 2 potential next client states: getDetails or Reserve. One of these URIs changes system state and the other doesn’t. Rightly, you use GET on one of the URIs and POST for the other (although the example doesn’t indicate which is which). The “getDetails” tag bothered me because in a non-trivial case there might be numerous operations available relative to the resource. Or, the resource might be a summarization (like a report). If the data has a thousand customers IDs and there are 10 or more links possible per customer – you get the point. Why offer up a bunch of links that the client won’t use and potentially outweighs the actual data? Is it even possible to determine the complete set of links?

What I failed to remember, until now, is that data and protocols are independent. There can be multiple protocols with the same data as the starting point – but not the same URI. The URI identifies what data the caller wants and under which protocol.

Data + protocol = resource -> URI

The protocol embodies the caller’s intentions and the server’s constraints. Protocols are easy to secure (URI = security descriptor) and they work like interfaces for data hiding. My mistake until now was in believing that:

• Each entity (I work on ERP apps and we still think about entities) is a resource that has a protocol defined by a system state sequence.
  
• The only useful links in resources are URIs for posting value changes or transitioning states
  
• Views of data representing many resources have no links because it’s difficult to generate a complete set and the caller’s intentions are unknown.

After working through Tim’s description, I’ve reached different conclusions about how to design a REST-style system:

• I still like defining a state sequence and change constraints for entities in the application. It’s a good programming model for ERP because the server-side state transitions are handy places to hang workflow logic. Also, entity/life-cycle thinking aligns well with the business operations.
  
• Systems have protocols that describe data interactions with reasonably specific purposes in mind. The views passed in and out of the system are resources.
  
• A URI identifies a resource, a protocol, and where in that protocol the caller sits.
  
• Resources contain links to other resources or invocation points specific to the protocol. This solves my issue with (potentially) large numbers of unhelpful links.
  
• Protocols buffer the application from conversations – protocols are the new “service layer”.
  
• Getting back to the first point, a “default” protocol can be derived from the server-side state sequence for an entity.

Protocols can carry links to places other than the initial server and resources can be more than just XML representations of entity data – think of the mash-up potential. The URI – via the protocol – tells the system what data to get, what format to use, what stylesheet to apply, and which links are useful to the caller. Adding new protocols to a system needs to be an easy thing to do.

I thought I knew REST and had taken a decent crack at applying it to ERP applications. This re-think has me wondering what else I haven’t figured out yet. In the meantime, I’m looking back at our web services, metadata, and customer-driven scenarios to see what features protocol definitions might need. The goal is to find a way to make it easy for our developers and our customers to create protocols. I’ll write up what I find.

WS-I and Transparency: WTF?

Over 300 people from 100 companies came to San Francisco in early April 2002 for the first ever WS-I community meeting. I was there. Like most of the attendees, I had never worked on a standards effort of any kind. But the "founding board" members of the WS-I – IBM, Microsoft, Oracle, SAP, HP, Intel, etc. – had each reached out to their customer bases to drum up participation. The carrot hung out the window for us smaller companies to sniff was that we could hang out with the big guys to work on web services standards and, importantly, we could get a WS-I conformance logo to grace our product.

It's exactly five years later and I'm in Delray Beach Florida for the WS-I Spring 2007 Community Meeting. The event's web page (for members) list says I'll be one of 32 people attending. Only 18 companies are being represented. The WS-I Board is made up of 11 companies, who, one would think, are obligated to participate. So the "community" beyond the leadership team will be me plus 6 other companies. Over 70 other companies pay dues to keep their up their membership, but I'll wager that many (like my company) have fallen out of good standing.

WHAT THE HELL HAPPENED?

Marketing – There are test programs that determine whether an application conforms to a WS-I profile. You don't need to belong to the WS-I to run them – or even claim you've run them successfully. You can just say your app conforms to a WS-I profile. It's up to someone else to call your bluff. With membership optional and conformance claims free for the taking, it's no wonder the membership is practically extinct. My company's dues are paid out the marketing budget. Hmmm.

Resources – A tenet behind the WS-I charter says that, for any profile, a set of sample applications and a test suite identifying test assertions will be developed on multiple platforms. The idea was to show best practices and demonstrate the potential interoperability benefits behind the effort. But developing this work is enormously expensive. Microsoft, IBM, SAP, Fujitsu, and others have deep pockets. But over time the contributions have declined to the point where the organization can't "afford" to charter new profile work. But there is a bigger challenge to the long-term solvency of the WS-I…

Transparency – This is the big one. To be fairer than necessary, the WS-I was conceived at a time when web services hype was still growing and the founding members were deeply suspicious of each other's motives (the players, I think: IBM, Microsoft, Oracle, HP, Intel, SAP, Accenture, BEA, and webMethods). The hype issue has almost disappeared. And the mutual suspicion problem is in a little better shape except for a lasting grudge within the WS-I Board over the fate of an effort 3 years ago to incorporate SOAP with attachments into the WS-I Basic Profile.

But there is a bigger problem. There are 2 WS-I organizations: the working groups and the WS-I Board. Of course, all organizations have boards that are independent and need to be able to deliberate privately. But the WS-I Board – and I mean the institution, not the individuals or their companies – is frankly an anathema in the face of the WS-I's public mission.

WS-I working groups deliberate within narrowly defined charters, but the discussions are open. Members are free to discuss and describe deliberations wherever they like. The only caveat – and it's significant – is that no "intellectual property" developed by working group members can be published as WS-I material without Board approval and (depending on the nature of the work) as membership-wide vote. Most members can live within those constraints. But the Board is another issue.

The WS-I Board was intentionally designed to be a vendor-driven black-box that controls what is advertised (with a bit of mockery) as a membership community. This isn't hyperbole (despite my usual tendencies). Here are the rules that are driving the WS-I straight into the ground:

- ALL WS-I Board proceedings are confidential – even to the membership. There was an effort by IBM a year ago to prove that Board members could discuss what happened in meetings even if the minutes were secret. But that theory was rejected by others in the Board as a reckless interpretation of the bylaws.

- Virtually all proposed Board actions – like approving a document for publication or launching profile work – requires 9 of the 11 Board members to vote "yes". Abstentions count as "no" votes (a rule which is potentially being addressed)

- Of the 11 Board positions, 9 are allocated on a permanent basis to "founding members". The other 2 are elected. Of those 2, one is effectively (and justifiably) locked up by Sun Microsystems, who was obviously blackballed from participating when the WS-I was formed. But the point is that there is no real opportunity for WS-I members to become part of the leadership.

- The Board determines what work is in scope for the WS-I, but has been utterly unable to declare the criteria behind making this determination. Part of the reason is that the Board is trapped by a wacky bylaw treatise inured with a declaration of unambiguous consensus. In other words, the board can't give its own membership, even internally, any feedback without 9 of 11 votes affirming both the act and the message.

How in the hell can members have any involvement in determining future profiles and other work for the WS-I when (a) there are precious (and highly subjective) rules about what work is in scope and (b) Board members can't agree about what, the bylaws allow them to say to their own membership? The WS-I is the epitome of dysfunction and the reason is mainly the misguided invention of that whacked-out cocoon in which the Board comfortably resides.

So, I'm participating in a Working Group chartered to come up with requirements for future WS-I profile work. To be fair, I've been away from the WS-I for a year, so I feel bad about making waves on the first day back. But I know the workings having chaired their XML Schema Planning WG in 2004-2005. Anyway, the Requirements WG is only allowed to submit draft Working Group charters for WS-I Board approval. Astonishingly, our Working Group can't even give feedback to OASIS or ask the W3C a question without a 60-day approval processes and WS-I Board vote. We can't contact the organizations that own the material the WS-I is considering for potential profiling efforts

I'm going to propose that the WS-I Requirements working group take several actions: (1) Questions and feedback to other organizations are deemed "non-material work", which means the whole WS-I membership does not need to vote before publication. (2) That approval to make materials developed by working groups or committees public requires a simple majority (I can be talked into a 2/3 margin) to be approved for publication. (3) That votes by the WS-I Board on work submitted by a working group or committee are made public to the membership (and by extension – the world). Of course the Board and the bylaws will have to yield and be amended, respectively.

Without these reforms (and others), the WS-I will continue its utter collapse into a small club of vendors seeking to satisfy a few constituents that, for now, are propping up the justification behind the entire organization. The rest of the world (spot the pun) will overtake the WS-I like callous covers a splinter on your heel. And while I strongly think there are much better programming models on the web for many situations, I also think there is a large place in the economy for WS-*. Large organizations will need the mechanisms behind WS-* as provable standards for to bind their touch points. And large IT companies will wave WS-I profiles, and the like, as bespoke competencies fulfill them.

Maybe the WS-I should be disbanded for the shear arrogance of its founders who squander the enthusiasm and contributions of the membership. But starting over also sucks.

SOA vs. Service-Oriented Applications

[Updated 16-Oct to fix a typo]

Service-Oriented Architecture (SOA-arch) and Service-Oriented Applications (SOA-app) are different animals with a common name. I think this is a major point of confusion that hasn't yet been widely recognized. Just like the fact that three points of view exist about what "workflow" means. Here is the difference and I've also included advice for building a service-oriented application.

Service-Oriented Architecture

Here is a quick SOA-arch diagram I built after looking at various resources on the Web:

Service-Oriented Architecture Layers

Those like me who work on commercial ERP applications find this a little humiliating. All that work to build an enterprise application – and we wind up at the bottom of the SOA stack? We clearly have thin skins. Moreover, this implies that one of the chief reasons that SOA-arch exists is because apps can't integrate with each other and we have to fool them into working together. SOA-arch also exists to crack apart applications and cobble them back together to match a business process not envisioned by the application's authors. It's an expensive and invasive process.

ESB providers will eternally demand credit for inventing SOA-arch. But I think we should think about the "day after" effect of web services. People began putting SOAP layers around their existing applications or building new applications from scratch on the WS-* stack. But getting some type interoperability only solves one of the many problems around integrating complex apps into a specific business. There was nothing SOAP could do to match an SAP-formatted employee record to an HR-XML version. SOAP can't make my application email the HR department if I browse http://www.monster.com from my desk. SOA-arch puts those things together. SOA-app gives enterprises a way to control and access data maintained by a given application.

Service-Oriented Applications

SOA-app is about designing applications aggressively to be good citizens in someone's SOA-arch implementation. Service-oriented applications have a basic stack diagram as well, I mentioned some months ago. I won't repeat that explanation, but here is a more colorful picture:

Service-Oriented Application Stack

SOA-app revolves around designing elements of the enterprise application troika – message, data, and code – specifically to make it much easier to incorporate the resulting application into your SOA-arch and also promote a dynamic system that tolerates change more effectively. So, my app might be at the bottom of your SOA-arch stack, but I'll make sure my app is your best friend rather than some opaque, inflexible, monster. SOA-app means changing some classic approaches and, in some cases, fighting the dev tools to make that happen. Here are some key ideas behind SOA-app:

• Make sure callers can mix business intent into a single unit of work. Suppose you allow callers to update customers or update suppliers. Make sure you also allow callers to update a customer and a supplier in a single call. Each business function is represented by a distinct message part format, so your posting function should be able to take an arbitrary list of message parts as input and return a corresponding set of message parts containing results (if any). The reason for this is that SOA-arch may have an entity data aggregation function that updates multiple entities at once. Wouldn't it be nice if your application could accept composite documents with less initial shredding?
  
• Make sure your messages tolerate unknown or incomplete content. If a caller adds extra data that some other part of the SOA-arch needs, just ignore it. For XML, many type serializers do this pretty well. But for binary serializers, this is practically impossible. On the flip side, make sure your messages make "everything optional". There are reasonable limits here. For example, when a caller wants to create a new customer but doesn't include a name or address. Again the reason is to make life easier for callers syncing fractions of records.
  
• Use an inversion of control framework that lets your implementers re-sequence or replace logic declaratively. Windows Workflow Foundation is pretty well designed for just this kind of role. Going this route obviously has a big impact on the overall programming model of the system. For one thing, routines that update data should basically not ever return anything directly to the caller because makes the coupling too tight between components. The scenario to keep in mind is when an implementer needs to suspend a function and execute an approval routing or clear some business semaphore (like a credit hold).
  
• Give callers reasonable access to the query processor. One of the problems with most interfaces – no matter the language or platform – is that callers often can't get the exact data they need unless some designer first thought to add the right function. One of the biggest requests I see is for a correlated sub-query that our product managers didn't think of. I'm not saying that you should allow callers to send SQL – or even fragments of SQL. But you should implement some mechanism that lets callers retrieve data by submitting a query tree that can be vetted by your framework and that you can secure.
  
• Stay very document-oriented. This is easier in business applications because the data lines up well with real-world business artifacts (orders, shipments, etc.). Many application designers over-design the interface library by building a matrix of functions required by caller roles. The problem is that the matrix won't have an entry for many interactions required in the SOA-arch. Come up with a CRUDx standard for your entities and use that to determine message formats (ideally, via a transform). The "x" is whatever critical behavior a particular entity may possess. Keep the number of members of "x" as small as possible. For ERP, we've been able to do it with a single "t" – for "transition" which means to move a record from one transactional state to another (order:new -> order:cancelled).
  

It's troubling that for the past year I've talked to many people about SOA without anyone realizing we were somewhat cross-purpose. I've wanted to talk about the above points for some time, but when I would go over them with other solutions architects, it seemed like we weren't on the same page. Hopefully distinguishing between SOA-arch and SOA-app will be useful going forward.

Diagnosis: XML Fetish

The apparent medical experts who wrote According to Microsoft Windows Communication Foundation – Hands-On, have – in Chapter 3 – diagnosed me as having an “XML Fetish”:

Yet practitioners of contract-first development, working in the XML Schema language in an XML editor, tend to become distracted from those core concerns and start to worry about exactly how the data is to be represented in XML. Consequently, they begin to debate, among other things, the virtues of various ways of encoding XML, and become highly suspicious of anything that might inhibit them from seeing and fiddling with XML. The XML becomes a fetish, falsely imbued with the true virtues of contract-first development, and, as Sigmund Freud wrote, "[s]uch substitutes are with some justice likened to the fetishes in which savages believe that their gods are embodied" (1977, 66).

Wow – I’m practically paranoid! I think I understand the points the authors were trying to get across: Let the XML serializers do their job and stay out of the way. The preaching and the lame attempt to bring Freud into the discussion really put me off.

To (badly) paraphrase Norma Desmond, XML is big. It’s the toolkits that got small. Experience tells me that you actually have to keep a sharp eye on the WSDL and XML emitted from just about any toolkit. The W3C has a working group basically dedicated around that fact.

Let’s say I have a service that accepts purchase order lines. Purchasers can buy things that they receive and stock (items) or services that will be rendered. Items have a code and a quantity; services have a free-form description and an estimated cost. How do I create a service description for an operation that gives the caller that kind of choice in what gets sent?

In XML Schema, I can simply say “xs:choice”.

In C# (or most languages), I can implement this choice using a number of tactics. The most obvious way to go is polymorphism: create a base class POLine and two descendent classes StockedItem and Service. The function prototype then takes the base class as an argument. Hmm – try that once and see what kind of interoperable service description you get. My bet is something like xs:any.

To be fair, .NET has added a ton of attribution to the XML Serialization mechanism. You can implement a choice construct that works pretty well. But the attributes you need to faithfully build such a C# types are very specific to .NET and probably even less intuitive than XML Schema (never thought I would say that). Further, you have no guarantee that Microsoft (or any toolkit vendor) won’t change the XML Schema constructs mapped to a class construct when new framework versions shipped. We had this happen between .NET 1.1 and .NET 2.0.

I tend to build data contracts using XML Schema first. Then I roll the schemas through the various class generators to see how good the output is. Then I build a service using the generated classes to see how good the WSDL will be. Then I build yet another set of types using the generated WSDLs to simulate what service consumers will experience. Finally and if all looks good, I throw all of it away except for the original schema, which gets incorporated into the service description directly and, sometimes, into the service code base via XML Schema validation. I don’t like building static types for message payloads or data domain members. But that’s a whole other topic!

ERP Data as Hypermedia

I work on ERP applications for a living, so I tend to see lots of ways business intent is manifested in an API. For the past few years, I’ve been trying out different architectural ideas to merge the success of the Web to enterprise application API development. The two aspects that have garnered the most payback have been (a) casting the data domain as a hypermedia set and (b) rationalizing constraints around GET, PUT, and POST to convey business intent and manage work. This blog entry is about (a). Making the data domain flexibly addressable is critical to workflow and EAI, which in turn makes it critical to SOA -- no matter how REST or WS-* might be utilized.

Most reasonably normalized databases already have clear semantics for referential data and child data. Reference columns link tables together by record ID. Child tables inherit the ID of their parent plus their own additional anonymous key value. The referential aspects form hypermedia-like links and the parent-child aspects provide document scope (not to mention the nice fit with XML).

Putting these together means that we can map a record as a URL using the same key values that identify records in a database. We can also extend the URL format across referential values. Here are some examples from an implementation we put out several years ago:

url://MySite/orders/101.xml returns the XML representation of order “101” in all its glory (details and all)

url://MySite/orders/101/1.xml returns just the first line item of order “101”.

url://MySite/orders/101/CustomerID.xml returns the ID of the customer

url://MySite/orders/101/CustomerID/Customer.xml returns the entire customer record for the customer identified in the “CustomerID” link

The last item is the most significant, because you can traverse the entire data domain – if the linkages exist – and access data many degrees of separation away from your starting point. We’ve been doing this for several years now in Business Process Management (BPM) architectures because the data one needs to make a business policy decision is rarely in the message. In other words, workflow-driven architectures need expression formats that bind the data in question with data already in the system.

For example, an administrator might set a policy to disallow orders for items belonging to a product class that itself is under approval hold by, say, the U.S. FDA.

url://message/orders/lines/items/holdnotices/fda.xml

Or, you don’t want to allow orders for customers on credit hold: url://message/orders/CustomerID/holdnotices/credit.xml

Note the “message” bit rather than “MySite” in the previous examples. The idea is to not only traverse entities in a database – it’s also to traverse from the message payload to the database in one expression. Non-programmers can actually read this and figure out what it’s doing. That capability is important in BPM because line managers control the policies in force. Line managers understand hierarchies like directory structures. The key is to keep the URN constructs uncluttered and simple.

That seems straightforward, but you have to work out how to pipeline referential constructs through the entity chain for multiple situations: inner-join, outer-join, existential, etc. You also have to work out how to constrain certain kinds of URLs that bring back too much data. Our initial work let you do something like this: url://MySite. It spit out every record in the database.

BTW, you can swap “.xml” for “.xsd” and get a schema for the results. You can also use “.htm” to get an HTML rendering. In fact, you can go to town with MIME types and style sheets to provide lots of viewable interpretations of whatever data is being sought. That’s pretty powerful.

Are GET and PUT Symmetrical in REST?

Does REST require symmetry? In other words, must the payload schema for messages I GET be the same as the schema for messages I PUT? I hope not, because (in my world) what I PUT is rarely a new, complete authoritative version of the data. In the work I've done, the PUT (or POST) payload explicitly indicates whether the item is new (in which case it is authoritative) or if I am changing an item that already exists.

The REST idea – in my mind – is that (a) you can address instances of data unambiguously and completely and (b) that what you GET and what you PUT (for a given URL) are semantically related -- but not necessarily exact copies.

Stephan's List: REST vs. SOAP

Stefan’s List is an interesting effort to categorize some very good bloggers into REST, SOAP, and (um) “either” camps. I know a few people on both sides of the isle and I don’t think any of them are on some sort of web-oriented death march. I mainly think about how business intent is most effectively expressed in message payloads – something neither REST nor WS-* are really helping with (it’s semantics after all).

My friend Paul Downey (http://blog.whatfettle.com/) of British Telecom (erroneously placed in the “wrong” camp in an earlier version of Stefan’s List) put together a brilliant slide deck (pdf) that unambiguously (and literally) illustrates his point of view about WS-*. I’ve been meaning to steal from it for some time. BTW, Paul is chairing a W3C working group to advise toolkits about mapping language constructs to XML Schema constructs – something that will help both sides of the WS isle. I think Paul’s working group will enable more XML Schema constructs to be supported in the mainstream, which in turn increases the vocabulary I can employ to represent my business intent.

REST Doesn’t Obviate WS-*
There is a VERY large IT constituency that relies heavily on modeling non-functional requirements. Pioneers have the luxury of trying these things out and mitigating the shortcomings as they go. Obviously, current enterprises want to extend their architectural core concepts into more open realms. But they need to prove that service-level requirements are achievable and that non-functional aspects can be modeled using approved best practices.

WS-* has grown around those needs. The specs provide CYA support for the big IT shops. I sometimes get upset that WS-* is no longer simple or particularly exciting. But people are often compelled into multilateral infrastructures and to make their complex interactions interoperable. There’s apparently a lot of money in achieving that.

PS. For the record, I use REST and tunnel it through SOAP and/or WS-* where needed.

2.0 is to SOA what SOA is to ESB

About a year ago, I put out a blog post describing ambitions for SOA as an architectural basis within a single application domain. It was about relating messages, code, and data bi-directionally using transforms. Formal descriptions for those transforms could then become a new order of patterns in software architecture. The goal wasn’t to solve the gap between describing requirements and cutting code. It was to link the overall solution concerns more intelligently and pave the way for more declarative aspects and dynamic features.

The mistake I made was in trying to put a moniker around the concept – SOA 2.0! At the time, the ESB community was pushing really hard to say ESB==SOA. I wanted to distinguish architecting integration solutions – the bread and butter of an ESB – from architecting the traditionally opaque integral applications within an enterprise.

But calling it SOA 2.0 was clearly a bad choice. It’s sort of nice to be near the top of a search result (Google: “SOA 2.0”). Maybe your experience will vary, but I cringe when I see what company I have on the same page (but I did beat Oracle by a few months!). The post has been linked by others (Hinchcliffe, Little) but not exactly the way I hoped. A rookie blogging mistake if there ever was one.

Workflow Flavors

“Workflow” came up during a panel discussion I was in at Microsoft’s Tech-Ed conference this year. The word itself – like many IT terms these days – is overloaded to the point where it is hard to distinguish what a given “workflow toolkit” is meant to achieve. When explaining workflow concepts, I’ve started by making sure the audience understands the differences between 3 major categories of workflow:

• Human workflow is where information is conveyed to real people for action or simple notification. Workflow systems that present documents to employees have been around for years – so human workflow wins the right to actually use the word “workflow”. The key issue for human workflow is to have a system where non-technical users can actually program the routing.

• Orchestration is collaboration between application domains (and, by extension, between enterprises). BizTalk, Sonic, and ESB’s are all orchestration tools even though their target markets and general features may differ widely.

• Service Agents route execution of logic within a specific application domain. This is the sweet spot for Workflow Foundation. Service agents are under-served by toolkit vendors, which is surprising given the demands for content-based logic routing in the SOA world.

The trick for WF is to prove it can actually become the primary message pump for an enterprise application. But what drives me crazy is when people think of WF as some sort of “BizTalk Light”, which does both products a disservice.

PS. I haven’t blogged in quite a few months for no reason other than not being sure what to go into. So I thought a gentle entry like this might get the wheels moving forward.

The W3C Schema Patterns WG is not Misguided

Dare Obasanjo’s recent post labeling the W3C W3C XML Schema Patterns for Databinding Working Group as misguided seems like an overreaction. The toolkit vendors put out bad XML Schema processors and invented those leaky abstractions in the (misguided?) rush to make XML painless and web services a transparent feature for programmers with typical skillsets and approaches.

Many (including me) think that moving to XML as a primary integration mechanism for applications should a great step forward. But many IT staff who actually have to link different apps together are complaining that their job is much harder now than it ever was. Industry consortia are having a hell of a time publishing good standardized schemas because the inclusion of some seemingly innocuous XML Schema features will unknowingly break constituent implementations.

You can’t tell developers to simply avoid statically typed languages (at least not yet). You also can’t tell developers to wait a bit longer and the toolkit vendors will somehow watertighten their abstractions in an interoperable way. You *can* tell developers to avoid leaky abstractions to process XML, but you get resistance (which is regrettable). Worse, developers have to sometimes fight their toolkits to even do get to the message payload.

So, I don’t know what is so wrong about the W3C trying to alleviate the situation by attempting to shine a light on issues that seriously impact users. It may perpetuate XML <-> OO binding, which many people — including me — think is a problematic strategy. But maybe the W3C can get the Infoset in more hands sooner. More people can walk before they run, if you will.

UPA is your Friend (Repost)

The W3C XML Schema specification has a rule called unique particle attribution (UPA) that confuses, well, many. And after 4+ years of XML Schema in the wild, it still amazes me how inconsistent toolkits handle the issue. One toolkit – I think XMLSpy – at one time rebelled against UPA by enforcing the rule only if the user wished it. Just this week, I found a discrepancy in handling UPA between SQL 2005 and .NET Framework 2.0 (more on that later).

One problem in trying to comply with the UPA rule is that it is hard to describe in words. Here are a couple of examples to get the point across. This schema type is perfectly fine:

<xs:element name="MyType">

<xs:complexType>

<xs:sequence>

<xs:element name="Foo" />

<xs:element name="Foo" />

</xs:sequence>

</xs:complexType>

</xs:element>

But this type violates UPA:

<xs:element name="MyType">

<xs:complexType>

<xs:sequence>

<xs:element name="Foo" minOccurs="0"/>

<xs:element name="Foo" />

</xs:sequence>

</xs:complexType>

</xs:element>

Why? Suppose a schema processor is checking a document that looks like this:

<MyType>

<Foo />

</MyType>

The processor can’t figure out which element declaration in the schema to match the element “Foo”. It must be able to find the match unambiguously. You might think that it just doesn’t matter: the document still fits the description. But it’s critical for a schema processor, because it must be able to align the current XML document node with exactly one schema declaration. This disconnect between how people feel they should be able to describe a document and how a schema processor really works is what’s really behind the criticism of UPA. But most documents and schemas are WAY more complicated than these examples, and the UPA rule keeps the validation mechanisms manageable. By the way, this change to the schema fixes the UPA problem – can you see why?

<xs:element name="MyType">

<xs:complexType>

<xs:sequence>

<xs:element name="Foo" />

<xs:element name="Foo" minOccurs="0"/>

</xs:sequence>

</xs:complexType>

</xs:element>

When the schema validator hits the first “Foo” element, it knows that “Foo” must have been declared in the first element declaration within “MyType”.

The UPA problem can come up just as result of schema factoring. I came across this situation when looking at a schema for an XML document containing a SQL expression tree:

<xs:element name="Subquery">

<xs:complexType>

<xs:choice>

<xs:sequence>

<xs:group ref="tns:grpColumn" minOccurs="1" maxOccurs="1" />

<xs:group ref="tns:grpQuery"/>

</xs:sequence>

<xs:sequence>

<xs:group ref="tns:grpColumn" minOccurs="1" maxOccurs="1" />

<xs:element name="ListItem" minOccurs="1" maxOccurs="unbounded">

<xs:complexType>

<xs:attributeGroup ref="attGrpScalarValue" />

</xs:complexType>

</xs:element>

</xs:sequence>

</xs:choice>

<xs:attributeGroup ref="tns:attGrpSubqueryItems" />

</xs:complexType>

</xs:element>

This schema type gives a choice of two sequences, both starting with an element group called “grpColumn”. So, when the schema validation processor encounters the group elements in an XML document, it can’t tell which choice is in play. Also, the schema spec designers did not want schema processors to look “past” the current node to try and resolve which schema node matches a document node. By the way, this is the schema construct that .NET Framework 2.0 compiles with no errors, but SQL 2005 (via MSXML 6.0) throws a UPA violation error.

I refactored the schema a little to eliminate the UPA violation:

<xs:element name="Subquery">

<xs:complexType>

<xs:sequence>

<xs:group ref="tns:grpColumn" minOccurs="1" maxOccurs="1" />

<xs:choice>

<xs:group ref="tns:grpQuery"/>

<xs:element name="ListItem" minOccurs="1" maxOccurs="unbounded">

<xs:complexType>

<xs:attributeGroup ref="attGrpScalarValue" />

</xs:complexType>

</xs:element>

</xs:choice>

</xs:sequence>

<xs:attributeGroup ref="tns:attGrpSubqueryItems" />

</xs:complexType>

</xs:element>

So again, you could argue that both schema examples are identical when it comes to describing an XML document. Why should UPA make me use a specific methodology? It’s because the people who created the XML Schema specification were also envisioning how schema validation tools would be created.

In the case above, UPA forced me – IMO – to create a better schema. So aside from just general unfamiliarity with UPA and inconsistent toolkit support, does UPA really do any harm? Schema authors sometimes want to allow users to add extra elements to their XML documents. Since you don’t know what the XML will look like ahead of time, it’s nice to use a wildcard like xs:any in the schema. The problem is that UPA restricts the occurrence of a wildcard element depending on the occurrence of the element previously declared.

For example, this is a legal sequence:

<xs:sequence>

<xs:element name="Foo"/>

<xs:any />

</xs:sequence>

However, you can’t change the occurrence of “Foo” to any value other than “1” (the default) or you violate UPA. UPA does not allow any wildcard to be next to an optional element. Also, you can’t put a wildcard before any element in the same namespace. Some feel this constrains extensibility in some cases.

But honestly, I haven’t found a situation where UPA caused pain that a workaround doesn’t fix. I also think you can extend schemas better by wrapping them rather than extending them. In other words, create a schema that imports the schema you want to extend, add some new elements and tie them to the original schema through key/keyref declarations. It’s really the only way for schema validation to work with content unknown at design time.

Islands of Metadata

Does anyone remember the term “islands of automation”? I’m not sure where it was coined, but it described - in the late '70s - the situation where an enterprise implemented multiple systems with little or no integration. Pretty much every enterprise application has a customer table. Having a CRM system, a logistics system, and, say, a financials system with individual ideas of a customer creates obvious problems.

The “islands of automation” problem was solved in two unsatisfying ways. One way was to install a single big applications suite. The customization costs were big and enterprises were forced to adapt their business to whatever functionality was baked-in the suite. The second way was to buy a set of so-called “best of breed” point solutions. But integrating mission-critical apps can be more expensive than customizing a single app. Neither approach avoided vendor lock-in – whether from an apps vendor or a systems integrator. But I guess that’s sort of the idea, isn’t it.

The metadata-based toolkits under development at Microsoft seem in a similar conundrum. On one side are the individual mechanisms like Indigo, Windows Workflow, DSL, and distributed solutions. Individually, they are interesting toolkits – but they have no awareness of each other. On the other side is Microsoft Business Framework, which uses a single, rather big pile of metadata to describe an application from the database up to the UI. Here we go again: Do I want disconnected best-of-breed metadata, or closed, monolithic metadata.


A Conceptual Leap at PDC 2005

I saw one very promising demonstration at the Microsoft PDC 2005 conference – one that broke a conceptual barrier. Don Box and Dharma Shukla showed Windows Workflow Foundation (WWF) acting as an agent for some Indigo (a post-Whidbey version) services. Indigo-tagged components were dragged into workflow design surface that in turn represented a specific service action. Voila! The service description was inferred from the message “needs” of the application components.

This is significant because loose coupling between application components and the agent layer of the (canonical) SOA stack means that the agent describes the services. Only the agent knows the complete content of the message. Only the agent knows how to distribute message parts to components.

The other thing I liked was that – for once – the metadata behind WWF wasn’t just source code behind a code generator. Microsoft likes to write tools that spit out reams of partial classes like shot from a Holland & Holland (and about as constructively). Maybe Microsoft feels that OO coupling and static types is how us minions want to develop systems. A more cynical explanation is that nothing sells an idea – at least internally – like good support for IntelliSense. But I digress.

The XOML files used in the WWF/Indigo future-ware demo were processed dynamically (or at least portrayed to be - it was a demo). Generated code may be all the rage, but I only like generated code in one usage pattern: transient assemblies. Here was a great example of an agent that was adept at managing work instances while all the while knowing that the processing plan can change at any moment.

I used to think I was old enough to just let Kool-Aid roll down my back. But seeing the WWF and Indigo teams pick up on this concept definitely brought gave me something to smile about.