Poking at ROA

+1 to Tim Ewald’s point that the ROA crowd might be pressing too hard to make PUT do some uncomfortable things. -1 (with all due respect) to Stefan Tilkov’s assertion that “anytime you find yourself adding words like “operation” to your representation, you’ve violated one of the core RESTful HTTP principles, which is that the intent should be communicated using the HTTP verb.” IMO, that’s an unfair litmus test for ROAishness.

Stefan was chiding a specific situation with respect to GData, which I do not know much about. But I do know that situations exist where I want to convey multiple “intentions” in a single physical call. You should be able to update a customer and a supplier in the same message -- to support arbitrary composition of the intent. What's the URI for that? That’s a rhetorical question, because a POSTing location and a schema indicating the message format are all you need.

I’m a big fan of ROA, but I’m worried that ROA fundamentalism will create a quagmire (shared by SOA) where all advice seems to be about what *not* to do. ROA makes it easy to bind the HTTP verb with your intent, but it doesn’t require you to do so. If I define a format for a message that can contain multiple “intentions” and then expose POST endpoint for processing messages, have I broken some Law of ROA? I don’t think so.

Or, like Tim asks, does ROA mean I have to PUT things in an imaginary basket and then PUT an imaginary thing into an imaginary place to make the basket get processed? No. There is no crime in using The Uniform Interface in a way that partners the payload, verb, and URI to dispatching logic or help give you a cleaner programming model.

Sure, it’s best to keep business process protocol stuff out of the data in your payload, which is what I think Stefan was really alluding to. That obviously gives you the ability to reuse a message format by isolating intention to the URI. But resources often have internal processes, like special state transitions, which may need to be manipulated via flags in the payload.

The free market rightly determined that WS-* is often too difficult to use and it probably doesn’t solve the problems you think it will. But if ROA forces people to use more effort or do unnatural acts to get a day’s work done, ROA will be out on its ass as well. In both cases, the good goes down with the bad.

REST by way of SOAP

It’s OK to love REST (as I do), but I try to not let it blind me completely from WS-*. As I’ve often said, choosing REST – or any resource-oriented architecture – means you’ve decided to adopt a uniform interface. That does not necessarily mean dropping WS-*. Unfortunately, SOAP 1.0 emphasized an RPC programming model even though a pure message-based model was included as well. SOAP v1.2 sort-of/kind-of attempted to make amends:

Some early uses of [SOAP 1.1] emphasized the use of this pattern as means for conveying remote procedure calls (RPC), but it is important to note that not all SOAP request-response exchanges can or need to be modelled [SIC] as RPCs.

So, SOAP from the beginning identified two payload models: the Document model indicated that the payload is simply a message. The RPC model meant that elements within the payload should be mapped to functions and parameters.

SOAP also included a basic type system – SOAP Encoding – to help implementers know how to serialize objects defined – as Don Box might put it – in languages that use dots. It had a rather short life because the W3C was about to deliver XML Schema and having 2 type systems in SOAP wasn’t going to be helpful. For some reason, using XML Schema to define a message format was called Literal.

So, SOAP messages could be considered RPC-Encoded, RPC-Literal, Document-Encoded, or Document-Literal. The WS-I Basic Profile working group unanimously banned Document-Encoded and RPC-Encoded because they were obsolete, leaving *-Literal in play.

But what people generally don’t know is that some members including Microsoft and IBM argued hard to go farther and eliminate RPC-Literal as well. Tim Ewald had said “the message is the task” meaning a payload identified with a namespace URI is all you need to dispatch a call. But others (like JAX-RPC users) argued that keeping RPC and was critical for the adoption of web services.

In an attempt to mollify the RPC advocates, Tim, Don Box, and Keith Ballenger went so far as to write up a spec for mapping object graphs to an XML tree so implementers could align their type builders and serializers using a common interpretation of a message schema. The idea was to standardize an object-graph to XML conversion and include it with the WS-I Basic Profile.

For bureaucratic reasons, the WS-I couldn’t publish it (but you can find it: USPTO Application 20040239674). BTW, think about how useful that (now patent-pending) spec would be today in standardizing JSON/XML conversions. It might have also helped out these guys [W3C XML Schema Patterns for Data Binding].

So, the SOAP Document/Literal approach was an early favorite approach to web services – and its closer to REST than many realize. It standardized on GET and POST, which certainly aligns with my thinking. The URL for a GET can easily be cast as a straight resource fetch. The URL for a POST is generally cast as a “processing endpoint”, which reflects real-world thinking, IMO (you never really PUT what you GET). That leaves dispatching – invoking the right code when a message is received.

Both REST and WS-* users can tell you it’s difficult to rely only on the value of a URL path to determine how to process a message. Sooner or later, you have to look at the payload. The original SOAP mechanism involved an HTTPHeader called SOAPAction, which was a hint instructing the application about how to process the message. Many REST implementations use the content-type HTTP header in exactly the same way (which technically violates the HTTP 1.1 spec, BTW).

Using an HTTP header value to indicate what your payload is makes some sense if your payload can be anything in any format. But if your payload is XML (or a well-formed JSON array), then why not dispatch the call based solely on the (qualified) name of the root element? The only real difference between SOAP and REST here is that SOAP has an envelope construct and your data goes inside an element called “body”. If it’s a non-SOAP call, don’t use an envelope – it’s OK.

What about the WSDL? Don’t use one unless you have the need to build RPC-oriented proxies for your service. All you really need is a schema for the message format and away you go. If your endpoint is HTTP (if you’re looking at REST, when is it not?), you don’t even need a proxy built for you at all – another reason REST is getting popular.

In my REST work, I’m keeping an eye out to avoid straying too far from the bits of WS-* that might come in hand sooner rather than later. Security is my biggest worry. I don’t want to architect around the notion that transport level security is all I can have. I might need to sign that XML and encrypt bits of it anyway. I might, someday, actually come across an intermediary processor. Maybe that header element can be useful.

Most of WS-* is now beyond my comprehension and I don’t see ever using major parts of the stack because the toolkits won’t support resource-orientation to the degree I need. But HTTP, SOAP, the WS-I Basic Profile, and XML Schema can be useful even in completely RESTful projects. If you want.

How Do Bandwagons Fit into a Project Plan?

That thunder behind you is the REST bandwagon comin’ your way. Anne Thomas-Manes at the Burton Group has proclaimed that the future of SOA lies in REST. I’m not saying the Burton group is itself just jumping on a bandwagon – but their declaration is significant enough to convince others to take a leap. The REST dust-up is in such a very early phase and it’s far from clear whether the bandwagon is going to help me or hurt me. David thinks he might jump on, but let someone else do the driving. Nothing wrong with that strategy, but does it feels like history has come around a bit, eh David?

I think the REST bandwagon, the long-time Lords of the Web, and the WS-* camps are heading toward a colossal and rather fun collision. There will be friction between the stewards of HTTP and those who warp it to meet their RESTful needs. The REST camp likes to say HTTP is all you need – and I agree. But don’t be fooled into thinking that the foresight of the authors of RFC-2616 and RFC-2396 took all this into account. The RESTafarians probably won't want to see their momentum slowed by the standards trust. And then someone is going to say, “Hello! Is that an XML document you’re handing me? Care to sign it?” The WS-* folks are already figuring out how to chase that ambulance.

I’ve wasted a lot of cycles worrying that the Lords of the Web are going to tell me I’m perverting the Web’s majesty. For whatit's worth, here is my advice. First, it’s OK to say you are doing REST even if you haven’t read the dissertation. Personally, I think the magic is in the URIs and the links embedded in the payload. But that’s just me. Second, use the HTTP methods however you see fit. I use GET and POST exclusively myself. I use the occasional URN and I don’t think it’s an evil thing. Some people use the HTTP media-type to indicate what the payload contains. But RFC-2616 “discourages” the use of media-type values that are unregistered (section 3.7). Just use media-type if it helps you. When my payloads are XML, I have a namespace (a URN, BTW) to indicate what it is. For HTML, I haven’t really settled on anything yet.

No matter how far ahead of the REST bandwagon you think you are, it *will* overtake you. I’m sure our marketing department will snap up REST as fast as we did “web services”. I blinked one day and the word “SOA” appeared on the collateral. But you don’t need a REST toolkit – use the bits you have now. More importantly, don’t let the toolkit vendors say “just make objects – we’ll hide the REST part for you”. Good musicians learn to improvise by first knowing their scales. If you know the fundamentals of data modeling, URIs, and HTTP you shouldn’t need WADL, WCF, O’Reilly, or the bandwagon to be successful.

Harry & David

Harry and David (not these guys) are blogging comments to each other about REST. I wasn't sure where to leave my own, so I guess it's here...

David wonders whether widely varying interpretations of REST is an interop killer. Now that is an interesting question. It’s arguable that if two systems have completely different notions of state transitions and URI constructs, interop will suffer and you’re back to writing glue – or pitching for standards. On the other hand, there are those who argue that RFC-2616 has everything you need to be “good enough”, which, conveniently for them, makes interoperability an app domain issue. I think the web wonks have a good point. But it’s hard to know if resource-orientation (sorry) is a way forward unless you first let go of interfaces. One thing that scares the crap out of me is that someone will wire URIs and BPEL together and declare “Mission Accomplished”.

In Perfect Land, you can look at a URI and know what it does. URIs aren’t overloaded by packing the query string. Behavior is consistent and payloads for a given URI have the same format no matter whether you GET, POST, or PUT. But the RFCs (rightly) do not attempt perfection – real-world experience shows you rarely POST exactly what you GET. Also, ambiguities in the specs used to be settled by the whims of browsers. But now those specs are being used to connect systems instead of browsers to servers. I’m not ready to look a trading partner in the eye and say “hmm, let’s see how Opera handles an HTTP 415”. I’ll instead go hide in a closet and do just that.

So, even with the flawed cast of characters you see a lot of whining about – HTTP, URI, XML, and even (gasp!) XML Schema – the pieces are there to build good systems that also make great constituents in anyone’s SOA. The specs, with one glaring exception, are easy to digest. My advice is to start by thinking about a URI strategy that people can follow intuitively. Don’t underestimate the value of a good URI set or the design skills it takes to build. I would also ignore worries about pissing off the Old-Hands of the Web by somehow not using the Web exactly as intended. Their crystal balls weren’t clearer than anyone else’s and these guys do put pants on the usual way (not that I’ve personally witnessed it).

And on Harry’s point that REST and CRUD, I agree that Tim was simply advising people not to limit their comprehension of REST around entities accessed via GET and PUT. REST resources are like views that may or may not be underpinned by an entity model. REST state transitions use URIs to label and invoke services which may or may not use an entity/CRUD programming model under the hood. I agree with Tim in that the presence of those URIs as links is what actually defines REST – NOT simply that you’ve labeled data with URIs.

REST Protocols are the Service Layer

Business operations look a lot like protocol state machines. One of the areas I’ve been looking at (for several years) is how to better leverage state machine thinking into data-driven applications design. I read the REST dissertation, but didn’t see how Roy Fielding’s notion of state transitions applied to my projects. A couple of years ago, David and I looked at how server-side state transitions could drive a service layer (and more). We talked about writing up a paper, but I flaked out so David went ahead with a post.

So, I read Tim Ewald’s posts about REST (first post is here) with a lot of interest. Tim’s description of REST is centered on client protocol state transitions and, critically, ensuring responses to GET requests include the URIs (links) that transition between client states. Up until now, I ignored the idea of URIs for client states that are not mapped to system state transitions on the server. I also questioned the wisdom of including links at all in REST-style responses. Here’s why.

A classic blunder in applications development is designing the programming model (and it's API) around a specific client interaction style. An acid test for an API is how well it works for different calling scenarios, especially those unknown at design time. On that second point, only time can tell you if you have a winner. SOA itself was invented largely to counter unsuitable APIs (and fill space on collateral). Service layers usefully loosen the coupling (you don’t need my binaries to call me) and serve as adapters between an application and the potentially numerous and radically different callers.

So, isn’t it a slippery slope to presumptuously inject URIs into messages for operations that might not apply to the client’s intention? In Tim’s example, the server returns a list of itineraries along with URIs that transition the client to one of 2 potential next client states: getDetails or Reserve. One of these URIs changes system state and the other doesn’t. Rightly, you use GET on one of the URIs and POST for the other (although the example doesn’t indicate which is which). The “getDetails” tag bothered me because in a non-trivial case there might be numerous operations available relative to the resource. Or, the resource might be a summarization (like a report). If the data has a thousand customers IDs and there are 10 or more links possible per customer – you get the point. Why offer up a bunch of links that the client won’t use and potentially outweighs the actual data? Is it even possible to determine the complete set of links?

What I failed to remember, until now, is that data and protocols are independent. There can be multiple protocols with the same data as the starting point – but not the same URI. The URI identifies what data the caller wants and under which protocol.

Data + protocol = resource -> URI

The protocol embodies the caller’s intentions and the server’s constraints. Protocols are easy to secure (URI = security descriptor) and they work like interfaces for data hiding. My mistake until now was in believing that:

• Each entity (I work on ERP apps and we still think about entities) is a resource that has a protocol defined by a system state sequence.
  
• The only useful links in resources are URIs for posting value changes or transitioning states
  
• Views of data representing many resources have no links because it’s difficult to generate a complete set and the caller’s intentions are unknown.

After working through Tim’s description, I’ve reached different conclusions about how to design a REST-style system:

• I still like defining a state sequence and change constraints for entities in the application. It’s a good programming model for ERP because the server-side state transitions are handy places to hang workflow logic. Also, entity/life-cycle thinking aligns well with the business operations.
  
• Systems have protocols that describe data interactions with reasonably specific purposes in mind. The views passed in and out of the system are resources.
  
• A URI identifies a resource, a protocol, and where in that protocol the caller sits.
  
• Resources contain links to other resources or invocation points specific to the protocol. This solves my issue with (potentially) large numbers of unhelpful links.
  
• Protocols buffer the application from conversations – protocols are the new “service layer”.
  
• Getting back to the first point, a “default” protocol can be derived from the server-side state sequence for an entity.

Protocols can carry links to places other than the initial server and resources can be more than just XML representations of entity data – think of the mash-up potential. The URI – via the protocol – tells the system what data to get, what format to use, what stylesheet to apply, and which links are useful to the caller. Adding new protocols to a system needs to be an easy thing to do.

I thought I knew REST and had taken a decent crack at applying it to ERP applications. This re-think has me wondering what else I haven’t figured out yet. In the meantime, I’m looking back at our web services, metadata, and customer-driven scenarios to see what features protocol definitions might need. The goal is to find a way to make it easy for our developers and our customers to create protocols. I’ll write up what I find.