REST URI Spaces and Information Reuse

I’m on a plane returning from the QCon 2008 conference in London. It was a top-notch event and among the great presentations, two things I learned stand out. The first was that I need want to learn Erlang. I spent some time with Erlang inventor Joe Armstrong, and had such good fun that I’ve already downloaded the bits and bought the book. Second, the REST rationale has really gelled and the proponents no longer see a need to argue their case – it’s time to mature the story.

In the REST track, it was said (repeatedly) that WS-* circumvented the “design” of the Web and thus ignored the Web’s innate capabilities (for example, Jim Webber’s slides, #14). Sure, architectures often encapsulate subsystems for the sake of abstraction, which brings me to my point. Just as WS-* tunnels the Web, component architectures often manage persistence by tunneling a relational database. The query processing power of the database is rarely directly accessible by the solution consumers.

My company sells ERP applications, which are used to run enterprises. Our customers demand commercial RDBMS products because, in their minds, our application maintains their data. That data must be in a store they understand and (feel they) control. These enterprises invariably invent ways, bypassing our code, to productively reuse their data in ways their vendor (us) did not foresee. Conceptually, I don’t see much difference between this kind of reuse and the serendipitous reuse described by Roy Fielding (and summarized nicely here [pdf] by Steve Vinoski). If their data was stuck in an opaque blob, accessible only through our APIs, we wouldn’t sell very many systems.

But like I mentioned earlier component architectures relegate relational databases to soulless “persistence stores” devoid of independent capabilities. The conventional wisdom for component design says data is private to the component and access only through a public interface. But I think people have tended to militantly privatize data, denying support to anyone daring to connect to the data on their own. That practice is one reason some solutions are opting away from an RDMS stores altogether and going with more OO-friendly stores.

The obvious problem with taking this tenet too literally is that access to data is limited to a preconceived set of interfaces. Reusing data in unforeseen ways is accomplished by sneaking around the application or rebuilding it to accommodate new requirements. Databases are rather smart engines designed under an assumption that data, once created, is then retrievable in countless, unforeseen ways. Nevertheless, architectures are burying perfectly good query processors under layers of abstractions, objects, and interfaces. Being pushed to the bottom of the stack is one reason that physical data models are actually devolving, IMO, but that’s another story.

The public “API” for a RESTful application is its URI address space. You can invent a list of URIs mapped to resources and state sequences all you like. But the reuse potential is limited to whatever your callers can get out of that URI space. Like REST, SQL databases have a uniform interface. But look at the practically unlimited variety of resources you can access. Obviously, a REST URI shouldn’t be a SQL statement and I’m not trying to shoehorn XQuery into a URI. All I’m saying is that a URI space can incorporate parent-child and relational characteristics from a data model – using relational database behavior as a guide. This has been a key aspect (for 8+ years, BTW) in developing URI strategies for our products.

The emerging specs and toolkits, like WADL and WCF, feature URI template constructs. But URI templates have no notion of resource linkages (parent, relational, or otherwise) and that limits their effectiveness. At QCon, there was little consensus that WADL was the right way to describe a REST application. But I think REST description languages resource types are coming and I’d like their creators to at least consider resource linkage features for URI templates. It’s all been done before.

Got my name in the Financial Times!

Only a few people know I like to hobby with macroeconomics during break-time around the office*. Most of my colleagues had begun the weekly yoga session in the area outside my office. So, the lights were out and the soothing music seeped under my door. I was reading the commentary in that day’s FT – the top op-ed piece essentially said that the US should welcome sovereign wealth funds without looking too intently under the hood.

Well, clearly someone had to say something. So, I channeled a little John Maynard K. and typed up my first-ever letter to an editor. And in the spirit of what can be done to “some of the people some of the time”, the FT printed my letter (requires free registration to read the whole letter**) in last Friday’s edition (at least in the US). I had no idea until I stumbled on it while on the train home from work. Cool!

* Not really.
** You can try this link until the FT scrolls the entry into the archives.

Poking at ROA

+1 to Tim Ewald’s point that the ROA crowd might be pressing too hard to make PUT do some uncomfortable things. -1 (with all due respect) to Stefan Tilkov’s assertion that “anytime you find yourself adding words like “operation” to your representation, you’ve violated one of the core RESTful HTTP principles, which is that the intent should be communicated using the HTTP verb.” IMO, that’s an unfair litmus test for ROAishness.

Stefan was chiding a specific situation with respect to GData, which I do not know much about. But I do know that situations exist where I want to convey multiple “intentions” in a single physical call. You should be able to update a customer and a supplier in the same message -- to support arbitrary composition of the intent. What's the URI for that? That’s a rhetorical question, because a POSTing location and a schema indicating the message format are all you need.

I’m a big fan of ROA, but I’m worried that ROA fundamentalism will create a quagmire (shared by SOA) where all advice seems to be about what *not* to do. ROA makes it easy to bind the HTTP verb with your intent, but it doesn’t require you to do so. If I define a format for a message that can contain multiple “intentions” and then expose POST endpoint for processing messages, have I broken some Law of ROA? I don’t think so.

Or, like Tim asks, does ROA mean I have to PUT things in an imaginary basket and then PUT an imaginary thing into an imaginary place to make the basket get processed? No. There is no crime in using The Uniform Interface in a way that partners the payload, verb, and URI to dispatching logic or help give you a cleaner programming model.

Sure, it’s best to keep business process protocol stuff out of the data in your payload, which is what I think Stefan was really alluding to. That obviously gives you the ability to reuse a message format by isolating intention to the URI. But resources often have internal processes, like special state transitions, which may need to be manipulated via flags in the payload.

The free market rightly determined that WS-* is often too difficult to use and it probably doesn’t solve the problems you think it will. But if ROA forces people to use more effort or do unnatural acts to get a day’s work done, ROA will be out on its ass as well. In both cases, the good goes down with the bad.

Software Architecture as Principles

So, I had The Talk with my 10-year old son today. He was a confused and even became a bit emotional as the gravity of the facts emerged. The discussion was, of course, about copyright law.

He had burned a CD containing tracks from his iTunes library to give to a friend as a birthday gift. So, we talked about how this was in fact stealing and that we should just go buy a new copy the music outright, etc. Here’s a sampling of from his questions during the discussion:

Q: What if my friend has some songs on his iTunes, but his CD burner is broken. I have the same songs, so can I burn them from my computer and give him the CD? A: Um, maybe that’s OK. I don’t know.

Q: How come it’s OK to lend out PlayStation disks? A: Easy (whew!), because while they borrow it, you are not using it (note to self: is that really legal?).

Q: So, can I burn a CD, give it to someone, and just not listen to the songs myself until I get the CD back? A: Um, you’re late for school – off you go.

Like software architecture (and legal systems), copyright is a principles-based rather than a rules-based concept because it’s impossible to precisely spell out, up-front, all actions that constitute non-compliance. The principle says you can’t disseminate the work of others without permission. Laws assign penalties to broad forms of violations like producing counterfeit software. Court precedents over time develop the more specific lists of what’s OK (backing up your iTunes) and not OK (giving out copies).

Software architecture works the same way. Software architectures are collections of principles that define guidance and boundaries for producing software systems. Lower-level guidance comes in the form of standards and practices for developing a software system that conforms to the principles in the architecture.

Principles-based regulation means that laws are enforced by examining the intent of people rather than reconciling deeds against a roster of specifically banned actions. The Enron scandal, it’s said, grew unnoticed for years because Enron parsed the regulations in bad faith and created scenarios that somehow convinced Author Anderson to sign off on audits. Enron and Arthur Anderson both knew that accounting principles (and the law) were being violated but felt relatively safe because any by-rote analysis of their accounts against the rules (as written) would come up clean.

UK regulators like to say they have no Enron situations because UK accounting standards are principles-based from the outset. I don’t know how true that is, but the infamous U.S. Sarbanes-Oxley Act of 2002 directs the U.S. Securities & Exchange Commission to examine the feasibility of moving the U.S. to a principles-based accounting standards system.

Getting (at last) back to software architecture, I work for an independent software vendor (ISV) in the enterprise resource planning (ERP) market. One of the characteristics of an ISV is that we are highly principles-based and generally don’t rely on thick volumes of specific details telling our engineers exactly what to do. Sure, we have standards around code and UI styles. But developers are taught the principles within the product architectures and the intent behind the functionality. That in turn helps prevent issues like “the spec didn’t specifically say the shipment quantity had to be a positive number”.

As we expanded product development overseas, we didn’t rely on outsourcing to contractors. We tried it, but it was too hard to convey the architectural principles and the business intent to a transient staff 13.5 time zones away. Without the principles as context, we had to type reams more specs and it the results consistently had to be reworked. We wondered whether our processes were just plain sloppy. But that wasn’t the case. Our development model just didn’t fit an environment where people came only with bespoken capabilities and never developed any lasting interest in our objectives.

Instead, we opened development offices staffed by our own full-time employees. That meant we could convey our architectural principles, development standards, and train the teams up on business functions like cost accounting and manufacturing. Permanent employees doing the work, no matter where in the world, turned out to be cheaper than just outsourcing to a contractor. More importantly, we realized much better agility.

The problem with developing long lists of rules is that they are expensive to maintain and easy to evade when the pressure is on. I would rather state a principle like “creating new records for purpose X MUST minimize the DB batch request count and NOT hold locks on Y, under typical loads, to significantly affect other processes” and then test for compliance rather than using, say, code reviews and a checklist to attempt to spot things that potentially violate the principles.
For us – a 400-person development organization – agility means efficiently releasing products year-after-year that incorporate rapidly changing markets. Believe it or not, technology shifts aren’t nearly as acute to us as market shifts from compliance and globalization. I never need to find 150 Ruby programmers on a moment’s notice. I need 150 people to understand how a chart of accounts works and how to make one that can be legally used worldwide.

So, while we don’t do many scrums in our product development cycle, we also don’t do waterfall management. The middle ground works because our jobs revolve around incremental evolution of a few systems. It’s an easy place for principles-based management to work. Software architecture perfection to me is nothing but a short list of general capabilities and a long list of non-functional requirements. Those are the principles. Standards and practices do the rest.

My son says he likes knowing how music is copyrighted. The thought he might be letting down people who really own the recordings actually horrified him (much more than me, anyway). By understanding the principles, he can (hopefully) figure out for himself what scenarios violate the law. Now, I guess we’ll have to do the “fair use” talk pretty soon, though. What other talks am I forgetting about?

Corporationlets

In the Financial Times on 13 July 2007 (European Edition, special insert), Linda Gratten asserts that collaboration is poised to upstage competition as a primary business strategy – that partnerships can create value more efficiently than pure competition. The X and Y generations were born wired to the Web, have been raised in an “everyone wins” mentality, and are averse to organizational hierarchies. So, once that new generation rises to corporate power, partnerships will form the dominant agent of business value creation. That last bit smacks me as unsubstantiated age discrimination – but creating value via joint ventures obviously works.

Since the focus of the piece was collaboration, describing Second Life as a vehicle for collaboration couldn’t be helped. Gratten mentions the “experiments” at IBM (“very successful”) to use Second Life for team interaction. Given that the GenY mastery of social networks is so entrenched, something akin to Second Life could become the wheel grease of tomorrow’s joint ventures. After all, any mass proclivity toward partnerships requires a place where participants can sniff each other out – an online speed-dating environment for stakeholders.

A gold rush has started to converge social networking and business strategy. The idea has moved beyond making money by connecting people together – that’s so last month. People need to make money for themselves by easily forming ad-hoc commercial partnerships. That kind of value creation can reap a huge economic benefit, especially if small businesses become enfranchised. So, the European Commission is trying to act on this vision.

The EU25 is home to 23 million individual businesses where more than 90% are “micro-enterprises”. In the EU15, only 10% of businesses have any computer integration with either side (supply and demand) of their supply chain*. Connecting these 23 million companies, so they can inexpensively form partnerships and new supply chains, is in the EC’s economic thinking. It also ties back to Gratten’s point about next-generation entrepreneurs choosing joint venture strategies over brute competition.

Micro-enterprises – like your corner laundry – are not generally capable enough to build inter-enterprise orchestrations. But there is one IT skill that the entire population has in-hand: Web browsing. So, the EU is smitten with the notion that Web 2.0, the Social Web, Second Life-ish solutions, SOA, and SaaS can all come together to form a solution – and so the European Commission is funding efforts [cordis.europa.eu] toward that goal. It’s a Grand Unification where businesses find each other and collaborate like Second Life players while the business transactions are easily, properly, and legally pumped across business entity boundaries.

It’s certainly my nature to drool over the software architecture challenges in building this dream. But my biggest worry, if I was picked to do the software estimate**, is determining when you’ve done enough to displace existing business practices.

Supply chain commerce, commodity purchasing, service industries, and consumer businesses all have buyers and sellers. But the operational characteristics of how buyers find sellers, how trust chains are established, how contracts are executed, and how payments are managed are highly evolved and optimized within each sector. Can a solution that undermines the efficiencies in business growth developed during the last century be successful?

Here’s an example: A few decades ago, businesses (especially manufacturers) kept many active suppliers on file. It was believed that having many suppliers for a widget meant more competition for your business kept purchase prices low. But it was later realized that reducing the number of widgets kept on-hand inventory improved profits much more than buying your widgets at rock-bottom price.

Just-in-time (JIT) manufacturing was invented for this reason. Material arrived at the precise moment and location where it would be consumed – which meant daily, weekly, or ad hoc replenishments. Perfection meant having zero widgets sitting in your warehouse. Purchasing strategies were simplified –you only have 1 or 2 suppliers on long-term contracts rather having to manage bidding cycles with multiple suppliers. But there were also the matters of trust and reliability. If a supplier failed to deliver, production would likely shutdown and employees would be furloughed.

The supply chain partnerships in JIT operations created more value than traditional brute competition. Manufacturers eschewed continuously fighting to get the best deal from a transient supplier to create strategic arrangements with a single supplier. Boeing and Airbus have gone even farther by having their suppliers share risk in major programs. Suppliers absorb some up-front costs in exchange for revenue to come later if the airplane sells well in the marketplace.

But doesn’t that fly in the face of the model the EU is envisioning? The goal is to foster easy, ad-hoc partnerships with extremely low cost of entry. However, manufacturers revolutionized their businesses by getting away from a scattered roster of transient. Is the EU is looking for technology to bring enterprises together despite entrenched, non-technical aspects?

The answer involves, as usual, a tipping point. Some sort of foundation technology – the goo in the Petri dish that feeds life – needs to be developed (no small task). Then, an entire population has to learn that partnering is feasible, online or not, and they have to know how to do it legally. Finally, participants have to know how to measure the quality of a trust chain.

From a business-to-business perspective, this is the final frontier of the Internet. It’s achievable and inevitable if the IT industry can get non-technical stakeholders into the game.

* These figures are from Eurostat via a keynote presentation by Christina Martinez of the European Commission at the 2007 WS-I Spring Plenary in Brussels.

** My estimating practices were pretty much codified here in 2005.

REST by way of SOAP

It’s OK to love REST (as I do), but I try to not let it blind me completely from WS-*. As I’ve often said, choosing REST – or any resource-oriented architecture – means you’ve decided to adopt a uniform interface. That does not necessarily mean dropping WS-*. Unfortunately, SOAP 1.0 emphasized an RPC programming model even though a pure message-based model was included as well. SOAP v1.2 sort-of/kind-of attempted to make amends:

Some early uses of [SOAP 1.1] emphasized the use of this pattern as means for conveying remote procedure calls (RPC), but it is important to note that not all SOAP request-response exchanges can or need to be modelled [SIC] as RPCs.

So, SOAP from the beginning identified two payload models: the Document model indicated that the payload is simply a message. The RPC model meant that elements within the payload should be mapped to functions and parameters.

SOAP also included a basic type system – SOAP Encoding – to help implementers know how to serialize objects defined – as Don Box might put it – in languages that use dots. It had a rather short life because the W3C was about to deliver XML Schema and having 2 type systems in SOAP wasn’t going to be helpful. For some reason, using XML Schema to define a message format was called Literal.

So, SOAP messages could be considered RPC-Encoded, RPC-Literal, Document-Encoded, or Document-Literal. The WS-I Basic Profile working group unanimously banned Document-Encoded and RPC-Encoded because they were obsolete, leaving *-Literal in play.

But what people generally don’t know is that some members including Microsoft and IBM argued hard to go farther and eliminate RPC-Literal as well. Tim Ewald had said “the message is the task” meaning a payload identified with a namespace URI is all you need to dispatch a call. But others (like JAX-RPC users) argued that keeping RPC and was critical for the adoption of web services.

In an attempt to mollify the RPC advocates, Tim, Don Box, and Keith Ballenger went so far as to write up a spec for mapping object graphs to an XML tree so implementers could align their type builders and serializers using a common interpretation of a message schema. The idea was to standardize an object-graph to XML conversion and include it with the WS-I Basic Profile.

For bureaucratic reasons, the WS-I couldn’t publish it (but you can find it: USPTO Application 20040239674). BTW, think about how useful that (now patent-pending) spec would be today in standardizing JSON/XML conversions. It might have also helped out these guys [W3C XML Schema Patterns for Data Binding].

So, the SOAP Document/Literal approach was an early favorite approach to web services – and its closer to REST than many realize. It standardized on GET and POST, which certainly aligns with my thinking. The URL for a GET can easily be cast as a straight resource fetch. The URL for a POST is generally cast as a “processing endpoint”, which reflects real-world thinking, IMO (you never really PUT what you GET). That leaves dispatching – invoking the right code when a message is received.

Both REST and WS-* users can tell you it’s difficult to rely only on the value of a URL path to determine how to process a message. Sooner or later, you have to look at the payload. The original SOAP mechanism involved an HTTPHeader called SOAPAction, which was a hint instructing the application about how to process the message. Many REST implementations use the content-type HTTP header in exactly the same way (which technically violates the HTTP 1.1 spec, BTW).

Using an HTTP header value to indicate what your payload is makes some sense if your payload can be anything in any format. But if your payload is XML (or a well-formed JSON array), then why not dispatch the call based solely on the (qualified) name of the root element? The only real difference between SOAP and REST here is that SOAP has an envelope construct and your data goes inside an element called “body”. If it’s a non-SOAP call, don’t use an envelope – it’s OK.

What about the WSDL? Don’t use one unless you have the need to build RPC-oriented proxies for your service. All you really need is a schema for the message format and away you go. If your endpoint is HTTP (if you’re looking at REST, when is it not?), you don’t even need a proxy built for you at all – another reason REST is getting popular.

In my REST work, I’m keeping an eye out to avoid straying too far from the bits of WS-* that might come in hand sooner rather than later. Security is my biggest worry. I don’t want to architect around the notion that transport level security is all I can have. I might need to sign that XML and encrypt bits of it anyway. I might, someday, actually come across an intermediary processor. Maybe that header element can be useful.

Most of WS-* is now beyond my comprehension and I don’t see ever using major parts of the stack because the toolkits won’t support resource-orientation to the degree I need. But HTTP, SOAP, the WS-I Basic Profile, and XML Schema can be useful even in completely RESTful projects. If you want.

Metadata Pathways (Rewrite)

I got a comment from Mark Little a couple of weeks ago about my 2005 blog entry Patterns for SOA 2.0 saying (nicely) that he couldn’t figure out what I was talking about. I completely agree – it’s like I took a quick discussion about metadata and the last-place entry in a Faux Faulkner contest and shoved them into that machine from The Fly.

I’ve been meaning to re-write the whole thing for a while. And earlier this week, I saw a comment from Jörg Schäfer here, which relates to what I was trying to say back on 2005:

Philosophical remark: Maybe what we need is a programming model that takes one of the few mathematically sound models of CS, namely relational theory, seriously (the other mathematically sound foundation is lambda calculus of course leading to functional programming). So rather than continuing with OO we need a proper programming abstraction for programming with relational algebra (entities, relations, RFD…).

I also saw David Ing asking Microsoft to please get on with whatever building 42 has cooking. I don’t know anything about that except to keep sunscreen with a high UV rating handy during the unveiling. The point is that all 3 examples (and I’m just guessing on the Microsoft one) are rethinking the abstractions used to define systems. Obviously, any benefits from improving the abstractions have great leverage for the rest of the system.

So, here goes another try:

If you take a typical approach to programming a data-driven application, you have a data access classes, business logic classes, (and now) service classes, client-side adapters, and UI classes. If it’s a large system you start breaking down common areas and define a framework. But in any case, you can wind up with a mass of interfaces, components, serialization, data binding, and event management.

It works initially – and you get great Intellisense support. But anyone who sells enterprise apps for a living knows that most any deviation from the initial functionality requires a rebuild of at least some application. The better solutions (like those from my employer :) ) go through significant hell trying to put the flexibility users want back into systems built on classic interface/component architectures and programming models. But that means fighting the toolkits and building lots of framework code. And on top of that, it turns out the classic interface-based programming model might not be the best way to go in the first place.

My point back in 2005 was this: Systems need to tolerate changing data models and changing behavior without downtime. Systems must accept varying message formats for a given business intent. Callers should be able invoke an arbitrary set of functions and specify the unit(s) of work. All data must be addressable by at least one human-readable identifier. Callers should not have to generate code or otherwise build proxies strongly typed to my application domain – but they can if they like (that’s a recent addition to the list).

I want to build systems around engines and metadata as much as possible. That doesn’t mean eliminating any work by a competent programmer. But I would like the work a programmer produces to stick to the explicit functional task at hand and be exquisitely reusable. I’m also not talking about DSLs or software factories (which I’m pretty much over with). But I like the idea of a system factory, where engines manage messages, data, and execution workflow – the primary colors of an enterprise system. The factory is a runtime that adapts the system as information about the application changes. The robustness of that system thus depends on the quality of its metadata and, crucially, the quality of coupling between metadata and the processes that consume metadata.

To get there, the industry needs to begin improving how abstractions are expressed through metadata – which is an architectural activity. Metadata now often spawns other metadata. Say I add a field to the customer entity in my data model (metadata). That infers a change to the message formats for my customer service – described in, say, XML Schema (metadata). My data model change also initiates a change to the physical data table (handled by an engine). The side-effects of metadata consumers are changes to system behavior, the generation of derived metadata, and many other possible actions.

Some attributes in my data model metadata affect message formats but not the physical tables (or vice-versa). In other words, data model attributes have aspects that drive processes that consume the metadata. For example, a concurrency model aspect tells the DDL engine to emit specialized columns to manage, say, record versioning. That same aspect also causes the message format description to include a required version value. The process that receives messages also sees this aspect and then knows to validate that messages actually the version value and checks to see if it is current.

Aspects might not just be name-value pairs, but complete metadata categories on their own. A good example is a life-cycle chart defined using a state sequence. As I’ve mentioned, I think you can derive all behavior (in my problem domain – ERP) by mapping CRUD constraints to a state chart – all of which can be invoked by a uniform interface. Putting my entity life-cycles into metadata and the understanding into (for example) the DDL and message description/processing engines is a pretty powerful approach because I can revise them to match business requirements as needed.

Aspects should be composable. For example, a Pat Hellend aspect in my data model metadata with values like resource, reference, and activity might combine the life-cycle and versioning aspects to make sure reference data stays read-only (affects the service behavior) and that resource data is versioned (affects message format, service behavior, and the physical data model).

Getting back to Jörg’s comment, let’s say that each attribute in a given metadata type represents a category A. You can then think of each aspect f as a morphism connecting A to another category B, where B is a consumer of the metadata type (f: A->B). As I mentioned, the target for a given aspect can be other metadata (like the message description), a running processor or engine (like the service layer), or anything else (like SQL DDL). So, “B” in this case is a collection of objects (like other metadata) or a behavior specification. I can also put an aspect g on category “B” to create a category “C” (g: B->C). That implies a compositional ability because for each element in “A” (a), you can get to a corresponding element in C by way of g(f(a)). If I have an engine that relies on metadata “C” and an aspect element a is changed, the engine knows it needs to execute the transforms g and f to get the revised member of “C”.

So, I have 2 questions for the world in looking to develop these lines of thought. The first is about dealing with the pervasive metadata pathways in complex applications. If we look at what all that metadata represents and how it affects a system, I think we could start to categorize metadata-oriented scenarios. In other words, we can identify software architectural patterns in the use of metadata – in both its content and the effects on systems. Does anyone agree?

The second question is about applying some sort of mathematical rigor to metadata architecture. The categories and morphisms I mentioned come from category theory, which may or may not have any promise as a basis for the formalism I’m looking for. I was an SDSU music major – what do I know? Category theory has been tried as a basis for programming languages without a lot of commercial success. But it also was helpful in connecting Cartesian Closed Categories and lambda-calculus (which is some of the math behind LINQ and F#). So, if metadata and metadata consumers are used to define systems, and we can cast these artifacts as collections of categories connected by morphisms, can some part of a mathematical discipline like category theory be employed to help formalize the way we define systems?

That was what I was trying to articulate in 2005 without success. I hope this comes off a little more clear.

REST Over HTTP

Harry Pierson asks a direct question about whether REST is necessarily bound to HTTP. I agree with Harry (and Anne Thomas-Manes as described here) that REST is an architectural style as much than anything else. And I also noticed this from the Alex James, who has a (self-described) rant on the subject. As much as the point is totally arguable and understandable, I think it’s an unwinnable battle.

I was at a WS-I meeting (the most productive one ever, BTW, because it was held here in Hawaii) a couple of years ago and casually asked Mark Nottingham (probably over a Monkey’s Lunch) about loosening the Basic Profile to allow for SOAP over TCP/IP. His answer was simply, “Why?” His point was this: Sure, you can drive SOAP over TCP/IP, Netware, whatever. But is there an interoperability goal in doing so? The answer is not really. You would need to own both sides of the pipe to make it work.

The IT industry is cozy with ports 80 and 8080. Opening up anything else practically requires an SEC 10-K disclosure. Come to think of it, we used to actually say “SOAP over HTTP” like they were independent and someone might actually use SOAP over SMTP.

I see the term “REST” in a similar situation. It’s tied to HTTP and it is probably impossible to wrest it away. I don’t have a problem with that – I’ll just learn to speak architecturally about “resource-orientation” and then about how to tie in “REST” out on the edge.

How Do Bandwagons Fit into a Project Plan?

That thunder behind you is the REST bandwagon comin’ your way. Anne Thomas-Manes at the Burton Group has proclaimed that the future of SOA lies in REST. I’m not saying the Burton group is itself just jumping on a bandwagon – but their declaration is significant enough to convince others to take a leap. The REST dust-up is in such a very early phase and it’s far from clear whether the bandwagon is going to help me or hurt me. David thinks he might jump on, but let someone else do the driving. Nothing wrong with that strategy, but does it feels like history has come around a bit, eh David?

I think the REST bandwagon, the long-time Lords of the Web, and the WS-* camps are heading toward a colossal and rather fun collision. There will be friction between the stewards of HTTP and those who warp it to meet their RESTful needs. The REST camp likes to say HTTP is all you need – and I agree. But don’t be fooled into thinking that the foresight of the authors of RFC-2616 and RFC-2396 took all this into account. The RESTafarians probably won't want to see their momentum slowed by the standards trust. And then someone is going to say, “Hello! Is that an XML document you’re handing me? Care to sign it?” The WS-* folks are already figuring out how to chase that ambulance.

I’ve wasted a lot of cycles worrying that the Lords of the Web are going to tell me I’m perverting the Web’s majesty. For whatit's worth, here is my advice. First, it’s OK to say you are doing REST even if you haven’t read the dissertation. Personally, I think the magic is in the URIs and the links embedded in the payload. But that’s just me. Second, use the HTTP methods however you see fit. I use GET and POST exclusively myself. I use the occasional URN and I don’t think it’s an evil thing. Some people use the HTTP media-type to indicate what the payload contains. But RFC-2616 “discourages” the use of media-type values that are unregistered (section 3.7). Just use media-type if it helps you. When my payloads are XML, I have a namespace (a URN, BTW) to indicate what it is. For HTML, I haven’t really settled on anything yet.

No matter how far ahead of the REST bandwagon you think you are, it *will* overtake you. I’m sure our marketing department will snap up REST as fast as we did “web services”. I blinked one day and the word “SOA” appeared on the collateral. But you don’t need a REST toolkit – use the bits you have now. More importantly, don’t let the toolkit vendors say “just make objects – we’ll hide the REST part for you”. Good musicians learn to improvise by first knowing their scales. If you know the fundamentals of data modeling, URIs, and HTTP you shouldn’t need WADL, WCF, O’Reilly, or the bandwagon to be successful.

Harry & David

Harry and David (not these guys) are blogging comments to each other about REST. I wasn't sure where to leave my own, so I guess it's here...

David wonders whether widely varying interpretations of REST is an interop killer. Now that is an interesting question. It’s arguable that if two systems have completely different notions of state transitions and URI constructs, interop will suffer and you’re back to writing glue – or pitching for standards. On the other hand, there are those who argue that RFC-2616 has everything you need to be “good enough”, which, conveniently for them, makes interoperability an app domain issue. I think the web wonks have a good point. But it’s hard to know if resource-orientation (sorry) is a way forward unless you first let go of interfaces. One thing that scares the crap out of me is that someone will wire URIs and BPEL together and declare “Mission Accomplished”.

In Perfect Land, you can look at a URI and know what it does. URIs aren’t overloaded by packing the query string. Behavior is consistent and payloads for a given URI have the same format no matter whether you GET, POST, or PUT. But the RFCs (rightly) do not attempt perfection – real-world experience shows you rarely POST exactly what you GET. Also, ambiguities in the specs used to be settled by the whims of browsers. But now those specs are being used to connect systems instead of browsers to servers. I’m not ready to look a trading partner in the eye and say “hmm, let’s see how Opera handles an HTTP 415”. I’ll instead go hide in a closet and do just that.

So, even with the flawed cast of characters you see a lot of whining about – HTTP, URI, XML, and even (gasp!) XML Schema – the pieces are there to build good systems that also make great constituents in anyone’s SOA. The specs, with one glaring exception, are easy to digest. My advice is to start by thinking about a URI strategy that people can follow intuitively. Don’t underestimate the value of a good URI set or the design skills it takes to build. I would also ignore worries about pissing off the Old-Hands of the Web by somehow not using the Web exactly as intended. Their crystal balls weren’t clearer than anyone else’s and these guys do put pants on the usual way (not that I’ve personally witnessed it).

And on Harry’s point that REST and CRUD, I agree that Tim was simply advising people not to limit their comprehension of REST around entities accessed via GET and PUT. REST resources are like views that may or may not be underpinned by an entity model. REST state transitions use URIs to label and invoke services which may or may not use an entity/CRUD programming model under the hood. I agree with Tim in that the presence of those URIs as links is what actually defines REST – NOT simply that you’ve labeled data with URIs.

REST Protocols are the Service Layer

Business operations look a lot like protocol state machines. One of the areas I’ve been looking at (for several years) is how to better leverage state machine thinking into data-driven applications design. I read the REST dissertation, but didn’t see how Roy Fielding’s notion of state transitions applied to my projects. A couple of years ago, David and I looked at how server-side state transitions could drive a service layer (and more). We talked about writing up a paper, but I flaked out so David went ahead with a post.

So, I read Tim Ewald’s posts about REST (first post is here) with a lot of interest. Tim’s description of REST is centered on client protocol state transitions and, critically, ensuring responses to GET requests include the URIs (links) that transition between client states. Up until now, I ignored the idea of URIs for client states that are not mapped to system state transitions on the server. I also questioned the wisdom of including links at all in REST-style responses. Here’s why.

A classic blunder in applications development is designing the programming model (and it's API) around a specific client interaction style. An acid test for an API is how well it works for different calling scenarios, especially those unknown at design time. On that second point, only time can tell you if you have a winner. SOA itself was invented largely to counter unsuitable APIs (and fill space on collateral). Service layers usefully loosen the coupling (you don’t need my binaries to call me) and serve as adapters between an application and the potentially numerous and radically different callers.

So, isn’t it a slippery slope to presumptuously inject URIs into messages for operations that might not apply to the client’s intention? In Tim’s example, the server returns a list of itineraries along with URIs that transition the client to one of 2 potential next client states: getDetails or Reserve. One of these URIs changes system state and the other doesn’t. Rightly, you use GET on one of the URIs and POST for the other (although the example doesn’t indicate which is which). The “getDetails” tag bothered me because in a non-trivial case there might be numerous operations available relative to the resource. Or, the resource might be a summarization (like a report). If the data has a thousand customers IDs and there are 10 or more links possible per customer – you get the point. Why offer up a bunch of links that the client won’t use and potentially outweighs the actual data? Is it even possible to determine the complete set of links?

What I failed to remember, until now, is that data and protocols are independent. There can be multiple protocols with the same data as the starting point – but not the same URI. The URI identifies what data the caller wants and under which protocol.

Data + protocol = resource -> URI

The protocol embodies the caller’s intentions and the server’s constraints. Protocols are easy to secure (URI = security descriptor) and they work like interfaces for data hiding. My mistake until now was in believing that:

• Each entity (I work on ERP apps and we still think about entities) is a resource that has a protocol defined by a system state sequence.
  
• The only useful links in resources are URIs for posting value changes or transitioning states
  
• Views of data representing many resources have no links because it’s difficult to generate a complete set and the caller’s intentions are unknown.

After working through Tim’s description, I’ve reached different conclusions about how to design a REST-style system:

• I still like defining a state sequence and change constraints for entities in the application. It’s a good programming model for ERP because the server-side state transitions are handy places to hang workflow logic. Also, entity/life-cycle thinking aligns well with the business operations.
  
• Systems have protocols that describe data interactions with reasonably specific purposes in mind. The views passed in and out of the system are resources.
  
• A URI identifies a resource, a protocol, and where in that protocol the caller sits.
  
• Resources contain links to other resources or invocation points specific to the protocol. This solves my issue with (potentially) large numbers of unhelpful links.
  
• Protocols buffer the application from conversations – protocols are the new “service layer”.
  
• Getting back to the first point, a “default” protocol can be derived from the server-side state sequence for an entity.

Protocols can carry links to places other than the initial server and resources can be more than just XML representations of entity data – think of the mash-up potential. The URI – via the protocol – tells the system what data to get, what format to use, what stylesheet to apply, and which links are useful to the caller. Adding new protocols to a system needs to be an easy thing to do.

I thought I knew REST and had taken a decent crack at applying it to ERP applications. This re-think has me wondering what else I haven’t figured out yet. In the meantime, I’m looking back at our web services, metadata, and customer-driven scenarios to see what features protocol definitions might need. The goal is to find a way to make it easy for our developers and our customers to create protocols. I’ll write up what I find.

WS-I and Transparency: WTF?

Over 300 people from 100 companies came to San Francisco in early April 2002 for the first ever WS-I community meeting. I was there. Like most of the attendees, I had never worked on a standards effort of any kind. But the "founding board" members of the WS-I – IBM, Microsoft, Oracle, SAP, HP, Intel, etc. – had each reached out to their customer bases to drum up participation. The carrot hung out the window for us smaller companies to sniff was that we could hang out with the big guys to work on web services standards and, importantly, we could get a WS-I conformance logo to grace our product.

It's exactly five years later and I'm in Delray Beach Florida for the WS-I Spring 2007 Community Meeting. The event's web page (for members) list says I'll be one of 32 people attending. Only 18 companies are being represented. The WS-I Board is made up of 11 companies, who, one would think, are obligated to participate. So the "community" beyond the leadership team will be me plus 6 other companies. Over 70 other companies pay dues to keep their up their membership, but I'll wager that many (like my company) have fallen out of good standing.

WHAT THE HELL HAPPENED?

Marketing – There are test programs that determine whether an application conforms to a WS-I profile. You don't need to belong to the WS-I to run them – or even claim you've run them successfully. You can just say your app conforms to a WS-I profile. It's up to someone else to call your bluff. With membership optional and conformance claims free for the taking, it's no wonder the membership is practically extinct. My company's dues are paid out the marketing budget. Hmmm.

Resources – A tenet behind the WS-I charter says that, for any profile, a set of sample applications and a test suite identifying test assertions will be developed on multiple platforms. The idea was to show best practices and demonstrate the potential interoperability benefits behind the effort. But developing this work is enormously expensive. Microsoft, IBM, SAP, Fujitsu, and others have deep pockets. But over time the contributions have declined to the point where the organization can't "afford" to charter new profile work. But there is a bigger challenge to the long-term solvency of the WS-I…

Transparency – This is the big one. To be fairer than necessary, the WS-I was conceived at a time when web services hype was still growing and the founding members were deeply suspicious of each other's motives (the players, I think: IBM, Microsoft, Oracle, HP, Intel, SAP, Accenture, BEA, and webMethods). The hype issue has almost disappeared. And the mutual suspicion problem is in a little better shape except for a lasting grudge within the WS-I Board over the fate of an effort 3 years ago to incorporate SOAP with attachments into the WS-I Basic Profile.

But there is a bigger problem. There are 2 WS-I organizations: the working groups and the WS-I Board. Of course, all organizations have boards that are independent and need to be able to deliberate privately. But the WS-I Board – and I mean the institution, not the individuals or their companies – is frankly an anathema in the face of the WS-I's public mission.

WS-I working groups deliberate within narrowly defined charters, but the discussions are open. Members are free to discuss and describe deliberations wherever they like. The only caveat – and it's significant – is that no "intellectual property" developed by working group members can be published as WS-I material without Board approval and (depending on the nature of the work) as membership-wide vote. Most members can live within those constraints. But the Board is another issue.

The WS-I Board was intentionally designed to be a vendor-driven black-box that controls what is advertised (with a bit of mockery) as a membership community. This isn't hyperbole (despite my usual tendencies). Here are the rules that are driving the WS-I straight into the ground:

- ALL WS-I Board proceedings are confidential – even to the membership. There was an effort by IBM a year ago to prove that Board members could discuss what happened in meetings even if the minutes were secret. But that theory was rejected by others in the Board as a reckless interpretation of the bylaws.

- Virtually all proposed Board actions – like approving a document for publication or launching profile work – requires 9 of the 11 Board members to vote "yes". Abstentions count as "no" votes (a rule which is potentially being addressed)

- Of the 11 Board positions, 9 are allocated on a permanent basis to "founding members". The other 2 are elected. Of those 2, one is effectively (and justifiably) locked up by Sun Microsystems, who was obviously blackballed from participating when the WS-I was formed. But the point is that there is no real opportunity for WS-I members to become part of the leadership.

- The Board determines what work is in scope for the WS-I, but has been utterly unable to declare the criteria behind making this determination. Part of the reason is that the Board is trapped by a wacky bylaw treatise inured with a declaration of unambiguous consensus. In other words, the board can't give its own membership, even internally, any feedback without 9 of 11 votes affirming both the act and the message.

How in the hell can members have any involvement in determining future profiles and other work for the WS-I when (a) there are precious (and highly subjective) rules about what work is in scope and (b) Board members can't agree about what, the bylaws allow them to say to their own membership? The WS-I is the epitome of dysfunction and the reason is mainly the misguided invention of that whacked-out cocoon in which the Board comfortably resides.

So, I'm participating in a Working Group chartered to come up with requirements for future WS-I profile work. To be fair, I've been away from the WS-I for a year, so I feel bad about making waves on the first day back. But I know the workings having chaired their XML Schema Planning WG in 2004-2005. Anyway, the Requirements WG is only allowed to submit draft Working Group charters for WS-I Board approval. Astonishingly, our Working Group can't even give feedback to OASIS or ask the W3C a question without a 60-day approval processes and WS-I Board vote. We can't contact the organizations that own the material the WS-I is considering for potential profiling efforts

I'm going to propose that the WS-I Requirements working group take several actions: (1) Questions and feedback to other organizations are deemed "non-material work", which means the whole WS-I membership does not need to vote before publication. (2) That approval to make materials developed by working groups or committees public requires a simple majority (I can be talked into a 2/3 margin) to be approved for publication. (3) That votes by the WS-I Board on work submitted by a working group or committee are made public to the membership (and by extension – the world). Of course the Board and the bylaws will have to yield and be amended, respectively.

Without these reforms (and others), the WS-I will continue its utter collapse into a small club of vendors seeking to satisfy a few constituents that, for now, are propping up the justification behind the entire organization. The rest of the world (spot the pun) will overtake the WS-I like callous covers a splinter on your heel. And while I strongly think there are much better programming models on the web for many situations, I also think there is a large place in the economy for WS-*. Large organizations will need the mechanisms behind WS-* as provable standards for to bind their touch points. And large IT companies will wave WS-I profiles, and the like, as bespoke competencies fulfill them.

Maybe the WS-I should be disbanded for the shear arrogance of its founders who squander the enthusiasm and contributions of the membership. But starting over also sucks.

SOA vs. Service-Oriented Applications

[Updated 16-Oct to fix a typo]

Service-Oriented Architecture (SOA-arch) and Service-Oriented Applications (SOA-app) are different animals with a common name. I think this is a major point of confusion that hasn't yet been widely recognized. Just like the fact that three points of view exist about what "workflow" means. Here is the difference and I've also included advice for building a service-oriented application.

Service-Oriented Architecture

Here is a quick SOA-arch diagram I built after looking at various resources on the Web:

Service-Oriented Architecture Layers

Those like me who work on commercial ERP applications find this a little humiliating. All that work to build an enterprise application – and we wind up at the bottom of the SOA stack? We clearly have thin skins. Moreover, this implies that one of the chief reasons that SOA-arch exists is because apps can't integrate with each other and we have to fool them into working together. SOA-arch also exists to crack apart applications and cobble them back together to match a business process not envisioned by the application's authors. It's an expensive and invasive process.

ESB providers will eternally demand credit for inventing SOA-arch. But I think we should think about the "day after" effect of web services. People began putting SOAP layers around their existing applications or building new applications from scratch on the WS-* stack. But getting some type interoperability only solves one of the many problems around integrating complex apps into a specific business. There was nothing SOAP could do to match an SAP-formatted employee record to an HR-XML version. SOAP can't make my application email the HR department if I browse http://www.monster.com from my desk. SOA-arch puts those things together. SOA-app gives enterprises a way to control and access data maintained by a given application.

Service-Oriented Applications

SOA-app is about designing applications aggressively to be good citizens in someone's SOA-arch implementation. Service-oriented applications have a basic stack diagram as well, I mentioned some months ago. I won't repeat that explanation, but here is a more colorful picture:

Service-Oriented Application Stack

SOA-app revolves around designing elements of the enterprise application troika – message, data, and code – specifically to make it much easier to incorporate the resulting application into your SOA-arch and also promote a dynamic system that tolerates change more effectively. So, my app might be at the bottom of your SOA-arch stack, but I'll make sure my app is your best friend rather than some opaque, inflexible, monster. SOA-app means changing some classic approaches and, in some cases, fighting the dev tools to make that happen. Here are some key ideas behind SOA-app:

• Make sure callers can mix business intent into a single unit of work. Suppose you allow callers to update customers or update suppliers. Make sure you also allow callers to update a customer and a supplier in a single call. Each business function is represented by a distinct message part format, so your posting function should be able to take an arbitrary list of message parts as input and return a corresponding set of message parts containing results (if any). The reason for this is that SOA-arch may have an entity data aggregation function that updates multiple entities at once. Wouldn't it be nice if your application could accept composite documents with less initial shredding?
  
• Make sure your messages tolerate unknown or incomplete content. If a caller adds extra data that some other part of the SOA-arch needs, just ignore it. For XML, many type serializers do this pretty well. But for binary serializers, this is practically impossible. On the flip side, make sure your messages make "everything optional". There are reasonable limits here. For example, when a caller wants to create a new customer but doesn't include a name or address. Again the reason is to make life easier for callers syncing fractions of records.
  
• Use an inversion of control framework that lets your implementers re-sequence or replace logic declaratively. Windows Workflow Foundation is pretty well designed for just this kind of role. Going this route obviously has a big impact on the overall programming model of the system. For one thing, routines that update data should basically not ever return anything directly to the caller because makes the coupling too tight between components. The scenario to keep in mind is when an implementer needs to suspend a function and execute an approval routing or clear some business semaphore (like a credit hold).
  
• Give callers reasonable access to the query processor. One of the problems with most interfaces – no matter the language or platform – is that callers often can't get the exact data they need unless some designer first thought to add the right function. One of the biggest requests I see is for a correlated sub-query that our product managers didn't think of. I'm not saying that you should allow callers to send SQL – or even fragments of SQL. But you should implement some mechanism that lets callers retrieve data by submitting a query tree that can be vetted by your framework and that you can secure.
  
• Stay very document-oriented. This is easier in business applications because the data lines up well with real-world business artifacts (orders, shipments, etc.). Many application designers over-design the interface library by building a matrix of functions required by caller roles. The problem is that the matrix won't have an entry for many interactions required in the SOA-arch. Come up with a CRUDx standard for your entities and use that to determine message formats (ideally, via a transform). The "x" is whatever critical behavior a particular entity may possess. Keep the number of members of "x" as small as possible. For ERP, we've been able to do it with a single "t" – for "transition" which means to move a record from one transactional state to another (order:new -> order:cancelled).
  

It's troubling that for the past year I've talked to many people about SOA without anyone realizing we were somewhat cross-purpose. I've wanted to talk about the above points for some time, but when I would go over them with other solutions architects, it seemed like we weren't on the same page. Hopefully distinguishing between SOA-arch and SOA-app will be useful going forward.

Diagnosis: XML Fetish

The apparent medical experts who wrote According to Microsoft Windows Communication Foundation – Hands-On, have – in Chapter 3 – diagnosed me as having an “XML Fetish”:

Yet practitioners of contract-first development, working in the XML Schema language in an XML editor, tend to become distracted from those core concerns and start to worry about exactly how the data is to be represented in XML. Consequently, they begin to debate, among other things, the virtues of various ways of encoding XML, and become highly suspicious of anything that might inhibit them from seeing and fiddling with XML. The XML becomes a fetish, falsely imbued with the true virtues of contract-first development, and, as Sigmund Freud wrote, "[s]uch substitutes are with some justice likened to the fetishes in which savages believe that their gods are embodied" (1977, 66).

Wow – I’m practically paranoid! I think I understand the points the authors were trying to get across: Let the XML serializers do their job and stay out of the way. The preaching and the lame attempt to bring Freud into the discussion really put me off.

To (badly) paraphrase Norma Desmond, XML is big. It’s the toolkits that got small. Experience tells me that you actually have to keep a sharp eye on the WSDL and XML emitted from just about any toolkit. The W3C has a working group basically dedicated around that fact.

Let’s say I have a service that accepts purchase order lines. Purchasers can buy things that they receive and stock (items) or services that will be rendered. Items have a code and a quantity; services have a free-form description and an estimated cost. How do I create a service description for an operation that gives the caller that kind of choice in what gets sent?

In XML Schema, I can simply say “xs:choice”.

In C# (or most languages), I can implement this choice using a number of tactics. The most obvious way to go is polymorphism: create a base class POLine and two descendent classes StockedItem and Service. The function prototype then takes the base class as an argument. Hmm – try that once and see what kind of interoperable service description you get. My bet is something like xs:any.

To be fair, .NET has added a ton of attribution to the XML Serialization mechanism. You can implement a choice construct that works pretty well. But the attributes you need to faithfully build such a C# types are very specific to .NET and probably even less intuitive than XML Schema (never thought I would say that). Further, you have no guarantee that Microsoft (or any toolkit vendor) won’t change the XML Schema constructs mapped to a class construct when new framework versions shipped. We had this happen between .NET 1.1 and .NET 2.0.

I tend to build data contracts using XML Schema first. Then I roll the schemas through the various class generators to see how good the output is. Then I build a service using the generated classes to see how good the WSDL will be. Then I build yet another set of types using the generated WSDLs to simulate what service consumers will experience. Finally and if all looks good, I throw all of it away except for the original schema, which gets incorporated into the service description directly and, sometimes, into the service code base via XML Schema validation. I don’t like building static types for message payloads or data domain members. But that’s a whole other topic!

ERP Data as Hypermedia

I work on ERP applications for a living, so I tend to see lots of ways business intent is manifested in an API. For the past few years, I’ve been trying out different architectural ideas to merge the success of the Web to enterprise application API development. The two aspects that have garnered the most payback have been (a) casting the data domain as a hypermedia set and (b) rationalizing constraints around GET, PUT, and POST to convey business intent and manage work. This blog entry is about (a). Making the data domain flexibly addressable is critical to workflow and EAI, which in turn makes it critical to SOA -- no matter how REST or WS-* might be utilized.

Most reasonably normalized databases already have clear semantics for referential data and child data. Reference columns link tables together by record ID. Child tables inherit the ID of their parent plus their own additional anonymous key value. The referential aspects form hypermedia-like links and the parent-child aspects provide document scope (not to mention the nice fit with XML).

Putting these together means that we can map a record as a URL using the same key values that identify records in a database. We can also extend the URL format across referential values. Here are some examples from an implementation we put out several years ago:

url://MySite/orders/101.xml returns the XML representation of order “101” in all its glory (details and all)

url://MySite/orders/101/1.xml returns just the first line item of order “101”.

url://MySite/orders/101/CustomerID.xml returns the ID of the customer

url://MySite/orders/101/CustomerID/Customer.xml returns the entire customer record for the customer identified in the “CustomerID” link

The last item is the most significant, because you can traverse the entire data domain – if the linkages exist – and access data many degrees of separation away from your starting point. We’ve been doing this for several years now in Business Process Management (BPM) architectures because the data one needs to make a business policy decision is rarely in the message. In other words, workflow-driven architectures need expression formats that bind the data in question with data already in the system.

For example, an administrator might set a policy to disallow orders for items belonging to a product class that itself is under approval hold by, say, the U.S. FDA.

url://message/orders/lines/items/holdnotices/fda.xml

Or, you don’t want to allow orders for customers on credit hold: url://message/orders/CustomerID/holdnotices/credit.xml

Note the “message” bit rather than “MySite” in the previous examples. The idea is to not only traverse entities in a database – it’s also to traverse from the message payload to the database in one expression. Non-programmers can actually read this and figure out what it’s doing. That capability is important in BPM because line managers control the policies in force. Line managers understand hierarchies like directory structures. The key is to keep the URN constructs uncluttered and simple.

That seems straightforward, but you have to work out how to pipeline referential constructs through the entity chain for multiple situations: inner-join, outer-join, existential, etc. You also have to work out how to constrain certain kinds of URLs that bring back too much data. Our initial work let you do something like this: url://MySite. It spit out every record in the database.

BTW, you can swap “.xml” for “.xsd” and get a schema for the results. You can also use “.htm” to get an HTML rendering. In fact, you can go to town with MIME types and style sheets to provide lots of viewable interpretations of whatever data is being sought. That’s pretty powerful.

Are GET and PUT Symmetrical in REST?

Does REST require symmetry? In other words, must the payload schema for messages I GET be the same as the schema for messages I PUT? I hope not, because (in my world) what I PUT is rarely a new, complete authoritative version of the data. In the work I've done, the PUT (or POST) payload explicitly indicates whether the item is new (in which case it is authoritative) or if I am changing an item that already exists.

The REST idea – in my mind – is that (a) you can address instances of data unambiguously and completely and (b) that what you GET and what you PUT (for a given URL) are semantically related -- but not necessarily exact copies.

Stephan's List: REST vs. SOAP

Stefan’s List is an interesting effort to categorize some very good bloggers into REST, SOAP, and (um) “either” camps. I know a few people on both sides of the isle and I don’t think any of them are on some sort of web-oriented death march. I mainly think about how business intent is most effectively expressed in message payloads – something neither REST nor WS-* are really helping with (it’s semantics after all).

My friend Paul Downey (http://blog.whatfettle.com/) of British Telecom (erroneously placed in the “wrong” camp in an earlier version of Stefan’s List) put together a brilliant slide deck (pdf) that unambiguously (and literally) illustrates his point of view about WS-*. I’ve been meaning to steal from it for some time. BTW, Paul is chairing a W3C working group to advise toolkits about mapping language constructs to XML Schema constructs – something that will help both sides of the WS isle. I think Paul’s working group will enable more XML Schema constructs to be supported in the mainstream, which in turn increases the vocabulary I can employ to represent my business intent.

REST Doesn’t Obviate WS-*
There is a VERY large IT constituency that relies heavily on modeling non-functional requirements. Pioneers have the luxury of trying these things out and mitigating the shortcomings as they go. Obviously, current enterprises want to extend their architectural core concepts into more open realms. But they need to prove that service-level requirements are achievable and that non-functional aspects can be modeled using approved best practices.

WS-* has grown around those needs. The specs provide CYA support for the big IT shops. I sometimes get upset that WS-* is no longer simple or particularly exciting. But people are often compelled into multilateral infrastructures and to make their complex interactions interoperable. There’s apparently a lot of money in achieving that.

PS. For the record, I use REST and tunnel it through SOAP and/or WS-* where needed.

2.0 is to SOA what SOA is to ESB

About a year ago, I put out a blog post describing ambitions for SOA as an architectural basis within a single application domain. It was about relating messages, code, and data bi-directionally using transforms. Formal descriptions for those transforms could then become a new order of patterns in software architecture. The goal wasn’t to solve the gap between describing requirements and cutting code. It was to link the overall solution concerns more intelligently and pave the way for more declarative aspects and dynamic features.

The mistake I made was in trying to put a moniker around the concept – SOA 2.0! At the time, the ESB community was pushing really hard to say ESB==SOA. I wanted to distinguish architecting integration solutions – the bread and butter of an ESB – from architecting the traditionally opaque integral applications within an enterprise.

But calling it SOA 2.0 was clearly a bad choice. It’s sort of nice to be near the top of a search result (Google: “SOA 2.0”). Maybe your experience will vary, but I cringe when I see what company I have on the same page (but I did beat Oracle by a few months!). The post has been linked by others (Hinchcliffe, Little) but not exactly the way I hoped. A rookie blogging mistake if there ever was one.

Workflow Flavors

“Workflow” came up during a panel discussion I was in at Microsoft’s Tech-Ed conference this year. The word itself – like many IT terms these days – is overloaded to the point where it is hard to distinguish what a given “workflow toolkit” is meant to achieve. When explaining workflow concepts, I’ve started by making sure the audience understands the differences between 3 major categories of workflow:

• Human workflow is where information is conveyed to real people for action or simple notification. Workflow systems that present documents to employees have been around for years – so human workflow wins the right to actually use the word “workflow”. The key issue for human workflow is to have a system where non-technical users can actually program the routing.

• Orchestration is collaboration between application domains (and, by extension, between enterprises). BizTalk, Sonic, and ESB’s are all orchestration tools even though their target markets and general features may differ widely.

• Service Agents route execution of logic within a specific application domain. This is the sweet spot for Workflow Foundation. Service agents are under-served by toolkit vendors, which is surprising given the demands for content-based logic routing in the SOA world.

The trick for WF is to prove it can actually become the primary message pump for an enterprise application. But what drives me crazy is when people think of WF as some sort of “BizTalk Light”, which does both products a disservice.

PS. I haven’t blogged in quite a few months for no reason other than not being sure what to go into. So I thought a gentle entry like this might get the wheels moving forward.

The W3C Schema Patterns WG is not Misguided

Dare Obasanjo’s recent post labeling the W3C W3C XML Schema Patterns for Databinding Working Group as misguided seems like an overreaction. The toolkit vendors put out bad XML Schema processors and invented those leaky abstractions in the (misguided?) rush to make XML painless and web services a transparent feature for programmers with typical skillsets and approaches.

Many (including me) think that moving to XML as a primary integration mechanism for applications should a great step forward. But many IT staff who actually have to link different apps together are complaining that their job is much harder now than it ever was. Industry consortia are having a hell of a time publishing good standardized schemas because the inclusion of some seemingly innocuous XML Schema features will unknowingly break constituent implementations.

You can’t tell developers to simply avoid statically typed languages (at least not yet). You also can’t tell developers to wait a bit longer and the toolkit vendors will somehow watertighten their abstractions in an interoperable way. You *can* tell developers to avoid leaky abstractions to process XML, but you get resistance (which is regrettable). Worse, developers have to sometimes fight their toolkits to even do get to the message payload.

So, I don’t know what is so wrong about the W3C trying to alleviate the situation by attempting to shine a light on issues that seriously impact users. It may perpetuate XML <-> OO binding, which many people — including me — think is a problematic strategy. But maybe the W3C can get the Infoset in more hands sooner. More people can walk before they run, if you will.