ERP Data as Hypermedia

I work on ERP applications for a living, so I tend to see lots of ways business intent is manifested in an API. For the past few years, I’ve been trying out different architectural ideas to merge the success of the Web to enterprise application API development. The two aspects that have garnered the most payback have been (a) casting the data domain as a hypermedia set and (b) rationalizing constraints around GET, PUT, and POST to convey business intent and manage work. This blog entry is about (a). Making the data domain flexibly addressable is critical to workflow and EAI, which in turn makes it critical to SOA -- no matter how REST or WS-* might be utilized.

Most reasonably normalized databases already have clear semantics for referential data and child data. Reference columns link tables together by record ID. Child tables inherit the ID of their parent plus their own additional anonymous key value. The referential aspects form hypermedia-like links and the parent-child aspects provide document scope (not to mention the nice fit with XML).

Putting these together means that we can map a record as a URL using the same key values that identify records in a database. We can also extend the URL format across referential values. Here are some examples from an implementation we put out several years ago:

url://MySite/orders/101.xml returns the XML representation of order “101” in all its glory (details and all)

url://MySite/orders/101/1.xml returns just the first line item of order “101”.

url://MySite/orders/101/CustomerID.xml returns the ID of the customer

url://MySite/orders/101/CustomerID/Customer.xml returns the entire customer record for the customer identified in the “CustomerID” link

The last item is the most significant, because you can traverse the entire data domain – if the linkages exist – and access data many degrees of separation away from your starting point. We’ve been doing this for several years now in Business Process Management (BPM) architectures because the data one needs to make a business policy decision is rarely in the message. In other words, workflow-driven architectures need expression formats that bind the data in question with data already in the system.

For example, an administrator might set a policy to disallow orders for items belonging to a product class that itself is under approval hold by, say, the U.S. FDA.

url://message/orders/lines/items/holdnotices/fda.xml

Or, you don’t want to allow orders for customers on credit hold: url://message/orders/CustomerID/holdnotices/credit.xml

Note the “message” bit rather than “MySite” in the previous examples. The idea is to not only traverse entities in a database – it’s also to traverse from the message payload to the database in one expression. Non-programmers can actually read this and figure out what it’s doing. That capability is important in BPM because line managers control the policies in force. Line managers understand hierarchies like directory structures. The key is to keep the URN constructs uncluttered and simple.

That seems straightforward, but you have to work out how to pipeline referential constructs through the entity chain for multiple situations: inner-join, outer-join, existential, etc. You also have to work out how to constrain certain kinds of URLs that bring back too much data. Our initial work let you do something like this: url://MySite. It spit out every record in the database.

BTW, you can swap “.xml” for “.xsd” and get a schema for the results. You can also use “.htm” to get an HTML rendering. In fact, you can go to town with MIME types and style sheets to provide lots of viewable interpretations of whatever data is being sought. That’s pretty powerful.

Are GET and PUT Symmetrical in REST?

Does REST require symmetry? In other words, must the payload schema for messages I GET be the same as the schema for messages I PUT? I hope not, because (in my world) what I PUT is rarely a new, complete authoritative version of the data. In the work I've done, the PUT (or POST) payload explicitly indicates whether the item is new (in which case it is authoritative) or if I am changing an item that already exists.

The REST idea – in my mind – is that (a) you can address instances of data unambiguously and completely and (b) that what you GET and what you PUT (for a given URL) are semantically related -- but not necessarily exact copies.

Stephan's List: REST vs. SOAP

Stefan’s List is an interesting effort to categorize some very good bloggers into REST, SOAP, and (um) “either” camps. I know a few people on both sides of the isle and I don’t think any of them are on some sort of web-oriented death march. I mainly think about how business intent is most effectively expressed in message payloads – something neither REST nor WS-* are really helping with (it’s semantics after all).

My friend Paul Downey (http://blog.whatfettle.com/) of British Telecom (erroneously placed in the “wrong” camp in an earlier version of Stefan’s List) put together a brilliant slide deck (pdf) that unambiguously (and literally) illustrates his point of view about WS-*. I’ve been meaning to steal from it for some time. BTW, Paul is chairing a W3C working group to advise toolkits about mapping language constructs to XML Schema constructs – something that will help both sides of the WS isle. I think Paul’s working group will enable more XML Schema constructs to be supported in the mainstream, which in turn increases the vocabulary I can employ to represent my business intent.

REST Doesn’t Obviate WS-*
There is a VERY large IT constituency that relies heavily on modeling non-functional requirements. Pioneers have the luxury of trying these things out and mitigating the shortcomings as they go. Obviously, current enterprises want to extend their architectural core concepts into more open realms. But they need to prove that service-level requirements are achievable and that non-functional aspects can be modeled using approved best practices.

WS-* has grown around those needs. The specs provide CYA support for the big IT shops. I sometimes get upset that WS-* is no longer simple or particularly exciting. But people are often compelled into multilateral infrastructures and to make their complex interactions interoperable. There’s apparently a lot of money in achieving that.

PS. For the record, I use REST and tunnel it through SOAP and/or WS-* where needed.

2.0 is to SOA what SOA is to ESB

About a year ago, I put out a blog post describing ambitions for SOA as an architectural basis within a single application domain. It was about relating messages, code, and data bi-directionally using transforms. Formal descriptions for those transforms could then become a new order of patterns in software architecture. The goal wasn’t to solve the gap between describing requirements and cutting code. It was to link the overall solution concerns more intelligently and pave the way for more declarative aspects and dynamic features.

The mistake I made was in trying to put a moniker around the concept – SOA 2.0! At the time, the ESB community was pushing really hard to say ESB==SOA. I wanted to distinguish architecting integration solutions – the bread and butter of an ESB – from architecting the traditionally opaque integral applications within an enterprise.

But calling it SOA 2.0 was clearly a bad choice. It’s sort of nice to be near the top of a search result (Google: “SOA 2.0”). Maybe your experience will vary, but I cringe when I see what company I have on the same page (but I did beat Oracle by a few months!). The post has been linked by others (Hinchcliffe, Little) but not exactly the way I hoped. A rookie blogging mistake if there ever was one.

Workflow Flavors

“Workflow” came up during a panel discussion I was in at Microsoft’s Tech-Ed conference this year. The word itself – like many IT terms these days – is overloaded to the point where it is hard to distinguish what a given “workflow toolkit” is meant to achieve. When explaining workflow concepts, I’ve started by making sure the audience understands the differences between 3 major categories of workflow:

• Human workflow is where information is conveyed to real people for action or simple notification. Workflow systems that present documents to employees have been around for years – so human workflow wins the right to actually use the word “workflow”. The key issue for human workflow is to have a system where non-technical users can actually program the routing.

• Orchestration is collaboration between application domains (and, by extension, between enterprises). BizTalk, Sonic, and ESB’s are all orchestration tools even though their target markets and general features may differ widely.

• Service Agents route execution of logic within a specific application domain. This is the sweet spot for Workflow Foundation. Service agents are under-served by toolkit vendors, which is surprising given the demands for content-based logic routing in the SOA world.

The trick for WF is to prove it can actually become the primary message pump for an enterprise application. But what drives me crazy is when people think of WF as some sort of “BizTalk Light”, which does both products a disservice.

PS. I haven’t blogged in quite a few months for no reason other than not being sure what to go into. So I thought a gentle entry like this might get the wheels moving forward.

The W3C Schema Patterns WG is not Misguided

Dare Obasanjo’s recent post labeling the W3C W3C XML Schema Patterns for Databinding Working Group as misguided seems like an overreaction. The toolkit vendors put out bad XML Schema processors and invented those leaky abstractions in the (misguided?) rush to make XML painless and web services a transparent feature for programmers with typical skillsets and approaches.

Many (including me) think that moving to XML as a primary integration mechanism for applications should a great step forward. But many IT staff who actually have to link different apps together are complaining that their job is much harder now than it ever was. Industry consortia are having a hell of a time publishing good standardized schemas because the inclusion of some seemingly innocuous XML Schema features will unknowingly break constituent implementations.

You can’t tell developers to simply avoid statically typed languages (at least not yet). You also can’t tell developers to wait a bit longer and the toolkit vendors will somehow watertighten their abstractions in an interoperable way. You *can* tell developers to avoid leaky abstractions to process XML, but you get resistance (which is regrettable). Worse, developers have to sometimes fight their toolkits to even do get to the message payload.

So, I don’t know what is so wrong about the W3C trying to alleviate the situation by attempting to shine a light on issues that seriously impact users. It may perpetuate XML <-> OO binding, which many people — including me — think is a problematic strategy. But maybe the W3C can get the Infoset in more hands sooner. More people can walk before they run, if you will.

UPA is your Friend (Repost)

The W3C XML Schema specification has a rule called unique particle attribution (UPA) that confuses, well, many. And after 4+ years of XML Schema in the wild, it still amazes me how inconsistent toolkits handle the issue. One toolkit – I think XMLSpy – at one time rebelled against UPA by enforcing the rule only if the user wished it. Just this week, I found a discrepancy in handling UPA between SQL 2005 and .NET Framework 2.0 (more on that later).

One problem in trying to comply with the UPA rule is that it is hard to describe in words. Here are a couple of examples to get the point across. This schema type is perfectly fine:

<xs:element name="MyType">

<xs:complexType>

<xs:sequence>

<xs:element name="Foo" />

<xs:element name="Foo" />

</xs:sequence>

</xs:complexType>

</xs:element>

But this type violates UPA:

<xs:element name="MyType">

<xs:complexType>

<xs:sequence>

<xs:element name="Foo" minOccurs="0"/>

<xs:element name="Foo" />

</xs:sequence>

</xs:complexType>

</xs:element>

Why? Suppose a schema processor is checking a document that looks like this:

<MyType>

<Foo />

</MyType>

The processor can’t figure out which element declaration in the schema to match the element “Foo”. It must be able to find the match unambiguously. You might think that it just doesn’t matter: the document still fits the description. But it’s critical for a schema processor, because it must be able to align the current XML document node with exactly one schema declaration. This disconnect between how people feel they should be able to describe a document and how a schema processor really works is what’s really behind the criticism of UPA. But most documents and schemas are WAY more complicated than these examples, and the UPA rule keeps the validation mechanisms manageable. By the way, this change to the schema fixes the UPA problem – can you see why?

<xs:element name="MyType">

<xs:complexType>

<xs:sequence>

<xs:element name="Foo" />

<xs:element name="Foo" minOccurs="0"/>

</xs:sequence>

</xs:complexType>

</xs:element>

When the schema validator hits the first “Foo” element, it knows that “Foo” must have been declared in the first element declaration within “MyType”.

The UPA problem can come up just as result of schema factoring. I came across this situation when looking at a schema for an XML document containing a SQL expression tree:

<xs:element name="Subquery">

<xs:complexType>

<xs:choice>

<xs:sequence>

<xs:group ref="tns:grpColumn" minOccurs="1" maxOccurs="1" />

<xs:group ref="tns:grpQuery"/>

</xs:sequence>

<xs:sequence>

<xs:group ref="tns:grpColumn" minOccurs="1" maxOccurs="1" />

<xs:element name="ListItem" minOccurs="1" maxOccurs="unbounded">

<xs:complexType>

<xs:attributeGroup ref="attGrpScalarValue" />

</xs:complexType>

</xs:element>

</xs:sequence>

</xs:choice>

<xs:attributeGroup ref="tns:attGrpSubqueryItems" />

</xs:complexType>

</xs:element>

This schema type gives a choice of two sequences, both starting with an element group called “grpColumn”. So, when the schema validation processor encounters the group elements in an XML document, it can’t tell which choice is in play. Also, the schema spec designers did not want schema processors to look “past” the current node to try and resolve which schema node matches a document node. By the way, this is the schema construct that .NET Framework 2.0 compiles with no errors, but SQL 2005 (via MSXML 6.0) throws a UPA violation error.

I refactored the schema a little to eliminate the UPA violation:

<xs:element name="Subquery">

<xs:complexType>

<xs:sequence>

<xs:group ref="tns:grpColumn" minOccurs="1" maxOccurs="1" />

<xs:choice>

<xs:group ref="tns:grpQuery"/>

<xs:element name="ListItem" minOccurs="1" maxOccurs="unbounded">

<xs:complexType>

<xs:attributeGroup ref="attGrpScalarValue" />

</xs:complexType>

</xs:element>

</xs:choice>

</xs:sequence>

<xs:attributeGroup ref="tns:attGrpSubqueryItems" />

</xs:complexType>

</xs:element>

So again, you could argue that both schema examples are identical when it comes to describing an XML document. Why should UPA make me use a specific methodology? It’s because the people who created the XML Schema specification were also envisioning how schema validation tools would be created.

In the case above, UPA forced me – IMO – to create a better schema. So aside from just general unfamiliarity with UPA and inconsistent toolkit support, does UPA really do any harm? Schema authors sometimes want to allow users to add extra elements to their XML documents. Since you don’t know what the XML will look like ahead of time, it’s nice to use a wildcard like xs:any in the schema. The problem is that UPA restricts the occurrence of a wildcard element depending on the occurrence of the element previously declared.

For example, this is a legal sequence:

<xs:sequence>

<xs:element name="Foo"/>

<xs:any />

</xs:sequence>

However, you can’t change the occurrence of “Foo” to any value other than “1” (the default) or you violate UPA. UPA does not allow any wildcard to be next to an optional element. Also, you can’t put a wildcard before any element in the same namespace. Some feel this constrains extensibility in some cases.

But honestly, I haven’t found a situation where UPA caused pain that a workaround doesn’t fix. I also think you can extend schemas better by wrapping them rather than extending them. In other words, create a schema that imports the schema you want to extend, add some new elements and tie them to the original schema through key/keyref declarations. It’s really the only way for schema validation to work with content unknown at design time.

Islands of Metadata

Does anyone remember the term “islands of automation”? I’m not sure where it was coined, but it described - in the late '70s - the situation where an enterprise implemented multiple systems with little or no integration. Pretty much every enterprise application has a customer table. Having a CRM system, a logistics system, and, say, a financials system with individual ideas of a customer creates obvious problems.

The “islands of automation” problem was solved in two unsatisfying ways. One way was to install a single big applications suite. The customization costs were big and enterprises were forced to adapt their business to whatever functionality was baked-in the suite. The second way was to buy a set of so-called “best of breed” point solutions. But integrating mission-critical apps can be more expensive than customizing a single app. Neither approach avoided vendor lock-in – whether from an apps vendor or a systems integrator. But I guess that’s sort of the idea, isn’t it.

The metadata-based toolkits under development at Microsoft seem in a similar conundrum. On one side are the individual mechanisms like Indigo, Windows Workflow, DSL, and distributed solutions. Individually, they are interesting toolkits – but they have no awareness of each other. On the other side is Microsoft Business Framework, which uses a single, rather big pile of metadata to describe an application from the database up to the UI. Here we go again: Do I want disconnected best-of-breed metadata, or closed, monolithic metadata.


A Conceptual Leap at PDC 2005

I saw one very promising demonstration at the Microsoft PDC 2005 conference – one that broke a conceptual barrier. Don Box and Dharma Shukla showed Windows Workflow Foundation (WWF) acting as an agent for some Indigo (a post-Whidbey version) services. Indigo-tagged components were dragged into workflow design surface that in turn represented a specific service action. Voila! The service description was inferred from the message “needs” of the application components.

This is significant because loose coupling between application components and the agent layer of the (canonical) SOA stack means that the agent describes the services. Only the agent knows the complete content of the message. Only the agent knows how to distribute message parts to components.

The other thing I liked was that – for once – the metadata behind WWF wasn’t just source code behind a code generator. Microsoft likes to write tools that spit out reams of partial classes like shot from a Holland & Holland (and about as constructively). Maybe Microsoft feels that OO coupling and static types is how us minions want to develop systems. A more cynical explanation is that nothing sells an idea – at least internally – like good support for IntelliSense. But I digress.

The XOML files used in the WWF/Indigo future-ware demo were processed dynamically (or at least portrayed to be - it was a demo). Generated code may be all the rage, but I only like generated code in one usage pattern: transient assemblies. Here was a great example of an agent that was adept at managing work instances while all the while knowing that the processing plan can change at any moment.

I used to think I was old enough to just let Kool-Aid roll down my back. But seeing the WWF and Indigo teams pick up on this concept definitely brought gave me something to smile about.

PDC BOF: Apps Design for SOA

I’m hosting a Birds of a Feather session at PDC Tuesday night 10:15PM. Here is the description:

Some build services to wrap legacy applications or integrate otherwise monolithic systems. Others say building new applications for service-orientation make those solutions agile. What does that mean? How should applications be designed to fit future service-oriented architectures? This is a brainstorm session -- no telling where it will lead.

Apocalyptic, no? I threw this together last Friday afternoon – almost on a whim – because there wasn’t really any other BOF sessions taking on the topic. It was accepted on Monday, which makes me feel bad for these guys, but they’ve got a backup plan.

A Canonical SOA Stack

In my last post I mentioned a Canonical SOA stack. So, I thought it would be good to just press on and propose a description of that animal. I know it’s probably beating a dead horse and many people are working on helpful descriptions of SOA. The recent work is good because I think we've turned a corner from the murkiness of Tech-Ed 2004 (San Diego). I recall lingering around the cabanas where, it seemed, SOA was defined only in terms of what it is not.

So, I'm risking coming off presumptuous in pitching a description of an SOA stack. But it's only a logical model to establish some boundaries between some (I think) obvious and independent concerns. There is quite a lot of discussion yet to come about SOA and this model could make for a handy map.

There are 4 basic layers separating the concerns of the Canonical SOA stack. At the bottom, quite expectedly, are application components which reside in a layer call the application domain. Next up the stack is the agent layer, which I think is at the heart of SOA. The agent layer is accessed by any number of representation channels which are endpoints with specific wire formats, invocation protocols, and service description flavors. At the top of the stack are conversation managers that provide for situations that span multiple invocations of service actions and probably have some state for managing conversation instances (sorry about the image quality - I'll get the hang of it).

I think this description is a simple as possible, although the top and bottom pairs of layers each form their own conspiracies. The bottom two layers compose functional capabilities to create callable service actions. The top two layers rationalize service actions into invocation models – stateless (e.g., SOAP, REST, proprietary mechanisms) and stateful (UI controllers, Orchestration Engines, etc.). This bears repeating: The bottom two layers formulate the solution's functionality by splitting requirements into two special categories. The top two layers provide the loose coupling and transport flexibility.


Application Domains and Agents

One tenet (ugh - the "t" word) of designing applications for an SOA is to segregate capabilities from policies when analyzing functional requirements. This is because we are betting that policies frequently change over time – at least more frequently than our capabilities need to evolve. So we want to isolate the hand-built code to the area of the solution that both seldom changes and tends to implement requirements that are tough to implement declaratively. Conversely, the bits that change most often are the collaborations and transforms that reflect business policies between functional areas.

As an example, not much has changed in the way debts and credits work in quite a while (except possibly at Enron). But the business processes that ultimately drive accounting transactions do change frequently. Agility (again, Enron notwithstanding) is valuable. For a packaged software company, having agile business processes in your products is these days is a competitive requirement.

Confession: I’m old-fashioned and still prefer nouns when describing an application domain.

Suppose a functional requirement dictates that when a sales order is confirmed, the order value must be recalculated, the order tables must updated and the inventory available-to-promise (ATP) balances must be adjusted. The rule requiring that the two nouns – sales order and inventory – collaborate shouldn’t be implemented in the order confirmation code. But you would be surprised how often collaboration is imperatively nested inline with the functional code. This means that policy changes require core code changes. Over time, you wind up with spaghetti.

The idea of factoring functions and collaborations from requirements is a classic concept commonly depicted in UML sequence diagrams. In SOA, the agent layer is responsible for executing collaborations. So collaboration logic needs to be extricated from the functional logic. I’m trying to stay away from specific design choices for now, but it makes sense to me to leverage the heck out of metadata in the agent layer. The more metadata you have and the richer it’s content, the more the agent can do as an engine. That means less hand-rolled code.

The ideal agent needs to be a fast, micro-orchestration engine that can validate messages, create transformations, execute application components, and delineate transactional and asynchronous boundaries. It’s a tall order, but the emergence of a robust agent layer coupled with stringent (yet simpler) coding practices is really the lynchpin of SOA.

Representation Channels and Conversation Managers

A representation channel is a mechanism providing access to service actions. Channels not only specify wire formats, but are also responsible for publishing descriptions of the service action set. Interoperability is achieved (in theory) when channels comply with industry standards like SOAP, SMTP, etc.

Some protocols and wire formats cannot convey the complete set of service actions. So it’s important to know where a service design might have difficulties with some channels in conveying the semantic intent (distinguishing what service you are calling) or the physical data (video streams over SOAP). It's important – and most people have already concluded this – that the presence of any specific communications channel does not an SOA make. You choose your set of data representation channels based on your anticipated caller patterns. Not all channels can handle all service actions.

The bottom three layers of the Canonical SOA Stack revolve around statelessness. If you look at BizTalk orchestrations or user interfaces, there is often a notion of a conversation instance (or session). Also, such facilities at the edges often need to aggregate and shred data for their own specific purposes. It might not be cost-justified to incorporate these requirements all the way down the stack.

The top of the stack – the conversation managers – implements stateful conversations that rely on the stateless model beneath. There is room for debate here because stateful conversation managers can just as easily connect directly to the agent layer. Maybe the top two layers ought to be merged, but for now I’m more comfortable leaving them separate. Let me know your thoughts!

Examples of conversation managers might include BizTalk orchestrations, user interface services, and pub/sub data staging. There are lots of facilities you could consider a conversation manager or a representation channel. Some channels are dedicated to a specific conversation manager. One example would be a publish/subscribe data staging area. Occasionally-connected clients – like Outlook on a notebook computer – also have custom conversation managers to stream just enough data to and from the server.


Summary

Many people think of service-orientation as fractal architecture, meaning that what one SOA domain might consider, an application component is really another SOA domain. This is sort of the Horton Hears a Who philosophy. Sure, inter-SOA-domain calls can happen in at least 3 of the 4 layers (not sure about channels). But this SOA description stays in it's own borders, for now, to keep the tiers clear.

Anyway, I wanted to start grounding the conversation about SOA and thought this would be helpful. At some point, someone is going to ask us to build a proper an SOA. I, for one, would like to be able to open Eclipse or Visual Studio some day and sort of know what to do next. Maybe having some named tiers can help propel design discussions. I think some interesting work is ahead to find metadata pathways from the bottom to the top of the SOA Stack. Such pathways host the transformation patterns I described in the last post.

Patterns for SOA 2.0

[23 June 2007: A hopefully clearer rewrite is here.] My dad is a research biochemist at an institute in Southern California. One of the (literally) cool things about his laboratory is the cold room. Going through the huge vault-like door, there are some tall glass tubes a couple of inches in diameter and several feet high. In the glass columns are various kinds of cloudy goo. Over time, colored bands of (I think) concentrated proteins and other molecular constituents appear. This event tended to cause some cordial celebration to occur, which I never really appreciated. But looking back, these scientists would sit back, wait, and then witness specialized things emerge naturally from a process of self-refinement.

Think about the move over the last 4 decades from monolithic applications to service-orientation. Instead of having gravity and osmosis do the work, we have trial, error, and debate. But software architecture is itself self-refining process nonetheless. By definition an architecture pattern is a reflection of an existing practice. I haven’t become nearly familiar with software architecture patterns enough to where I can recite them at cocktail parties. But it seems like the classic process of describing an architecture in terms of patterns is not helping us pin down the formal properties of SOA. Some argue that this proves SOA isn’t real because it can’t be described.

But maybe we are just looking for patterns in the wrong way. Physical SOA implementations will probably use well-known patterns like observer-subject, content-based routing and pipelines. However, I think that describing a modern SOA requires a deeper analysis centered on transformational patterns rather than classic software architectural or integration patterns.

SOA, well at least a 1.0 version, has been around for quite a while as any systems integrator will tell you for $225 an hour. Enterprise service buses, object brokers, and other agent-oriented have been successfully fooling monolithic applications into working with each other for years. Web services also put agent-oriented systems and services into people’s faces rather suddenly and on a mass scale. But version 1.0 of SOA was geared primarily toward aggregating otherwise inert systems and providing some new communication channels.

But now, many see a need for a more modern SOA, which I’ll call SOA 2.0 – where frameworks, applications, agents and communication channels understand each other more deeply – ideally using more aspect-oriented approaches. In short, the new SOA is about building a smarter stack and designing applications to take advantage of new constructs that (we hope) promote agility and simplicity.

At its core, SOA 2.0 uses graph transformation mathematics to convey semantics throughout the SOA stack’s layers as executable logic. At each layer of the stack, you define some semantic categories (a.k.a. viewpoints or aspects) and develop transformation patterns that produce hard rules that specific SOA layers can execute dynamically. Semantic category assignments are stored in metadata and transforms are implemented as engines or (less appealingly) as code generators.

This is what makes the architecture stack much smarter – semantics of the underlying application requirements are pulled through the solution mechanically. More importantly, as application semantics evolve over time, the solution itself evolves automatically from top to bottom. The idea is to get deployment overhead to approach zero.

Here is an admittedly simplistic example (italics represent some canonical SOA layers): In the data domain you could define a semantic category called “data representation” that describes whether a domain entity is resource data (customers, parts), activity data (orders, timesheets), or reference data (sales analysis), which is read-only. You then define a transform template called CRUD that produces interfaces for invoking functions in the application domain.

The template can be smart enough, for example, to avoid producing operations for reference data entities other than “Retrieve” (the “R” in CRUD). In turn, the agent layer has a transform template to create sets of actions (conceptually like a SOAPAction) from the application domain’s interface set. Service communication channels implement physical endpoints for WS-*, REST, etc. These channels use transforms to produce message processors and service descriptions from the agent action set. Finally, you might have 1 or more conversation managers that manage specialized state and data formatting capabilities for special purposes like driving user interfaces, BizTalk orchestrations, or Office Integration.

So, in each successive layer of a SOA 2.0 stack, new semantic categories and transformation templates are applied that may use artifacts residing in an adjacent layer to affect behavior. This transform pipelining approach creates a turnkey engine to project application capabilities into a sort of super-API. In the real world, however, these transforms are obviously not as simplistic as the CRUD example above. This is why we now need to begin building a library of reusable semantic categories and their corresponding transformation patterns.

Transformation rules – when described in mathematical terms – can be proven complete, unlike many hand-rolled, imperative programming approaches. In fact, this might be an entry requirement for adding candidate patterns to a future library. I think this approach beats the current entry barrier – prove something’s been done at least three times and voila – it’s a pattern!

Seriously, transformations are deterministic and flexible. They can be manifested as independent engines or in static code generation. They can be domain-specific languages (DSLs). Once implemented into an SOA stack, changes to the application domain can affect the entire solution predictably and automatically. Transformation rules can distinguish breaking changes from compatible evolution. So, the agent can, for example, know when to create a new action-set and when to simply alter descriptions of an existing action-set.

So to summarize, a key area in the next evolution of services architecture is about having the architecture stack aware of certain semantics in the application requirements. The stack can adapt itself to changes in the requirements by executing directed transforms against metadata. And now is the time to start identifying patterns of transformations that link layers of the SOA together intelligently. The mathematical world is well aware that graph transformations and category logic both relate well to computer science. Unfortunately, I majored in music. So, I need to get some help from more mathematically-astute people to see if all this actually matches up. Maybe a new color band is emerging from that goo in the glass column.

Dancing DOM to DOM

Several years ago, I was involved in a project for my day job where the client was browser based and the server was called strictly using SOAP. Even though the server was built on .NET, we decided not to create .NET types (classes or datasets) to represent parts of the SOAP messages. Instead, we just defined types using XML Schema and associated those types with message parts using a custom attribute.

The client was the browser (JavaScript) and the server received non-scalar message parts as .NET XmlDocument instances. So, the programming environment on both sides was DOM-oriented. Data access on both ends was done using predominantly XPath, SelectNode(s), and other DOM API calls.

I was later asked how the development team adjusted to a world without strong types. Was there a riot? Actually, there were worries about type safety, conversion between XML simple types and C# intrinsic types (yes, dates had to be closely managed), or the lack of a nice object hierarchy.

There was one complaint, though: no Intellisense.

Is that all we need to really send XML, web services, etc. to some next level of adoption? So, for all of the Visual Studio and Eclipse dudes out there, how hard could it be to attribute some DOM type with a schema QName and then get some IDE help when creating an XPath statement? If it’s doable, let me know!

One other issue was that Intranet single-sign-on is still hard and the solutions are very platform-specific. WS-Security is probably a great next step, but unless it gets implemented in the browsers (part of XMLHTTPRequest?) it will have to come by way of a plug-in.

Some people look at the DOM API and see a big stick of ugly. But the complaining dies down after realizing that XML messages have interesting, interoperable capabilities to compose messages. The skillsets are also reusable (meaning lucrative) in web programming.

XML Schema Profile for Language Types?

I attended the W3C Workshop on XML Schema 1.0 User Experiences (the program and minutes are here). My job was to represent the position of the WS-I, where I chair the XML Schema Work Plan Working Group. It was my first W3C event. It went well and I appreciate the W3C having a public event like this. What I didn’t expect was so much new support for profiling the XML Schema spec.

Just to restate the problem: XML Schema has constructs not easily expressible in most programming languages or databases. But I (and others) don’t see this is as an interoperability problem – it’s a toolkit problem. Despite some shortcomings, XML Schema can unambiguously describe the format of an XML document. So, the idea of creating an XML Schema “profile” to improve web service interoperability doesn’t add up to me.

Microsoft proposed that toolkit vendors should fully support the XML Schema spec (whew!), but that a number of “domain-specific” profiles would be “useful”. SAP said that XML Schema profiles are “inevitable and useful for language bindings, business vocabularies, and for specific user communities”.

So, the W3C might start looking into if/when/how to develop a set of domain-specific profiles. Is this a good idea? Are web services such a domain unto itself? Will it be ironic when schema profiles are authored using xs:redefine?* On the surface, it sounds useful, but let me overreact in my (usual) conflicted manner anyway.

On one hand, it would be nice if the Microsoft, IBM, Sun, and BEA toolkits could accurately reconstitute types from each other’s generated schema documents. And if you ask them, this is what they say their customers want.

But what if the WS-I gets pressure to add XML Schema profile conformance to the Basic Profile? It’s easy to see how this would happen. There are WS-I members who insist problems using schema in programming languages is a huge problem in developing web services. Also, the WS-I Board is dominated by toolkit vendors who have permanent seats and would like some sort of collective win to come of all this.

This could cause already tenuously adopted standards to diverge altogether. Many will not like a standards organization (whichever one) dictating a programming model, which in effect such a profile would do. And we all know there are already major, public web service implementers who more or less avoid the WS-* world altogether.

I am hopeful that RPC semantics will fade significantly from web services development by the time a language binding profile for XML Schema makes it to toolkits. Loose coupling means much more than “you don’t need my binaries to call my code”. It’s about building services that can evolve robustly and safely. One day, toolkit vendors will see that binding messages (or their parts) to static types – while seemingly the Holy Grail today – actually gets in the way of loose coupling and dynamic systems.

Ironically, XML is probably more natural than an object graph in representing a lot of real-world data. In fact, schema-relational mapping is actually easier to accomplish than object-relational mapping. Isn't it strange that people complain about the overhead of XML Schema validation, but have no problem with having their messages shredded value-by-value into a statically-typed object graph which is then re-shredded into a bunch of SQL? Why so much focus on the lowest-value part of the process?

Improving toolkits in order to make XML processing more amenable to today’s object-oriented approaches is NOT** the way to achieve better service-oriented solutions. Concocting a subset of the XML Schema specification just to provide more convenient programming experiences is a capitulation in the name of the wrong goal.

* This bit of attempted humor is homage to my friend David Ing, who also has seemingly proven that footnotes are indispensable in blogs.

** An earlier version of this post was missing this rather important word!

Test Post

This is a test post. However, feel free to comment.