The W3C Schema Patterns WG is not Misguided

Dare Obasanjo’s recent post labeling the W3C W3C XML Schema Patterns for Databinding Working Group as misguided seems like an overreaction. The toolkit vendors put out bad XML Schema processors and invented those leaky abstractions in the (misguided?) rush to make XML painless and web services a transparent feature for programmers with typical skillsets and approaches.

Many (including me) think that moving to XML as a primary integration mechanism for applications should a great step forward. But many IT staff who actually have to link different apps together are complaining that their job is much harder now than it ever was. Industry consortia are having a hell of a time publishing good standardized schemas because the inclusion of some seemingly innocuous XML Schema features will unknowingly break constituent implementations.

You can’t tell developers to simply avoid statically typed languages (at least not yet). You also can’t tell developers to wait a bit longer and the toolkit vendors will somehow watertighten their abstractions in an interoperable way. You *can* tell developers to avoid leaky abstractions to process XML, but you get resistance (which is regrettable). Worse, developers have to sometimes fight their toolkits to even do get to the message payload.

So, I don’t know what is so wrong about the W3C trying to alleviate the situation by attempting to shine a light on issues that seriously impact users. It may perpetuate XML <-> OO binding, which many people — including me — think is a problematic strategy. But maybe the W3C can get the Infoset in more hands sooner. More people can walk before they run, if you will.

UPA is your Friend (Repost)

The W3C XML Schema specification has a rule called unique particle attribution (UPA) that confuses, well, many. And after 4+ years of XML Schema in the wild, it still amazes me how inconsistent toolkits handle the issue. One toolkit – I think XMLSpy – at one time rebelled against UPA by enforcing the rule only if the user wished it. Just this week, I found a discrepancy in handling UPA between SQL 2005 and .NET Framework 2.0 (more on that later).

One problem in trying to comply with the UPA rule is that it is hard to describe in words. Here are a couple of examples to get the point across. This schema type is perfectly fine:

<xs:element name="MyType">

<xs:complexType>

<xs:sequence>

<xs:element name="Foo" />

<xs:element name="Foo" />

</xs:sequence>

</xs:complexType>

</xs:element>

But this type violates UPA:

<xs:element name="MyType">

<xs:complexType>

<xs:sequence>

<xs:element name="Foo" minOccurs="0"/>

<xs:element name="Foo" />

</xs:sequence>

</xs:complexType>

</xs:element>

Why? Suppose a schema processor is checking a document that looks like this:

<MyType>

<Foo />

</MyType>

The processor can’t figure out which element declaration in the schema to match the element “Foo”. It must be able to find the match unambiguously. You might think that it just doesn’t matter: the document still fits the description. But it’s critical for a schema processor, because it must be able to align the current XML document node with exactly one schema declaration. This disconnect between how people feel they should be able to describe a document and how a schema processor really works is what’s really behind the criticism of UPA. But most documents and schemas are WAY more complicated than these examples, and the UPA rule keeps the validation mechanisms manageable. By the way, this change to the schema fixes the UPA problem – can you see why?

<xs:element name="MyType">

<xs:complexType>

<xs:sequence>

<xs:element name="Foo" />

<xs:element name="Foo" minOccurs="0"/>

</xs:sequence>

</xs:complexType>

</xs:element>

When the schema validator hits the first “Foo” element, it knows that “Foo” must have been declared in the first element declaration within “MyType”.

The UPA problem can come up just as result of schema factoring. I came across this situation when looking at a schema for an XML document containing a SQL expression tree:

<xs:element name="Subquery">

<xs:complexType>

<xs:choice>

<xs:sequence>

<xs:group ref="tns:grpColumn" minOccurs="1" maxOccurs="1" />

<xs:group ref="tns:grpQuery"/>

</xs:sequence>

<xs:sequence>

<xs:group ref="tns:grpColumn" minOccurs="1" maxOccurs="1" />

<xs:element name="ListItem" minOccurs="1" maxOccurs="unbounded">

<xs:complexType>

<xs:attributeGroup ref="attGrpScalarValue" />

</xs:complexType>

</xs:element>

</xs:sequence>

</xs:choice>

<xs:attributeGroup ref="tns:attGrpSubqueryItems" />

</xs:complexType>

</xs:element>

This schema type gives a choice of two sequences, both starting with an element group called “grpColumn”. So, when the schema validation processor encounters the group elements in an XML document, it can’t tell which choice is in play. Also, the schema spec designers did not want schema processors to look “past” the current node to try and resolve which schema node matches a document node. By the way, this is the schema construct that .NET Framework 2.0 compiles with no errors, but SQL 2005 (via MSXML 6.0) throws a UPA violation error.

I refactored the schema a little to eliminate the UPA violation:

<xs:element name="Subquery">

<xs:complexType>

<xs:sequence>

<xs:group ref="tns:grpColumn" minOccurs="1" maxOccurs="1" />

<xs:choice>

<xs:group ref="tns:grpQuery"/>

<xs:element name="ListItem" minOccurs="1" maxOccurs="unbounded">

<xs:complexType>

<xs:attributeGroup ref="attGrpScalarValue" />

</xs:complexType>

</xs:element>

</xs:choice>

</xs:sequence>

<xs:attributeGroup ref="tns:attGrpSubqueryItems" />

</xs:complexType>

</xs:element>

So again, you could argue that both schema examples are identical when it comes to describing an XML document. Why should UPA make me use a specific methodology? It’s because the people who created the XML Schema specification were also envisioning how schema validation tools would be created.

In the case above, UPA forced me – IMO – to create a better schema. So aside from just general unfamiliarity with UPA and inconsistent toolkit support, does UPA really do any harm? Schema authors sometimes want to allow users to add extra elements to their XML documents. Since you don’t know what the XML will look like ahead of time, it’s nice to use a wildcard like xs:any in the schema. The problem is that UPA restricts the occurrence of a wildcard element depending on the occurrence of the element previously declared.

For example, this is a legal sequence:

<xs:sequence>

<xs:element name="Foo"/>

<xs:any />

</xs:sequence>

However, you can’t change the occurrence of “Foo” to any value other than “1” (the default) or you violate UPA. UPA does not allow any wildcard to be next to an optional element. Also, you can’t put a wildcard before any element in the same namespace. Some feel this constrains extensibility in some cases.

But honestly, I haven’t found a situation where UPA caused pain that a workaround doesn’t fix. I also think you can extend schemas better by wrapping them rather than extending them. In other words, create a schema that imports the schema you want to extend, add some new elements and tie them to the original schema through key/keyref declarations. It’s really the only way for schema validation to work with content unknown at design time.

Islands of Metadata

Does anyone remember the term “islands of automation”? I’m not sure where it was coined, but it described - in the late '70s - the situation where an enterprise implemented multiple systems with little or no integration. Pretty much every enterprise application has a customer table. Having a CRM system, a logistics system, and, say, a financials system with individual ideas of a customer creates obvious problems.

The “islands of automation” problem was solved in two unsatisfying ways. One way was to install a single big applications suite. The customization costs were big and enterprises were forced to adapt their business to whatever functionality was baked-in the suite. The second way was to buy a set of so-called “best of breed” point solutions. But integrating mission-critical apps can be more expensive than customizing a single app. Neither approach avoided vendor lock-in – whether from an apps vendor or a systems integrator. But I guess that’s sort of the idea, isn’t it.

The metadata-based toolkits under development at Microsoft seem in a similar conundrum. On one side are the individual mechanisms like Indigo, Windows Workflow, DSL, and distributed solutions. Individually, they are interesting toolkits – but they have no awareness of each other. On the other side is Microsoft Business Framework, which uses a single, rather big pile of metadata to describe an application from the database up to the UI. Here we go again: Do I want disconnected best-of-breed metadata, or closed, monolithic metadata.


A Conceptual Leap at PDC 2005

I saw one very promising demonstration at the Microsoft PDC 2005 conference – one that broke a conceptual barrier. Don Box and Dharma Shukla showed Windows Workflow Foundation (WWF) acting as an agent for some Indigo (a post-Whidbey version) services. Indigo-tagged components were dragged into workflow design surface that in turn represented a specific service action. Voila! The service description was inferred from the message “needs” of the application components.

This is significant because loose coupling between application components and the agent layer of the (canonical) SOA stack means that the agent describes the services. Only the agent knows the complete content of the message. Only the agent knows how to distribute message parts to components.

The other thing I liked was that – for once – the metadata behind WWF wasn’t just source code behind a code generator. Microsoft likes to write tools that spit out reams of partial classes like shot from a Holland & Holland (and about as constructively). Maybe Microsoft feels that OO coupling and static types is how us minions want to develop systems. A more cynical explanation is that nothing sells an idea – at least internally – like good support for IntelliSense. But I digress.

The XOML files used in the WWF/Indigo future-ware demo were processed dynamically (or at least portrayed to be - it was a demo). Generated code may be all the rage, but I only like generated code in one usage pattern: transient assemblies. Here was a great example of an agent that was adept at managing work instances while all the while knowing that the processing plan can change at any moment.

I used to think I was old enough to just let Kool-Aid roll down my back. But seeing the WWF and Indigo teams pick up on this concept definitely brought gave me something to smile about.

PDC BOF: Apps Design for SOA

I’m hosting a Birds of a Feather session at PDC Tuesday night 10:15PM. Here is the description:

Some build services to wrap legacy applications or integrate otherwise monolithic systems. Others say building new applications for service-orientation make those solutions agile. What does that mean? How should applications be designed to fit future service-oriented architectures? This is a brainstorm session -- no telling where it will lead.

Apocalyptic, no? I threw this together last Friday afternoon – almost on a whim – because there wasn’t really any other BOF sessions taking on the topic. It was accepted on Monday, which makes me feel bad for these guys, but they’ve got a backup plan.

A Canonical SOA Stack

In my last post I mentioned a Canonical SOA stack. So, I thought it would be good to just press on and propose a description of that animal. I know it’s probably beating a dead horse and many people are working on helpful descriptions of SOA. The recent work is good because I think we've turned a corner from the murkiness of Tech-Ed 2004 (San Diego). I recall lingering around the cabanas where, it seemed, SOA was defined only in terms of what it is not.

So, I'm risking coming off presumptuous in pitching a description of an SOA stack. But it's only a logical model to establish some boundaries between some (I think) obvious and independent concerns. There is quite a lot of discussion yet to come about SOA and this model could make for a handy map.

There are 4 basic layers separating the concerns of the Canonical SOA stack. At the bottom, quite expectedly, are application components which reside in a layer call the application domain. Next up the stack is the agent layer, which I think is at the heart of SOA. The agent layer is accessed by any number of representation channels which are endpoints with specific wire formats, invocation protocols, and service description flavors. At the top of the stack are conversation managers that provide for situations that span multiple invocations of service actions and probably have some state for managing conversation instances (sorry about the image quality - I'll get the hang of it).

I think this description is a simple as possible, although the top and bottom pairs of layers each form their own conspiracies. The bottom two layers compose functional capabilities to create callable service actions. The top two layers rationalize service actions into invocation models – stateless (e.g., SOAP, REST, proprietary mechanisms) and stateful (UI controllers, Orchestration Engines, etc.). This bears repeating: The bottom two layers formulate the solution's functionality by splitting requirements into two special categories. The top two layers provide the loose coupling and transport flexibility.


Application Domains and Agents

One tenet (ugh - the "t" word) of designing applications for an SOA is to segregate capabilities from policies when analyzing functional requirements. This is because we are betting that policies frequently change over time – at least more frequently than our capabilities need to evolve. So we want to isolate the hand-built code to the area of the solution that both seldom changes and tends to implement requirements that are tough to implement declaratively. Conversely, the bits that change most often are the collaborations and transforms that reflect business policies between functional areas.

As an example, not much has changed in the way debts and credits work in quite a while (except possibly at Enron). But the business processes that ultimately drive accounting transactions do change frequently. Agility (again, Enron notwithstanding) is valuable. For a packaged software company, having agile business processes in your products is these days is a competitive requirement.

Confession: I’m old-fashioned and still prefer nouns when describing an application domain.

Suppose a functional requirement dictates that when a sales order is confirmed, the order value must be recalculated, the order tables must updated and the inventory available-to-promise (ATP) balances must be adjusted. The rule requiring that the two nouns – sales order and inventory – collaborate shouldn’t be implemented in the order confirmation code. But you would be surprised how often collaboration is imperatively nested inline with the functional code. This means that policy changes require core code changes. Over time, you wind up with spaghetti.

The idea of factoring functions and collaborations from requirements is a classic concept commonly depicted in UML sequence diagrams. In SOA, the agent layer is responsible for executing collaborations. So collaboration logic needs to be extricated from the functional logic. I’m trying to stay away from specific design choices for now, but it makes sense to me to leverage the heck out of metadata in the agent layer. The more metadata you have and the richer it’s content, the more the agent can do as an engine. That means less hand-rolled code.

The ideal agent needs to be a fast, micro-orchestration engine that can validate messages, create transformations, execute application components, and delineate transactional and asynchronous boundaries. It’s a tall order, but the emergence of a robust agent layer coupled with stringent (yet simpler) coding practices is really the lynchpin of SOA.

Representation Channels and Conversation Managers

A representation channel is a mechanism providing access to service actions. Channels not only specify wire formats, but are also responsible for publishing descriptions of the service action set. Interoperability is achieved (in theory) when channels comply with industry standards like SOAP, SMTP, etc.

Some protocols and wire formats cannot convey the complete set of service actions. So it’s important to know where a service design might have difficulties with some channels in conveying the semantic intent (distinguishing what service you are calling) or the physical data (video streams over SOAP). It's important – and most people have already concluded this – that the presence of any specific communications channel does not an SOA make. You choose your set of data representation channels based on your anticipated caller patterns. Not all channels can handle all service actions.

The bottom three layers of the Canonical SOA Stack revolve around statelessness. If you look at BizTalk orchestrations or user interfaces, there is often a notion of a conversation instance (or session). Also, such facilities at the edges often need to aggregate and shred data for their own specific purposes. It might not be cost-justified to incorporate these requirements all the way down the stack.

The top of the stack – the conversation managers – implements stateful conversations that rely on the stateless model beneath. There is room for debate here because stateful conversation managers can just as easily connect directly to the agent layer. Maybe the top two layers ought to be merged, but for now I’m more comfortable leaving them separate. Let me know your thoughts!

Examples of conversation managers might include BizTalk orchestrations, user interface services, and pub/sub data staging. There are lots of facilities you could consider a conversation manager or a representation channel. Some channels are dedicated to a specific conversation manager. One example would be a publish/subscribe data staging area. Occasionally-connected clients – like Outlook on a notebook computer – also have custom conversation managers to stream just enough data to and from the server.


Summary

Many people think of service-orientation as fractal architecture, meaning that what one SOA domain might consider, an application component is really another SOA domain. This is sort of the Horton Hears a Who philosophy. Sure, inter-SOA-domain calls can happen in at least 3 of the 4 layers (not sure about channels). But this SOA description stays in it's own borders, for now, to keep the tiers clear.

Anyway, I wanted to start grounding the conversation about SOA and thought this would be helpful. At some point, someone is going to ask us to build a proper an SOA. I, for one, would like to be able to open Eclipse or Visual Studio some day and sort of know what to do next. Maybe having some named tiers can help propel design discussions. I think some interesting work is ahead to find metadata pathways from the bottom to the top of the SOA Stack. Such pathways host the transformation patterns I described in the last post.

Patterns for SOA 2.0

[23 June 2007: A hopefully clearer rewrite is here.] My dad is a research biochemist at an institute in Southern California. One of the (literally) cool things about his laboratory is the cold room. Going through the huge vault-like door, there are some tall glass tubes a couple of inches in diameter and several feet high. In the glass columns are various kinds of cloudy goo. Over time, colored bands of (I think) concentrated proteins and other molecular constituents appear. This event tended to cause some cordial celebration to occur, which I never really appreciated. But looking back, these scientists would sit back, wait, and then witness specialized things emerge naturally from a process of self-refinement.

Think about the move over the last 4 decades from monolithic applications to service-orientation. Instead of having gravity and osmosis do the work, we have trial, error, and debate. But software architecture is itself self-refining process nonetheless. By definition an architecture pattern is a reflection of an existing practice. I haven’t become nearly familiar with software architecture patterns enough to where I can recite them at cocktail parties. But it seems like the classic process of describing an architecture in terms of patterns is not helping us pin down the formal properties of SOA. Some argue that this proves SOA isn’t real because it can’t be described.

But maybe we are just looking for patterns in the wrong way. Physical SOA implementations will probably use well-known patterns like observer-subject, content-based routing and pipelines. However, I think that describing a modern SOA requires a deeper analysis centered on transformational patterns rather than classic software architectural or integration patterns.

SOA, well at least a 1.0 version, has been around for quite a while as any systems integrator will tell you for $225 an hour. Enterprise service buses, object brokers, and other agent-oriented have been successfully fooling monolithic applications into working with each other for years. Web services also put agent-oriented systems and services into people’s faces rather suddenly and on a mass scale. But version 1.0 of SOA was geared primarily toward aggregating otherwise inert systems and providing some new communication channels.

But now, many see a need for a more modern SOA, which I’ll call SOA 2.0 – where frameworks, applications, agents and communication channels understand each other more deeply – ideally using more aspect-oriented approaches. In short, the new SOA is about building a smarter stack and designing applications to take advantage of new constructs that (we hope) promote agility and simplicity.

At its core, SOA 2.0 uses graph transformation mathematics to convey semantics throughout the SOA stack’s layers as executable logic. At each layer of the stack, you define some semantic categories (a.k.a. viewpoints or aspects) and develop transformation patterns that produce hard rules that specific SOA layers can execute dynamically. Semantic category assignments are stored in metadata and transforms are implemented as engines or (less appealingly) as code generators.

This is what makes the architecture stack much smarter – semantics of the underlying application requirements are pulled through the solution mechanically. More importantly, as application semantics evolve over time, the solution itself evolves automatically from top to bottom. The idea is to get deployment overhead to approach zero.

Here is an admittedly simplistic example (italics represent some canonical SOA layers): In the data domain you could define a semantic category called “data representation” that describes whether a domain entity is resource data (customers, parts), activity data (orders, timesheets), or reference data (sales analysis), which is read-only. You then define a transform template called CRUD that produces interfaces for invoking functions in the application domain.

The template can be smart enough, for example, to avoid producing operations for reference data entities other than “Retrieve” (the “R” in CRUD). In turn, the agent layer has a transform template to create sets of actions (conceptually like a SOAPAction) from the application domain’s interface set. Service communication channels implement physical endpoints for WS-*, REST, etc. These channels use transforms to produce message processors and service descriptions from the agent action set. Finally, you might have 1 or more conversation managers that manage specialized state and data formatting capabilities for special purposes like driving user interfaces, BizTalk orchestrations, or Office Integration.

So, in each successive layer of a SOA 2.0 stack, new semantic categories and transformation templates are applied that may use artifacts residing in an adjacent layer to affect behavior. This transform pipelining approach creates a turnkey engine to project application capabilities into a sort of super-API. In the real world, however, these transforms are obviously not as simplistic as the CRUD example above. This is why we now need to begin building a library of reusable semantic categories and their corresponding transformation patterns.

Transformation rules – when described in mathematical terms – can be proven complete, unlike many hand-rolled, imperative programming approaches. In fact, this might be an entry requirement for adding candidate patterns to a future library. I think this approach beats the current entry barrier – prove something’s been done at least three times and voila – it’s a pattern!

Seriously, transformations are deterministic and flexible. They can be manifested as independent engines or in static code generation. They can be domain-specific languages (DSLs). Once implemented into an SOA stack, changes to the application domain can affect the entire solution predictably and automatically. Transformation rules can distinguish breaking changes from compatible evolution. So, the agent can, for example, know when to create a new action-set and when to simply alter descriptions of an existing action-set.

So to summarize, a key area in the next evolution of services architecture is about having the architecture stack aware of certain semantics in the application requirements. The stack can adapt itself to changes in the requirements by executing directed transforms against metadata. And now is the time to start identifying patterns of transformations that link layers of the SOA together intelligently. The mathematical world is well aware that graph transformations and category logic both relate well to computer science. Unfortunately, I majored in music. So, I need to get some help from more mathematically-astute people to see if all this actually matches up. Maybe a new color band is emerging from that goo in the glass column.

Dancing DOM to DOM

Several years ago, I was involved in a project for my day job where the client was browser based and the server was called strictly using SOAP. Even though the server was built on .NET, we decided not to create .NET types (classes or datasets) to represent parts of the SOAP messages. Instead, we just defined types using XML Schema and associated those types with message parts using a custom attribute.

The client was the browser (JavaScript) and the server received non-scalar message parts as .NET XmlDocument instances. So, the programming environment on both sides was DOM-oriented. Data access on both ends was done using predominantly XPath, SelectNode(s), and other DOM API calls.

I was later asked how the development team adjusted to a world without strong types. Was there a riot? Actually, there were worries about type safety, conversion between XML simple types and C# intrinsic types (yes, dates had to be closely managed), or the lack of a nice object hierarchy.

There was one complaint, though: no Intellisense.

Is that all we need to really send XML, web services, etc. to some next level of adoption? So, for all of the Visual Studio and Eclipse dudes out there, how hard could it be to attribute some DOM type with a schema QName and then get some IDE help when creating an XPath statement? If it’s doable, let me know!

One other issue was that Intranet single-sign-on is still hard and the solutions are very platform-specific. WS-Security is probably a great next step, but unless it gets implemented in the browsers (part of XMLHTTPRequest?) it will have to come by way of a plug-in.

Some people look at the DOM API and see a big stick of ugly. But the complaining dies down after realizing that XML messages have interesting, interoperable capabilities to compose messages. The skillsets are also reusable (meaning lucrative) in web programming.

XML Schema Profile for Language Types?

I attended the W3C Workshop on XML Schema 1.0 User Experiences (the program and minutes are here). My job was to represent the position of the WS-I, where I chair the XML Schema Work Plan Working Group. It was my first W3C event. It went well and I appreciate the W3C having a public event like this. What I didn’t expect was so much new support for profiling the XML Schema spec.

Just to restate the problem: XML Schema has constructs not easily expressible in most programming languages or databases. But I (and others) don’t see this is as an interoperability problem – it’s a toolkit problem. Despite some shortcomings, XML Schema can unambiguously describe the format of an XML document. So, the idea of creating an XML Schema “profile” to improve web service interoperability doesn’t add up to me.

Microsoft proposed that toolkit vendors should fully support the XML Schema spec (whew!), but that a number of “domain-specific” profiles would be “useful”. SAP said that XML Schema profiles are “inevitable and useful for language bindings, business vocabularies, and for specific user communities”.

So, the W3C might start looking into if/when/how to develop a set of domain-specific profiles. Is this a good idea? Are web services such a domain unto itself? Will it be ironic when schema profiles are authored using xs:redefine?* On the surface, it sounds useful, but let me overreact in my (usual) conflicted manner anyway.

On one hand, it would be nice if the Microsoft, IBM, Sun, and BEA toolkits could accurately reconstitute types from each other’s generated schema documents. And if you ask them, this is what they say their customers want.

But what if the WS-I gets pressure to add XML Schema profile conformance to the Basic Profile? It’s easy to see how this would happen. There are WS-I members who insist problems using schema in programming languages is a huge problem in developing web services. Also, the WS-I Board is dominated by toolkit vendors who have permanent seats and would like some sort of collective win to come of all this.

This could cause already tenuously adopted standards to diverge altogether. Many will not like a standards organization (whichever one) dictating a programming model, which in effect such a profile would do. And we all know there are already major, public web service implementers who more or less avoid the WS-* world altogether.

I am hopeful that RPC semantics will fade significantly from web services development by the time a language binding profile for XML Schema makes it to toolkits. Loose coupling means much more than “you don’t need my binaries to call my code”. It’s about building services that can evolve robustly and safely. One day, toolkit vendors will see that binding messages (or their parts) to static types – while seemingly the Holy Grail today – actually gets in the way of loose coupling and dynamic systems.

Ironically, XML is probably more natural than an object graph in representing a lot of real-world data. In fact, schema-relational mapping is actually easier to accomplish than object-relational mapping. Isn't it strange that people complain about the overhead of XML Schema validation, but have no problem with having their messages shredded value-by-value into a statically-typed object graph which is then re-shredded into a bunch of SQL? Why so much focus on the lowest-value part of the process?

Improving toolkits in order to make XML processing more amenable to today’s object-oriented approaches is NOT** the way to achieve better service-oriented solutions. Concocting a subset of the XML Schema specification just to provide more convenient programming experiences is a capitulation in the name of the wrong goal.

* This bit of attempted humor is homage to my friend David Ing, who also has seemingly proven that footnotes are indispensable in blogs.

** An earlier version of this post was missing this rather important word!

Test Post

This is a test post. However, feel free to comment.