SOA vs Mobile APIs

I recently read an article from Bill Appleton of Dream Factory with the provocative title SOA is not a Mobile Backend.

It raised quite a few good points that were in the back of my mind for more than a year.

Basically, what is the difference between SOA and API?

inclusion-229302_640[1]To an extent it is largely the domain of the buzzword department but as you think about it, it is more profound.

SOA really is an Enterprise Creature.  It’s a system integration strategy (despite what SOA purist will tell you).  As Bill mentions in his article, SOA also typically comes with its heavy Enterprise artillery:  Enterprise Service Bus, XML Message Translation, Service Locator, etc.  .  But it also comes with a set of useful practices:  domain knowledge, reusable services, etc.  .

API is an internet beast.  How do you interact with a service on the cloud?  You talk to its API.  API are typically simpler in terms of protocols:  HTTP, REST, JSON, simple messages, etc.  .  They are also messy:  is an API about manipulating a specific entity or doing a consisting set of functionalities?

To me, they spawn from the same principles, i.e. standard interface to exchange information / commands in a loosely couple way between remote components.  SOA is the Enterprise & earlier result of those principles.  API is the internet / mobile later result.

SOA was tried by some big enterprises, forged by comity with expensive consultants and executives trying to solve the next 10 years problem.

Integration_logo[1]API was put forward by a myriad of small companies and consumed by even more entities.  They figured out the simplest way to expose / consume services quickly and learned from each other.  In a few years a few set of practices were observed and documented and standards are even emerging.

Bill, in his article, contrasts the approaches in a way that reminds me of the old SOA debate of top-bottom vs bottom-top approaches, that is, do you discover your services by laying down your business processes and drilling down discover a need for services or by looking at your applications and which services they expose and hope that one day you can reuse them?

There is a lot of that in the issues brought by Bill around APIs.  Like in SOA if you just spawn new APIs ‘On demand’, you’ll end up with a weird mosaic with overlapping concepts or functionalities.  I agree that practices developed from SOA can definitely helped.  Service Taxonomy, for instance, forces you to think of how your services will align and where their boundaries will be drawn before you start.

But for an organization, I believe it is nearly a forced therapy to implement one or two APIs, experiment them in full operation before you can start having serious discussion around other SOA aspects.  Once you’ve tried it, you can have a much more informed discussion about what changes and at which in a service (while discussing versioning), what type of security rules make sense and a bunch of other aspects.

Otherwise you fall victim of the good old analysis paralysis and will host meetings after meetings for deciding about something because everyone has a different a priori perspective on it.

 

So my suggestion is yes, API are a little messier, but experimenting with them, even if you end up with a lot of rework, will bring much value to your organization.  So go, and create simple API, expose them and operate them!

Description of your REST API: Swagger & RAML

About a year ago I did a post about WADL. I hope you didn’t build a billion dollars business model around that initiative because it hasn’t left the ground yet :(

WADL, or Web Application Description Language for the initiated, was aimed to be what WSDL is for SOAP Web Services but for REST.

You see, the thing with REST that is so vastly superior to SOAP is its simplicity. You do not need a longish incantation of the XML Gods to cast a REST API. You simply need to be able to craft simple XML (or JSON) documents following an HTTP REST semantics.

One of the main problem with REST is its lack of self-description. Once you go beyond the proverbial calculator (always a super good example breaking about all distributed computing patterns) and that you distribute your API around, you will start to have Word or PDF or HTML documents describing your API. This is fine in the age of artisanal API but once you want consumer to plug and play your API, you would want them to do an “Add API reference” in Visual Studio or their favorite IDE.

Entered WADL. Actually, exit WADL: it was submitted to W3C in 2009 (by Sun Microsystems, oh yeah) and the W3C refused to standardize it.

Let me tell you about two contenders then: Swagger & RAML.

I heard about Swagger on Channel9, so I guess you could say that parts of Microsoft are looking into it.

Swagger has a very good head start in tooling with a lot of support in many target languages / platforms. It also produces a simple yet compelling UI.

You can look at the full specifications here, but basically it’s a JSON document describing your API. As a side-effect, Swagger enables the description of JSON document and therefore act as a sort of JSON Schema.

RAML is relatively similar to Swagger. Its format is YAML which is arguably more human-readable than JSON.

RAML seems to be less popular than Swagger but has several advantages, mainly:

  • Simpler to use to define a non-existing API (API design)
  • Existing features to manage repetition in an API (quite frequent)

At this point it really isn’t clear if either one will end up triumphing or if both will crash and burn. It is interesting to see the interest into an API description language is increasing though so I am confident that within years we will have a standard.

UPDATE:  It would appear that API Blueprint also is a contender.

Large Projects

There is something about large projects that you’ll never find, hence never learn, in smaller projects. The complexity, both technical and in terms of people dynamics, creates an all new set of challenges.

I read the article I Survived an ERP Implementation – Top 10 Gems of Advice I Learned the Hard Way at the beginning of the week. I was interested by the title since although I often work in companies where the ERP occupies a central place (don’t they always?), I’ve never been part of the implementation of an ERP.

As I read the article though I found much similarities between what the author was saying about the dynamics of an ERP implementation and large projects I’ve been on.

I therefore recommend it even if you don’t plan implementing an ERP anytime soon ;)

For instance, here are comments I would throw on the top of my head for some of her gems around ERP that apply to large projects in general:

10. Don’t be fooled by the system sales team. If they tell you “of course our system can do that” or “absolutely, with small modifications”, have your technical experts talk to their technical experts. Go in with eyes wide open.

 

Overselling a product isn’t a monopoly of the ERP sub industry, unfortunately!

Any complex product can easily be oversold by sampling the feature sheet of said product and matching it with a project’s requirement. The devil often is in the detail and on a large project, you can’t dive into all little details straight from the beginning.

If you are the technical expert or the one assigned to evaluate a product here are a couple of tips:

  • Ask questions, lots of questions
  • You won’t be able to cover everything so try to do horizontal and vertical sweeps: walkthrough an entire business process, look at an entire slice of data, look at an end-to-end identity journey, etc.

9. Whatever time period (and budget) you think will be required to go-live, you are most likely underestimating it. It’s tough enough to be immersed in an implementation, but continually pushing back the go-live date is deflating to the entire organization.

 

Large projects take more time than small projects, right? ;) Well, no, they take even longer!

Large projects have explicit steps that are implicit or much smaller scale on smaller projects: data migration, change management, business process optimization, user experience, etc. . Each take a life of its own and shouldn’t be underestimated.

8. There is no possible way you can over-communicate. Regardless of forum, of timeliness, of method, there is no such thing as too much.

 

Large projects have more people involved and last longer, hence give time for staff to churn. Your message will get distorted through layers of team, time, etc. . So repeating your message ensure that pieces of it will reach their destination.

6. Data is sexy. Learn to love it; treat it with respect and care. It’s the backbone of a successful implementation. You don’t want to to experience go-live with a broken back.

 

Amen.

 

5. A lack of change management will bite you in the butt. Post go-live, the speed at which you get through the hangover period will be heavily dependent on how well you managed change throughout the project.

 

You think the finish line is the delivery of the project? No, it’s the user acceptance of a new product, the large user base. If they reject it, whatever you have done won’t matter.

 

1. ERP implementations are equal parts politics and emotions. Ignore the effect of either of these at your own peril.

 

Expectations, perceptions, unspoken assumptions… ghosts that can harm you as much as the real thing. Do not ignore them!

WADL in a bottle eating noodles

In my last entry about REST web services I talked about its biggest weakness for me: the lack of description model of REST services.

The idea of hitting an HTTP endpoint as a shot in the dark is for me quite a leap of faith, and very likely an invitation to spend hours troubleshooting.

But despair no more, enters WADL! If it sounds like WSDL, it’s because it’s essentially the same acronym:

Web Service Definition Language -> WSDL

Web Application Description Language -> WADL

So WADL aims to be the WSDL of REST.

But… it was submitted the W3C in December 2009 by Sun Microsystem… one month before it was acquired by Oracle. Since them, it hasn’t budged… coincidence?

No other parties seem to have backed it, so it seems deemed to join the junkyard of unilateral attempt at standardizing global assets!

You can look up at an example on Wikipedia.

Maybe we’ll have another standard one day. Or maybe it’s a non-issue and I’m the only one to worry about it.

REST style with Hypermedia APIs

Once upon a time there was SOAP. SOAP really was a multi-vendor response to CORBA. It even share the same type of acronym, derived from object. Objects are so 90’s dude… The S in SOAP stands for Simple by the way. Have a go at a bare WSDL and try to repeat in your head that it is simple…

Then REST came along. I remember reading about REST back in 2002. It was a little after Roy Fielding‘s seminal article (actually his PhD thesis). Then there were a few articles about how SOAP bastardized the web and how XML RPC was so much better. But like the VHS vs Betamax battle before, the winner wasn’t going to be chosen on technical prowess. At least not at the beginning.

Then I stopped hearing about REST in 2003 and started seeing SOAP everywhere. We implemented it like COM+ interfaces really. A classic in the .NET community was to through Datasets on the wire via SOAP services. That really was a great way to misuse a technology… Ah… the youth… (a tear).

Microsoft tried to correct the trajectory by introducing WCF which enforced, or at least strongly suggest, a more SOA approach with a stronger focus on contracts and making boundaries more explicit. But somehow it was too late… something else was brewing beneath the SOA world…

In 2007, REST came back into fashion but now it was mainstream, i.e. people didn’t understand it, misquote it and threw it everywhere. Basically, it was: cool man, no more bloody contracts, I just send you an XML document, it’s so much simpler! Which of course works awesomely for 2-3 operations, then you start to get lost without a service repository because there are no explicit documentation!

If you see a parallel with the No-SQL movement (cool man, no more bloody schema, I just throw data in a can without ceremony, it’s so much simpler), I got no idea what you are talking about.

Anyway, if it wasn’t obvious, I’m not at all convinced that REST services solve that many issues by themselves. Ok, they don’t require a SOAP stack which make them appealing for a broader reach (read browser & mobile). But without the proverbial Word document next to you to know which service to call and to do something with, they aren’t that easy to use.

Then, finally, came Hypermedia API… I’ve a few articles about those, including the very good Designing and Implementing Hypermedia APIs by Mike Amundsen. I found in Hypermedia APIs the same magic I found when looking at HTML the first time: simple, intuitive & useful.

Hypermedia APIs are basically REST Web Services where you have one (or few) entry doors operations and from which you can find links to other operations. For instance, a list operation would return a list of items and each item would contain a URL pointing to the detail of the item. Sounds familiar? That’s how a portal (or dashboard) work in HTML.

Actually, you already know the best Hypermedia API there is: OData. With OData, you group many entities under a service. The root operation returns you a list of entities with a URL to an operation listing the instances of those entities.

The magic with Hypermedia APIs is that you just need to know your entry points and then the service becomes self-documented. It replaces a meta data entry (a la WSDL) with the service content itself.

The difference between now and the 2000’s when SOAP was developed is that now we really do need Services. We need them to integrate different systems within and across companies.

SOAP failed to deliver because of its complexity but mostly because it’s a nightmare to interoperate (ever tried to get a System.DateTime .NET type into a Java system? Sounds trivial, doesn’t it?).

REST seems easier on the surface because it’s just XML (or JSON). But you do lose a lot. The meta-data but also the WS-* protocols. Ok it was nearly impossible to interoperate with those but at least there was a willingness, a push, to standardise on things such as security & transactions. With REST, you’re on your own. You want atomicity between many operations? No worries, I’ll bake that into my services! It won’t look like any else you’ve ever seen or are likely to see though.

Mostly, you lose the map. You lose the ability to say ‘Add Web Reference’ and have your favorite IDE pump the metadata and generate nice strongly type proxies that will show up in intellisense as you interact with the proxy. Sounds like a gadget but how much is Intellisense responsible for the discovery of APIs for you? For me, it must be above %80.

Hypermedia API won’t give you Intellisense, but it will guide you in how to use the API. If you use it in your designs, you’ll also quickly find out that it will drive you to standardise on representations.

Applied SOA: Part 9–Service Versioning

This is part of a series of blog post about Applied SOA. The past blog entries are:

In this article, I’ll cover Service Versioning.

In SOA a Service usually has many consumers & it should be able to evolve independently of its consumers.  By that I mean that when a Service evolves, consumers shouldn’t break.  Otherwise we would have a hard time to evolve services.

We can do quite a bit of changes to a Service without breaking existing consumers logic. For instance:

  • Changing Service implementation details, i.e. not its behaviour, e.g. tap to another Database, add logging, route information to different systems, etc.
  • Adding a new Service Operation to a Service
  • Adding optional data-items in message payload

What needs to remain stable for consumers to continue to operate normally is called the Service contract:  Service behaviours and information exchange patterns, i.e. schemas & bindings.  This definition of Service contract is broader than WCF Service Contracts.

Service Contract can be backward compatible, i.e. a new Service Contract can support the same consumer base (for instance when we add a Service Operation without altering existing ones).

We need to version Service Contracts in order to accommodate for breaking changes.  When an evolution of a Service requires to introduce breaking change in its Contract, we need to support the current version until all consumers can be moved to the new version.

We want to keep track of two types of Service Versions:  breaking ones and non-breaking changes.  Breaking change versions require multiple versions to co-exist in order to support different consumers while non-breaking change versions are useful to determine the capacity of a Service.

A typical versioning scheme is to track breaking change versions with major version number while non-breaking versions are tracked with minor version number.  This is the scheme used by Windows Azure REST services for instance.

Concurrent Service Versions

Maintaining more than one Service Contract version at the same time is notoriously hard.  Typically you introduce breaking changes because either a business process or a back-end system imposes that change on you.  Those changes are often hard to burry under a legacy Service Contract version.

That problem isn’t unlike its equivalent in the component world.  How do you evolve a component?  You version its interface and try to keep old interface backward compatible as long as you can before deprecating them.

Service Behaviours

Just a word about Service Behaviours:  those are encapsulated in Service Contract along operation signatures.  Behaviours are any logic surfaced to the consumer, as opposed to implementation details which aren’t surfaced.

For example, if a Service encapsulates access to a Database, a consumer won’t know which Database the Service read / writes from.  The exact Database is therefore an implementation detail.

On the other hand, if a Service operation takes an integer in input and validates the input as being greater than zero by raising a fault if that pre-condition isn’t met, that becomes a behaviour.  A consumer depends on that validation:  if a change requires the input to be greater than 5, consumer sending values between 1 and 5 will break.

Service Behaviours are hard to document with tools.  For instance, WCF Service Contracts do not capture pre & post conditions.  This typically require documentation on the side.