Simple Workflow Service - Amazon Adding One Enterprise Brick At Time

Yesterday, Amazon announced a new orchestration service called Simple Workflow Service. I would encourage you to read the announcement on Werner's blog where he explains the need, rationale, and architecture. The people I spoke to had mixed reactions. One set of people described this as a great idea and were excited that the developers can now focus on writing domain-specific code as opposed to writing plumbing code to orchestrate their actual code. The other set of people felt that this service creates a new cloud lock-in making it difficult for the developers to switch from one cloud to another as well as being able to interoperate at the orchestration level.

I believe this is a brilliant idea for a variety of reasons. Orchestration has always been painful. Ask the developers who have been involved in managing task execution across a cluster that required them to code for load balancing, handling exceptions, restarting hung processes, tracking progress etc. This is not a core competency the most developers have but they do end up writing such code due to lack of better alternative. The frameworks such as WS-BPEL were not designed to run in cloud-like environments and there has been no single standard REST orchestration framework out there that people could use.

From a vendor's perspective, I admire Amazon's ability to keep innovating via such services that differentiate them as a leading cloud vendor. As computing becomes more and more commodity, competing based on price alone isn't a good idea. If you're a cloud vendor you need to go above and beyond the traditional IaaS attributes even though you excel in all of them. I also see PaaS eventually bleeding into IaaS as IaaS continues to become a commodity. As far as PaaS goes, federated or otherwise, we're barely scratching the surface.

I don't see this service as a cloud lock-in but it certainly makes EC2 more attractive and sticky. I would be concerned if Amazon were to force the developers to use their SWS for orchestration. This is their version of how they think orchestration should be done and the developers can opt in if they want. And kudos to them to think beyond their cloud. The folks who worry about cloud lock-ins also talk about Amazon not following the standards. I believe that we should not create standards for the sake of creating standards. I am a believer in first showing that something works and later, if there's enough interest, figure out a way to standardize it. All these talks about standard-first even before you write that first line of code doesn't make any sense.

It's yet to be seen how this service turns out, but this is a huge step forward for getting more enterprise software customers onboard. Orchestration is one of the most chronic problems of enterprise software and with the challenges of a hybrid landscape to be able to orchestrate across on-premise and cloud-based solutions, this service is certainly a step in the right direction. Right Scale has been using a Ruby workflow Ruote for their workflow needs and now they orchestrate these workflows using SWS  to achieve fault tolerance and concurrency. As you can see, Amazon has opened up a gold mine for start-ups. The back-end execution has always been challenging. Now, there is an opportunity to write your own enterprise grade workflow engine or scheduler that runs in the cloud.

SOAP may finally REST

Lately I have observed significant movement in two transformational trends - adoption of REST over SOAP and proliferation of non-relational persistence options. These two trends complement each other and they are likely to cause disruption sooner than later.

The enterprise software that required complex transactions, monitoring, and orchestration capabilities relied on the SOAP-based architecture and standards to realize their SOA efforts. The consumer web on the other side raced towards embracing RESTful interfaces since they were simple to set up and consume. There are arguments on both the sides. However, lately the market forces have taken the side of REST even if REST has significant drawbacks in the areas such as security and transactions. This once again proves that a simple and good enough approach that conforms to loose contracts outweighs a complex solution that complies to stricter standards even if it means compromising certain critical features. The web is essentially an unreliable stateless medium and any attempts to regulate it is less likely to work in our favor.

Many argue that the self-describing standards for SOAP are its strength over the RESTful services that lacks such features. However designing a RESTful service is fairly trivial since it allows to learn and experiment by being iterative unlike a relatively complex upfront learning process associated with the SOAP-based architecture. There has been a flurry of activities in the messaging middleware by Google that makes these RESTful interface even more compelling. This includes Google Wave Federation and PubSubHubbub. The developers are more likely to prefer these messaging protocols against SOAP and that would mean more RESTful APIs in the Pushbutton Web. Easy consumability reduces the initial adoption barrier and that's the key to success in many cases.

Since I last blogged about the continuum of the database on the cloud from schemaless to full-schema new persistence options have emerged such as RethinkDB and HadoopDB and many debates have spurred questioning the legacy of the RDBMS. For a cloud-like environment the statelessness, ad hoc persistence design, and instantaneous horizontal scale go well with the RESTful architecture. The growing popularity of SimpleDB and CouchDB along with many discussions on how to achieve CRUD with REST signal that the persistence is becoming more RESTful and schemaless.

I was convinced quite some back that REST was certainly the future for the consumer web but the latest trends have made me believe that the REST will see its adoption in the enterprise software accelerated much sooner than I had originally expected. This is like Java and Internet; the organizations embraced Java and the Internet at the same. The same will be true for the cloud and REST. When the companies consider moving to the cloud they will reconsider their SOA and persistence strategy and will likely adopt REST and alternate persistence models.

The cloud might be the last nail in the SOAP coffin.