Watch this presentation from Randy Shoup, distinguished engineer at eBay.
He goes on talking about the lessons learned from his organization to scale a service.
He brings in 10 lessons:
- Partition Everything
- Asynchrony Everywhere
- Automate Everything
- Everything Fails
- Embrace Inconsistency
- Expect (R)evolution
- Dependencies Matter
- Respect Authority
- Never Enough Data
- Custom Infrastructure
Those are valuable lessons for any scalable cloud deployed services.
Probably the most original aspect of the presentation for me was his discussion about changes in systems. This is a topic rarely discussed because it is complicated and isn’t supported natively by different Cloud Platforms: how do you deploy a new version of a service when you need to deploy a new storage (e.g. schema) and processing on that storage.
eBay’s solution is that you keep both versions of the storage around and perform what they call dual writing: the systems write on both versions of the storage. They then keep both versions until they are convinced they won’t roll back the new service.
Otherwise, very interesting talk full of gems!