2007-06-22

Distributed Computing Failures

I went to a talk several months ago given by Alan Robins, a Principal Engineer in the distributed systems engineering group at Amazon. The title of the talk was something like, "Performance and Availability" but the focus of the talk was much more on the how and why of what distributed computing technologies had failed at Amazon. It was really interesting. I took about three pages of notes, and they're more or less verbatim below. I wish he had released the slides, there was a lot of really good information that I was unable to get down on paper.
  • Technologies
    • XA Distributed transactions (two phase commit)
      • TP monitors such as Tuxedo
    • RPC
      • DCOM, DCE, CORBA, RMI, EJB
    • Stateful Remote Objects
      • RMI
  • Many dimensions to consider beyond performance and availability
    • Performance (TPS/Host Latency, etc)
    • Availability: How many nines (time up/total time)?
    • Scalability: How much effort to scale?
    • Distributability: How much effort for multiple data centers?
    • Evolvability: Effort to extend and mutate
    • TCO: Hardware, licensing, dev or integration, operations and maintenance
      • Reconsider performance and availability relative to TCO!
  • Distributed Transactions: Atomic transactions across multiple transactional resources
    • Example: Customer changes primary address and hits customer and address db
    • Dark Side
      • Expensive. Reduces scalability of db server
      • Latency of commit is 5x more over normal transactions
      • Reduce throughput of application
      • If any resources are down, nothing can happen. Reduces availability
    • Alternate to XA
      • Be optimistic, commit what work you can
      • Do no harm: order commits such that if failure occurs you can live with inconsistent state
      • Compensate: undo previous commit or queue up rest of work for later
      • Design for failure: Minimize cross db foreign key refs, even denormalize
      • Tolerate dangling references and inconsistencies
  • Remote Procedure Calls: make a function call like it's local, but it's not
    • Example: calculate shipping charges on a customer order
    • Dark Side
      • Binary formats create dependencies
      • Evolving API forces client side rebuilds. Expect to evolve.
      • Service owners must run multiple versions of their software.
      • RPC tightly couples availability requirements.
      • Many fine grained requests have high latency over global distances
    • Alternative to RPC
      • Document passing paradigm
        • Self describing wire format (XML)
        • Evolution without affecting old clients possible
        • Good for asynchronous message passing
        • RPC model still possible
      • SOAP Problems
        • Large messages
        • Expensive to parse and build DOMs
  • Stateful Remote Objects (CORBA, EJB)
    • Problem being solved: supports ? for clients, clients can make many fine grained calls, keeps data model on server, complex data model not transferred
    • Dark Side
      • Mapping client session to stateful server is complex.
      • Servers must keep state for each client (reduces scalability)
      • Server failure fails a lot of clients (reduced availability)
    • Alternative: Stateless servers with persistence store
      • Servers handle each request independently
      • Use data in request to establish context
      • Return results to caller
      • Advantage: High performance, high availability, scales great
      • Disadvantage: Pushes state onto data store
  • Asynchronous messaging, Once-only delivery
    • Problem being solved: service developers don't worry about dupes. They can just do what the request wants. Reduces application logic and complexity of handling dupes.
    • Example: Customer 1-clicks on an item
    • Dark Side
      • Almost impossible to guarantee. In order to ? everything must be transactional
        • Double clicks happen all the time
    • Alternative to Only-Once
      • Idempotence (quality of something that has the same effect if used multiple times at once): dupes handled correctly with respect to application.
      • Advantages: simple, enables more scalability and availability. Simplifies clients.
      • Disadvantages: Requires services to check their db. Sometimes service has to build look aside cache.
  • In order delivery: service doesn't worry about temporal discontinuities
    • Example: Order adds A, adds B, adds C, deletes B, submits order
    • Dark Side
      • Very difficult for infrastructure to manage total ordering.
      • Tight coupling.
      • Can't deliver message until current message delivered.
      • Eliminates availability and scalability
    • Alternative: best effort delivery
      • Developers deal with out of order messaging. Requires event to have a time stamp or sequence id.
      • Advantages: high throughput, optimistic delivery policy (deliver events when you can), very high availability
      • Disadvantage: Application developers must deal with out of order messages
  • Stored Procedures
    • Easy for developers to write RPC type applications
    • DBAs can ensure db resources are used efficiently
    • Complex logic performed without moving data across the wire
    • Dark Side
      • Database resources are the most expensive
      • Creates scaling limitations
      • Low performance
      • Application now split between application server and database
    • Alternative
      • Use a database for what it's good for: relational queries and updates
      • Keep business logic on server
  • Centralized Database
    • ACID model is easy to program against, ensures consistency
    • Reads after write guaranteed to reflect write
    • Provides single synchronization point for all applications
    • Provides richest set of capabilities
    • Example: Customer information database
    • Dark Side
      • Doesn't scale
      • Doesn't lend to global distribution
      • Most labor intensive
      • Least available
    • Alternative: Lightweight operation datastores/caches (e.g. bdb)
      • Datastore distributed geographically
      • Updates propagate via asynchronous messaging
      • Read operations are done locally
      • Updates done locally then write back to central or peer
      • Disadvantages
        • Inconsistencies: Read after write not absolutely guaranteed
        • Partitions can cause multiple versions to exist on different peers
        • Requires distributed group management (DHT)
  • The nature of distributed systems
    • Nodes fail
    • Networks partition
    • Data centers go down
  • There is a tradeoff between availability and consistency
    • Via distribution and redundancy you gain availability, scalability and performance but lose consistency
    • Strive for eventual consistency
  • Embrace failure: build in availability
  • Accept inconsistency.
    • Apology oriented development
  • If you are a developer, deal with these things: Potential inconsistencies considering race conditions
  • Infrastructure cant hide
  • Model applications as event driven process. Include all info needed in each message. Prop, repl, cache. This provides high performance and high availability
There are talks of this nature fairly regularly at UW and Seattle University, I encourage you to go when you can. This was one of the most informational talks I have ever been to, and it was free.