Half-Day Tutorial: Cloud Storage
Monday 17th August, Afternoon Session
Cheng Huang, Microsoft Research / Azure
Kiran-Kumar Muniswamy-Reddy, Amazon Web Services
The tutorial will take place in Sherfield Building, room SALC 7. For directions inside Imperial College check the campus map (building number 20).
12:30-14:00 buffet lunch
14:00 Afternoon Tutorial start
15:30-16:00 coffee break
17:30 Tutorial finish
Cloud storage has become such a convenient utility that we are now seeing it leveraged for networking research. The presenter has been consulting for three networking projects in Microsoft Research (one on SDN, another on abnormally detection and the third on Internet measurement), which all use various cloud storage services as building blocks. Answering questions regarding what and why often involves explaining how cloud storage systems work internally. This experience suggests that a tutorial might be welcome by the broad SIGCOMM audience.
Based on the working experiences in Amazon Web Services and Microsoft Azure, the presenters have compiled a list of open research problems. Exposing these problems to the broad SIGCOMM audience will hopefully attract networking researchers to help solve them.
The tutorial illustrates the internals of cloud storage systems. It covers patterns and best practices using cloud storage services in terms of cost,latency, and throughput. It also presents a list of open research problems derived from the working experiences in Amazon Web Services and Microsoft Azure.
Part I: Introduction to Cloud Storage (15 minutes)
- Cloud storage primitives in commercial services
- Cloud storage availability, durability and performance requirements
Part II: Internals of Cloud Storage (2 hours)
1: Distributed File System
- Replication: Chain Replication and CRAQ
- Consensus: Paxos and Egalitarian Paxos
2: Metadata Management
- Centralized metadata management
- Distributed metadata management in Ceph, Farsite, FDS and Swift
3: Erasure Coding
- Modern erasure coding for storage:
- LRC in Microsoft Azure and Regenerating Code
- Erasure coding across data centers: Facebook F4
- Data consistency protocols: eventual, strict and tunable
- Data consistency models in DynamoDB, S3 and MongoDB
- Performing transactions on existing NoSQL Systems
- Trade-offs with server-side transactions
Part III: Best Practices using Cloud Storage (30 minutes)
- Understanding service semantics & scalability targets
- Efficient NoSQL operations
- Plan for application failures: check-pointing state and backing up critical data
Part IV: Open Problems in Cloud Storage (30 minutes)
Cheng Huang is a research scientist and architect at Microsoft, where he is architecting and implementing the next generation Azure Storage system. He has invented technologies that have been shipped in many Microsoft products, including Azure, Windows, Office, Xbox and Bing, creating commercial impact in the order of hundreds of millions of dollars. He received a best paper at USENIX ATC in 2012, the Microsoft TCN Storage Technical Achievement award in 2013 and the IEEE Communications Society & Information Theory Society joint paper award in 2014.
Kiran-Kumar Muniswamy-Reddy is a Senior Engineer on Amazon DynamoDB, a fully managed NoSQL store that provides fast and predictable performance. He was involved in launching the product and has worked on launching various internal improvements and external customer facing features for DynamoDB. Most recently, he led the effort to launch online indexing ability for DynamoDB. In a previous life, he was an Academic and got his PhD from Harvard University.