"There's an incredible amount of depth and thinking in the practices described here, and it's impressive to see it all in one place."
-Win Treese, coauthor of Designing Systems for Internet Commerce
The Practice of Cloud System Administration, Volume 2, focuses on "distributed" or "cloud" computing and brings a DevOps/SRE sensibility to the practice of system administration. Unsatisfied with books that cover either design or operations in isolation, the authors created this authoritative reference centered on a comprehensive approach.
Case studies and examples from Google, Etsy, Twitter, Facebook, Netflix, Amazon, and other industry giants are explained in practical ways that are useful to all enterprises. The new companion to the best-selling first volume, The Practice of System and Network Administration, Second Edition, this guide offers expert coverage of the following and many other crucial topics:
Designing and building modern web and distributed systems
- Fundamentals of large system design
- Understand the new software engineering implications of cloud administration
- Make systems that are resilient to failure and grow and scale dynamically
- Implement DevOps principles and cultural changes
- IaaS/PaaS/SaaS and virtual platform selection
Operating and running systems using the latest DevOps/SRE strategies
- Upgrade production systems with zero down-time
- What and how to automate; how to decide what not to automate
- On-call best practices that improve uptime
- Why distributed systems require fundamentally different system administration techniques
- Identify and resolve resiliency problems before they surprise you
Assessing and evaluating your team's operational effectiveness
- Manage the scientific process of continuous improvement
- A forty-page, pain-free assessment system you can start using today
Autorentext
Thomas A. Limoncelli is an internationally recognized author, speaker, and system administrator with more than twenty years of experience at companies like Google, Bell Labs, and StackExchange.com.
Strata R. Chalup has more than twenty-five years of experience in Silicon Valley, focusing on IT strategy, best-practices, and scalable infrastructures at firms that include Apple, Sun, Cisco, McAfee, and Palm.
Christina J. Hogan has more than twenty years of experience in system administration and network engineering, from Silicon Valley to Italy and Switzerland. She has a master's degree in computer science, a doctorate in aeronautical engineering, and has been part of a Formula 1 racing team.
Inhalt
Preface xxiii
About the Authors xxix
Introduction 1
Part I: Design: Building It 7
Chapter 1: Designing in a Distributed World 9
1.1 Visibility at Scale 10
1.2 The Importance of Simplicity 11
1.3 Composition 12
1.4 Distributed State 17
1.5 The CAP Principle 21
1.6 Loosely Coupled Systems 24
1.7 Speed 26
1.8 Summary 29
Exercises 30
Chapter 2: Designing for Operations 31
2.1 Operational Requirements 31
2.2 Implementing Design for Operations 45
2.3 Improving the Model 48
2.4 Summary 49
Exercises 50
Chapter 3: Selecting a Service Platform 51
3.1 Level of Service Abstraction 52
3.2 Type of Machine 56
3.3 Level of Resource Sharing 62
3.4 Colocation 65
3.5 Selection Strategies 66
3.6 Summary 68
Exercises 68
Chapter 4: Application Architectures 69
4.1 Single-Machine Web Server 70
4.2 Three-Tier Web Service 71
4.3 Four-Tier Web Service 77
4.4 Reverse Proxy Service 80
4.5 Cloud-Scale Service 80
4.6 Message Bus Architectures 85
4.7 Service-Oriented Architecture 90
4.8 Summary 92
Exercises 93
Chapter 5: Design Patterns for Scaling 95
5.1 General Strategy 96
5.2 Scaling Up 98
5.3 The AKF Scaling Cube 99
5.4 Caching 104
5.5 Data Sharding 110
5.6 Threading 112
5.7 Queueing 113
5.8 Content Delivery Networks 114
5.9 Summary 116
Exercises 116
Chapter 6: Design Patterns for Resiliency 119
6.1 Software Resiliency Beats Hardware Reliability 120
6.2 Everything Malfunctions Eventually 121
6.3 Resiliency through Spare Capacity 124
6.4 Failure Domains 126
6.5 Software Failures 128
6.6 Physical Failures 131
6.7 Overload Failures 138
6.8 Human Error 141
6.9 Summary 142
Exercises 143
Part II: Operations: Running It 145
Chapter 7: Operations in a Distributed World 147
7.1 Distributed Systems Operations 148
7.2 Service Life Cycle 155
7.3 Organizing Strategy for Operational Teams 160
7.4 Virtual Office 166
7.5 Summary 167
Exercises 168
Chapter 8: DevOps Culture 171
8.1 What Is DevOps? 172
8.2 The Three Ways of DevOps 176
8.3 History of DevOps 180
8.4 DevOps Values and Principles 181
8.5 Converting to DevOps 186
8.6 Agile and Continuous Delivery 188
8.7 Summary 192
Exercises 193
Chapter 9: Service Delivery: The Build Phase 195
9.1 Service Delivery Strategies 197
9.2 The Virtuous Cycle of Quality 200
9.3 Build-Phase Steps 202
9.4 Build Console 205
9.5 Continuous Integration 205
9.6 Packages as Handoff Interface 207
9.7 Summary 208
Exercises 209
Chapter 10: Service Delivery: The Deployment Phase 211
10.1 Deployment-Phase Steps 211
10.2 Testing and Approval 214
10.3 Operations Console 217
10.4 Infrastructure Automation Strategies 217
10.5 Continuous Delivery 221
10.6 Infrastructure as Code 221
10.7 Other Platform Services 222
10.8 Summary 222
Exercises 223
Chapter 11: Upgrading Live Services 225
11.1 Taking the Service Down for Upgrading 225
11.2 Rolling Upgrades 226
11.3 Canary 227
11.4 Phased Roll-outs 229
11.5 Proportional Shedding 230
11.6 Blue-Green Deployment 230
11.7 Toggling Features 230
11.8 Live Schema Changes 234
11.9 Live Code Changes 236
11.10 Continuous Deployment 236
11.11 Dealing with Failed Code Pushes 239
11.12 Release Atomicity 240
11.13 Summary 241
Exercises 241
Chapter 12: Automation 243
12.1 Approaches to Automation 244
12.2 Tool Building versus Automation 250
12.3 Goals of Automation 252
12.4 Creating Automation 255
12.5 How to Automate 258
12.6 Language Tools 258
12.7 Software Engineering Tools and Techniques 262
12.8 Multitenant Systems 270
12.9 Summary 271
Exercises 272
Chapter 13: Design Documents 275
13.1 Design Documents Overview 275
13.2 Design Document Anatomy 277
13.3 Template 279
13.4 Document Archive 279
13.5 Review Workflows 280
13.6 Adopting Design Documents 282
13.7 Summary 283
Exercises 284
Chapter 14: Oncall 285
14.1 Designing Oncall 285
14.2 Being Oncall 294
14.3 Between Oncall Shifts 299
14.4 Periodic Review of Alerts 302
14.5 Being Paged Too Much 304
14.6 Summary 305