Coding Blocks

It’s finally time to learn what Site Reliability Engineering is all about, while Jer can’t speak nor type, Merkle got one (!!!), and Mr. Wunderwood is wrong.

The full show notes for this episode are available at https://www.codingblocks.net/episode181.

Survey Says

So, DevOps is a culture, but SRE is a job title?

Reviews

Thanks for the review “Amazon Customer”! (You, er, we know who you are.)

Site Reliability Engineering

  • Site Reliability Engineering: How Google Runs Production Systems is a collections of essays, from Google’s perspective, released in 2016 … and it’s free. (sre.google)
  • There’s a free workbook to go along with it too. (sre.google)
  • But how is SRE as a career? (GlobalDots.com)
    • Career Advancement Score (out of 10): 9
    • Median Base Salary: $200,000
    • Job Openings (YoY growth): 1,400+ (72%)
  • These essays are what one company did, that company being Google.
  • The book is told from the perspective of people within the company.

It is about scaling a business process, rather than just the machinery.

Site Reliability Engineering: How Google Runs Production Systems
  • Their tale should be used for emulating, not copying.
  • 40-90% of your effort is after you have deployed a system.
  • The notion that once your software is “stable”, the easy part starts is just plain wrong.
  • Yeah, but what is a Site Reliability Engineering role?
    • It’s engineers who apply the principles of computer science and engineering to the design and development of computing systems, usually large distributed ones.
    • It includes writing software for those systems.
    • Including building all the additional pieces those systems need, i.e. backups, load balancers, etc.
  • Reliability … the most fundamental feature of any product?
    • Software doesn’t matter much if it can’t be used.
    • Software need only to be reliable “enough”.
      • Once you’ve accomplished this, you spend time building more features or new products.
  • SRE’s also focus on operating services on top of the distributed computing systems. Examples include:
    • Storage,
    • Email, and
    • Search.
  • Reliability is regarded as the primary focus of the SRE.
  • The book was largely written to help the community as a whole by exposing what Google did to solve the post deploy problems as well as to help define what they believe the role and function is for an SRE.
  • They also call out in the book that they hope the information in the book will work for small to large businesses. Even though they know small businesses don’t have the budget and manpower of larger businesses, the concepts here should help any software development shop.

However, we acknowledge that smaller organizations may be wondering how they can best use the experience represented here: much like security, the earlier you care about reliability, the better.

Site Reliability Engineering: How Google Runs Production Systems
  • It’s less costly to implement the beginnings of lightweight reliability support early in the software process rather than introduce something later that’s not present at all or has no foundation.
  • Who was the first SRE? Maybe Margaret Hamilton? (Wikipedia)
  • The SRE way:
    • Thoroughness,
    • Dedication,
    • Belief in the value of preparation and documentation, and
    • Awareness of what could go wrong, and the strong desire to prevent it.

Hope is not a strategy.

Site Reliability Engineering: How Google Runs Production Systems

Chapter 1 – Introduction

The famous “SRE Book” from Google
  • Consider the sysadmin approach to system management:
    • The sysadmins run services and respond to events and updates as they happen.
    • Teams typically grow as the capacity is needed.
    • Usually the skills for a product developer and a sysadmin are different, therefore they end up on different teams, i.e. a development team and an operations team (i.e. the sysadmins).
    • This approach is easy to implement.
  • Disadvantages of the sysadmin approach:
    • Direct costs that are not subtle and are easy to see.
      • As the size and complexity of the services managed by the operations team grows, so does the operations team.
      • Doesn’t scale well because manual intervention with regards to change management and process updates requires more manpower.
    • Indirect costs that are subtle and often more costly than the direct costs.
      • Both teams speak about things with different vocabularies (i.e. no ubiquitous language from back in the DDD days).
      • Each team has different assumptions about risk and possibilities for technical solutions.
      • Each team has different assumptions about target level of product stability.
  • Due to these differences, these teams usually end up in conflict.
    • How quickly should software be released to production?
      • Developers want their features out as soon as possible for their customers.
      • Operations teams want to make sure the software won’t break and be a pain to manage in production.
  • A developer always wants their software released as fast as possible.
  • An ops person would want to minimize the amount of changes to ensure the system is as stable as possible.
  • This results in trench warfare between the two groups!
    • Operations introduces launch and change gates, such as test for every problem that’s ever happened.
    • Development teams introduce fewer changes and introduce more feature flags, such as sharding the features so they’re not beholden to the launch review.

What exactly is Site Reliability Engineering, as it has come to be defined at Google? My explanation is simple: SRE is what happens when you ask a software engineer to design an operations team.

Site Reliability Engineering: How Google Runs Production Systems

Google’s Approach to this Problem?

  • Focus on hiring software engineers to run their products (not sysadmins).
    • Create systems to accomplish the work that would have historically been done by sysadmins.
  • SRE can be broken down into two main categories:
    • 50-60% are Google software engineers, that is people who were hired via the standard hiring procedure.
    • 40-50% are candidates who were very close to the Google software engineer qualifications but didn’t quite make the original cut.
      • Additionally, they had skills that would be very valuable for SRE’s but not as common in typical software engineers, like Unix system internals and networking knowledge.
  • SREs believe in building software to solve complex technical problems.
    • Google has tracked the progress career-wise of the two groups and have found very little difference in their performance over time.
  • Software engineers get bored by nature doing repetitive work and are mentally geared towards automating problems with software solutions.
  • SRE teams must be focused on engineering.
  • Traditional ops groups scale linearly by service size, hiring more people to do the same tasks over and over.
  • For this reason, Google puts a 50% utilization cap on SRE’s doing traditional ops work.
    • This ensures the SRE team has time to automate and stabilize the software through means of automation.
    • Over time, as the SRE team has automated most of the tasks, their operations workload should be reduced to minimal amounts as the software runs and heals itself.
  • The goal is that the other 50% of the SRE’s time is on development.
  • Only way to maintain those rates is to measure them.
  • Google has found that SRE teams are cheaper than traditional ops teams with fewer employees because they know the systems well and prevent problems.

… we want systems that are automatic, not just automated.

Site Reliability Engineering: How Google Runs Production Systems

Challenges

  • Hiring is hard and the SRE role competes with product teams.
  • Pager duty!
  • Requires developer skills as well as system engineering.
  • This is a new discipline.
  • Requires strong management to protect the budgets, such as stopping releases, respecting the 50% rules, etc.

One could view DevOps as a generalization of several core SRE principles to a wider range of organizations, management structures, and personnel. One could equivalently view SRE as a specific implementation of DevOps with some idiosyncratic extensions.

Site Reliability Engineering: How Google Runs Production Systems

Tenants of SRE

  • Availability
  • Latency
  • Performance
  • Efficiency
  • Change Management
  • Monitoring
  • Emergency Response
  • Capacity Planning

Durable Focus on Engineering

  • In order to keep time for project work, SREs should receive a maximum of 2 events per 8-12 hour on-call shift.
  • This low volume allows the engineer to spend adequate time for accuracy, cleanup, and postmortem.
  • More than events that mean you have a problem to solve or more SREs to hire, less and you have too many SREs.
  • Postmortems should be written for all significant incidents, whether paged or not.
  • Non-paged work might be even more important since it can point to a hole in the monitoring.
  • Cultivate a blame-free postmortem culture.

Max Change Velocity

  • An error budget is an interesting way to balance innovation and reliability.
  • Too many problems and you need to slow down and focus more on reliability, not enough problems and you’re probably gold plating.
  • Ever have a manager push back on tech-debt? Maybe they aren’t aware of this balance? What can you do to quantity it?
  • 100% uptime is generally considered to not be worth it, as gets more expensive as you get closer to the mark and your customers generally don’t have 100% uptime, so it’s wasteful.
  • What is the right reliability number though? That’s a business decision.
    • What downtime percentage will the users allow, based on their usage of the product?
    • How critical is your service? Is there a workaround?
    • How well does the experience degrade?
  • What could a team do if there’s not anymore room in the budget?
  • What if there’s too much?

Monitoring

  • Monitoring is how to track the system’s health and availability.
    • Classic approach was to have an alert get sent when some event or threshold is crossed.
      • This is flawed though because anything that requires human intervention is by it’s very definition, not automated and introduces latency.
      • Software should be interpreting and people should only be involved when the software can’t do what it needs to do.
  • Three types of valid monitoring:
    • Alerts – a person needs to take immediate action.
    • Tickets – a person needs to take action but not immediately. The event cannot automatically be handled but can wait a few days to be resolved.
    • Logging – nobody needs to do anything. The logs should only be viewed if something prompts them to do so.

Reliability is a function of mean time to failure (MTTF) and mean time to repair (MTTR).

Site Reliability Engineering: How Google Runs Production Systems

Emergency Response

  • The best metric for determining effectiveness of an emergency response is the MTTR, i.e. how quickly things got back into a healthy state.
  • People add latency. Even if there are more failures, a system that can avoid emergencies that require people to do something, will still have higher availability.
    • Thinking through problems before they happen and creating a playbook resulted in 3x improvement in MTTR as opposed to “winging it”.
    • On call SRE’s always have on-call playbooks while also doing exercises they dub the Wheel of Misfortune to prepare for on call events.

Change Management

  • 70% of outages are due to changes in a live system.
  • Best practices:
    • Progressive rollouts,
    • Quickly and accurately detecting problems, and
    • Ability to rollback safely when something goes wrong.
  • Removing people from the loop, the practices above help improve release velocity and safety.

Demand Forecasting and Capacity Planning

  • Forecasting helps you ensure service availability and keep costs in check and understood.
  • Be sure to account for both organic growth, i.e. normal usage, and inorganic growth, such as launches, marketing, etc.
  • Three mandatory steps:
    • Accurate organic forecast, extending beyond the leadtime for adding capacity,
    • Accurate incorporation of inorganic demand sources, and
    • Regular load testing.

Provisioning

  • The faster provisioning is, the later you can do it.
  • The later you can do it, the less expensive it is.
  • Not all scaling is created equally. Adding a new instance may be cheap but repartitioning can be very risky and time consuming.

Efficiency and Performance

  • Since SRE are in charge of provisioning and usage, they are close to the costs.
  • It’s important to maximize resources, which fundamentally affect the success of the project.
  • Systems get slower as load is added, and slowness can also be viewed as a loss of capacity.
  • There is a balance between cost and speed. SREs are responsible for defining and maintaining SLOs.

Resources we Like

  • Links to Google’s free books on Site Reliability Engineering (sre.google)
  • Why is SRE Becoming 2021’s Hottest Hire? (GlobalDots.com)
  • How much money do SREs make? (Gremlin.com)
  • Margaret Hamilton (software engineer) (Wikipedia)

Tip of the Week

  • Don’t reinvent the wheel, if you’re in Java. Guava is a collection of utilities that solve common problems, courtesy of Google. (GitHub)
  • From the mindset of RTFM: There are some interesting flags you can pass for git cherry-pick … and other tools you might use. (git-scm.com)
  • You can use CTRL+NUM on Windows or CMD+NUM on macOS to navigate between tabs in Chrome. (support.google.com)
Direct download: coding-blocks-episode-181.mp3
Category:Software Development -- posted at: 8:01pm EDT

We’re living through the tail end, maybe?, of the Great Resignation, so we dig into how that might impact software engineering careers while Allen is very somber, Joe’s years are … different, and Michael pronounces each hump.

The full show notes for this episode are available at https://www.codingblocks.net/episode180.

Sponsors

  • Mergify – Save time by automating your pull requests and securing the code merge using a merge queue.

Survey Says

What's most important to you when you're looking for another job?

Take the survey at: https://www.codingblocks.net/episode180.

Reviews

Thanks for the review Chuck Rugged (or is it Rugged?).

What is “the great resignation”?

  • The Great Resignation is an ongoing economic trend where a lot of people started quitting their jobs in 2021 and peaked at 3% unemployment (up roughly 50% from the pre-COVID unemployment average).
  • Primarily, but not exclusively, in the US, but also trended in Europe, China, India, Australia as well.
  • Some interesting factors:
    • High worker demand and labor shortages.
    • High unemployment.
    • Employees between 30 and 45 years old have had the greatest increase in resignation rates, with an average increase of more than 20% between 2020 and 2021.
    • Resignation rates actually dropped for people in their 20s.
    • Tech and healthcare led the trend, 4.5% for US, 3.6% for healthcare.
    • Reasons cited included stagnant wages and working conditions.

Why is this a big deal?

  • Hiring is expensive! Think of thinks like referral fees, recruiter’s percentage, takes a while for people to become productive, onboarding, etc.
  • What does this mean for working conditions? More remote, better compensation, more flexibility, etc.?
  • Why do people change jobs?
    • Promotion,
    • Work life balance,
    • Compensation,
    • Flexibility,
    • Leaving a bad environment, and/or
    • Better company

What can you gain?

  • Salary bands, FAANG vs local vs remote vs startup
  • The “TC” (total compensation) Trap
    • Restricted Stock Units vs Options
  • Top paying companies, by level (levels.fyi)
  • Comparing levels across orgs (levels.fyi)

About those levels

  • Senior engineers are senior developers who may specialize in a specific area, oversee projects, and manage junior developers.
  • Principal Engineer is a highly experienced engineer who oversees a variety of projects from start to finish.
  • Staff engineer is a senior, individual contributor role in a software engineering organization. There is no “one” kind of staff engineer and many fall into one of four archetypes: Tech Lead, Architect, Solver, and Right Hand. (staffeng.com)
  • Is there a hiring level cap? What does that mean?

What can you lose?

  • The people,
  • The grass isn’t always greener,
  • Seniority (don’t be the “At X we …” person), and/or
  • Comfort

Resources we Like

  • Great Resignation (Wikipedia)
  • The Great Resignation: Data and analysis show it’s not as great as screaming headlines suggest (NevadaCurrent.com)
  • The Great Resignation is here. What does that mean for developers? (stackoverflow.blog)
  • Who Is Driving the Great Resignation? (hbr.org)
  • What Do Software Developers Want Out of Their Next Job? (insights.dice.com)
  • How the Government Measures Unemployment (bls.gov)
  • Salary comparisons (levels.fyi)

Tip of the Week

  • Did you know you can expand or collapse all the files in a pull request on GitHub? Press Alt + Click on any file chevron in the pull request to collapse or expand them all! (github.blog)
  • Kotlin code for using Google Cloud! (cloud.google.com)
  • Thanks to Dave Follett for sharing How to securely erase your hard drive or SSD! (pcworld.com)
  • Thanks to Fuzzy Muffin for sharing Nvchad, a nice face for Neovim (Nvim) that adds some nice features, like directory access and tabs. (nvchad.github.io)
  • How do you merge two Git repositories? (Stack Overflow)
  • Use git-sizer to get various statistics about your repository. (GitHub)
  • How to find/identify large commits in git history? (Stack Overflow)
  • Then forget about BFG and filter-branchgit filter-repo is the way to remove large files from your Git repo (GitHub)
  • Use --shallow-exclude to exclude commits found in the supplied ref in either (or both) your git clone (git-scm.com) or git fetch operations. (git-scm.com)
  • Limit your git push operation “up to” a commit by using the format git push <remote name> <commit ID>:refs/heads/<branch name>. (If the <branch name> already exists on the <remote name>, you can leave off the refs/heads/ portion. (git-scm.com)
Direct download: coding-blocks-episode-180.mp3
Category:Software Development -- posted at: 8:39pm EDT

1