0:02My name is Neela Rad and I am a product manager here at Cloudflare. I actually oversee a good portion of our disaster recovery and business continuity strategy, primarily related to the services that you all receive as customers or potential customers. Hopefully you leave here with an inspired ambition to partner with us. I also oversee a safe change. And what I mean by that is how we validate what you are receiving from Cloudflare is not going to break you as a customer. And similarly, what controls do you have as a customer to make sure that you're also implementing those same best practices for yourselves? Before I put myself on the spot for the next 20 minutes, I think it's only fair to get an assessment from the audience here.

0:49How many of you feel confident that if something were to go very, very, very bad with your service? this is tomorrow, you and your entire team would know exactly what to do and there would be no issue. Thank you, Brandon. One Cloudflare person. I appreciate that. So hopefully I'm not going to answer all those questions for you today. I would love to. I wish I could. But I'm hoping that you're giving you a little bit of visibility into how we think about resilience at Cloudflare, gives you some questions to ask internally, and maybe some ideas of how you can actually test that. your own company. So how do we think about resilience at Cloudflare? Fundamentally, there's four main pillars that I bucket my work into because I think frameworks are helpful in terms of thinking about where to even start when you're talking about a disaster.

1:35My job is not to magically predict what the disaster is going to be. It is to make sure that we actually are prepared to respond in the event that there is one. And those could range pretty dramatically, right? So having these principles. allows us to anchor on what our response is going to be. High availability is at the core of how we design a Cloudflare. And what that means is that we are always on. We shouldn't have a system that is failing because of an individual component, a service, or infrastructure. So we need to design with those in mind. We need to have automated failovers. We need health-aware routing. And that's where high availability as part of our resilience strategy comes in.

2:20play. Next, there is safe change. You are going to hear a full lot about safe change from me today. But fundamentally, what I am trying to make sure that we do there is that we are continuing to move fast, that we're continuing to innovate and deliver that innovation that you have come to expect from Cloudflare while still staying reliable. And then we sort of move to, well, what happens when something goes wrong? And that's where disaster recovery and business continuity come into play. And disaster recovery is making sure that we actually have predictable and efficient recovery after a disaster. That means that we know how we are going to do it and we can do that. And the business continuity plan goes hand in hand with that because what it allows us to do is actually ensure that every single person who is going to be responding in the event of a disaster can do so efficiently, effectively, and with as little panic as possible.

3:13What I didn't tell you earlier is I actually spent the last 20 years of my career, personally doing crisis management. And I found that that actually comes into play quite a bit nicely in this role, because if you are an engineer responding for your first P0 or even P1, maybe even a P2, that can be a pretty alarming experience. And part of my role is to ensure that that experience is not alarming, and that there's predictability and operational rigor around how we respond. And finally, probably the most fun part of my job is the chaos engineering program that we run. And that is essentially continuously validating that everything that we are building actually operates the way that we expect it to, both in staging environments and production. So that's just introducing chaos to the business and seeing how we respond.

3:59So let's talk about each one of those components in a little bit more detail. High availability. Again, we talked about always on architecture. You've heard a lot about how we are built. We've got our control plane, which is essentially where you're configuring. configurations. That's the simplest way to think about it. And the data plane. Our data plane is globally redundant and it's locally autonomous. And that means that one data center failing has no impact on any other data center. We also have automated global and local traffic steering. There's a lot of sessions about that. So if you're interested to hear more, I'd encourage you to listen to some of those. But that essentially means that we shift traffic between data centers to load balance. and there's no impact to the customer experience. On the control plane side, we have a primary and a secondary architecture there.

4:49And we have three, not redundant necessarily, but they load balance across the three for our control plane, both in our North American region as well as our European data center. This allows us both to implement an active, active architecture. So for some of our most critical services, as well as to also ensure that we have we have the ability to fail over in the event of a loss of one specific region. We also have high availability across those three. So, if we were to lose a single data center in a single region, we also could survive within that region by load balancing against the two surviving regions. So, that's what we're talking about when we're talking about high availability. There's also parts of this program that you may not see represented here, which is, for example, that we validate that our third-party dependencies also have a high availability.

5:36ability posture. So we're making sure that we don't have a single point of failure across our third party dependencies. And we do that regularly. We audit it on quarterly basis and ensure that we're building again with that high availability in mind. Safe change. And this is about deploying continuously, deploying safely. There's a lot that goes into safe change. So I am not going to cover all of that. What I will tell you is I did an exercise recently with our teams to say, let's validate our maturity posture against all of these areas, do we have investments in every single one of them? The answer is yes. That's the good news. The better news is the opportunity that we have is that we can continue to invest in every single one of these. And as the world evolves and changes, we need to continuously revisit every single one of these areas and continue to make those investments and mature.

6:25We've invested very heavily over the last year in outcome validation. And what we're looking at there is did that act change actually work that means that we implemented health signals we implemented um we improved our health signals we improved stabilization windows control comparisons uh and we are now really shifting to focus heavily and our investments on blast radius doesn't mean that we are taking our foot off the pedal on any of these but it does mean that we want to have a really focused investment in blast radius reduction and so um breaking BlastRadius into two different areas. I'm going to talk, I would love to have an entire session on customer control. We're going to talk less about that in this session, but I wanted to give you some insight to say part of BlastRadius reduction is making sure that you as a customer also have control over the changes that you're making on Cloudflare and that you can control BlastRadius for yourselves.

7:20What I'm going to be talking about more today is how is Cloudflare making sure that we're holding ourselves accountable for the changes that we are making that could impact you as a customer. and continuously reducing blast radius there. And so there's three specific ways that we've invested in the last six months. And this, again, is not a new investment, but it's something where we've more heavily invested and decided to focus on. The first is CI. And that is we've implemented Codex checkpoints to enforce requirements. This is all about how do we make sure that the code looks good before it even gets to starting the deployment cycle. So we've implemented pretty heavily there. We now can identify violations with agents that we've implemented. We have large -scale tools that allow us to correct MRs or create MRs across multiple repositories.

8:09So, for example, I have the ability to actually identify patterns that I consider higher risk in terms of architecture or ways that we are building, and then just address those by pushing out MRs and having engineers have a much simpler experience of figuring out how to implement or correct any issues that we're seeing. We have deployment safety infrastructure, and that's really where I'm going to focus on today. And then we're going to talk about service level segmentation. So deployment safety infrastructure, just to take you a little bit back in time. Before 2023, a lot of our code releases were actually manual edge releases. And is anyone in this vote willing to disclose that you are in a similar situation today? All right, no brave volunteers here. But we have materially improved that posture in the last, up until last year, we were continuously investing in how do we ensure that we have continuous edge deployment.

9:04That not only allowed us to move faster with automated deployments, but it also allowed more unversioned, it also allowed us to make sure that we were getting things to the edge quicker and identifying issues quicker, which is a good thing to do. We still, however, had unversioned change sources that were still present. And so one of the major investments we've made this year is actually allowing that even faster deployment. So significant infrastructure developments, massive high scale use cases, and then taking all of those unversioned change sources that we had and bundling that in and allowing us to actually make sure that everything is going through this rigor. process, but also a scalable one that allows us to continue to move fast.

9:57That means that we also had to build in some pretty important infrastructure into how we did it. So health media deployments, that means real-time monitoring of deployment health. It means that we have automatic rollback when we see degradation. It means a next -gen deployment platform that's scaling to 100,000 or more. releases per day with pluggable architecture so that means that any service can easily onboard and leverage the service. It means configuration changes that get the same health checks as code and it means one-click recovery so that we can go back to a last known state pretty quickly and reduce incident time to recovery. The data shows that it's working which is great. What we were able to show is that we were able to actually increase our deployment.

10:45velocity by 900 percent during this time period. And then what I'm talking about is Q3 of last year through Q3 of this year. And reduce incidents that are P0, P1, and P2 by 50 percent. So what that means, interestingly, our P3 incidents went up. But that's the trend that we want to see at this stage, because what that means is we are identifying issues earlier in the deployment cycle before they impact you as a customer. and customer impacting incidents have gone down significantly while we actually increased velocity.

11:22So we talked about deployments. I am now going to transition to talk about service level segmentation. essentially is a way for us, again, to reduce blast radius for you as a customer. And what you see here at the top is that all of our... traffic is going through health-mediated deployments, which we talked about. So we will go from 1%, and these are just arbitrary numbers to give you an idea, to 5 % of traffic to 25 % of traffic when we're releasing a change. That being said, that 1%, maybe 5 % of your traffic at this point in time. And that's not a great experience if you are the person that has 5 % of your traffic in that 1 % that we are deploying. And so what we are actually actively working is ensuring that we have lower-risk traffic that is receiving the changes first before we deploy to other traffic.

12:18So we are now able to actually isolate the experience and ensure that our traffic is going to a subset of traffic first and then higher-risk traffic next.

12:33We're not stopping there. So the next layer up there is where... customers, we're also giving them the ability to now tag their traffic. And so you'll be able to assign a color cohort to your traffic. And what this essentially allows you to do is validate that you don't have sequential deployments and that change that we're releasing is not going to impact your properties at the same time. You may have three different production zones and want to give them different colors. You will be able to do that. And that means that then you can validate that when we are deploying, Your red traffic will not be impacted at the same time as your, I'm choosing random colors and you can't fault me on this, but your green traffic or your blue traffic. And this allows you to make sure that, again, blast radius is mitigated. And any issues that you experience, if it is even to make it to this stage, given all the prevents that we have put in place prior to, are now isolated to a specific area or traffic.

13:34All right. Things inevitably are going to go wrong though. And when they do, we want to make sure that we have a plan and we can execute it. So this is essentially kind of the wheel that I follow continuously at Cloudflare. I am consistently making sure that I'm assessing our resilience posture. We'll talk about chaos engineering next. We then test and we validate. And finally, we remediate and we evolve. I consider it a win when I identify issues, when we do testing or when I identify issues and we do assessments, because what that means is that we are not waiting until an incident or a production event to actually identify and remediate. And so part two of the components of that is just regularly assessing disaster recovery posture through tests, through assessments, through services who are coming for proactive engagement.

14:30And then we've got our business continuity guide, which many of you have seen, and that is essentially our blueprint for what do we do in the event of things going wrong. This is all tested and validated through our chaos engineering program. And we have a lot of different tests that we do. This is just a subset to give you an idea that on a weekly basis, there are multiple tests that are happening in our production environment across our teams. sure that our resilience posture is exactly what we expect it to be. Some of the more common ones that you might hear me talk about are cut testing. And what we do during a cut test is we essentially, in our production environment, simulate the loss of a single colo within a region, and we validate that we are load balancing across the two surviving colos.

15:15So we will do that in both regions and ensure that there is, it's essentially a very, very minimal. or no impact to you as a customer as we do that. We do latency and injection testing. We regularly do failover testing where we simulate the loss of an entire region. And we do those typically quarterly or ad hoc as needed. And then we do tabletop exercises as well. We actually have our next major tabletop exercise coming up in November where we are, as a business, validating that we know how to operate exactly as expected based on the business continuity plan. So- we will, there will be no impact to you as a customer. This is purely a tabletop exercise, meaning that we have all of our teams come in and collaborate and they find out in the moment what's going on and they, we validate that everyone knows what to do, how to do it, when to do it.

16:10And then finally, we do break glass drills. This year, we actually have been doing them quarterly. This is an opportunity for engineers and anyone responding in the event of a disaster to practice other response mechanisms that we. built in place to allow them to break glass for the tooling that they need to remediate any incident. In the new year, we're actually moving to an automated model where your team could be selected at random and you would need to participate in a break glass drill to validate, again, that you know how to break glass. All of these, like I said, crisis work. This is designed to make sure that when bad things happen, which inevitably they will, we know how to respond. We can respond quickly. We can respond efficiently. we can respond well. And so these are our pillars.

16:56I only had 20 minutes, so I wanted to walk you through as much as I possibly could in as short a period of time as possible. But again, the goal of high availability and safe change is to make sure that we're doing that preventative work to identify issues, to reduce blast radius, to make it so that issues do not get experienced by our customers. But then we also want to make sure that disaster recovery and business continuity means that if something does go wrong, whether Cloudflare or otherwise, we know how to respond quickly and efficiently for our customers. And then again, the chaos engineering is also, let's figure out what else we need to know. What are we not thinking about? And just because we think we've built this correctly, is it actually working the way that we expect it to? And so in a very quick 17 minutes, I think that was, that's what business building resilience systems at Cloudflare looks like.

17:46Thank you all so much.