0:03My name is Henry. I'm a principal security engineer in the detection response team with Cloudflare and today I'm going to talk you through a system we built called Argus and what I'm going to go through is explain to you what Argus is. It's pretty much a platform for our detection response team. We predominantly use machine learning detections over large security data sets and then I'm taking you through what goes under the hood there, take you through some of our detections that we use to build on Cloudflare gateway logs and then the last part is about security language models, how we train the LLMs to help us in investigation, what data sets we train on.

0:56and how we use it in our triage, in our detection, our generation as well.

1:04Here pretty much is the basic flow through Argus. We get some data through, we apply some rules, some ML detection, try and calculate the risk score, some of our models, and based on that we send an action to our cert team.

1:26Why we build Argus, so we build Argus to solve some of our problems that we face, usually we have large data sets, large time windows, we use some batch clearing that creates some delays in the results. We use, usually SQL, and then going away from the... is fixed detections to more anomalous, more of anomaly behavior things. And what we try and do is give evidence to people at the end on how to use these results to try out and find out what is malicious security wise and what's not.

2:16Before August, this is how our our detection cycle pretty much worked. We had a detection written in SQL and it's good when it's launched at the time, however there's a drift, things change, so these rules become quite brittle. So what we tried to do in Argus is to build a lot of anomaly detection on top anomalous behaviours so we don't have a fixed rule and based on that. we can detect when things are anomalous and possibly bad and also not to miss things that are that we usually run based on SQL queries. Over here is pretty much a diagram to what is Argus, the ARP platform.

3:07We run it in Kubernetes cluster in one of our data centers. Pretty much

3:19so over here is our data sources here is our detection compute engine we use two compute engines we use spark and slink i'm not sure if you any of you heard of that of these two engines spark is very good at doing ai and ml on a very large scale it's very good for batch processing of slink which is this one It's very good at streaming. So part of our detections is that we try and reduce the lag between the events and alert firing. We got it now to around three seconds lag between an event being triggered and an alert firing.

4:06Here are these, these are like the tools that we use for our ML pipelines. MLflow tracks our experiments, our parameters for each detection. Feast is our feature store, so every time we need to run the model, we don't have to calculate that from scratch. Jupyter Notebook for doing our testing, our development of our rules. Airflow is DAG and we use Elastic to view the results of the alerts. So that's pretty much what Argus is. We built a UI on top of that, which sits here, which shows all of those have some sort of UI and then ties them all up here so an analyst or a detection engineer can use that UI to look at things and try and find out what's going on in terms of day-to-day operation.

5:07Over here, I'm going to take... take you through the architecture so the slide that I showed earlier I'm just going to focus a bit more detail on each and one of those boxes I showed and yeah and just take you through the process of how we run the detection build it from scratch and how does the process look like inside Argus. As I mentioned these are the different components I touched upon earlier We use Spark for building for our batch process, our large data sets. We use Flink for streaming. We use Feast as a feature store for ourselves in Cloudflare, MLflow.

5:54And then we pass the results through a trained LLM, which I've trained on, which I'll touch on later in the topic. And that helps. in filtering some of the results. We usually think is that anomalous doesn't mean security bad-wise, so what we do is that we try and filter that through the results. So whatever we can send out to the cert team or to the investigation is a lot less than the number of the volume of what's anomalous.

6:31So over here, uh this this how the process starts so we read a data set say for example our gateway logs um we read probably the last around last 30 days worth of data we extract some features of it so whether it's ip address location where people work uh we around probably have around 40 features we try and extract from the from from that detection We train our data sets on that, we get the model, an anomaly detection model, and once we're happy with the results, we have hyperparameters. What I've been trying to do is similar for Flink, so we read the events, you read the model from there, and then extract the features from Feast, I used the same model to do this.

7:29and then we have a risk score. That risk score determines whether it's going to be sent for search investigation, whether it's going to increase the next time run, not as the next time you run risk score, and send all of that as evidence to our detection and response team.

7:52This is how our ML pipeline, Through in Argus, we pretty much go through the data and then we built and do some pre -processing. We load the feature store, a feature matrix if we have it. We train, we validate on top of that. If the model is having a drift, not optimized, too many results coming out, we... We send it back for re -evaluation to optimize the model. If it's approved, we have an approved bundle. This has a version, a hash of sorts, and that gets deployed.

8:41And that's how the process works, pretty much in a semi-autonomous way for our ML pipeline.

8:53Now I'm going to talk a bit more into some of the algorithms that we use and some of the models that we use in practice.

9:02This is a famous diagram on what anomaly detection looks like. So pretty much we take a baseline or a feature, we put it on here and then we measure against a recent. window if if there's an outlier we detect on that if it's above a certain threshold if it's not we we just we just leave it as it is

9:39over here these are some of the models that we use so the k -means measures distance A Gaussian mixed semester, which is GMM, usually uses the likelihood of something happening. PCA is used for computing error. We pass that through some LLM and we use these different models and we create a composite risk scoring, measuring different aspects for different user behavior. that we have a score and that score plus the evidence from each of these models gets categorized whether it's severe critical high low medium and based on that we can get to do some automations or some stuff down the line or send it for in further investigation

10:42these are some of our models that we use on some of our gateway logs for the gateway logs we use k -means gmm for dns because it has host names we use term frequency inverse document frequency so that turns these domains into some wording and some matrices and then and then based on that we calculate a risk call and then And then based on that, we do interesting things for measuring the anomalous behavior between people based on the job family, where they work, if they're an engineer, if they're non -technical.

11:29And then we can do a bit of interesting clustering, compare people based on different features and different behavior. One last thing that we've been also experimenting with is using... LLMs for our CASB, for our DLP logs and what it does is we use a fast LLM that grabs the logs, grabs the actual rules themselves and review that and based on that review it will have a risk call whether it's worth investigating further, whether some of these DLP rules need some further optimisation or whether it's it's benign and there's nothing you can do.

12:14These DLP logs, we have very large volumes of them, so no human can pretty much have the time to do that. So we have a two -step process of using a fast LLM for filtering first and then a more advanced one to review the outcome of the first one. And these are some of the things that we run in August to help us with our decisions. detections. I touched earlier now upon the LLM side and in section here I'm going to talk you through what we built on training of our security LLMs for detection and response and how that process looks like. Pretty much the reason for that is to train the data.

13:06train the security LLMs in here is pretty much to give a consistent behavior and feedback and then also train it on a lot of our security data and then the right taxonomy, the right schema, also absentation. So it can only speak based on evidence. Over here, we pretty much much been doing different types of training, CPT, which is continuous pre -training, supervised fine -tuning, direct preference optimization, and then reinforced fine -tuning. And based on each of these methods, I train on different types of data to help the model do two things.

13:59Our first one is based on... So if the results come, then read the model and then give some recommendation on how to optimize it. The second part is the triage part, which is it reads the results, the alerts, and based on that it can decide whether it's malicious, requires more security review, or whether it's safe to drop.

14:31So, over here for the process of training the LLMs, we also have been training two different types of LLMs. A fast one, which can run very fast, needs the least amount of GPU to run, and then that fast rank here pretty much goes through, has a very small context, can run very quickly. quickly, and then you can do that in line with your detection and your alerting pipeline. So in a way, it acts like a very basic triage. And then the biggest part of training it is to make sure that it can abstain from making a decision if it doesn't have enough data supported in its corpus.

15:22Hallucination is a big thing. And then trying to get a very... lightweight LLM to abstain when it doesn't have the information and how you train about doing that. What we trained on so far, we used a lot of public security data material, detection structures, so there are a lot of repositories based on S -Query, based on different types of Sigma, calco, different types of rules, some cases. These have been mostly also open source. Some of our material from inside Cloudflare, do some quality checks on our data, deduplication, and then divide them to different categories of data sets.

16:20For the C- GPT, which is the bulk of the data that goes to your heavy type of data corpus, I think I've been training around 5 billion records on that. The supervised fine-tuning is usually very fast. That takes a few days to train on, and then that usually can change with time. And then those two pretty much between the DPO. and the reinforced fine-tuning is to make sure that you tweak the behavior of the model to pretty much respond in the way that you want it to be.

17:06Over here, I think I touched upon that, that each stage has a distinct job. This is probably the one that takes the longest and most resources. That usually takes around the one. week of training uh to get this done um the supervised fine tuning takes around a few days two to three days to run um this these granting tools and changing the the behavior of the model that's usually a few hours the testing on that there are some benchmarks there's some things between open source and industry standard ones and ones that i've been internally built to to test the data on that usually takes around a day so an experiment like that to train a model we usually take around 10 10 days minimum to sometimes two weeks so pretty much it's been an interesting process and now i'm going to discuss some of the results that we've seen it does improve a lot the outcome the model so these are different tasks that that train the model on so triage networking and web knowledge you can see here so this is the untrained model and then the trained models you can see here that you we pretty much got very very interesting uh improvements in the model's performance

18:38um i think our position went to from 0 .02 to 90 percent and the correction absentations which is very important which you don't want your model to hallucinate it abstained when it didn't know 119 out of 120 so it tells you that your models if you train it on what you want you can get very good very interesting and very big improvements on the performance of the model And finally, I want to share with everyone, this is like a summary of what we've built in the last year, Argus, which is one platform, fast delivery, we can move things quite fast, we can improve the models, better detections based on machine learning.

19:39improvements for our security trained models and have the ability to improve the models and help our analysts in terms of their work and day -to -day. And I think that's it.