Blog Setting up Detection-as-Code the Hard Way

Setting up Detection-as-Code the Hard Way

Detection-as-Code

Ethan Smart

June 25, 2026

8 min

I'm a believer in detection-as-code. Version control, peer review, CI/CD, the same engineering discipline we apply to product code applied to the thing standing between us and the next breach. That part isn't controversial anymore. Every mature security team I talk to is either doing it or actively trying to.

What's controversial, or at least under-discussed, is how much work it actually takes to set up. And, more importantly, what you still don't have when you're done.

This post walks through what it takes to wire up a real detection-as-code pipeline for Sumo Logic Cloud SIEM by hand: YAML rules in a Git repo, a linter, GitHub Actions, Terraform deploys, the whole thing. Working code is in the companion repo. Clone it, point it at your tenant, and you've got a starter pipeline running this afternoon.

Then I'll tell you, plainly, what that pipeline doesn't do and what we built Rilevera to do instead.

‍

Why detection-as-code matters (briefly)

Skip this section if you're already sold.

The pre-DaC world looks like this: detection content lives in a SIEM console. People edit rules in a web UI. Changes are made in the dead of night because someone got paged at 2 a.m. Nobody knows who changed what. The "tuning" history is a chat-thread archaeology project. Eventually a rule gets disabled "temporarily" and nobody remembers it for six months.

The DaC world looks like product engineering: rules live in Git. Changes go through pull requests. CI lints them. Merges deploy automatically. Rollback is git revert. Audit is git log. The operating principle is if it's not in the repo, it doesn't exist.

Sumo Logic themselves have a good guide on this, and it's the right starting point. What no vendor guide tells you is how much glue you need to write, and what you're still missing at the end.

‍

What we're building

A repo that:

Stores Sumo Logic Cloud SIEM rules in YAML.
Lints them with a Python validator (schema + sanity checks).
Unit-tests the validator itself.
Compiles YAML → Terraform HCL using the SumoLogic/sumologic provider.
Runs terraform plan on every PR.
Runs terraform apply on merge to main.
Keeps Terraform state in S3 with OIDC, so no static cloud creds ever touch the repo.

Five Sumo CSE rule types exist: match, threshold, aggregation, chain, first-seen, outlier (docs). We'll wire up the two most common ones (match, threshold) end to end. The others follow the same pattern.

Cost estimate: a senior detection engineer, focused, two to three weeks to get a first version solid, plus ongoing maintenance forever.

Let's go.

‍

Step 1: Decide what your YAML actually looks like

This is the first decision that bites you later, so think about it for real.

You have two options:

Author rules directly as Terraform HCL. Honest about what's happening underneath. Detection engineers have to learn HCL, which most of them don't want to.
Author rules in your own YAML schema, compile to HCL at deploy time. What most teams end up doing. Cleaner authoring surface, but you now own a compiler.

I went with the YAML route in the companion repo. Here's the shape:

1id: aws-cloudtrail-disable-logging
2name: AWS CloudTrail - Logging disabled or deleted
3type: match
4enabled: true
5severity: 8
6expression: |
7  metadata_vendor = 'AWS' AND
8  metadata_product = 'CloudTrail' AND
9  (fields.eventName = "DeleteTrail" OR
10  fields.eventName = "StopLogging" OR
11  fields.eventName = "UpdateTrail")
12entity_selectors:
13  - entity_type: _username
14    expression: user_username
15  - entity_type: _ip
16    expression: srcDevice_ip
17signal:
18  name_expression: "AWS CloudTrail logging disabled by {{user_username}}"
19  summary_expression: "Action observed that disables or alters AWS CloudTrail logging."
20  description_expression: |
21    A CloudTrail API call was observed that disables, deletes, or
22    alters a CloudTrail trail. This is a common adversary technique
23    for impairing defenses (MITRE T1562.008).
24tags: [_mitreAttackTactic:TA0005, _mitreAttackTechnique:T1562.008]
25metadata:
26  owner: detection-engineering@example.com
27  runbook: https://runbooks.example.com/aws-cloudtrail-disable-logging
28  references: [https://attack.mitre.org/techniques/T1562/008/]
29is_prototype: false

‍

The fields are designed to map cleanly onto the Sumo sumologic_cse_match_rule Terraform resource. The expression syntax is Sumo's own — see Cloud SIEM Rules Syntax. Notice the metadata block: owner, runbook, references. The Sumo schema doesn't require these, but if you don't enforce them, every detection eventually has a mystery author and no link to the runbook.

Friction point #1. The Sumo rule expression language is its own DSL with its own functions and metadata field naming (metadata_vendor, fields.eventName, srcDevice_ip). It's not Splunk SPL. It's not KQL. It's not Sigma. Your detection engineers need to learn it, and your linter can't really validate it locally since the parser lives server-side in Sumo. Anything beyond a balanced-paren check is best-effort.

Step 2: Write the linter

You can't author detections as code without a linter unless you enjoy finding out at deploy time that you forgot to set the severity.

The companion repo's validator is about 200 lines of Python. It does:

JSON Schema validation of the YAML shape — required fields, allowed enum values (rule type, entity type), severity in [1, 10], ID kebab-case, metadata.owner shaped like an email.
Rule-type-specific required fields — threshold and aggregation rules also need group_by, window, count.
A balanced-paren check on the rule expression.
A "did you anchor on vendor/product?" check — Sumo's docs strongly recommend every rule reference metadata_vendor or metadata_product so it doesn't run against unrelated logs.

The schema is plain jsonschema — no clever framework. Here's the relevant part:

RULE_SCHEMA = {
  "type": "object",
  "required": [
    "id", "name", "type", "enabled", "severity",
    "expression", "entity_selectors", "signal", "tags", "metadata",
  ],
  "properties": {
    "id": {"type": "string", "pattern": "^[a-z0-9][a-z0-9-]{2,63}$"},
    "type": {"type": "string", "enum": [
      "match", "threshold", "aggregation",
      "chain", "first_seen", "outlier",
    ]},
    "severity": {"type": "integer", "minimum": 1, "maximum": 10},
    # ...
  },
}
The repo ships with a deliberately broken rule (rules/_example_failing_rule.yml) so the linter has something to yell about. Running it:

$ python -m validator.validate rules/

FAIL rules/_example_failing_rule.yml
  - [schema] <root>: 'entity_selectors' is a required property
  - [schema] severity: 42 is greater than the maximum of 10
  - [schema] type: 'magic' is not one of ['match', 'threshold', ...]

OK rules/aws_cloudtrail_disable_logging.yml
OK rules/aws_cloudtrail_root_console_login.yml
OK rules/okta_brute_force_single_ip.yml

Validated 4 rule(s): 3 passed, 1 failed.

‍

That's the minimum bar. If you skip this step, every typo becomes a CI failure during terraform plan instead of a fast local error.

Friction point #2. "Validation" in the SIEM-as-code world has three distinct meanings and the gap between them matters:

Syntactic validation — does the YAML parse and have the right shape? This is what our linter does. Cheap. Fast. Catches typos.
Schema validation against the SIEM — does the rule expression reference fields that actually exist in your records? Our linter can't do this locally. terraform plan partially can.
Behavioral validation — does this rule actually fire on the events it's supposed to fire on? Nothing in this pipeline checks that. Hold on to this thought; we'll come back to it.

Step 3: Compile YAML → Terraform

Now you need to turn each YAML rule into a Sumo Terraform resource. The compiler in the repo is another ~150 lines of Python. It reads each validated YAML, picks the right resource type (sumologic_cse_match_rule vs sumologic_cse_threshold_rule etc.), and emits HCL into terraform/generated/.

For a threshold rule, the output looks like this:

resource "sumologic_cse_threshold_rule" "okta_brute_force_single_ip" {
  name = "Okta - Repeated failed logins from a single source IP"
  description_expression = <<-EOT
    Multiple failed Okta authentication attempts were observed
    from a single source IP within a short window...
  EOT
  expression = <<-EOT
    metadata_vendor = 'Okta' AND
    metadata_product = 'Okta' AND
    metadata_deviceEventId = 'user.session.start' AND
    outcome = 'FAILURE'
  EOT
  enabled = true
  tags = ["_mitreAttackTactic:TA0006", "okta", "identity"]
  entity_selectors {
    entity_type = "_ip"
    expression  = "srcDevice_ip"
  }
  severity_mapping {
    type    = "constant"
    default = 6
  }
  group_by_fields = ["srcDevice_ip"]
  limit       = 10
  window_size = "PT10M"
}

‍

You'll want terraform/generated/ in .gitignore. The YAML is the source of truth; the HCL is a build artifact.

Friction point #3. Every Sumo CSE rule type is a different Terraform resource with its own field names. count in your YAML becomes limit in the threshold resource. window becomes window_size. The mapping is fiddly, and the Terraform provider docs are the authoritative source — not the Sumo UI, and not the general Sumo API docs. Plan to spend an afternoon staring at the registry pages for each rule type.

Step 4: Wire up Terraform state

Local Terraform state files are fine for tinkering. For a real detection-as-code pipeline they aren't an option. Multiple engineers need to plan and apply, and you need an audit trail.

The right setup is:

An S3 bucket for Terraform state, with a least-privilege IAM policy.
An OIDC provider in AWS pointed at GitHub Actions (https://token.actions.githubusercontent.com).
An IAM role that trusts that OIDC provider, with the S3 policy attached.
No static AWS credentials anywhere. Ever.

Sumo's guide has the step-by-step for setting up the IAM side, and it's correct as far as it goes. Block out an afternoon for the AWS setup if you've never done OIDC-to-GitHub-Actions before.

In the repo, the backend is configured at terraform init time so the same code can target dev and prod tenants:

terraform init \
  -backend-config="bucket=$TF_STATE_BUCKET" \
  -backend-config="region=$TF_STATE_REGION" \
  -backend-config="key=terraform-state/prod.tfstate"

‍

Friction point #4. Multi-environment promotion (dev → staging → prod) isn't free. You'll need separate Sumo tenants, separate state keys, separate GitHub environments with separate secrets, and some convention for how a change graduates between them. The Sumo guide gestures at this with feature-branch → PR → main → manual promote, but the actual implementation is yours to build.

Step 5: The GitHub Actions workflow

The workflow in the repo has three jobs:

lint — runs on every PR and push. yamllint over rules/, then python -m validator.validate, then pytest -q against the validator's own unit tests.
plan — runs on PRs. Compiles YAML → HCL, assumes the OIDC role, runs terraform plan, uploads the plan as an artifact for reviewers.
deploy — runs only on push to main. Gated by a GitHub environment: prod so a human can approve. Compiles, plans, applies.

permissions:
  id-token: write  # required for OIDC
  contents: read

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.12' }
      - run: pip install -r requirements.txt
      - uses: ibiqlik/action-yamllint@v3
        with: { file_or_dir: rules/ }
      - run: python -m validator.validate rules/
      - run: pytest -q

  plan:
    runs-on: ubuntu-latest
    needs: lint
    if: github.event_name == 'pull_request'
    # ... see repo for full job

‍

That's the whole pipeline. Total moving parts: a YAML schema, a Python validator, a Python compiler, a Terraform module, an S3 backend with OIDC, three secrets and three variables in GitHub, and a workflow file. Probably 600 lines of code and a thousand lines of YAML and HCL once you've covered all the rule types.

Friction point #5. The pipeline now exists. That's not the same as the pipeline actually catching things. Read on.

‍

What the pipeline doesn't do

Now the honest part.

We built a pipeline that gives you, faithfully:

Syntactic confidence. The YAML parses and has the right shape.
A deploy trigger. Merge to main, the change lands in Sumo.
An audit trail. Git history of who changed which rule and why.

That is all it gives you. And once you've shipped your first ten detections through it, the gap between what the pipeline tells you and what's actually true in production starts to show up.

Specifically, the hard-way pipeline cannot tell you:

Whether the detection is actually firing on the events it's supposed to fire on. "It deployed" is not "it works." The CloudTrail disable-logging rule above will deploy cleanly even if the CloudTrail integration was misconfigured three weeks ago and the rule hasn't seen a matching record since. A green CI badge is indistinguishable from broken silence.

Whether the logs each detection depends on are still flowing. Sources stop ingesting. Collectors crash. IAM permissions drift. The rule keeps "passing" CI and is silently a no-op.

Whether the upstream log schema has changed under you. AWS renames a field. Okta restructures an event format. The detection still parses, still deploys, still validates — and quietly stops firing on half of what it used to fire on. Schema drift is the most common silent killer of detection content, and it's invisible to anything that only inspects the rule itself.

Whether someone bypassed your pipeline. Sumo's UI lets people edit rules in the console directly. Unless you have a process and an alert in place, your main branch can be cleanly merged and totally out of sync with production. The "source of truth" claim of detection-as-code only holds if drift is actively detected; otherwise it's an aspiration.

You can build instrumentation around each of these. Synthetic event injection. Per-source log-flow heartbeats. Schema snapshots and diffs. Nightly drift checks with terraform plan (Sumo's guide actually recommends this, it's a good idea). Each one is a project on its own. None of them is the pipeline you just spent two weeks building.

This is the gap we built Rilevera to close.

‍

The Rilevera contrast

Rilevera isn't another way to write detections. The world has enough of those. Rilevera focuses on the outcomes of detection-as-code — verifying it's actually working in production, not just that the YAML committed cleanly.

The four blind spots above, in order:

1. End-to-end detection validation. Rilevera answers the question the hard-way pipeline can't: does this rule actually fire on the events it's supposed to fire on, in the live SIEM? Not "does it pass a unit test against a synthetic payload," but does it fire when the adversary actually does the thing. This is the headline capability.

2. Log source health. Rilevera continuously checks that the logs each detection depends on are still flowing. The CloudTrail-disable rule above is meaningless if CloudTrail stopped ingesting two weeks ago. The hard-way pipeline can't tell you that. Rilevera can.

3. Schema drift detection. When an upstream log source changes its schema — a field renamed or a format restructured — detections that depend on those fields silently break. Rilevera recognizes that drift. (This is the more advanced piece of the platform.)

4. Unauthorized-change detection. Rilevera catches when someone edits a rule directly in the SIEM console outside the git/PR flow, so your repo doesn't drift from production reality. The "source of truth" claim of detection-as-code only holds if drift is actively detected; otherwise it's an aspiration.

On top of those four, Rilevera also gives you:

An in-product IDE for editing detections, with built-in versioning. No need to wire up VS Code + GitHub + a separate validator.
An automated, out-of-the-box peer review system. PR-style review for detections without having to build a GitHub Actions workflow yourself.

The same three detections in this post — CloudTrail disable, root console login, and Okta brute-force — take roughly one click each in Rilevera, deployed and instrumented. Setup is about an hour, not two weeks. And when one of them stops firing because Okta restructured an event payload, you find out the same week instead of during your next incident.

If that's the kind of thing you'd rather not build yourself, come see what we're doing.

‍

Build the hard-way pipeline anyway

Genuine recommendation, no irony: if you've never built one of these, spend a weekend on the companion repo. You'll learn what a real detection-as-code pipeline costs, where the seams are, and what questions to ask any vendor — including us — about what they're actually solving for. The walkthrough above gets you 80% of the way there; the Sumo guide covers the AWS/OIDC dance.

Then, when you're staring at the green CI badge for the third week in a row and wondering whether any of your rules have fired this month, you'll know exactly what you're shopping for.

Companion repo: sumo-detection-as-code/. Working Sumo CSE detections, a Python linter, a YAML→Terraform compiler, a GitHub Actions workflow, and a deliberately broken rule so the linter has something to fail on.

‍