Suppression

What Is Suppression?

Most of a log stream is the same few lines, over and over. Auditty recognises a repeating pattern after about 25 occurrences, forwards the first ones, and from then on replaces the repeats with one summary per window: how many, over what time, which ids varied, and a link to the originals. The volume your platform indexes falls by 80 to 95%; what you can find in it does not.

When Does Suppression Happen?

The Summary Envelope

Every suppression summary, in JSON, logfmt, or plain text, carries the same unified envelope. Rather than a copy of one suppressed line, it states, in one clean record: a readable message, how many events were suppressed, the window they spanned, exactly which correlation keys varied under ids (and their values), and a backfill_url to the exact originals in the Vault.

Plain Text Logs

For plain text or unstructured logs, Auditty emits a single-line summary: a readable message (the masked template) followed by the bracketed envelope. Every id is its own greppable | name=values segment listing the complete distinct set for the window.

connection timeout after <decimal> ms [auditty: suppressed 100 over 3s (12:38:00-12:38:03) | namespace:prod | workload:api-gateway | fingerprint:f7e8d9c0 | host=10.0.0.5,10.0.0.7,10.0.0.9,10.0.0.11,10.0.0.13 | backfill_url:https://<your-hivemind-url>/backfill/replay?token=<TOKEN>&sig=<SIG>]

JSON Logs

For JSON-formatted logs, Auditty emits a clean, standalone object: a readable message plus the auditty envelope. It does not copy the original event's fields; the exact originals live in the Vault, one click away via backfill_url:

{
  "message": "Alert rule evaluated",

  "auditty": {
    "suppressed": 85,
    "window": { "start": "2026-01-15T09:03:00Z", "end": "2026-01-15T09:03:06Z" },
    "duration": "6s",
    "fingerprint": "f7e8d9c052931784",
    "class": "http",
    "namespace": "monitoring",
    "workload": "grafana",
      "ids": {
      "org_id":   ["1"],
      "rule_uid": ["cdyy38hxcuadcd", "edyy38fwgaz28a", "b2c3d4e5f6a1b7"]
    },
    "backfill_url": "https://<your-hivemind-url>/backfill/replay?token=<TOKEN>&sig=<SIG>"
  }
}

Logfmt Logs

For logfmt (key=value) logs, Auditty emits the same envelope as one flat logfmt line: the masked template of the whole logfmt line under msg= (every key, values as placeholders; msg by logfmt convention, not message), then every envelope key namespaced under the auditty.* prefix. Each id is auditty.ids.<name>, carrying its complete comma-separated value set. Any logfmt parser auto-facets the fields.

msg="host=<ip> level=warn msg=connection timeout after <num> ms rule_uid=<id>" auditty.suppressed=100 auditty.start=2026-01-15T09:03:00Z auditty.end=2026-01-15T09:03:06Z auditty.duration=6s auditty.namespace=monitoring auditty.workload=grafana auditty.fingerprint=f7e8d9c052931784 auditty.ids.host=10.0.0.5,10.0.0.7,10.0.0.9 auditty.ids.rule_uid=cdyy38hxcuadcd,edyy38fwgaz28a,b2c3d4e5f6a1b7 auditty.backfill_url="https://<your-hivemind-url>/backfill/replay?token=<TOKEN>&sig=<SIG>"

The msg= value is the masked template of the whole logfmt line, keys in sorted order (varying spans as <num>/<decimal>/<str>/<id>, …); the auditty.* keys mirror the JSON envelope exactly, so a single set of parsing rules covers both structured formats.

Declaring Named Correlation Keys

Auditty indexes searchable ids automatically: shape-detected correlation keys (uuid, hex trace/RayID, prefixed id, long token, IP, MAC, JWT) and values under id-ish key names (trace_id=abc, request_id, …) land in ids under their real name. For a known schema (the correlation keys your team searches on every day), declare them in identity_keys so they are always indexed from JSON and logfmt lines at the highest priority, even a plain word or bare number, even on very wide logs dominated by other fields (dots and dashes in the name become underscores in ids: X-Request-Id is X_Request_Id). (To index a specific magnitude such as bytes_sent, use promote_numeric_fields.)

policy:
  suppression:
    # Correlation keys that must always appear in summaries, by exact
    # (case-insensitive) field name. Matched anywhere in the log, top-level
    # or nested, and never crowded out by other fields.
    identity_keys: [RayID, X-Speleo-Trace-Id, trace_id, request_id]

Once declared, a key shows up like any other id (auditty.ids.RayID) and is queryable in your SIEM immediately.

Platform Parsing Rules

JSON logs are auto-extracted by all major platforms, no setup needed (the nested auditty.ids.* become searchable attributes automatically). For logfmt, a built-in key-value parser auto-extracts every auditty.* key, including per-id values like auditty.ids.trace_id, so a custom rule is rarely needed. For plain text, add a Grok rule to extract the fixed envelope (suppressed, window, namespace, workload, fingerprint, backfill_url, …); the per-id | name=values segments are best consumed from the JSON or logfmt formats, or matched individually if you know the id name.

Datadog

Add a Grok Parser processor in Logs → Configuration → Pipelines. Datadog uses lowercase matchers.

auditty_rule ^.*\[auditty: suppressed %{number:auditty.suppressed} over %{data:auditty.duration} \(%{data:auditty.window}\)( \| namespace:%{notSpace:auditty.namespace})?( \| workload:%{notSpace:auditty.workload})?( \| node:%{notSpace:auditty.node})?( \| severity:%{word:auditty.severity})?( \| class:%{notSpace:auditty.class})?( \| fingerprint:%{word:auditty.fingerprint})?( \| anomaly:%{notSpace:auditty.anomaly})?%{data}( \| backfill_url:%{data:auditty.backfill_url})?\]

auditty_logfmt_rule %{data} auditty\.suppressed=%{number:auditty.suppressed} auditty\.start=%{notSpace:auditty.start} auditty\.end=%{notSpace:auditty.end} auditty\.duration=%{notSpace:auditty.duration}%{data}( auditty\.backfill_url="%{data:auditty.backfill_url}")?

Splunk

Use rex for inline extraction, or set KV_MODE = auto in props.conf for logfmt.

index=your_index "[auditty: suppressed"
| rex "\[auditty: suppressed (?<auditty_suppressed>\d+) over (?<auditty_duration>[^ ]+) \((?<auditty_window>[^)]+)\)"
| rex "namespace:(?<auditty_namespace>[^ |\]]+)"
| rex "workload:(?<auditty_workload>[^ |\]]+)"
| rex "node:(?<auditty_node>[^ |\]]+)"
| rex "fingerprint:(?<auditty_fingerprint>[^ |\]]+)"
| rex "class:(?<auditty_class>[^ |\]]+)"
| rex "severity:(?<auditty_severity>[^ |\]]+)"
| rex "anomaly:(?<auditty_anomaly>[^ |\]]+)"
| rex "backfill_url:(?<auditty_backfill_url>[^ \]]+)"
| rex "\| host=(?<auditty_host>[^|\]]+)"

index=your_index "auditty.suppressed="
| kv

Elastic / Kibana

Use an ingest pipeline with a Grok processor (plain text) or KV processor (logfmt). Elastic uses uppercase matchers.

PUT _ingest/pipeline/auditty-suppression
{
  "processors": [{
    "grok": {
      "field": "message",
      "patterns": [
        "\\[auditty: suppressed %{NUMBER:auditty.suppressed} over %{DATA:auditty.duration} \\(%{DATA:auditty.window}\\)(?: \\| namespace:%{NOTSPACE:auditty.namespace})?(?: \\| workload:%{NOTSPACE:auditty.workload})?(?: \\| node:%{NOTSPACE:auditty.node})?(?: \\| severity:%{WORD:auditty.severity})?(?: \\| class:%{NOTSPACE:auditty.class})?(?: \\| fingerprint:%{WORD:auditty.fingerprint})?(?: \\| anomaly:%{NOTSPACE:auditty.anomaly})?%{DATA}(?: \\| backfill_url:%{DATA:auditty.backfill_url})?\\]"
      ],
      "ignore_missing": true, "ignore_failure": true
    }
  }]
}

PUT _ingest/pipeline/auditty-suppression-logfmt
{
  "processors": [{
    "kv": {
      "field": "message", "field_split": " ", "value_split": "=",
      "ignore_missing": true, "ignore_failure": true
    }
  }]
}

Grafana Loki

Loki extracts at query time. For logfmt, the built-in | logfmt parser auto-extracts all auditty.* fields (dots become underscores).

{namespace="production"} |= "[auditty: suppressed"
  | regexp `\[auditty: suppressed (?P<auditty_suppressed>\d+) over (?P<auditty_duration>[^ ]+) \((?P<auditty_window>[^)]+)\)`
  | regexp `namespace:(?P<auditty_namespace>[^ |\]]+)`
  | regexp `workload:(?P<auditty_workload>[^ |\]]+)`
  | regexp `fingerprint:(?P<auditty_fingerprint>[^ |\]]+)`

{namespace="production"} |= "auditty.suppressed"
  | logfmt
  | auditty_suppressed != ""

New Relic

Add a Grok parsing rule in Logs → Parsing. New Relic uses uppercase matchers and underscores in attribute names.

%{GREEDYDATA}\[auditty: suppressed %{NUMBER:auditty_suppressed} over %{DATA:auditty_duration} \(%{DATA:auditty_window}\)( \| namespace:%{NOTSPACE:auditty_namespace})?( \| workload:%{NOTSPACE:auditty_workload})?( \| node:%{NOTSPACE:auditty_node})?( \| severity:%{WORD:auditty_severity})?( \| class:%{NOTSPACE:auditty_class})?( \| fingerprint:%{WORD:auditty_fingerprint})?( \| anomaly:%{NOTSPACE:auditty_anomaly})?%{DATA}( \| backfill_url:%{DATA:auditty_backfill_url})?\]

%{GREEDYDATA}auditty\.suppressed=%{NUMBER:auditty_suppressed} auditty\.start=%{NOTSPACE:auditty_start} auditty\.end=%{NOTSPACE:auditty_end} auditty\.duration=%{NOTSPACE:auditty_duration}%{GREEDYDATA}( auditty\.backfill_url="%{DATA:auditty_backfill_url}")?

Configuration

Key settings in your Edge configuration:

policy:
  suppression:
    summary_window: 600s    # Time window (default: 10 minutes)
    summary_count: 100      # Count threshold (default: 100)
    summary_min_count: 25   # Minimum count to emit (default: 25)

Per-namespace / per-workload overrides: The window and minimum count are cluster-wide by default, but timestamp-sensitive or bursty streams (e.g. CDN or security logs that matter most during an incident) can use a shorter window for near-real-time visibility while noisy workloads keep the longer default. Add overrides under suppression:

policy:
  suppression:
    summary_window: 600s         # cluster-wide default
    overrides:
      - namespace: cdn           # all workloads in the "cdn" namespace
        summary_window: 30s
        summary_count: 50        # emit after 50 events (overrides the cluster-wide count)
      - namespace: cdn           # this workload gets an even shorter window
        workload: edge-proxy
        summary_window: 10s
        summary_min_count: 10

Proxy / non-Kubernetes sources: namespace is optional. OTLP / Edge Proxy streams have no namespace, so key the override by service (or workload) alone, at least one of namespace, workload, or service must be set. service is an alias for workload, so either names the same stream:

policy:
  suppression:
    summary_window: 600s
    overrides:
      - service: checkout-api    # no namespace: matches this service in any namespace
        summary_window: 5s
        summary_min_count: 10

Suppression grouping key: each window is scoped per source, so each stream is summarised on its own. For Kubernetes logs that scope is namespace + workload; for non-Kubernetes sources that have no namespace (e.g. Edge Proxy / OTLP senders), Auditty keys the window off the service attribute instead, so distinct services are summarized independently.

Renaming the Summary Key

By default Auditty namespaces every summary field under a top-level auditty key: the JSON object key ("auditty": { … }), the logfmt prefix (auditty.namespace=…), and the plain-text tag ([auditty: suppressed …]). If your SIEM dashboards and monitors already key on a house-standard namespace, you can rename it with summary_key so no downstream queries need to change:

policy:
  suppression:
    summary_key: auditty   # top-level key for summary fields (default: "auditty")

Only the top-level key changes, nested fields (suppressed, window, ids, backfill_url, …) keep their names. The value must be a safe identifier (^[A-Za-z][A-Za-z0-9_]*$: letters, digits, underscore; no dots, spaces, or quotes) so it stays a single logfmt key segment and a valid JSON/plain-text key. The setting is hot-reloadable.

Forwarding the Suppressed Stream (Archival)

By default suppressed events are replaced by summaries on the primary stream and the originals are kept in the Vault for backfill. If you maintain your own downstream archive (e.g. a second Vector pipeline), you can instead have Auditty forward the full suppressed stream to your primary destination tagged suppressed=true, so you can route it to cold storage and drop it from your index.

policy:
  suppression:
    tag_suppressed: true       # forward suppressed events tagged suppressed=true
    suppressed_tag: suppressed # tag key (default: "suppressed")

Promoting Numeric Fields

The summary ids object indexes only searchable correlation keys, so bare magnitudes and counters (bytes_sent, duration_ms, committed) are kept in full in the Vault but are not listed in ids by default; they are context, not identifiers. If a specific stream emits a magnitude you genuinely want searchable inline, promote it per namespace/workload to add it to ids for that stream:

policy:
  suppression:
    overrides:
      - namespace: tracon
        promote_numeric_fields: [bytes_sent, bytes_received, duration]

Total-Recall Search Index (1.7.1)

Shipped in Edge 1.7.1 / Auditty 1.7.1. ids deliberately indexes only correlation keys. If your requirement is search on anything (measurements, dimensions, status codes, every value of every extracted field, directly in your log platform), enable the Total-Recall search index. Every summary then also carries vals: the complete deduplicated set of distinct values for every extracted field in the window (up to 32 fields per line; the raw line is always in the Vault), plus the same tokens in the summary message so bare free-text search gets a hit.

policy:
  suppression:
    search_index:
      enabled: true                  # off by default; hot-reloadable
      channel: both                  # attributes | text | both
      max_value_len: 256             # bytes; longer values (blobs) stay Vault-only
      max_bytes_per_window: 131072   # 128 KiB per suppression window
      max_chunk_bytes: 65536         # 64 KiB per emitted summary line
      max_total_bytes: 16777216      # 16 MiB node-wide budget
    overrides:
      - namespace: payments
        search_index: true   # or scope it to specific streams only

Every summary then carries vals next to ids: the same tokens are also appended to the summary message when channel includes text:

{
  "message": "payment authorized tr_9f2ab 38 41 2210 ord_58201 ...",
  "auditty": {
    "suppressed": 8000,
    "ids":  { "trace_id": ["tr_9f2ab", "..."] },
    "vals": {
      "trace_id":   ["tr_9f2ab", "..."],
      "latency_ms": [38, 41, 2210],
      "order_id":   ["ord_58201", "..."],
      "status":     [200, 503]
    },
    "backfill_url": "https://..."
  }
}

Memory is capped, and every cap has a defined overflow: a value that does not fit inline is in the Vault.

Track what the index costs in egress bytes and billable lines per workload with auditty_search_index_bytes_total and auditty_search_index_spill_lines_total.

Outlier Passthrough

A promoted numeric field is searchable, but a genuinely anomalous value inside an otherwise-repetitive log line (a real latency spike buried in request completed) is still only recoverable by querying the Vault after the fact. Outlier passthrough closes that gap: when a promoted field’s value falls far outside its fingerprint’s own learned normal range, Edge forwards that one event through in full instead of suppressing it. No Vault query needed, no operator action required.

processor:
  fingerprint:
    outlier:
      enabled: true
      z_score_threshold: 4.0   # standard deviations from the learned mean to count as an outlier
      warmup_samples: 30       # observations required before gating activates for a field
      flood_guard_window: 1m   # per-fingerprint passthrough budget window
      flood_guard_max: 10      # max passthroughs per fingerprint per window
      max_tracked_fields: 8    # distinct numeric field names tracked per fingerprint
      categorical_enabled: true    # also gate never-seen values on low-cardinality string fields
      max_categorical_values: 32   # distinct values remembered per categorical field

Passthroughs are counted in auditty_outlier_detected_total and mirrored in auditty_suppression_skipped_total{reason="outlier_passthrough"} for observability.

Preserving Public IPs & Emails

By default the identity ledger captures private IPs but skips public IPs and email addresses: public IPs are personal data under GDPR and high-cardinality, and emails are PII, so promoting them into summaries is a deliberate, per-stream choice. Opt in per namespace/workload when you need them searchable inline (e.g. in Datadog) rather than only in the Vault:

policy:
  suppression:
    overrides:
      - namespace: edge
        preserve_public_ips: true   # capture routable client IPs inline
        preserve_emails: true       # capture email addresses inline (PII)

Both default to false; private-IP capture is unchanged. Enabling these increases the cardinality of the inline summary, so scope them to the namespaces that need them.

Summary

• Suppression starts when the same log pattern repeats ~25 times
• Summaries are emitted when any threshold is hit: 100 events, 10 minutes (once summary_min_count is met), 50 distinct identity rows, or window eviction
• Suppression stops automatically when log frequency decreases
• One unified envelope across JSON, logfmt, and plain text, a readable message (the log’s masked template) plus auditty with suppressed, window, ids, and backfill_url
• Real id names, no noise: every id lists its complete value set inline, under the name it has in your logs
• Every original one click away: every correlation key is searchable inline, and every original line, context fields included, opens from the Vault via backfill_url
• Need to search on anything? The opt-in Total-Recall search index puts every distinct value of every extracted field into the summary itself, faceted and free-text searchable in your platform