Vault (Log Archive)

What Is Vault?

The Vault is where the suppressed lines go: a complete archive, in your own cloud storage, of exactly the log lines a summary stands for, in open columnar files you can query with any SQL engine. Forwarded lines are not duplicated into it; they already reach your platform. Every summary carries a link that opens its originals from the Vault in one click, and can replay them into your log platform when you need them there.

Enabling Vault

Add the vault configuration under the configMap section in your values.yaml:

configMap:
  forwarder:
    vault:
      enabled: true
      provider: "s3"
      bucket: "my-log-archive"
      prefix: "logs"              # Optional: object key prefix
      region: "us-east-1"         # AWS region (default: us-east-1)

configMap:
  forwarder:
    vault:
      enabled: true
      provider: "gcs"
      bucket: "my-log-archive"
      prefix: "logs"

configMap:
  forwarder:
    vault:
      enabled: true
      provider: "azure"
      bucket: "my-log-container"   # Azure Blob Storage container name
      prefix: "logs"
      azure_storage_account: "mystorageaccount"

S3-Compatible Storage (MinIO)

Vault also supports any S3-compatible storage backend:

configMap:
  forwarder:
    vault:
      enabled: true
      provider: "s3"
      bucket: "my-log-archive"
      s3_endpoint: "https://minio.internal:9000"
      s3_force_path_style: true    # Required for MinIO
      region: "us-east-1"

Durability and Retention Settings

The defaults archive every suppressed line without tuning. These are the knobs for how the archive behaves while cloud storage is slow or unreachable, and how long it is kept:

configMap:
  forwarder:
    vault:
      flush_interval: "5m"      # longest a batch waits before it is archived
      buffer_size: 10000        # or this many lines, whichever comes first
      spool_max_mb: 1024        # disk per node for lines the Vault has not taken yet
      buffer_wal: true          # lines in a batch survive an Edge pod crash
      cloud_buffer_wal: true    # ...and the loss of the node itself
      cloud_buffer_wal_interval: "30s"
      retention_days: 0         # 0: keep the archive forever
  policy:
    suppression:
      require_archive: true     # suppress only while the Vault is taking writes

• flush_interval / buffer_size: a backfill link works the moment the summary is written, before its batch is archived, so there is no reason to archive more often than the default; larger batches make fewer, more efficient archive files
• spool_max_mb: how much the node’s disk holds for the archive while cloud storage is unreachable (the chart’s daemonSet.spoolDir, which survives a pod restart and a node reboot); it is uploaded when storage returns. With require_archive on, suppression has already paused by then and every new line is going to your platform, so the bound only ever concerns lines suppressed before storage went away; if it is reached, the Edge pod says so once, with the amount
• buffer_wal / cloud_buffer_wal: lines accepted for the archive but not yet uploaded are kept on disk and, every cloud_buffer_wal_interval, copied to the bucket under <prefix>/_wal/, so they outlive the node they were written on; the hourly archive maintenance folds that copy into the archive
• retention_days: the compactor deletes archived data older than this; unset means the archive grows forever and your bucket’s own lifecycle rules are the only bound

Retrieving Archived Logs

Every suppression summary written by Edge includes a backfill_url field. Click the link to open the Replay Viewer with all original suppressed logs for that pattern and time window.

Every backfill link opens on the lines it names. Auditty suppresses a line only once the Vault can hold it (require_archive, on by default): while cloud storage is unreachable, or the license has expired, Edge pauses suppression and sends every line straight to your log platform, resuming the moment a write to the Vault lands. A summary is only ever written for lines that are archived. Lines accepted for the archive wait on the node’s own disk (up to spool_max_mb, 1 GiB by default) across pod restarts and node reboots, and cloud_buffer_wal keeps a copy in the bucket itself, so the path from suppression to archive to backfill holds end to end.

{
  "message": "Health check passed",
  "auditty": {
    "suppressed": 100,
    "window": { "start": "2026-01-15T09:00:00Z", "end": "2026-01-15T09:05:00Z" },
    "duration": "5m0s",
    "fingerprint": "a1b2c3d4",
    "backfill_url": "https://<your-hivemind-url>/backfill/replay?token=<TOKEN>&sig=<SIG>"
  }
}

Programmatic / Agentic Backfill API (JSON)

Raw (including suppressed) archived logs are available as JSON for scripts, agents, and compliance tooling, no browser required. For automated/agentic workflows use the versioned public API under /api/v1/backfill/*. These endpoints return JSON only and carry a stability guarantee (see below); build against them rather than the HTML backfill page or the UI’s internal routes.

Authentication is the token + sig pair already embedded in every suppression summary’s backfill_url (signed, tenant-scoped, and a capability in itself). No session or separate API key is required, so an agent can go straight from a Datadog/Splunk record to its originals.

Read flow: poll status until ready, then page through the records:

# token + sig come from the summary's backfill_url
GET /api/v1/backfill/status?token=<TOKEN>&sig=<SIG>
{ "status": "ready" }            # retry while "loading"

GET /api/v1/backfill/page?token=<TOKEN>&sig=<SIG>&page=1&sort=desc
{
  "records": [
    {
      "timestamp": "2026-01-01T00:00:01.123Z",
      "namespace": "payments",
      "workload": "api",
      "fingerprint": "a1b2c3d4",
      "node_id": "node-1",
      "source_uri": "...",
      "suppressed": true,
      "log_line": "Health check passed",
      "preserved_json": "{...}"
    }
  ],
  "totalCount": 100,
  "hasMore": false,
  "page": 1,
  "pageSize": 100,
  "queryTimeMs": 7
}
# paginate while hasMore === true

Restore flow: list targets, then replay the originals back into your platform:

GET /api/v1/backfill/integrations?token=<TOKEN>&sig=<SIG>
[ { "id": 12, "platform": "datadog", "name": "DD - prod" } ]

POST /api/v1/backfill/replay
{ "token": "<TOKEN>", "sig": "<SIG>", "integrationId": 12 }

{ "totalSent": 100, "totalFailed": 0, "durationMs": 842 }
# already replayed (idempotent): { "alreadyReplayed": true, "totalSent": 100, ... }
# pass "force": true to re-send

Alternative: by fingerprint (no summary needed). To query the archive for any fingerprint and date range, use GET /api/vault/logs?fingerprint=…&startDate=YYYY-MM-DD&page=1. This call is authenticated with your Auditty session (not a backfill token) and returns { status, fingerprint, start, end, records } with the same record shape; { status: "loading", retryAfterMs } (HTTP 202) while a cold fingerprint is being fetched, 503 while the archive is unavailable.

Field Predicates: Filtering by Value

The summary's ids object indexes only searchable correlation keys; bare magnitudes and measurements (latencies.kong, bytes_sent) are archived in full but aren't listed there. A field predicate lets you filter a backfill down to only the rows where a specific field in preserved_json matches a value or comparison, no need to promote the field first, and no need to page through every archived record by hand.

In the Replay Viewer, use the filter box above the table: enter a field name (dotted paths like latencies.kong work directly, no need to escape the dot), pick an operator, and a value.

latencies.kong > 100    # only rows where the kong measurement exceeded 100

This works with the paginated table view and the time histogram alike, and with the JSON API (GET /api/v1/backfill/page) once the predicate is baked into the token you pass.

Global Search: Field Names and Words

The Vault Explorer’s search box accepts free text, not just fingerprint hashes, and one search is answered from two places at once. Click any match to open its archived logs in the Replay Viewer, where a field predicate narrows to the values you are after.

Vault Explorer (Auditty UI)

Beyond following backfill links from summaries, you can browse all archived logs directly in Auditty. Navigate to Vault in the Auditty dashboard:

Replay to Platform

The Replay Viewer lets you send archived logs directly back to your logging platform. Supported platforms include Datadog, Splunk, Elasticsearch, Coralogix, Grafana Loki, New Relic, Sumo Logic, and any custom endpoint via Webhook. This is useful when you need the suppressed logs re-ingested for investigation or compliance.

Query the Archive with SQL (Open Table Format)

The Vault is a standard columnar archive in your bucket under a Hive-style partition layout (tenant/year/month/day/hour), open by construction, no proprietary format. In Vault Explorer, click Query with SQL, pick your engine, and copy the generated setup statement for your registered bucket. The full archive (suppressed originals included) is queryable with SQL, Spark, or any engine that reads open columnar data, with partition pruning and no Auditty components in the read path.

• Amazon Athena: partition projection: no crawler, no MSCK REPAIR, new vault files are queryable the moment they land
• Trino / Presto: hive connector external table
• Apache Iceberg (Spark SQL): creates an Iceberg table and registers the existing vault files in place via add_files (no copy, no rewrite)

Raw Archive (self-managed, full-fidelity)

Some teams need a complete, untouched copy of every log line in storage they control, independent of Auditty, for compliance or long-term retention. The raw archive is an optional, node-local copy: it writes the complete pre-suppression stream (every line, suppressed or not) to plain-text files alongside the reduced stream that flows to your observability backend.

configMap:
  forwarder:
    raw_archive:
      enabled: true
      directory: /var/log/auditty/raw   # node-local base dir (mount as a hostPath/volume)

The directory layout mirrors your source paths, so one recursive glob captures every stream. Point your downstream agent at it:

[sources.auditty_raw]
type    = "file"
include = ["/var/log/auditty/raw/**/*.log"]

Summary

• Vault archives every suppressed log line to cloud storage; non-suppressed events already reach your normal destination and are not duplicated into Vault
• Multi-cloud support: AWS S3, Google Cloud Storage, Azure Blob Storage, and S3-compatible backends
• Instant recovery via backfill links in suppression summaries or Auditty Vault Explorer
• Suppression summaries include a signed backfill URL to view original logs instantly
• One-click replay: Send archived logs directly to your logging platform from the Replay Viewer
• Suppress with confidence: every suppressed original is archived to your own storage, and while storage is unreachable suppression pauses and your platform receives every line
• Automatic optimization: Archived files are consolidated in the background for fast retrieval, no configuration needed
• Raw archive (optional): Write the complete pre-suppression stream to node-local plain-text files for a downstream agent to ship to storage you control, independently of Auditty