Detecting Data Exfiltration via Kubernetes Events

The event below was created by a user, not the kubelet. Both the image size and the source fields are attacker-controlled:

LAST SEEN   TYPE     REASON   OBJECT                   MESSAGE
5m          Normal   Pulled   pod/nginx-7d9b4c-xk9p2   Successfully pulled image "nginx:1.21.6" in 1.565s (1.565s including waiting). Image size: 190503180520 bytes.
6m          Normal   Pulled   pod/nginx                Successfully pulled image "nginx:1.21.6" in 7.425s (7.425s including waiting). Image size: 134469729 bytes.

Both events share reason: Pulled and the same image name. The malicious one references a pod that does not exist, and the image size is approximately 177GB against the real 128MB for nginx:1.21.6. Neither is a reliable automated signal. The reliable signal is in the audit log.

How Detection Works

Detection depends on two independent signals. The audit log answers who created an event. The event content answers what was written. At Metadata level the audit log captures the creator identity but not the message or numeric field values.

Signal 1: Creator Identity

Every event creation produces a ResponseComplete audit entry. An alert fires when an event is created by an identity outside the known controller allowlist. This works at Metadata level and requires no special audit policy changes.

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "3489e8b5-0ad7-4189-a85c-d6e404d43076",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/production/events?fieldManager=kubectl-client-side-apply&fieldValidation=Strict",
  "verb": "create",
  "user": {
    "username": "jane",
    "groups": ["system:masters", "system:authenticated"],
    "extra": { "authentication.kubernetes.io/credential-id": ["X509SHA256=492cca92bcc2c74153290f6e3343e5d84a3498ab011963797f6545e681ac70d0"] }
  },
  "sourceIPs": ["203.0.113.45"],
  "userAgent": "kubectl/v1.35.3 (darwin/arm64) kubernetes/6c1cd99",
  "objectRef": {
    "resource": "events",
    "namespace": "production",
    "name": "nginx-7d9b4c-xk9p2.18a45e7e41549f71",
    "apiVersion": "v1"
  },
  "responseStatus": { "metadata": {}, "code": 201 },
  "requestReceivedTimestamp": "2026-04-11T16:39:18.982281Z",
  "stageTimestamp": "2026-04-11T16:39:18.985681Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": ""
  }
}

The user.username is jane, not a controller service account. That is the trigger. The source.component: kubelet value inside the event object is not visible here and carries no weight. The API server stores whatever the creator submits.

Signal 2: Event Content

The first path is to query events directly via the API. This shows the full content of every event currently in etcd:

kubectl get events -n <namespace> -o json \
  | jq '.items[] | {name: .metadata.name, reason: .reason, message: .message, source: .source}'

{
  "name": "nginx-7d9b4c-xk9p2.18a45e7e41549f71",
  "reason": "Pulled",
  "message": "Successfully pulled image \"nginx:1.21.6\" in 1.565s (1.565s including waiting). Image size: 190503180520 bytes.",
  "source": { "component": "kubelet", "host": "worker-node-1" }
}

The data is in the numeric value 190503180520. A1Z26 decoding maps each two-digit pair to a letter: 19=S, 05=E, 03=C, 18=R, 05=E, 20=T. The limitation of this path is that events expire out of etcd. Direct inspection only works while the event is still present.

The second path is to raise the audit policy to Request level for the events resource. This captures the full request body into the audit log at write time, before the event expires:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: Request
    resources:
      - group: ""
        resources: ["events"]
    verbs: ["create", "patch"]
  - level: Metadata
    resources:
      - group: ""
        resources: ["*"]

With this policy, the audit record includes requestObject containing the full event body, captured regardless of whether the event later expires:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Request",
  "auditID": "5e2c019d-1608-4285-ba69-f1e1941a234e",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/production/events?fieldManager=kubectl-client-side-apply&fieldValidation=Strict",
  "verb": "create",
  "user": {
    "username": "jane",
    "groups": ["system:masters", "system:authenticated"],
    "extra": { "authentication.kubernetes.io/credential-id": ["X509SHA256=492cca92bcc2c74153290f6e3343e5d84a3498ab011963797f6545e681ac70d0"] }
  },
  "sourceIPs": ["203.0.113.45"],
  "objectRef": {
    "resource": "events",
    "namespace": "production",
    "name": "nginx-7d9b4c-xk9p2.18a45e7e41549f71",
    "apiVersion": "v1"
  },
  "responseStatus": { "metadata": {}, "code": 201 },
  "requestObject": {
    "reason": "Pulled",
    "message": "Successfully pulled image \"nginx:1.21.6\" in 1.565s (1.565s including waiting). Image size: 190503180520 bytes.",
    "source": { "component": "kubelet", "host": "worker-node-1" },
    "type": "Normal"
  },
  "requestReceivedTimestamp": "2026-04-11T16:39:18.982281Z",
  "stageTimestamp": "2026-04-11T16:39:18.985681Z",
  "annotations": { "authorization.k8s.io/decision": "allow", "authorization.k8s.io/reason": "" }
}

A SIEM rule scanning requestObject.message for numeric values outside the plausible container image size range catches A1Z26-encoded payloads without knowing the cipher.

Numeric anomalies in message

Attackers may encode data using A1Z26 or similar ciphers within numeric values in the message field:

{
  "requestObject": {
    "reason": "Pulled",
    "message": "Successfully pulled image \"nginx:1.21.6\" in 1.565s (1.565s including waiting). Image size: 190503180520 bytes.",
    "source": { "component": "kubelet", "host": "worker-node-1" },
    "type": "Normal"
  }
}

The value 190503180520 is far outside the plausible range for a container image size and contains encoded data.

Image digest channel

An attacker encodes a credential such as an AWS access key ID as hex and embeds it in the sha256: position of a pinned image digest:

Successfully pulled image "nginx:1.21.6@sha256:414b4941494f53464f444e4e374558414d504c45000000000000000000000000" in 1.565s (1.565s including waiting). Image size: 134469729 bytes.

The digest 414b4941494f53464f444e4e374558414d504c45000000000000000000000000 hex-decodes to AKIAIOSFODNN7EXAMPLE padded with null bytes. The message format, image size, and digest length are all correct. Detection at Request level requires extracting the digest from requestObject.message and comparing it against the known real digest for that image tag. Any mismatch is the signal.

Hex-encoded data in reportingInstance

This field is not displayed by kubectl get events. In -o wide output it appears in the SOURCE column as kubelet, <value> alongside the component name, but the hex string blends into the wide table and is easy to overlook. Only -o json surfaces it cleanly as a dedicated field. An attacker stores a hex-encoded service account token fragment here while the visible event message remains a normal pull result:

{
  "requestObject": {
    "message": "Successfully pulled image \"order-service:v2.4.1\" in 2.103s (2.103s including waiting). Image size: 134469729 bytes.",
    "reportingComponent": "kubelet",
    "reportingInstance": "65794a68624763694f694a53557a49314e694973496d74705a434936496b4e4b596e6857566b4a5a626a45336444464d513052334f4863774e6c6c5a52454e7a4d304e55634846785a30316b53456b7453453835646c6b6966512e65794a68645751694f6c73696148523063484d364c79397264574a..."
  }
}

The reportingInstance value hex-decodes to the first chunk of a service account JWT token for system:serviceaccount:production:deployer. Three consecutive events carry the full token across three chunks, reassembled on retrieval. Real kubelet events set reportingInstance to the node hostname. Any value that is not a valid cluster hostname warrants inspection.

Detection requires an explicit query. The default event list will not surface this:

kubectl get events -n <namespace> -o json \
  | jq '.items[] | select(.reportingInstance != null and (.reportingInstance | length) > 253) | {name: .metadata.name, reportingInstance}'

The hostname length limit of 253 characters is the reliable threshold. Hex-encoded payloads are at minimum hundreds of characters a 933-character JWT produces 1866 hex characters, split across three chunks of ~622 each. A hostname regex is not sufficient because hex strings consist entirely of [0-9a-f], which passes any [a-z0-9] pattern.

The query above only works while the event is still in etcd. The audit log at Metadata level genuinely hides reportingInstance, the requestObject is absent and the canary value does not appear anywhere in the audit entry. At Request level the audit log captures requestObject.reportingInstance permanently at write time, before the event expires. A SIEM rule on that field with the same length threshold catches the channel regardless of event retention:

grep '"resource":"events"' /var/log/kubernetes/audit.log \
  | grep '"verb":"create"' \
  | jq 'select((.requestObject.reportingInstance // "" | length) > 253) | {user: .user.username, name: .objectRef.name, reportingInstance: .requestObject.reportingInstance}'

Detection Queries

Assuming API server audit logs are shipped to Loki with the label {job="k8s-audit"}.

Detect non-controller event creation

Filter for event creation by identities outside the known controller allowlist:

logcli query '{job="k8s-audit"} |= "resource":"events" |= "verb":"create" !~ "username":"system:serviceaccount:kube-system:"' \
  --output=jsonl \
  | jq -r '.line | fromjson | {user: .user.username, event: .objectRef.name, timestamp: .requestReceivedTimestamp}'

{
  "user": "jane",
  "event": "nginx-7d9b4c-xk9p2.18a45e7e41549f71",
  "timestamp": "2026-04-11T16:39:18.982281Z"
}

Detect anomalous reportingInstance length

Monitor for reportingInstance values exceeding the hostname length limit of 253 characters:

logcli query '{job="k8s-audit"} |= "resource":"events" |= "verb":"create"' \
  --output=jsonl \
  | jq -r '.line | fromjson | select((.requestObject.reportingInstance // "" | length) > 253) | {user: .user.username, event: .objectRef.name, reportingInstance: .requestObject.reportingInstance}'

Known Legitimate Event Writers

Any event creation from an identity outside this list warrants investigation:

Audit Policy Requirements

A minimal audit policy that captures both creator identity and event content:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: Metadata
    resources:
      - group: ""
        resources: ["events"]
  - level: Request
    verbs: ["create", "patch"]
    resources:
      - group: ""
        resources: ["events"]

Metadata level captures the creator identity. Request level is required to inspect requestObject for encoded data in the message or reportingInstance fields.