Detecting kubectl debug Activity via Audit Logs

kubectl debug runs in two modes. The first attaches an ephemeral container directly to a running pod without restarting it. The second creates a new pod on a specific node with host access. Both modes produce audit events that do not appear during normal workload operations.

Signal 1: Ephemeral Container Injection

When kubectl debug runs against a running pod, kubectl first reads the current pod spec, then patches the pods/ephemeralcontainers subresource to inject the debug container. No controller or scheduler touches this subresource during normal cluster operation. A patch event against it is always user-initiated.

1. Initial pod read

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/production/pods/order-service-589fc77b9d-82k28",
  "verb": "get",
  "user": {
    "username": "jane",
    "groups": ["system:masters", "system:authenticated"],
    "extra": { "authentication.kubernetes.io/credential-id": ["X509SHA256=492cca92bcc2c74153290f6e3343e5d84a3498ab011963797f6545e681ac70d0"] }
  },
  "sourceIPs": ["203.0.113.45"],
  "userAgent": "kubectl/v1.35.3 (darwin/arm64) kubernetes/6c1cd99",
  "objectRef": {
    "resource": "pods",
    "namespace": "production",
    "name": "order-service-589fc77b9d-82k28",
    "apiVersion": "v1"
  },
  "responseStatus": { "metadata": {}, "code": 200 },
  "requestReceivedTimestamp": "2026-04-11T15:47:09.499970Z",
  "stageTimestamp": "2026-04-11T15:47:09.503218Z",
  "annotations": { "authorization.k8s.io/decision": "allow", "authorization.k8s.io/reason": "" }
}

2. Ephemeral container injection

Arrives milliseconds later from the same user and source IP:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/production/pods/order-service-589fc77b9d-82k28/ephemeralcontainers",
  "verb": "patch",
  "user": {
    "username": "jane",
    "groups": ["system:masters", "system:authenticated"],
    "extra": { "authentication.kubernetes.io/credential-id": ["X509SHA256=492cca92bcc2c74153290f6e3343e5d84a3498ab011963797f6545e681ac70d0"] }
  },
  "sourceIPs": ["203.0.113.45"],
  "userAgent": "kubectl/v1.35.3 (darwin/arm64) kubernetes/6c1cd99",
  "objectRef": {
    "resource": "pods",
    "namespace": "production",
    "name": "order-service-589fc77b9d-82k28",
    "apiVersion": "v1",
    "subresource": "ephemeralcontainers"
  },
  "responseStatus": { "metadata": {}, "code": 200 },
  "requestReceivedTimestamp": "2026-04-11T15:47:09.553192Z",
  "stageTimestamp": "2026-04-11T15:47:09.560775Z",
  "annotations": { "authorization.k8s.io/decision": "allow", "authorization.k8s.io/reason": "" }
}

The subresource: ephemeralcontainers field in objectRef is the primary signal. This is the event to alert on.

3. Log retrieval

Once the container starts, kubectl fetches session output via pods/log:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "c3d4e5f6-a7b8-9012-cdef-123456789012",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/production/pods/order-service-589fc77b9d-82k28/log",
  "verb": "get",
  "user": {
    "username": "jane",
    "groups": ["system:masters", "system:authenticated"],
    "extra": { "authentication.kubernetes.io/credential-id": ["X509SHA256=492cca92bcc2c74153290f6e3343e5d84a3498ab011963797f6545e681ac70d0"] }
  },
  "sourceIPs": ["203.0.113.45"],
  "userAgent": "kubectl/v1.35.3 (darwin/arm64) kubernetes/6c1cd99",
  "objectRef": {
    "resource": "pods",
    "namespace": "production",
    "name": "order-service-589fc77b9d-82k28",
    "apiVersion": "v1",
    "subresource": "log"
  },
  "responseStatus": { "metadata": {}, "code": 200 },
  "requestReceivedTimestamp": "2026-04-11T15:47:14.591913Z",
  "stageTimestamp": "2026-04-11T15:47:14.598203Z",
  "annotations": { "authorization.k8s.io/decision": "allow", "authorization.k8s.io/reason": "" }
}

The pods/log access from the same user and source IP arriving seconds after the pods/ephemeralcontainers patch corroborates that the debug session ran and produced output.

Signal 2: Node Debug Pod Creation

kubectl debug node/<node> produces a pod with these characteristics:

Name follows the pattern node-debugger-<node-name>-<random-suffix>, generated by kubectl
Namespace is wherever the caller specifies with -n, defaulting to default
No ownerReferences — not managed by any controller
Persists in Completed state after the session ends and must be removed manually

Pod creation event

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "d4e5f6a7-b8c9-0123-defa-234567890123",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/default/pods",
  "verb": "create",
  "user": {
    "username": "jane",
    "groups": ["system:masters", "system:authenticated"],
    "extra": { "authentication.kubernetes.io/credential-id": ["X509SHA256=492cca92bcc2c74153290f6e3343e5d84a3498ab011963797f6545e681ac70d0"] }
  },
  "sourceIPs": ["203.0.113.45"],
  "userAgent": "kubectl/v1.35.3 (darwin/arm64) kubernetes/6c1cd99",
  "objectRef": {
    "resource": "pods",
    "namespace": "default",
    "name": "node-debugger-worker-node-1-qqxwt",
    "apiVersion": "v1"
  },
  "responseStatus": { "metadata": {}, "code": 201 },
  "requestReceivedTimestamp": "2026-04-11T15:46:26.960836Z",
  "stageTimestamp": "2026-04-11T15:46:26.967515Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "",
    "pod-security.kubernetes.io/enforce-policy": "privileged:latest"
  }
}

The pod-security.kubernetes.io/enforce-policy: privileged:latest annotation confirms the pod was admitted under the privileged policy, consistent with a node debug pod that mounts the host filesystem.

Pod deletion event

When the pod is removed, the audit log records a delete event for the same pod name:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "e5f6a7b8-c9d0-1234-efab-345678901234",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/default/pods/node-debugger-worker-node-1-qqxwt",
  "verb": "delete",
  "user": {
    "username": "jane",
    "groups": ["system:masters", "system:authenticated"],
    "extra": { "authentication.kubernetes.io/credential-id": ["X509SHA256=492cca92bcc2c74153290f6e3343e5d84a3498ab011963797f6545e681ac70d0"] }
  },
  "sourceIPs": ["203.0.113.45"],
  "userAgent": "kubectl/v1.35.3 (darwin/arm64) kubernetes/6c1cd99",
  "objectRef": {
    "resource": "pods",
    "namespace": "default",
    "name": "node-debugger-worker-node-1-qqxwt",
    "apiVersion": "v1"
  },
  "responseStatus": { "metadata": {}, "code": 200 },
  "requestReceivedTimestamp": "2026-04-11T15:51:11.595603Z",
  "stageTimestamp": "2026-04-11T15:51:11.602418Z",
  "annotations": { "authorization.k8s.io/decision": "allow", "authorization.k8s.io/reason": "" }
}

The create and delete pair for a node-debugger-* pod from the same user within a short window is the full audit footprint of a node debug session.

Detection Queries

Assuming API server audit logs are shipped to Loki with the label {job="k8s-audit"}.

Detect ephemeral container injection

Filter for patch events against the pods/ephemeralcontainers subresource:

logcli query '{job="k8s-audit"} |= "subresource":"ephemeralcontainers" |= "verb":"patch"' \
  --output=jsonl \
  | jq -r '.line | fromjson | {user: .user.username, pod: .objectRef.name, timestamp: .requestReceivedTimestamp}'

Detect node debug pod creation

Filter for pod creation events matching the node-debugger- prefix:

logcli query '{job="k8s-audit"} |= "verb":"create" |= "resource":"pods" |= "node-debugger-"' \
  --output=jsonl \
  | jq -r '.line | fromjson | {user: .user.username, pod: .objectRef.name, timestamp: .requestReceivedTimestamp}'

Known Legitimate Debug Users

Not every debug session is hostile. The following are common authorized uses:

Identity	Typical use
Cluster operators	Troubleshooting application issues in production
Break-glass users	Emergency access during incidents
SRE teams	Investigating node-level resource exhaustion

The detection signal is debug activity outside these patterns. A developer account using kubectl debug on a production pod without a corresponding incident ticket should trigger investigation.

Audit Policy Requirements

A minimal audit policy that captures both signals:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: Metadata
    resources:
      - group: ""
        resources: ["pods"]

Metadata level is sufficient for both signals. The subresource field in objectRef identifies ephemeral container injection, and the node-debugger- prefix in objectRef.name identifies node debug pods.

Limitations

Both signals depend on audit logging capturing the pods resource at Metadata level or higher. A minimal policy that covers both:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: Metadata
    resources:
      - group: ""
        resources: ["pods"]

The node-debugger- prefix is generated by the kubectl client. A user creating a node debug pod via a raw API call can choose any pod name, bypassing that signal. The pods/ephemeralcontainers subresource signal is not bypassable in the same way because the subresource is fixed by the Kubernetes API regardless of the client used.