Consuming violations using Pubsub

Feature State: Gatekeeper version v3.13+ (alpha)

❗ This feature is alpha, subject to change (feedback is welcome!).

Description

This feature pushes audit violations to a pubsub service. Users can subscribe to pubsub service to consume violations.

To gain insights into different methods of obtaining audit violations and the respective trade-offs for each approach, please refer to Reading Audit Results.

Enabling Gatekeeper to export audit violations

Install prerequisites such as a pubsub tool, a message broker etc.

Setting up audit with pubsub enabled

In the audit deployment, set the --enable-pub-sub flag to true to publish audit violations. Additionally, use --audit-connection (defaults to audit-connection) and --audit-channel(defaults to audit-channel) flags to allow audit to publish violations using desired connection onto desired channel. --audit-connection must be set to the name of the connection config, and --audit-channel must be set to name of the channel where violations should get published.

A ConfigMap that contains provider and config fields in data is required to establish connection for sending violations over the channel. Following is an example ConfigMap to establish a connection that uses Dapr to publish messages:

apiVersion: v1
kind: ConfigMap
metadata:
  name: audit-connection
  namespace: gatekeeper-system
data:
  provider: "dapr"
  config: |
    {
      "component": "pubsub"
    }

• provider field determines which tool/driver should be used to establish a connection. Valid values are: dapr
• config field is a json object that configures how the connection is made. E.g. which queue messages should be sent to.

Available Pubsub drivers

Dapr: https://dapr.io/

Quick start with publishing violations using Dapr and Redis

Prerequisites

1. Install Dapr

   To install Dapr with specific requirements and configuration, please refer to Dapr docs.

   [!IMPORTANT]

   • Make sure to set SIDECAR_DROP_ALL_CAPABILITIES environment variable on dapr-sidecar injector pod to true to avoid getting PodSecurity violation errors for the injected sidecar container as Gatekeeper by default requires workloads to run with restricted policy. If using helm charts to install Dapr, you can use --set dapr_sidecar_injector.sidecarDropALLCapabilities=true.
   • Additionally, configure appropriate seccompProfile for sidecar containers injected by Dapr to avoid getting PodSecurity violation errors. We are setting required Dapr annotation for audit pod while deploying Gatekeeper later in this quick start to avoid getting PodSecurity violation error.

   Dapr is installed with mtls enabled by default, for more details on the same please refer to Dapr security.

2. Install Redis

   Please refer to this guide to install Redis.

   Redis is used for example purposes only. Dapr supports many different state store options. To install Redis with TLS, please refer to this doc.

Configure a sample subscriber to receive violations

1. Create fake-subscriber namespace and redis secret

   kubectl create ns fake-subscriber
   # creating redis secret in subscriber namespace to allow Dapr sidecar to connect to redis instance
   kubectl get secret redis --namespace=default -o yaml | sed 's/namespace: .*/namespace: fake-subscriber/' | kubectl apply -f -

2. Create Dapr pubsub component

   kubectl apply -f - <<EOF
   apiVersion: dapr.io/v1alpha1
   kind: Component
   metadata:
     name: pubsub
     namespace: fake-subscriber
   spec:
     type: pubsub.redis
     version: v1
     metadata:
     - name: redisHost
       value: redis-master.default.svc.cluster.local:6379
     - name: redisPassword
       secretKeyRef: 
         name: redis
         key: redis-password
   EOF

   Please use this guide to properly configure Redis pubsub component for Dapr.

3. Deploy subscriber application

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: sub
     namespace: fake-subscriber
     labels:
       app: sub
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: sub
     template:
       metadata:
         labels:
           app: sub
         annotations:
           dapr.io/enabled: "true"
           dapr.io/app-id: "subscriber"
           dapr.io/enable-api-logging: "true"
           dapr.io/app-port: "6002"
       spec:
         containers:
         - name: go-sub
           image: fake-subscriber:latest
           imagePullPolicy: Never

   [!IMPORTANT] Please make sure fake-subscriber image is built and available in your cluster. Dockerfile to build image for fake-subscriber is under gatekeeper/test/fake-subscriber.

Configure Gatekeeper with Pubsub enabled

1. Create Gatekeeper namespace, and create Dapr pubsub component and Redis secret in Gatekeeper's namespace (gatekeeper-system by default). Please make sure to update gatekeeper-system namespace for the next steps if your cluster's Gatekeeper namespace is different.

   kubectl create namespace gatekeeper-system
   kubectl get secret redis --namespace=default -o yaml | sed 's/namespace: .*/namespace: gatekeeper-system/' | kubectl apply -f -
   kubectl apply -f - <<EOF
   apiVersion: dapr.io/v1alpha1
   kind: Component
   metadata:
     name: pubsub
     namespace: gatekeeper-system
   spec:
     type: pubsub.redis
     version: v1
     metadata:
     - name: redisHost
       value: redis-master.default.svc.cluster.local:6379
     - name: redisPassword
       secretKeyRef:
         name: redis
         key: redis-password
   EOF

2. To upgrade or install Gatekeeper with --enable-pub-sub set to true, --audit-connection set to audit-connection, --audit-channel set to audit-channel on audit pod.

   # auditPodAnnotations is used to add annotations required by Dapr to inject sidecar to audit pod
   echo 'auditPodAnnotations: {dapr.io/enabled: "true", dapr.io/app-id: "audit", dapr.io/metrics-port: "9999", dapr.io/sidecar-seccomp-profile-type: "RuntimeDefault"}' > /tmp/annotations.yaml
   helm upgrade --install gatekeeper gatekeeper/gatekeeper --namespace gatekeeper-system \
   --set audit.enablePubsub=true \
   --set audit.connection=audit-connection \
   --set audit.channel=audit-channel \
   --values /tmp/annotations.yaml

   Note: Verify that after the audit pod is running there is a Dapr sidecar injected and running along side manager container.

3. Create connection config to establish a connection.

   kubectl apply -f - <<EOF
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: audit-connection
     namespace: gatekeeper-system
   data:
     provider: "dapr"
     config: |
       {
         "component": "pubsub"
       }
   EOF

   Note: Name of the connection configMap must match the value of --audit-connection for it to be used by audit to publish violation. At the moment, only one connection config can exists for audit.

4. Create the constraint templates and constraints, and make sure audit ran by checking constraints. If constraint status is updated with information such as auditTimeStamp or totalViolations, then audit has ran at least once. Additionally, populated TOTAL-VIOLATIONS field for all constraints while listing constraints also indicates that audit has ran at least once.

   kubectl get constraint
   NAME                 ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
   pod-must-have-test                        0

5. Finally, check the subscriber logs to see the violations received.

   kubectl logs -l app=sub -c go-sub -n fake-subscriber 
   2023/07/18 20:16:41 Listening...
   2023/07/18 20:37:20 main.PubsubMsg{ID:"2023-07-18T20:37:19Z", Details:map[string]interface {}{"missing_labels":[]interface {}{"test"}}, EventType:"violation_audited", Group:"constraints.gatekeeper.sh", Version:"v1beta1", Kind:"K8sRequiredLabels", Name:"pod-must-have-test", Namespace:"", Message:"you must provide labels: {\"test\"}", EnforcementAction:"deny", ConstraintAnnotations:map[string]string(nil), ResourceGroup:"", ResourceAPIVersion:"v1", ResourceKind:"Pod", ResourceNamespace:"nginx", ResourceName:"nginx-deployment-58899467f5-j85bs", ResourceLabels:map[string]string{"app":"nginx", "owner":"admin", "pod-template-hash":"58899467f5"}}

Violations

The audit pod publishes violations in following format:

{
  "id": "2023-07-18T21:21:52Z",
  "details": {
    "missing_labels": [
      "test"
    ]
  },
  "eventType": "violation_audited",
  "group": "constraints.gatekeeper.sh",
  "version": "v1beta1",
  "kind": "K8sRequiredLabels",
  "name": "pod-must-have-test",
  "message": "you must provide labels: {\"test\"}",
  "enforcementAction": "deny",
  "resourceAPIVersion": "v1",
  "resourceKind": "Pod",
  "resourceNamespace": "nginx",
  "resourceName": "nginx-deployment-cd55c47f5-2b84x",
  "resourceLabels": {
    "app": "nginx",
    "pod-template-hash": "cd55c47f5"
  }
}