Cloud Control Operator

Cloud Control Operator is a default component part of any Userplane & Controlplane deployment. The operator is responsible for ongoing management activities for which Kubernetes itself has no functionality. Currently the operator contains the following modules:

Feature	Description
Readiness Based Primary	Responsible for selection of a primary Pod to be used in Service objects which are configured to be sticky towards a single Pod
Secrets Manager	Responsible for server-side generation of Secrets for which no static or preprovisioned value is present
Serverside Manager	Responsible for server-side management of miscellaneous objects
STEK Manager	Responsible for generation & rotation of STEK keys used by dnsdist `DoH` & `DoT` listeners

Readiness Based Primary

The Readiness Based Primary (RBP) module constantly monitors the Pods which belong to Deployments & Statefulsets with a label cloudcontrol.powerdns.com/rbp-primary: true and applies an algorithm which ensures that for each of these Deployments & Statefulsets there is always 1 Pod which has a label cloudcontrol.powerdns.com/rbp-service: true present. This same label is added as a selector to the corresponding Service object (for example, the Auth API Service) to achieve stickiness towards that single Pod.

The algorithm used to determine the Pod which should receive the traffic is based on the following rules:

The Pod must be in Ready state
The Pod is preferred to be from the most recent iteration of the Deployment or Statefulset during rolling upgrades
If the currently selected Pod is still eligible based on above rules, it remains the selected Pod (ie: No swapping when there is no reason for it)

The components currently utilizing the RBP functionality are as follows:

Each of these has a toggle disableSticky which can be set to true to disable the RBP functionality. Disabling RBP means the traffic distribution will be controlled by the default Service behaviour on the Kubernetes cluster, likely round-robin.

Warning

The current set of Services which utilize the RBP functionality has been carefully curated. It is strongly recommended to avoid setting the disableSticky parameter to true unless there is a good reason to do so.

Secrets Manager

The Secrets Manager is a module which is responsible for generating Secrets for which no specific value has been configured via static value or preprovisioned Secret. Examples of such Secret values which may need to be generated are the API key for Authoritative Server instances (which can be overridden by users) or the Secret which Cloud Control agents use to authenticate themselves on the internal NATS mesh.

This mechanism exists for 2 reasons:

It allows Cloud Control to be in line with the server-side approach which GitOps tooling like Argo CD require and Helm is moving towards in the future
It improves security by preventing Helm from having to generate a value on a client outside of the Kubernetes cluster and then having to communicate that value to the cluster to be injected into a Secret object

To implement this feature, Helm & Secrets Manager work together as follows:

Helm is responsible for creating a ConfigMap which contains a definition of the Secrets which the Secrets Manager will be responsible for. These ConfigMaps have the same naming convention as the Deployments / Statefulsets which utilize them, with a suffix of -ccsm, for example the auth-testauth deployment will be accompanied by a ConfigMap named auth-testauth-ccsm. Inside this ConfigMap, a secretdef item contains the definitions needed by the Secrets Manager.

The auth-testauth-ccsm example (an Authoritative Server instance set named testauth) would have the following contents if no API key is specified during deployment:

auth-testauth-ccsm:
  entries:
    - key: api_key
      generator: alphanumeric
      length: 32
      legacy:
        secret: auth-testauth
        key: api_key

Each of these ConfigMaps containing definitions of Secrets (identified using a specific label) are constantly monitored by the Secrets Manager. The Secrets Manager ensures a Secret is present with the name of the root node (auth-testauth-ccsm in the above example) and key value pairs based on the definitions under the entries node.

As suggested by the above example definition, there are several ways the contents of a Secret key can be decided upon by the Secrets Manager:

Generated: If there is no existing generated value and there are no prioritized sources of obtaining the value, a new one is generated using the generator specified under generator with length length.
Legacy: To facilitate upgrades from Cloud Control releases without Secrets Manager support, the Legacy mechanism is used to ensure any values previously generated client side by Helm are copied into the new Secrets Manager controlled Secret. This exists to ensure these upgrades do not experience unexpected changes to generated Secret values.
Static: In some cases a static value may accompany a Generated/Legacy value. For example: A static username could accompany a generated password.

Serverside Manager

The Serverside Manager is a module which is responsible for managing miscellaneous objects for which direct control via Helm and/or GitOps tooling is not desirable. The following Kubernetes objects are currently managed via this module:

EndpointSlices created for Resolver configurations

Helm and/or the GitOps tooling is responsible for creating a ConfigMap which contains a definition of the objects which the Serverside Manager will be responsible for. These ConfigMaps have the same naming convention as the instance sets which utilize them, with a suffix of -ssm, for example a Resolver instance set named testresolv will be accompanied by a ConfigMap named resolver-testresolv-ssm. Inside this ConfigMap, a ssmdef item contains the definitions needed by the Serverside Manager to manage the corresponding Kubernetes objects.

Each of these ConfigMaps containing definitions of objects are constantly monitored by the Serverside Manager, which ensures the corresponding Kubernetes objects are always present & updated accordingly if changes are detected.

STEK Manager

STEK keys are used by dnsdist DoH and DoT listeners to attempt to mitigate the overhead incurred while negotiating new TLS sessions by enabling the resumption of previously negotiated sessions. A detailed description of this mechanism can be found in the dnsdist documentation.

To simplify the usage of STEK keys in a Cloud Control deployment, the STEK Manager assumes responsibility for generating a single set of STEK keys for each dnsdist instance set and rotating them when necessary. Since all dnsdist Pods belonging to the same instance set apply the same set of STEK keys, session resumption is enabled across all dnsdist Pods within that instance set.

If you wish to override this behaviour by providing your own set of STEK keys, both DoH and DoT listeners have a stekSecret parameter which can be used to specify your own Secret to grab STEK keys from (changes to this Secret are monitored at runtime and STEK keys are updated in dnsdist accordingly if modifications are detected).

Configuration Reference

Cloud Control Operator can be configured as top-level item in the Userplane, Controlplane & Monitoring Helm Charts. For example:

operator:
  nodeSelector:
    mylabel: somevalue

Above will apply the node selector to the Operator Deployment.

The full list of parameters which can be used to configure Cloud Control Operator:

Parameter	Type	Default	Description
`affinity`	`k8s:`Affinity		pod affinity (Kubernetes docs: Affinity and anti-affinity). If unset, a default anti-affinity is applied using `antiAffinityPreset` to spread pods across nodes
`antiAffinityPreset`	`string`	`"preferred"`	pod anti affinity preset. Available options: `"preferred"` `"required"`
`containerSecurityContext`	`k8s:`SecurityContext	`allowPrivilegeEscalation: false readOnlyRootFilesystem: true seccompProfile: type: RuntimeDefault capabilities: drop: - "ALL"`	SecurityContext applied to each container
`hostNetwork`	`boolean`	`false`	Use host networking for pods
`logLevel`	`string`	`"info"`	Verbosity of logging for the operator container. Available options: `"debug"` `"info"` `"warn"` `"error"`
`minReadySeconds`	`integer`		Minimum number of seconds for which a newly created pod should be ready without any of its containers crashing, for it to be considered available. Applies Kubernetes defaults when not set, which is 0 seconds
`nodeSelector`	`k8s:`NodeSelector	`{}`	Kubernetes pod nodeSelector
`podAnnotations`	`k8s:`Annotations	`{}`	Annotations to be added to each pod
`podLabels`	`k8s:`Labels	`{}`	Labels to be added to each pod
`podPreemptionPolicy`	`string`		Policy for preempting pods with lower priority
`podPriorityClassName`	`string`		Name of PriorityClass to be assigned to each pod
`podRuntimeClassName`	`string`		Name of RuntimeClass to be assigned to each pod
`podSchedulerName`	`string`		Name of scheduler responsible for dispatching pods
`podSecurityContext`	`k8s:`PodSecurityContext	`fsGroup: 953 runAsUser: 953 runAsGroup: 953 runAsNonRoot: true`	SecurityContext applied to each pod
`podTerminationGracePeriodSeconds`	`integer`		Duration after which terminating pods are stopped immediately if they do not exit gracefully
`progressDeadlineSeconds`	`integer`		The maximum time in seconds for a deployment to make progress before it is considered to be failed. Applies Kubernetes defaults when not set, which is 600 seconds
`rbp`	Readiness Based Primary		Tuning parameters for the Readiness Based Primary module
`resources`	`k8s:`Resources	`limits: cpu: 250m memory: 256Mi`	Resources allocated to the operator container if `resourceDefaults` (global) is `true`
`secretsmanager`	Secrets Manager		Tuning parameters for the Secrets Manager module
`serversidemanager`	Serverside Manager		Tuning parameters for the Serverside Manager module
`stekmanager`	STEK Manager		Tuning parameters for the STEK Manager module
`tolerations`	List of `k8s:`Tolerations	`[]`	Kubernetes pod Tolerations
`topologySpreadConstraints`	List of `k8s:`TopologySpreadConstraint	`[]`	Kubernetes pod topology spread constraints

Readiness Based Primary

The full list of parameters which can be used to configure this module:

Parameter	Type	Default	Description
`heartbeat`	`integer`	`5`	Number of seconds between consecutive heartbeats. For each heartbeat the module will check all relevant objects and perform synchronization if necessary. Note: The heartbeat is a fallback mechanism, primary method of synchronization is based on events emitted by the Kubernetes API which the module acts upon instantly. Lowering this value will lead to higher load on the Kubernetes API server.

For example:

operator:
  rbp:
    heartbeat: 10

Secrets Manager

The full list of parameters which can be used to configure this module:

Parameter	Type	Default	Description
`heartbeat`	`integer`	`60`	Number of seconds between consecutive heartbeats. For each heartbeat the module will check all relevant objects and perform synchronization if necessary. Note: The heartbeat is a fallback mechanism, primary method of synchronization is based on events emitted by the Kubernetes API which the module acts upon instantly. Lowering this value will lead to higher load on the Kubernetes API server.

For example:

operator:
  secretsmanager:
    heartbeat: 40

Serverside Manager

The full list of parameters which can be used to configure this module:

Parameter	Type	Default	Description
`heartbeat`	`integer`	`60`	Number of seconds between consecutive heartbeats. For each heartbeat the module will check all relevant objects and perform synchronization if necessary. Note: The heartbeat is a fallback mechanism, primary method of synchronization is based on events emitted by the Kubernetes API which the module acts upon instantly. Lowering this value will lead to higher load on the Kubernetes API server.

For example:

operator:
  serversidemanager:
    heartbeat: 45

STEK Manager

The full list of parameters which can be used to configure this module:

Parameter	Type	Default	Description
`heartbeat`	`integer`	`60`	Number of seconds between consecutive heartbeats. For each heartbeat the module will check all relevant objects and perform synchronization if necessary. Note: The heartbeat is a fallback mechanism, primary method of synchronization is based on events emitted by the Kubernetes API which the module acts upon instantly. Lowering this value will lead to higher load on the Kubernetes API server.

For example:

operator:
  stekmanager:
    heartbeat: 50