Multi-Region Operations Guide

What You'll Learn

By the end of this guide, you will understand how to:

• Organize PowerDNS infrastructure by geographic region using label taxonomy
• Customize the default regional dashboards for your organization
• Extend navigation menus with organization-specific items
• Implement REGO authorization policies for regional access control
• Deploy a complete multi-region SPOG configuration
• Use advanced filter expressions for multi-dimensional organization
• Apply best practices for managing large-scale regional deployments

This is an integration guide that demonstrates how SPOG's label-based architecture enables flexible geographic organization of your global DNS infrastructure.

The Business Scenario

You are a platform engineer at a global company managing PowerDNS infrastructure across multiple continents. Your organization has:

• Americas operations: DNS clusters in US East, US West, and US Central
• Europe operations: DNS clusters in EU West, EU Central, and EU North
• Asia Pacific operations: DNS clusters in AP North, AP South, and AP Southeast

Each region has: - Production, staging, and development environments - Both authoritative and recursor DNS servers - Regional teams responsible for day-to-day operations - Compliance requirements for data residency (e.g., EU data must stay in EU) - Different SLA tiers (critical, standard, experimental)

Your goals are to: - Provide regional directors with visibility into their entire region - Give regional operators access to manage non-production environments in their region - Enable global administrators to view and manage all regions - Ensure compliance by restricting access based on geographic boundaries - Create regional dashboards that automatically filter and display the right clusters - Organize navigation menus hierarchically by region for intuitive access

This guide shows you how to implement this complete multi-region deployment using SPOG's integrated architecture.

1
2
3

# Set your credentials
export REGISTRY_USERNAME="your-username"
export REGISTRY_PASSWORD="your-password"

1

export GLASS_HOSTNAME="glass.example.com"  # Replace with your domain

1

export NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')

1
2
3
4

# Login to Helm registry
helm registry login registry.open-xchange.com \
  --username "$REGISTRY_USERNAME" \
  --password "$REGISTRY_PASSWORD"

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# Controlplane namespace
kubectl create namespace controlplane
kubectl create secret docker-registry registry-credentials \
  --docker-server=registry.open-xchange.com \
  --docker-username="$REGISTRY_USERNAME" \
  --docker-password="$REGISTRY_PASSWORD" \
  --namespace=controlplane

# Create namespaces for each regional cluster
for ns in us-east-prod us-west-prod eu-west-prod ap-north-prod us-east-staging; do
  kubectl create namespace $ns
  kubectl create secret docker-registry registry-credentials \
    --docker-server=registry.open-xchange.com \
    --docker-username="$REGISTRY_USERNAME" \
    --docker-password="$REGISTRY_PASSWORD" \
    --namespace=$ns
done

1
2
3

helm install powerdns-crds \
  oci://registry.open-xchange.com/cloudcontrol/powerdns-crds \
  --version "3.1.13"

1
2
3
4
5
6

helm install region-controlplane \
  oci://registry.open-xchange.com/cloudcontrol/controlplane \
  --version "3.1.13" \
  --namespace controlplane \
  --set global.imagePullSecretsList[0]=registry-credentials \
  -f controlplane-quickstart.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

# US East Production
helm install us-east-prod \
  oci://registry.open-xchange.com/cloudcontrol/powerdns \
  --version "3.1.13" \
  --namespace us-east-prod \
  --set global.imagePullSecretsList[0]=registry-credentials \
  -f powerdns-quickstart.yaml

# US West Production
helm install us-west-prod \
  oci://registry.open-xchange.com/cloudcontrol/powerdns \
  --version "3.1.13" \
  --namespace us-west-prod \
  --set global.imagePullSecretsList[0]=registry-credentials \
  -f powerdns-quickstart.yaml

# EU West Production
helm install eu-west-prod \
  oci://registry.open-xchange.com/cloudcontrol/powerdns \
  --version "3.1.13" \
  --namespace eu-west-prod \
  --set global.imagePullSecretsList[0]=registry-credentials \
  -f powerdns-quickstart.yaml

# AP North Production
helm install ap-north-prod \
  oci://registry.open-xchange.com/cloudcontrol/powerdns \
  --version "3.1.13" \
  --namespace ap-north-prod \
  --set global.imagePullSecretsList[0]=registry-credentials \
  -f powerdns-quickstart.yaml

# US East Staging
helm install us-east-staging \
  oci://registry.open-xchange.com/cloudcontrol/powerdns \
  --version "3.1.13" \
  --namespace us-east-staging \
  --set global.imagePullSecretsList[0]=registry-credentials \
  -f powerdns-quickstart.yaml

Understanding Multi-Region Architecture

Multi-region operations in SPOG are built on four integrated components that work together to provide flexible geographic organization.

The diagram above shows how the four label dimensions (region, environment, role, tier) create a flexible taxonomy for organizing clusters. Different combinations allow for precise filtering in dashboards and access control policies.

How Regional Organization Works

1. Labels Define Geography: Each cluster is labeled with region, environment, role, and tier
2. Dashboards Filter by Region: Filter expressions like region = "us-east" automatically show relevant clusters
3. Navigation Organizes Hierarchically: Menus group dashboards by continent → region → environment
4. REGO Controls Access: Policies map user roles to regional permissions

Why Organize by Region?

Regional organization provides multiple benefits:

• Compliance: Meet data residency requirements (GDPR, data sovereignty)
• Performance: Route teams to geographically close infrastructure
• Team Structure: Align SPOG views with organizational boundaries
• Disaster Recovery: Isolate and manage regional incidents
• Latency Optimization: Direct operations teams to local clusters
• Cost Management: Track and optimize regional infrastructure costs

Label-Based Flexibility

Unlike rigid hierarchies, SPOG's label system allows:

• Multi-dimensional organization: Filter by region AND environment AND role simultaneously
• Dynamic grouping: Change how you organize clusters without redeployment
• Overlapping views: Same cluster can appear in multiple regional dashboards
• Custom taxonomies: Add dimensions like customer, tier, cost-center as needed

Example Cluster Labeling

Each cluster is organized using four-dimensional labeling:

Labels are the foundation that enables:

User Types and Access Levels

This deployment defines seven user types with different regional access patterns:

Access Decision Logic

Authorization follows a multi-dimensional decision tree:

Step 1: Label Your Clusters by Region

The foundation of multi-region operations is applying consistent labels to your user plane clusters. Labels are configured during glass-instrumentation deployment and define each cluster's regional identity.

Multi-region deployment uses four label dimensions configured in glass-instrumentation values.yaml:

• region: Geographic location (us-east, eu-west, ap-north)
• environment: Deployment stage (production, staging, development)
• role: DNS service type (authoritative, recursor)
• tier: Service criticality (critical, standard, experimental)

Use lowercase, hyphenated format for consistency (e.g., region: "us-east" not region: "US_EAST").

Example: Regional Cluster Configurations

Here are examples showing how to label clusters for different regions:

US East Production Authoritative:

1
2
3
4
5
6
7

clusterId: "pdns-us-east-prod-01"

labels:
  region: "us-east"          # Geographic location - Americas East Coast
  environment: "production"  # Deployment stage - live customer-facing services
  role: "authoritative"      # DNS service type - PowerDNS Authoritative server
  tier: "critical"           # Service criticality - mission-critical, highest SLA

US West Production Recursor:

1
2
3
4
5
6
7

clusterId: "pdns-us-west-prod-01"

labels:
  region: "us-west"          # Geographic location - Americas West Coast
  environment: "production"  # Deployment stage - live customer-facing services
  role: "recursor"           # DNS service type - PowerDNS Recursor server
  tier: "critical"           # Service criticality - mission-critical, highest SLA

EU West Production Authoritative:

1
2
3
4
5
6
7

clusterId: "pdns-eu-west-prod-01"

labels:
  region: "eu-west"          # Geographic location - Europe West (Ireland, UK)
  environment: "production"  # Deployment stage - live customer-facing services
  role: "authoritative"      # DNS service type - PowerDNS Authoritative server
  tier: "standard"           # Service criticality - standard SLA

Asia Pacific Production Recursor:

1
2
3
4
5
6
7

clusterId: "pdns-ap-north-prod-01"

labels:
  region: "ap-north"         # Geographic location - Asia Pacific North (Japan, Korea)
  environment: "production"  # Deployment stage - live customer-facing services
  role: "recursor"           # DNS service type - PowerDNS Recursor server
  tier: "standard"           # Service criticality - standard SLA

US East Staging Authoritative:

1
2
3
4
5
6
7

clusterId: "pdns-us-east-staging-01"

labels:
  region: "us-east"          # Geographic location - Americas East Coast
  environment: "staging"     # Deployment stage - pre-production validation
  role: "authoritative"      # DNS service type - PowerDNS Authoritative server
  tier: "standard"           # Service criticality - standard SLA

Integration Point: Labels Enable Everything

Labels are the foundation that enables: - Step 2: Dashboard filtering by region - Step 3: Navigation organization - Step 4: REGO authorization policies

Without proper labels, none of the subsequent steps will work correctly.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

# US East Production
helm install glass-instrumentation \
  oci://registry.open-xchange.com/cc-glass/glass-instrumentation \
  --version "1.0.0" \
  --namespace us-east-prod \
  --set global.imagePullSecretsList[0]=registry-credentials \
  -f multi-region-labels-us-east-prod.yaml

# US West Production
helm install glass-instrumentation \
  oci://registry.open-xchange.com/cc-glass/glass-instrumentation \
  --version "1.0.0" \
  --namespace us-west-prod \
  --set global.imagePullSecretsList[0]=registry-credentials \
  -f multi-region-labels-us-west-prod.yaml

# EU West Production
helm install glass-instrumentation \
  oci://registry.open-xchange.com/cc-glass/glass-instrumentation \
  --version "1.0.0" \
  --namespace eu-west-prod \
  --set global.imagePullSecretsList[0]=registry-credentials \
  -f multi-region-labels-eu-west-prod.yaml

# AP North Production
helm install glass-instrumentation \
  oci://registry.open-xchange.com/cc-glass/glass-instrumentation \
  --version "1.0.0" \
  --namespace ap-north-prod \
  --set global.imagePullSecretsList[0]=registry-credentials \
  -f multi-region-labels-ap-north-prod.yaml

# US East Staging
helm install glass-instrumentation \
  oci://registry.open-xchange.com/cc-glass/glass-instrumentation \
  --version "1.0.0" \
  --namespace us-east-staging \
  --set global.imagePullSecretsList[0]=registry-credentials \
  -f multi-region-labels-us-east-staging.yaml

Step 2: Customize Regional Dashboards

Concept: Dashboard Filtering and Widgets

Dashboards use filter expressions to select which clusters to display. Each dashboard contains one or more widgets that visualize cluster data:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# Example: Multi-Region Dashboard Configuration
globalConfig:
  dashboards:
    # Global Overview Dashboard - All regions
    americas-region:
      title: "Americas Region"
      description: "PowerDNS clusters in US regions"
      url: /regions/americas
      requires:
        - "dashboard_americas_region"
      graphs:
        - widget: "cc-state-readiness-heatmap"
          title: "Americas Cluster Health"
          args:
            filter: 'region in ("us-east", "us-west", "us-central")'

        - widget: "cc-state-tree-table"
          title: "Americas Clusters"
          args:
            filter: 'region in ("us-east", "us-west", "us-central")'

    # Europe Regional Dashboard

This creates a dashboard showing only clusters in the Americas region (US East, US West, US Central).

Filter Expression Basics

The filter argument in each widget uses a query language to select clusters:

Dashboards use various widget types: tree-table (detailed inspection), heatmap (visual health overview), pie-charts (status distribution), and cytoscape (topology graphs).

Example Regional Dashboard Configurations

Europe Regional Overview:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# Example: Multi-Region Dashboard Configuration
globalConfig:
  dashboards:
    # Global Overview Dashboard - All regions
    europe-region:
      title: "Europe Region"
      description: "PowerDNS clusters in European regions"
      url: /regions/europe
      requires:
        - "dashboard_europe_region"
      graphs:
        - widget: "cc-state-readiness-heatmap"
          title: "Europe Cluster Health"
          args:
            filter: 'region in ("eu-west", "eu-central", "eu-north")'

        - widget: "cc-state-tree-table"
          title: "Europe Clusters"
          args:
            filter: 'region in ("eu-west", "eu-central", "eu-north")'

    # Asia Pacific Regional Dashboard

Shows all clusters where region label is in Europe (eu-west, eu-central, eu-north) using: - Tree table for detailed cluster state - Readiness heatmap for quick visual health check

Specific Region Dashboard (US East):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# Example: Multi-Region Dashboard Configuration
globalConfig:
  dashboards:
    # Global Overview Dashboard - All regions
    us-east:
      title: "US East"
      description: "PowerDNS clusters in US East region"
      url: /regions/us-east
      requires:
        - "dashboard_americas_region"
      graphs:
        - widget: "cc-state-tree-table"
          title: "US East Clusters"
          args:
            filter: 'region = "us-east"'

        - widget: "cc-state-readiness-pie-charts"
          title: "US East Readiness Distribution"
          args:
            filter: 'region = "us-east"'

    # Production Clusters Across All Regions

Shows only US East clusters with: - Tree table view for detailed inspection - Pie chart showing readiness distribution

Combined Filter Dashboard (Production Americas):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# Example: Multi-Region Dashboard Configuration
globalConfig:
  dashboards:
    # Global Overview Dashboard - All regions
    production-americas:
      title: "Production - Americas"
      description: "Production PowerDNS clusters in US regions"
      url: /production/americas
      requires:
        - "dashboard_americas_region"
      graphs:
        - widget: "cc-state-tree-table"
          title: "Americas Production Clusters"
          args:
            filter: 'environment = "production" and region in ("us-east", "us-west", "us-central")'

    # Authoritative Servers - Multi-Region

Combines multiple label dimensions: region AND environment, showing only production clusters in the Americas.

Integration Point: Dashboards Use Labels

Notice the connection:

• Step 1 labeled clusters with region: "us-east", environment: "production"
• Step 2 creates dashboards with filter: 'region in ("us-east", "us-west")'
• Result: Dashboards automatically display only matching clusters

This is the first integration point where labels drive behavior. The next step adds navigation so users can easily access these dashboards.

Step 3: Extend Navigation for Custom Dashboards

The default navigation includes comprehensive menus for all built-in dashboards. This step shows how to add custom menus for organization-specific dashboards created in Step 2.

Concept: Navigation Hierarchy and Menu Structure

SPOG navigation uses a three-level hierarchical structure:

1
2
3
4
5

Menu (e.g., "Regional Views")
  └─ Section (e.g., "Americas")
      ├─ Item: "Americas Overview" → /regions/americas
      ├─ Item: "US East" → /regions/us-east
      └─ Item: "US West" → /regions/us-west

Why Three Levels? - Prevents overwhelming users with flat lists - Groups related dashboards logically (Global → Region → Specific Location) - Provides clear navigation paths - Mirrors organizational structure

Example Navigation Configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82

# Example: Multi-Region Navigation Configuration
globalConfig:
  navigation:
    menus:
      # Regional Dashboards Menu
      - name: "Regional Views"
        sections:
          # Americas Section
          - name: "Americas"
            items:
              - name: "Americas Overview"
                url: "/regions/americas"
              - name: "US East"
                url: "/regions/us-east"
              - name: "US West"
                url: "/regions/us-west"
              - name: "US Central"
                url: "/regions/us-central"

          # Europe Section
          - name: "Europe"
            items:
              - name: "Europe Overview"
                url: "/regions/europe"
              - name: "EU West"
                url: "/regions/eu-west"
              - name: "EU Central"
                url: "/regions/eu-central"
              - name: "EU North"
                url: "/regions/eu-north"

          # Asia Pacific Section
          - name: "Asia Pacific"
            items:
              - name: "APAC Overview"
                url: "/regions/apac"
              - name: "AP North"
                url: "/regions/ap-north"
              - name: "AP South"
                url: "/regions/ap-south"
              - name: "AP Southeast"
                url: "/regions/ap-southeast"

          # Global Section
          - name: "Global"
            items:
              - name: "Global Overview"
                url: "/global"

      # Operations Menu
      - name: "Operations"
        sections:
          # Production Operations
          - name: "Production"
            items:
              - name: "Production - Global"
                url: "/production"
              - name: "Production - Americas"
                url: "/production/americas"
              - name: "Production - Europe"
                url: "/production/europe"
              - name: "Production - APAC"
                url: "/production/apac"

          # Service Type Operations
          - name: "By Service Type"
            items:
              - name: "Authoritative Servers"
                url: "/authoritative"
              - name: "Recursor Servers"
                url: "/recursor"

          # Critical Infrastructure
          - name: "Critical Infrastructure"
            items:
              - name: "Critical Clusters"
                url: "/critical"
              - name: "US Production Auth"
                url: "/us-prod-auth"

# Note: The default "Tools" menu (with DNS Query and Cache Management)
# is automatically appended after these custom menus.

Navigation Structure Explained

The example creates two main menus:

1. Regional Views Menu

Organizes dashboards by geographic location:

Americas Section: - Americas Overview → /regions/americas (all US regions) - US East → /regions/us-east (specific region) - US West → /regions/us-west (specific region) - US Central → /regions/us-central (specific region)

Europe Section: - Europe Overview → /regions/europe (all EU regions) - EU West → /regions/eu-west (specific region) - EU Central → /regions/eu-central (specific region) - EU North → /regions/eu-north (specific region)

Asia Pacific Section: - APAC Overview → /regions/apac (all AP regions) - AP North → /regions/ap-north (specific region) - AP South → /regions/ap-south (specific region) - AP Southeast → /regions/ap-southeast (specific region)

Global Section: - Global Overview → /global (all clusters worldwide)

2. Operations Menu

Organizes dashboards by operational concerns:

Production Section: - Production - Global → /production (all regions, production only) - Production - Americas → /production/americas (Americas production) - Production - Europe → /production/europe (Europe production) - Production - APAC → /production/apac (APAC production)

By Service Type Section: - Authoritative Servers → /authoritative (all authoritative) - Recursor Servers → /recursor (all recursor)

Critical Infrastructure Section: - Critical Clusters → /critical (critical tier across regions) - US Production Auth → /us-prod-auth (US production authoritative servers)

Navigation Best Practices

The diagram below shows both geographic and functional hierarchy patterns:

Geographic Hierarchy Pattern (Regional Views menu) organizes by physical location, ideal for teams managing infrastructure by region.

Functional Hierarchy Pattern (Operations menu) organizes by operational concerns, ideal for teams focused on service types or environments.

The default Tools menu (with DNS Query and Cache Management) is automatically appended after your custom menus.

Navigation URL Mapping

Navigation item URLs link to dashboard URL patterns:

1
2
3
4
5
6
7
8
9

# Dashboard configuration (Step 2)
dashboards:
  americas-region:
    url: /regions/americas

# Navigation item (Step 3)
items:
  - name: "Americas Overview"
    url: "/regions/americas"

When a user clicks "Americas Overview": 1. Glass UI routes to /regions/americas 2. Finds dashboard with matching url: /regions/americas 3. Loads dashboard and renders widgets with region in ("us-east", "us-west", "us-central") filter

Integration Point: Navigation Links to Dashboards

Notice the connection:

• Step 2 created dashboard with url: /regions/americas
• Step 3 creates navigation item with url: "/regions/americas"
• Result: Clicking navigation item takes user to regional dashboard showing only Americas clusters

This is the second integration point. The next step adds access control so only authorized users can access these regions.

Step 4: Configure Regional Authorization

REGO policies control who can access which regional clusters. Policies match user roles against cluster labels to enforce regional access boundaries.

REGO policies control regional access by evaluating user roles against cluster labels. The policy engine receives user and cluster context as input, then evaluates specific permission rules:

1
2
3
4
5

# Input structure for regional authorization
{
  "user": {"roles": ["americas", "production", "operator"]},
  "cluster": {"labels": {"region": "us-east", "environment": "production"}}
}

When checking if a user can perform an operation (e.g., restart_instance_set), the policy engine queries the corresponding rule directly (e.g., data.pdns_permissions.restart_instance_set) using this input context.

Note: The user data structure depends on how the OIDC server provides user data.

Regional Role System

Regional roles map geographic access (global, americas, europe, apac), service types (authoritative, recursor), environments (production, staging, development), and operations (observer for read-only, operator for write, admin for full access).

Example User Configurations

Americas Director:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# Example: Multi-Region REGO Authorization Policies
policy:
  # Static users configuration with regional roles
  staticUsers:
    # Global administrator - full access to all regions
    americas-director:
      id: "americas-director"
      password: "secret"
      sub: "americas-director@company.com"
      name: "Americas Regional Director"
      roles:
        - "americas"               # Americas regional access
        - "authoritative"          # Access authoritative clusters
        - "recursor"               # Access recursor clusters
        - "production"             # Access production
        - "staging"                # Access staging
        - "observer"               # Can read logs and check DNS
        - "operator"               # Can restart instances

    # Europe Regional Director - full access to European regions

This Americas Director has: - Access to all Americas regions (americas role) - Both authoritative and recursor clusters - Production and staging environments - Operator permissions (can restart instances)

Europe Viewer (Production Read-Only):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# Example: Multi-Region REGO Authorization Policies
policy:
  # Static users configuration with regional roles
  staticUsers:
    # Global administrator - full access to all regions
    europe-viewer:
      id: "europe-viewer"
      password: "secret"
      sub: "europe-viewer@company.com"
      name: "Europe Production Viewer"
      roles:
        - "europe"                 # Europe regional access
        - "authoritative"          # Access authoritative clusters
        - "recursor"               # Access recursor clusters
        - "production"             # Access production only
        - "observer"               # Can read logs and check DNS

    # APAC Content Manager - can clear cache in APAC non-production

This Europe Viewer has: - Access to Europe region only (europe role) - Both authoritative and recursor clusters - Production environment only - Observer permissions (read-only, can read logs)

Global Administrator:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# Example: Multi-Region REGO Authorization Policies
policy:
  # Static users configuration with regional roles
  staticUsers:
    # Global administrator - full access to all regions
    global-admin:
      id: "global-admin"
      password: "secret"
      sub: "admin@company.com"
      name: "Global Administrator"
      roles:
        - "admin"                  # Admin bypass role
        - "global"                 # Access all regions
        - "authoritative"          # Access authoritative clusters
        - "recursor"               # Access recursor clusters
        - "production"             # Access production
        - "staging"                # Access staging
        - "development"            # Access development

    # Americas Regional Director - full access to Americas regions

The Global Admin has the admin role which bypasses all authorization checks.

SPOG uses three REGO packages: pdns_permissions (operation permissions like read_logs, restart_instance_set), pdns_global_flags (dashboard and navigation access permissions), and user (helper functions for regional access checks, cluster role matching, and environment access verification).

REGO Policy Breakdown

The policies define three packages:

1. pdns_permissions.rego - Operation Permissions

This package defines well-known operations that services check:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# Basic connection - all authenticated users
connect if true

# Read cluster state - requires regional access + role + environment
read if user.can_see_cluster

# Read logs - requires observer role + cluster access
read_logs if user.can_observe_cluster

# Clear cache - requires content-manager role + cluster access
clear_cache if user.can_manage_cache

# Restart instances - requires operator role + cluster access
restart_instance_set if user.can_manage_instances

# Delete pod - requires operator role + cluster access
delete_pod if user.can_manage_instances

# DNS check - requires observer role + cluster access
dns_check if user.can_observe_cluster

Each permission references functions from the user package that evaluate access based on roles and cluster labels. The admin bypass is built into the helper functions themselves (see user.rego below).

2. pdns_global_flags.rego - Dashboard and Navigation Permissions

This package defines custom permission flags for UI access control:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# Global overview dashboard - admin and global roles
dashboard_global_overview if "admin" in input.user.roles
dashboard_global_overview if "global" in input.user.roles

# Production global dashboard - admin and users with production role
dashboard_production_global if "admin" in input.user.roles
dashboard_production_global if "global" in input.user.roles
dashboard_production_global if "production" in input.user.roles

# Americas regional dashboards
dashboard_americas_region if "admin" in input.user.roles
dashboard_americas_region if "global" in input.user.roles
dashboard_americas_region if "americas" in input.user.roles

# Europe regional dashboards
dashboard_europe_region if "admin" in input.user.roles
dashboard_europe_region if "global" in input.user.roles
dashboard_europe_region if "europe" in input.user.roles

# APAC regional dashboards
dashboard_apac_region if "admin" in input.user.roles
dashboard_apac_region if "global" in input.user.roles
dashboard_apac_region if "apac" in input.user.roles

These flags are referenced by dashboard requires fields to control visibility. Users with admin or global roles can access all dashboards. Regional users (americas, europe, apac) can only access their respective regional dashboards.

3. user.rego - Regional Authorization Logic

This package contains the core regional authorization logic:

Regional Access Mapping:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# Regional access - Americas (US regions)
has_matching_region if {
  some region_label in input.cluster.labels.region
  region_label in ["us-east", "us-west", "us-central"]
  "americas" in roles
}

# Regional access - Europe (EU regions)
has_matching_region if {
  some region_label in input.cluster.labels.region
  region_label in ["eu-west", "eu-central", "eu-north"]
  "europe" in roles
}

# Regional access - Asia Pacific (AP regions)
has_matching_region if {
  some region_label in input.cluster.labels.region
  region_label in ["ap-north", "ap-south", "ap-southeast"]
  "apac" in roles
}

# Regional access - Global (all regions)
has_matching_region if {
  "global" in roles
}

This rule checks if the user's regional role grants access to the cluster's region. For example: - User roles: ["americas", "authoritative", "production", "operator"] - Cluster labels: {region: "us-east", environment: "production", role: "authoritative"} - Evaluation: "americas" role maps to us-east/us-west/us-central → TRUE

Cluster Role Access:

1
2
3
4
5
6

# User must have role matching cluster role label
has_matching_cluster_role if {
  some role in roles
  some cluster_role in input.cluster.labels.role
  role == cluster_role
}

This rule checks if any of the user's roles matches the cluster's role label (authoritative or recursor): - User roles: ["americas", "authoritative", "production", "operator"] - Cluster labels: {role: "authoritative"} - Evaluation: "authoritative" in roles equals "authoritative" in cluster labels → TRUE

Environment Access:

1
2
3
4
5
6

# User must have environment role
has_matching_environment if {
  some env in roles
  some cluster_env in input.cluster.labels.environment
  env == cluster_env
}

This rule checks if the cluster's environment is in the user's roles: - User roles: ["americas", "authoritative", "staging", "development", "operator"] - Cluster labels: {environment: "production"} - Evaluation: "production" not in roles → FALSE (user cannot access production)

Cluster Visibility Rule:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# User can see cluster if region AND role AND environment match
can_see_cluster if {
  has_matching_region
  has_matching_cluster_role
  has_matching_environment
}

# Admin bypass - admins can see clusters with just region and role
can_see_cluster if {
  admin
  has_matching_region
  has_matching_cluster_role
}

All three dimensions must match for visibility. This implements the three-way gating strategy: 1. Must be authorized for region 2. Must have cluster role type 3. Must have environment access

Observer Capabilities:

1
2
3
4
5
6
7
8
9

# Can read logs and check DNS if observer role + visibility
can_observe_cluster if {
  can_see_cluster
  "observer" in roles
}

can_observe_cluster if {
  admin
}

Observer role grants read-only access: view logs, check DNS, read cluster state. Requires cluster visibility first. Admins have observer capabilities on all clusters.

Content Manager Capabilities:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# Can clear cache if content-manager role + visibility
can_manage_cache if {
  can_see_cluster
  "content-manager" in roles
}

# Content manager non-prod - can clear cache in non-production only
can_manage_cache if {
  can_see_cluster
  "content-manager-non-prod" in roles
  some env in input.cluster.labels.environment
  env != "production"
}

can_manage_cache if {
  admin
}

Content manager role grants cache management operations. The non-prod variant restricts this to non-production environments only. Requires cluster visibility first.

Operator Capabilities:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# Can restart instances and delete pods if operator role + visibility
can_manage_instances if {
  can_see_cluster
  "operator" in roles
}

# Operator non-prod - can restart/delete in non-production only
can_manage_instances if {
  can_see_cluster
  "operator-non-prod" in roles
  some env in input.cluster.labels.environment
  env != "production"
}

can_manage_instances if {
  admin
}

Operator role grants infrastructure management: restart instances, delete pods. The non-prod variant restricts this to non-production environments only. Requires cluster visibility first. Admins have operator capabilities on all clusters.

Complete REGO Policy Configuration

Here's the complete policy configuration with all three packages:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170

# Example: Multi-Region REGO Authorization Policies
policy:
  # Static users configuration with regional roles
  policies:
    # Dashboard and Navigation Permission Flags
    # These are custom flags for controlling UI access
    pdns_global_flags.rego: |
      package pdns_global_flags

      # Global overview dashboard - admin and global roles
      dashboard_global_overview if "admin" in input.user.roles
      dashboard_global_overview if "global" in input.user.roles

      # Production global dashboard - admin and users with production role
      dashboard_production_global if "admin" in input.user.roles
      dashboard_production_global if "global" in input.user.roles
      dashboard_production_global if "production" in input.user.roles

      # Americas regional dashboards
      dashboard_americas_region if "admin" in input.user.roles
      dashboard_americas_region if "global" in input.user.roles
      dashboard_americas_region if "americas" in input.user.roles

      # Europe regional dashboards
      dashboard_europe_region if "admin" in input.user.roles
      dashboard_europe_region if "global" in input.user.roles
      dashboard_europe_region if "europe" in input.user.roles

      # APAC regional dashboards
      dashboard_apac_region if "admin" in input.user.roles
      dashboard_apac_region if "global" in input.user.roles
      dashboard_apac_region if "apac" in input.user.roles

    # Permission definitions
    pdns_permissions.rego: |
      package pdns_permissions

      import data.user

      # Basic connection allowed for all authenticated users
      connect if true

      # Read permissions - requires matching region, role, and environment
      read if user.can_see_cluster

      # Read logs - requires observer role + cluster access
      read_logs if user.can_observe_cluster

      # Clear cache - requires content-manager role + cluster access
      clear_cache if user.can_manage_cache

      # Restart instances - requires operator role + cluster access
      restart_instance_set if user.can_manage_instances

      # Delete pod - requires operator role + cluster access
      delete_pod if user.can_manage_instances

      # DNS check - requires observer role + cluster access
      dns_check if user.can_observe_cluster

    # User authorization logic with regional access control
    user.rego: |
      package user

      roles := input.user.roles

      # Role flags
      admin if "admin" in roles
      operator if "operator" in roles
      viewer if "viewer" in roles

      # Regional access - Americas (US regions)
      has_matching_region if {
        some region_label in input.cluster.labels.region
        region_label in ["us-east", "us-west", "us-central"]
        "americas" in roles
      }

      # Regional access - Europe (EU regions)
      has_matching_region if {
        some region_label in input.cluster.labels.region
        region_label in ["eu-west", "eu-central", "eu-north"]
        "europe" in roles
      }

      # Regional access - Asia Pacific (AP regions)
      has_matching_region if {
        some region_label in input.cluster.labels.region
        region_label in ["ap-north", "ap-south", "ap-southeast"]
        "apac" in roles
      }

      # Regional access - Global (all regions)
      has_matching_region if {
        "global" in roles
      }

      # Cluster role access - user role matches cluster role label
      has_matching_cluster_role if {
        some role in roles
        some cluster_role in input.cluster.labels.role
        role == cluster_role
      }

      # Environment access - user has environment role
      has_matching_environment if {
        some env in roles
        some cluster_env in input.cluster.labels.environment
        env == cluster_env
      }

      # Can see cluster - must match region, role, and environment
      can_see_cluster if {
        has_matching_region
        has_matching_cluster_role
        has_matching_environment
      }

      # Admin bypass - admins can see clusters with just region and role
      can_see_cluster if {
        admin
        has_matching_region
        has_matching_cluster_role
      }

      # Observer permissions - can read logs and check DNS
      can_observe_cluster if {
        can_see_cluster
        "observer" in roles
      }

      can_observe_cluster if {
        admin
      }

      # Content manager permissions - can clear cache
      can_manage_cache if {
        can_see_cluster
        "content-manager" in roles
      }

      # Content manager non-prod - can clear cache in non-production only
      can_manage_cache if {
        can_see_cluster
        "content-manager-non-prod" in roles
        some env in input.cluster.labels.environment
        env != "production"
      }

      can_manage_cache if {
        admin
      }

      # Operator permissions - can restart/delete instances
      can_manage_instances if {
        can_see_cluster
        "operator" in roles
      }

      # Operator non-prod - can restart/delete in non-production only
      can_manage_instances if {
        can_see_cluster
        "operator-non-prod" in roles
        some env in input.cluster.labels.environment
        env != "production"
      }

      can_manage_instances if {
        admin
      }

Authorization Flow Examples

Let's trace complete authorization decisions:

Example 1: Americas Director Accesses US East Production

Scenario: Americas Director tries to restart instance on US East production cluster

Input (passed to policy engine when querying data.pdns_permissions.restart_instance_set):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

{
  "user": {
    "roles": ["americas", "authoritative", "recursor", "production", "staging", "operator", "observer"]
  },
  "cluster": {
    "labels": {
      "region": "us-east",
      "environment": "production",
      "role": "authoritative"
    }
  }
}

Evaluation:

Outcome: Americas Director CAN restart instances on US East production cluster.

Example 2: Americas Director Tries to Access EU West

Scenario: Americas Director tries to view logs from EU West cluster

Input (passed to policy engine when querying data.pdns_permissions.read_logs):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

{
  "user": {
    "roles": ["americas", "authoritative", "recursor", "production", "staging", "operator", "observer"]
  },
  "cluster": {
    "labels": {
      "region": "eu-west",
      "environment": "production",
      "role": "authoritative"
    }
  }
}

Evaluation:

Outcome: Americas Director CANNOT access EU West cluster - wrong region, even though they have production and observer roles.

Integration Point: Policies Protect Regional Access

Notice the complete integration across all four steps:

• Step 1 labeled clusters with region: "us-east", environment: "production"
• Step 2 created dashboards with filter: 'region in ("us-east", "us-west")'
• Step 3 created navigation linking to those dashboards
• Step 4 implements REGO policies checking regional roles match cluster labels

Result: Only users with americas role (or global/admin role) can: - See "Regional Views → Americas" navigation menu - Access /regions/americas dashboard - View Americas clusters in widgets - Perform operations on Americas clusters (if they also have operator role and environment access)

This is the complete security model enforcing regional isolation.

1
2
3
4
5
6
7
8
9

helm install glass-ui \
  oci://registry.open-xchange.com/cc-glass/glass-ui \
  --version "1.0.0" \
  --namespace controlplane \
  --set ui.ingress.host="$GLASS_HOSTNAME" \
  --set global.imagePullSecretsList[0]=registry-credentials \
  -f multi-region-dashboard.yaml \
  -f multi-region-navigation.yaml \
  -f multi-region-rego.yaml

1
2
3
4
5
6
7
8
9

helm install glass-ui \
  oci://registry.open-xchange.com/cc-glass/glass-ui \
  --version "1.0.0" \
  --namespace controlplane \
  --set global.imagePullSecretsList[0]=registry-credentials \
  --set "ui.config.nats.serverUrl=ws://localhost:8222" \
  -f multi-region-dashboard.yaml \
  -f multi-region-navigation.yaml \
  -f multi-region-rego.yaml

1
2

kubectl port-forward svc/glass-ui 8080:80 -n controlplane &
kubectl port-forward svc/glass-nats 8222:8080 -n controlplane &

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

helm install glass-ui \
  oci://registry.open-xchange.com/cc-glass/glass-ui \
  --version "1.0.0" \
  --namespace controlplane \
  --set global.imagePullSecretsList[0]=registry-credentials \
  --set ui.service.type=NodePort \
  --set ui.service.nodePort=31080 \
  --set "ui.config.nats.serverUrl=ws://${NODE_IP}:31222" \
  -f multi-region-dashboard.yaml \
  -f multi-region-navigation.yaml \
  -f multi-region-rego.yaml

What You Accomplished

Step 1: Labeled Clusters by Region - Applied region, environment, role, and tier labels to user plane clusters - Created geographic organization foundation for filtering and access control - Established multi-dimensional cluster taxonomy

Step 2: Customized Regional Dashboards - Learned how default dashboards cover common regional views (Americas, Europe, APAC) - Built custom dashboards for organization-specific needs beyond defaults - Configured filters like region in ("us-east", "us-west") expressions

Step 3: Extended Navigation for Custom Dashboards - Learned how default navigation includes comprehensive regional menus - Added custom menus for organization-specific dashboards - Understood how custom menus appear before default menus

Step 4: Implemented Regional Authorization - Defined regional access control mapping (americas → US regions, europe → EU regions, apac → AP regions) - Implemented three-way access verification (region + cluster role + environment) - Created role-based operation permissions (observer, operator, content-manager) - Configured static users with different regional access patterns and environment restrictions

Try It Yourself

Now that you understand how the components work together, here are some modifications you can try to deepen your understanding.

Add a New Region

Add a "Middle East" region with clusters in Dubai and Tel Aviv. You'll need to:

1. Create a new cluster with Middle East labels
2. Add a Middle East dashboard to your dashboard configuration
3. Add a navigation menu section for the Middle East region
4. Add REGO policies for regional access mapping and dashboard permissions
5. Add a Middle East operator user to test access

Change Dashboard Filters

Modify existing dashboards to show different cluster subsets. Try creating filters that:

• Show only critical tier clusters globally
• Show production recursors across all regions
• Exclude experimental tier from a regional view
• Combine multiple criteria (region, environment, tier)

Add a Cross-Region Dashboard

Create a dashboard showing critical infrastructure across all regions. This requires:

• A dashboard configuration filtering by tier
• REGO rules that grant access to regional directors from all regions

Implement Time-Based Access Control

Restrict production access to business hours only. Consider:

• How to differentiate production from non-production environments
• Using an additional role (e.g., after-hours-operator) for production access
• Note: This is a conceptual exercise—actual time-based access requires external time input

Add a Read-Only Global User

Create a user that can view all regions but cannot perform any operations. Think about:

• Which role grants access to all regions
• Which roles to include for cluster visibility
• Which operation roles to omit

Create Environment-Specific Regional Views

Add dashboards that combine region and environment filters. Create:

• An "Americas Production" dashboard showing only production clusters in US regions
• An "Americas Non-Production" dashboard showing staging and development clusters

Consider how to structure the filter expressions and which REGO permissions to require.

Refactoring with Placeholders

As your multi-region deployment grows, you may notice repetitive dashboard definitions—one for each region with nearly identical structure. SPOG supports :placeholder syntax that lets you create generic, reusable configurations.

Understanding Placeholders

SPOG supports two placeholder syntaxes for different purposes:

:paramName - For Display and Filtering

Used in dashboard titles, descriptions, and filter expressions:

• URL pattern: /regions/:region defines a route parameter named region
• When user visits /regions/us-east, the :region placeholder becomes "us-east"
• When user visits /regions/eu-west, the :region placeholder becomes "eu-west"

{{ paramName }} - For Permissions

Used in the requires field to create dynamic permission names:

• requires: ["dashboard_region_{{ region }}"]
• When user visits /regions/us-east, this checks for permission dashboard_region_us_east
• When user visits /regions/eu-west, this checks for permission dashboard_region_eu_west

This means a single dashboard definition can serve all regions while maintaining proper access control.

Before: Repetitive Dashboard Definitions

The original configuration requires a separate dashboard for each region:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

# One dashboard per region - repetitive!
us-east:
  title: "US East"
  url: /regions/us-east
  graphs:
    - widget: "cc-state-tree-table"
      args:
        filter: 'region = "us-east"'

us-west:
  title: "US West"
  url: /regions/us-west
  graphs:
    - widget: "cc-state-tree-table"
      args:
        filter: 'region = "us-west"'

eu-west:
  title: "EU West"
  url: /regions/eu-west
  graphs:
    - widget: "cc-state-tree-table"
      args:
        filter: 'region = "eu-west"'

# Need to add another dashboard for EACH new region...

After: Generic Dashboard with Placeholders

With placeholders, one definition works for all regions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# Single dashboard serves ALL regions
region-detail:
  title: ":region Region"
  description: "PowerDNS clusters in :region"
  url: /regions/:region
  requires:
    - "dashboard_region_{{ region }}"  # {{ region }} replaced with URL param
  graphs:
    - widget: "cc-state-readiness-heatmap"
      title: ":region Cluster Health"
      args:
        filter: 'region = ":region"'

    - widget: "cc-state-tree-table"
      title: ":region Clusters"
      args:
        filter: 'region = ":region"'