TopN Reporter

TopN Reporter can be used to create reports from the protobuf messages generated by dnsdist & recursor and distributed by dstoredist.

A TopN Reporter has 3 crucial parts:

Streams: Input sources for data distributed via dstoredist
Reports: The reports which should be generated based on a stream
Storage: Locations where reports should be stored

The relationship between these is as follows:

A report is based on the data from a single stream, multiple reports can be generated from the same stream
All reports are stored in all configured storage, unless otherwise specified on the configuration of the storage

A diagram of a potential TopN configuration:

Image title — Overview of potential TopN configuration

To allow dstoredist instance to discover & communicate with TopN streams, a Service object will be created for each stream. The above example will therefore have 2 services available:

The configuration of dstoredist to discover these TopN streams can be found in the dstoredist documentation.

Configuration Reference

Instance Sets

Instances of TopN Reporter can be defined under the root node topns. Each instance set should be a key-value pair, with:

key: Name of the instance set
value: Dictionary holding the configuration of the instance set

topns:
  mytopn:
    < TopN Reporter configuration >

In a minimal configuration you will need to have at least 1 stream, 1 report and 1 storage defined:

topns:
  mytopn:
    < TopN Reporter configuration >
    streams:
      - name: myStream
        < stream configuration >
    reports:
      - name: myReport
        stream: myStream
        < report configuration >
    storage:
      - name: myStorage
        backend: elastic
        < storage configuration >

Parameters which can be used to configure TopN instances for a specific instance set are shown in the below table.

Parameter	Type	Default	Description
`affinity`	`k8s:`Affinity		pod affinity (Kubernetes docs: Affinity and anti-affinity). If unset, a default anti-affinity is applied using `antiAffinityPreset` to spread pods across nodes
`agentLogLevel`	`string`	`"info"`	Verbosity of logging for the agent container. Available options: `"debug"` `"info"` `"warn"` `"error"`
`agentLogFormat`	`string`	`"text"`	Format of logging for the agent container. Available options: `"text"` `"json"`
`agentResources`	`k8s:`Resources	`limits: cpu: 250m memory: 256Mi`	Resources allocated to the agent container if `resourceDefaults` (global) is `true`
`antiAffinityPreset`	`string`	`"preferred"`	pod anti affinity preset. Available options: `"preferred"` `"required"`
`containerSecurityContext`	`k8s:`SecurityContext	`allowPrivilegeEscalation: false readOnlyRootFilesystem: true seccompProfile: type: RuntimeDefault capabilities: drop: - "ALL"`	SecurityContext applied to each container
`hostNetwork`	`boolean`	`false`	Use host networking for pods
`nodeSelector`	`k8s:`NodeSelector	`{}`	Kubernetes pod nodeSelector
`podAnnotations`	`k8s:`Annotations	`{}`	Annotations to be added to each pod
`podDisruptionBudget`	`k8s:`PodDisruptionBudgetSpec	`{}`	Spec of PodDisruptionBudget to be applied to deployment
`podLabels`	`k8s:`Labels	`{}`	Labels to be added to each pod
`podSecurityContext`	`k8s:`PodSecurityContext	`fsGroup: 953 runAsUser: 953 runAsGroup: 953 runAsNonRoot: true`	SecurityContext applied to each pod
`resources`	`k8s:`Resources	`limits: cpu: 1 memory: 2Gi`	Resources allocated to the topn container if `resourceDefaults` (global) is `true`
`reports`	List of Report	`[]`	Reports configuration
`serviceLabels`	`k8s:`Labels	`{}`	Labels to be added to each service
`storage`	List of Storage	`[]`	Storage configuration
`streams`	List of Stream	`[]`	Streams configuration
`tolerations`	List of `k8s:`Tolerations	`[]`	Kubernetes pod Tolerations

Report

Parameters which can be used to configure a TopN report:

Parameter	Type	Required	Default	Description
`entries`	`integer`		`1000`	Maximum number of entries to include in the report
`field`	`string`		`"qname"`	Field to use as the key for the report (see below for possible values)
`interval`	`go:`DurationString		`"300s"`	How often to generate the report (longer interval means longer in-memory storage of data = higher memory usage by TopN)
`name`	`string`	`yes`		Name of the report
`stream`	`string`	`yes`		Name of an input stream defined on this TopN instance

"field" can take the following values:

field	Description
`qname`	The lowercase DNS question name
`qname/raw`	The raw qname, not converted to lowercase
`qname/suffix`	The public suffix of the qname (e.g. .com, .co.uk, etc.)
`qname/suffix+1`	The public suffix plus one label (e.g. example.com, example.co.uk, etc.)
`qname/tld`	The TLD (e.g. com, uk, etc.)
`requestorid`	The subscriber’s username
`ip/prefix32/prefix64`	The IP address of the client, with the IP address aggregated to the v4/v6 prefix specified. For example ip/32/128 would perform no aggregation of v4 or v6 IPs.

Storage

Parameters which can be used to configure a TopN storage backend:

Parameter	Type	Required	Default	Description
`backend`	`string`	`yes`		Type of storage backend. Available options: `"elastic"` See below for more configuration options specific to each backend.
`name`	`string`	`yes`		Name of the storage backend
`reports`	`[]string`			List of Report names (all reports will be stored if this list is empty)
`retry_max`	`integer`		`0`	Maximum number of retries in case of connection errors or HTTP-500
`tlsconfig`	StorageTLSConfig		`{}`	TLS configuration options for the storage backend

Backend: Elastic

Additional parameters which are available on storage backends with backend: elastic. These should be attributes of the storage item itself. For example:

topns:
  mytopn:
    streams:
      < streams >
    reports:
      < reports >
    storage:
      - name: myStorage
        backend: elastic
        url: https://my.elastic.endpoint:9200/
        elastic_index_template: "{{.ReportName}}-{{.TimestampDate}}"

Parameter	Type	Required	Default	Description
`credentialsSecretName`	`string`			Name of a pre-existing Kubernetes Secret containing a username & password for authentication with Elastic
`credentialsSecretPasswordKey`	`string`		`"username"`	If `credentialsSecretName` is specified: name of the item inside the Secret which holds the username
`credentialsSecretUsernameKey`	`string`		`"password"`	If `credentialsSecretName` is specified: name of the item inside the Secret which holds the password
`elastic_id_template`	`string`			Template used to render the Elastic IDs. Randomly generated if not configured
`elastic_index_template`	`string`		`{{.ReportName}}`	Template used to render the name of the index to use in Elastic to store the reports
`elastic_single_doc`	`boolean`		`false`	Store a report in a single document in Elastic
`password`	`string`			Password to use for authentication with Elastic
`url`	`string`	`yes`		Base URL of Elastic instance
`username`	`string`			Username to use for authentication with Elastic

Storage TLS Config

Parameters to configure TLS options, these should be child attributes of the tlsconfig node. For example:

topns:
  mytopn:
    streams:
      < streams >
    reports:
      < reports >
    storage:
      - name: myStorage
        backend: elastic
        url: https://my.elastic.endpoint:9200/
        tlsconfig:
          insecure_skip_verify: true

Parameter	Type	Default	Description
`ca`	`string`		CA in PEM format to use for validation
`caSecretName`	`string`		Name of a pre-existing Kubernetes Secret with a data item named `ca.crt` containing the CA in PEM format to use for validation
`insecure_skip_verify`	`boolean`	`false`	Skip validation of the storage backend's certificate chain and hostname

Stream

Parameters which can be used to configure a TopN stream:

Parameter	Type	Required	Default	Description
`name`	`string`	`yes`		Name of the stream
`title`	`string`			Display friendly name of the stream
`upstream_sampling`	`integer`		`1`	Sampling value used in the dstoredist destination which populates this stream