dstore-dist

Root Node Config

A minimal dstore-dist configuration (missing the details of the destination and route configuration) could be:

listen:
  - "127.0.0.1:2000"
  - "[::1]:2000"
tls_listen:
  - addr: "127.0.0.1:2001"
    tlsconfig:
      cert_file: /etc/certs/cert.pem
      key_file: /etc/certs/cert.key
log:
  level: Warn
ipsets:
  ipset1:
    file: /etc/ipset1.txt
    poll_interval: 10s
destinations:
  mydestination1:
    <destination configuration>
  mydestination2:
    <destination configuration>
routes:
  myroute1:
    <route configuration>
  myroute2:
    <route configuration>

The following YAML key-values are supported for configuration at the root node:

Parameter	Type	Default	Description
listen	List of string		The addresses that dstore-dist will listen on for new protobuf messages. The value is a list of address:port strings, in either v4 or v6 format. IPv6 addresses must be placed in square brackets like this [::1]. You can omit the address to listen on all local addresses.
tls_listen	List of TLS Listen		The addresses that dstore-dist will listen on for new protobuf messages. The value is a list of address:port strings, in either v4 or v6 format. IPv6 addresses must be placed in square brackets like this [::1]. You can omit the address to listen on all local addresses.
http_address	string		Listen address for HTTP webserver for Prometheus metrics and status page. Value is an address:port string, using the same format as for listen addresses.
history_num_batches	int		Number of batches to keep in memory for the history API. A batch is capped at 1 MB by default, see batch_buffer_size for the actual size. If this is unset or set to 0, the history API is disabled. The history API can be accessed at /api/history. By default this URL will print additional usage information, including how to switch to Protobuf or JSON output format, and how to filter on certain fields.
http_api_key	string		If set, an X-API-Key header matching this key is required to access the optional HTTP history API.
log	Log Config
conn_timeout	go:DurationString		The idle timeout for incoming connections to dstore-dist. Defaults to no timeout, i,e. only applies if explicitly configured.
batch_buffer_size	int	1048576	The size in bytes of the buffer(s) used to batch messages internally. Normally the default of 1048576 (1MB) will suffice, but when using Kafka, making this value around 90% of the value of the kafka max_msg_size is a good idea, to avoid fragmentation of the messages when sending to Kafka.
ipsets	Map of IP Set
debug	boolean	false	Enable/disable very verbose debug logging
destinations	Map of Destination		Each map key is the name of a destination, and the value is a Destination
routes	Map of Route		Each map key is the name of a route, and the value is a Route

Log Config

Log Configuration is as follows:

Parameter	Type	Default	Description
text	boolean	false	If true, enable text-based logging. The default is JSON logging.
stdout	boolean	false	By default logging will go to stderr, set this to true send to stdout instead.
level	string		Defaults to Info. Can also be set to Warn, Error or Debug. For debug level to take effect, the -debug flag must also be set.

TLS Listen

You can configure a TLS Listener as follows:

Parameter	Type	Default	Description
addr	string		The address:port to listen on
tlsconfig	TLS Config		Configuration of TLS parameters

TLS Config

Parameter	Type	Default	Description
insecure_skip_verify	boolean	false	Controls whether a client verifies the server's certificate chain and hostname.
ca_file	string		Optional CA file to use (PEM).
ca	string		Optional CA to use specified as a string in PEM format.
add_system_ca_pool	boolean	false	Adds the system CA pool if private CAs are enabled, when set.
cert_file	string		Optional certificate file to use (PEM).
cert	string		Optional certificate to use specified as a string in PEM format.
key_file	string		Optional key file to use (PEM).
key	string		Optional key to use specified as a string in PEM format.
watch_certs	boolean	false	If true, enables background reloading of certifcate files.
watch_certs_poll_interval	go:DurationString	5s	If watch_certs is true, how often to check for changes.

IPSet

Parameter	Type	Required	Default	Description
file	string	yes		The path to a file containing newline separated IP prefixes (v4 or v6) which can be used for filtering events with various filters.
poll_interval	go:DurationString		5s	How often to check the file for changes.

An example IPSet file is shown below:

127.0.0.1/16
128.243.0.0/16
# Comments beginning with # are allowed
fe80::1cc0:3e8c:119f:c2e1/18

Destination

The following parameters can be used to configure a destination:

Parameter	Type	Default	Required	Description
type	string	pdns		Type of destination. Available options: "pdns" "kafka" "storage" "topn". See below for more configuration options specific to each destination type.
blackhole	string			Optional CA file to use (PEM).
burst	string			Optional certificate file to use (PEM).
rate	boolean	false		Adds the system CA pool if private CAs are enabled, when set.
sample	string			Optional CA to use specified as a string in PEM format.

Destination: pdns

Additional parameters are available on dstore-dist destinations with type: pdns (or no type). These should be attributes of the destination item itself. For example:

destinations:
  mydestination:
    type: pdns
    addresses:
      - myhost.example.com:1234

Parameter	Type	Required	Default	Description
addresses	List of string	yes		List of addresses of downstream servers supporting the pdns protobuf protocol. Should be either IP:port or host:port
connect_timeout	go:DurationString		"5s"	How long to wait before timing out connection attempts
distribute	string		roundrobin	Distribution algorithm when multiple addresses are configured. Available options: "roundrobin" "sharded"
framing	string		"16bit"	How to frame protobuf messages. Available options: "16bit" "32bit" "repeated"
tlsconfig	TLS Config		{}	TLS configuration options
use_tls	boolean		false	If true, attempt to connect to the addresses using TLS
write_timeout	go:DurationString		"5s"	How long to wait before timing out write attempts

Destination: kafka

Additional parameters are available on destinations with type: kafka. These should be nested attributes inside a kafka: item of the destination item itself. For example:

destinations:
  mydestination:
    type: kafka
    kafka:
      addresses:
        - kafka.endpoint.local:9092
      topic: mytopic

The nested kafka: attribute takes the following parameters:

Parameter	Type	Required	Default	Description
addresses	List of string	yes		List of addresses of kafka endpoints. Should be either IP:port or host:port
async	boolean		false	If true, dstore-dist writes to Kafka never block and all responses from Kafka are ignored
balancer	string		"roundrobin"	Balancer used to distribute Kafka messages amongst partitions. Available options: "roundrobin" "leastbytes" "fnv-1a" "crc32" "murmur2"
batch_size	integer		10000	Number of messages which will constitute a batch. dstore-dist will wait for new messages until either the batch size is reached, or the batch_timeout is exceeded
batch_timeout	go:DurationString		1ms	Timeout before an incomplete batch is written to Kafka
compression	string		""	Compression codec to use. Available options: "gzip" "snappy" "lz4" "zstd" No compression is performed when empty
instance_name	string			If configured, a header will be added to each Kafka message with this value
json_encode	boolean		false	If true, JSON encode the data before sending to Kafka
max_attempts	integer		2	Number of times a message will be attempted to send to kafka
max_msg_size	integer		900000	Maximum size of a kafka message (in bytes). Cannot be lower than 65536.
num_workers	integer		2	Number of concurrent workers that will process protobuf messages and send them to kafka
read_timeout	go:DurationString		10s	Timeout for reads from Kafka
required_acks	string		"one"	How many acks are required from kafka. Available options: "one" "all" "none"
single_msgs	boolean		false	If true, each Kafka message will only contain a single protobuf message
tlsconfig	TLS Config		{}	TLS configuration options
topic	string	yes		Name of Kafka topic to send messages to
use_tls	boolean		false	If true, attempt to connect to the addresses using TLS
write_timeout	go:DurationString		10s	Timeout for writes to Kafka

Destination: storage

Additional parameters are available on destinations with type: storage. These should be nested attributes inside a storage: item of the destination item itself. For example:

destinations:
  mydestination:
    type: storage
    storage:
      type: s3
      encoding: json
      options:
        secretName: myS3secret
        endpoint_url: https://my.s3.endpoint.local
        bucket: myBucket
        region: myRegion

The nested storage: attribute takes the following parameters:

Parameter	Type	Required	Default	Description
type	string		"s3"	Type of storage backend. Available options: "s3" "fs"
encoding	string		"protobuf"	Encoding used for the files. Available options: "protobuf" "json" "bind"
flush_interval	go:DurationString		300s	Time between consecutive flushes to storage (only if max_size is not reached before this interval)
max_size	integer			Maximum size in bytes of the file (before compression). Defaults to dstore-dist's toplevel configuration item batchBufferSize. If batchBufferSize is not set the value will be 1048576
num_workers	integer		2	Number of concurrent workers that will process protobuf messages and attempt to store them
options	Storage Options	yes		Configuration of the storage backend
use_compression	boolean		false	If true, compress files using gzip compression

Storage Options

For storage with type: s3 the following can be configured under options:

Parameter	Type	Required	Default	Description
access_key	string			S3 access key. Use of secretName is recommended for security reasons
bucket	string	yes		Name of the S3 bucket
endpoint_url	string	yes		Endpoint of the S3 service to connect to
region	string		"us-east-1"	Region used when connecting to the S3 endpoint
secret_key	string			S3 secret key. Use of secretName is recommended for security reasons
secretAccessKey	string		"access_key"	If secretName is specified: name of the item inside the Secret which holds the access key
secretName	string			Name of a pre-existing Kubernetes Secret containing a username & password for authentication with Elastic
secretSecretKey	string		"secret_key"	If secretName is specified: name of the item inside the Secret which holds the secret key
tlsconfig	TLS Config		{}	TLS configuration options

For storage with type: fs the following can be configured under options:

Parameter	Type	Required	Default	Description
root_path	string	yes		The path to a directory in which the files will be stored

Route

Parameters which can be used to configure a dstore-dist route:

destinations:
  mydestination:
    type: pdns
    addresses:
      - another.dstoredist.endpoint.local:1234
routes:
  myroute:
    destinations:
      - mydestination

Parameter	Type	Required	Default	Description
append_tags	List of string		[]	List of tags which will be appended to each message for this route
destinations	List of string	yes		List of names of destinations, these must have been configured on this dstore-dist instance
filters	List of Filter		{}	Filters to restrict which messages are sent for this route

Filters

By default, all events will be sent to a particular destination, however configuring filters for a route allows only events matching the filter to be sent.

For example the following filters configuration ensures that only events that contain the query name foo.com and are sent over IPv4 transport are sent to the destinations listed under myroute:

destinations:
  mydestination:
    type: pdns
    addresses:
      - another.dstoredist.endpoint.local:1234
routes:
  myroute:
    destinations:
      - mydestination
    filters:
      - qname: foo.com
      - is_ipv4: true

Note that the top-level filters are joined with an implicit and filter, meaning that all filters have to match for a message to reach the specified destination(s).

Matching Filters

Matching filters are used to match events based on specific information in the query/response. The list of possible filters is listed below:

Parameter	Type	Default	Description
has_deviceid	boolean		Matches if the message has a deviceid field
has_policy	boolean		Matches if message has an appliedPolicy field in the response
has_requestorid	boolean		Matches if message has a requestorID fields in the response
has_tags	boolean		Matches if message has any tags set
is_ipv4	boolean		Matches if the DNS query was received over IPv4
is_ipv4	boolean		Matches if the DNS query was received over IPv4
is_response	boolean		Matches if the message is a response message (as opposed to a request message)
is_tcp	boolean		Matches if the DNS query was received over TCP
is_udp	boolean		Matches if the DNS query was received over UDP
latency_msec	integer		Matches the latency field in the message, in milliseconds
latency_msec_gte latency_msec_gt latency_msec_lte latency_msec_lt	integer		These perform integer comparisons on the latency_msec field in the message
latency_msec_not	integer		This is simply the inverse of latency_msec
qname	string		The value is a domain name; the filter matches if the query qname is an exact match (case-insensitive) for the specified domain name.
qname_sub	string		The value is a domain name; the filter matches if the query qname is a subdomain of the specified domain name. Matches are case-insensitive.
qtype	string or integer		The value matches the query resource record type of the request. It can be specified as a string or an integer, as any string will be converted to an integer using the mapping specified in https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml. If the type is very new, you may need to use the integer version.
qtype_gte qtype_gt qtype_lte qtype_lt	string or integer		These perform integer comparisons on qtype, after converting the type to an integer
qtype_not	string or integer		This is simply the inverse of qtype
tag	string		Match a specific tag
tag_prefix	string		Match the start of a tag
from_in_ipset	string		Match all messages where the from IP is in a range contained in the named IP set
reqsubnet_in_ipset	string		Match all messages where the 'origRequestedSubnet' IP is in a range contained in the named IP set
rcode	integer		Match the response code in the message (only for messages that contain a response)
rcode_gte rcode_gt rcode_lte rcode_lt	integer		These perform integer comparisons on the response code in the message
rcode_not	integer		This is simply the inverse of rcode

Boolean Logic Filters

The and, or and not filters are used to combine or invert matching filters to create more complex filter patterns.

Parameter	Type	Default	Description
and	List of Filter		Applies a logical AND to all the specified filters
or	List of Filter		Applies a logical OR to all the specified filters
not	Filter		Inverts the specified filter

For example:

routes:
  myroute:
    destinations:
      - mydestination
    filters:
      - tag: REQUIRED_TAG
      - or:
        - tag: GAMBLING
        - tag: FASHION
      - not:
          qtype: foo.com