Changelog

1.5.0

Released: May 3rd, 2023

  • python: update requirements.txt to eliminate setuptools failures

    References: pull request 384

  • docs: upload docs to docs.powerdns.com as part of build

    References: pull request 380, pull request 383, pull request 382

  • dstore-dist: compression support for Kafka

    References: pull request 378

  • go: Update Go to v1.20.2

    References: pull request 378

  • dstore-dist: connect and write timeouts for PDNS destinations

    References: pull request 381

1.4.4

Released: October 20nd, 2022

  • dnspb2json:
    • Include the dnspb2json tool as dstore-dist-dnspb2json in the dstore-dist RPM package.
    • -b option to record raw protobuf streams instead of outputting JSON
    • -L option to output one JSON object per line for easy grep and processing

    References: pull request 371

  • dstore-dist: null Kafka key for better load balancing

    References: pull request 374

  • ev_aggregator: fix elasticsearch webhook username/password config flag parsing

    References: pull request 370

  • dstore-dist: add filters and JSON output for new protobuf fields

    New filters include:

    • tag_prefix
    • socket_protocol
    • is_newly_observed_domain
    • policy_type
    • policy_kind
    • validation_state and is_validation_state_bogus
    • trace_event and trace_match (see event-trace-enabled recursor setting)
    • meta_key, meta_key_string and meta_key_int

    The example config has been updated with examples for the new filters.

    dnspb2json now also supports a -d option that will output JSON in the same format as dstore-dist uses to write to Kafka queues. This format is different from the default format, e.g. it uses snake_case instead of camelCase.

    References: pull request 373

1.4.3

Released: April 22nd, 2022

1.4.3-rc1

Released: March 18th, 2022

  • dstore-dist: Support appending arbitrary tags to messages (route specific)

    References: pull request 366

  • dstore-dist: Build and test on Oracle Linux 8 instead of Centos 8

    References: pull request 367

  • dstore-dist: Add TLS support for both inbound and outbound traffic topn: Add TLS support for inbount traffic

    References: pull request 365

  • report: Support reporting not just on a per-user basis, but also per-device.

    References: pull request 363

  • dstore-ev-aggregator: add support for (now mandatory) “cat:” and “rule:” prefixes while reading tag list

    References: pull request 361

  • dstore-dist, topn: Add them configured to the development enviroment

    References: pull request 364, pull request 369

  • Fix and update regression tests

    References: pull request 362

1.4.2

Released: October 15, 2021

1.4.2-rc1

Released: October 1, 2021

  • top-n: support reporting by source IP

    References: pull request 359

  • top-n: improve templates and provide example configuration files for dstore-dist, topn-reporter and kibana

    References: pull request 349

  • ev_aggregator: performance improvements

    References: pull request 343

  • dstore-dist: add is_incoming_response and is_outgoing_query filters

    References: pull request 357

  • dstore-dist: exit with non-zero code upon error at startup

    References: pull request 348

  • golang: upgrade protobuf dependency to a more recent version

    References: pull request 353

  • egateway: document HTTP API

    References: pull request 352

  • Add a documentation target to the CI

    References: pull request 350

1.4.1

Released: March 12, 2021

1.4.1-beta2

Released: February 23, 2021

1.4.1-beta1

Released: February 23, 2021

  • dstore-dist: add file and syslog destinations support for logging

    References: pull request 346

  • dstore-dist: improve unit tests for protobuf/kafka integration

    References: pull request 345

1.4.0

Released: January 14, 2021

1.4.0-beta3

Released: December 18, 2020

  • build: update alpine linux base images and centos fixes

    References: pull request 344

1.4.0-beta2

Released: December 16, 2020

  • dstore-dist: add support and tooling to measure and report Top N domains

    References: pull request 342

1.4.0-beta1

Released: November 27, 2020

  • dstore-dist: support writing dns messages as JSON for Kafka destinations

    References: pull request 339

  • all: support new Device ID format while keeping backward compatibility

    References: pull request 340

1.4.0-alpha2

Released: November 13, 2020

  • dstore-dist: add TLS support for Kafka destination

    References: pull request 332

  • dstore-dist: add support for blackhole destination

    References: pull request 329

  • doc: fix pdf generation

    References: pull request 337

  • python-dist: upgrade Twisted package

    References: pull request 327

  • dstore-dist: kafka: allow multiple dnsmessage per kafka message

    References: pull request 333

1.4.0-alpha1

Released: October 27, 2020

  • dstore-dist: add Kafka support as destination for outgoing messages

  • dstore-dist: add sampling and rate limiting support

    References: pull request 314

  • dstore-dist: allow filtering for qname and subdomains of qname

    References: pull request 323

1.3.3

Released: July 27, 2020

  • Use unique build IDs for debug files.

    References: pull request 324

1.3.2

Released: July 22, 2020

1.3.2-beta2

Released: July 17, 2020

  • Also build release packages for CentOS 8.

1.3.2-beta1

Released: July 17, 2020

  • Add support for centos 8

    References: pull request 322

1.3.2-alpha1

Released: May 14, 2020

  • Fix compilation issues with recent versions of the build chain

    References: pull request 311, pull request 310

  • Add override file to the dstore-report-api gunicorn config

    References: pull request 316

  • Update Event Aggregator to handle protobuf for non-filtered DNS queries

    References: pull request 318

1.3.1

Released: February 11, 2020

  • Build and use our own python distribution.

    References: pull request 306

  • Sharding support in dstore-dist and other improvements.

    References: pull request 305

  • Aggregation feature for ev_aggregator.

    References: pull request 307

1.3.0

Released: November 19, 2019

1.3.0-beta3

Released: November 6, 2019

  • report-api: Make sure a default value PROM_STATS_DIR is defined.

    References: pull request 302

  • report-api: Fix dstore install on RHEL 7.

    References: pull request 300

1.3.0-beta2

Released: October 23, 2019

  • event-aggregator: Add support to Prometheus metrics.

    References: pull request 278

  • report-api: Add support to Prometheus metrics.

    References: pull request 279

1.3.0-beta1

Released: July 30, 2019

  • egateway: Allow searching by device name.

    References: pull request 288

  • text2tcp: Close the connection gracefully in order to avoid issues on the server end.

    References: pull request 264

  • Fix several issues that came up deploying the dstore-1.3.0 alphas:

    • dstore-ev-aggregator: fix an issue preventing Redis authentication to work correctly when the password is specified in the config file instead of command line,
    • dstore-report-api: handle API queries correctly when usernames (and possibly other fields) can be tokenised,
    • dstore-ev-aggregator: add a retry mechanism to gracefully handle situations where Redis connections are broken.

    References: pull request 289

  • dstore-report-api: Fix OpenAPI spec error where user_id was specified instead of username.

    References: pull request 287

1.3.0-alpha6

Released: June 21, 2019

  • Fix a couple of nits for event aggregator and dstore_alert

    References: pull request 285

  • UI: remove queries without response stat

    References: pull request 284

  • Improves code readability

    References: pull request 265

  • egateway: upgrade HTTP handling to libh2o

    References: pull request 250

1.3.0-alpha5

Released: June 14, 2019

  • Fix in protobuf split function and batch pool handling.

    References: pull request 266

  • Web UI: export results as CSV

    References: pull request 262

  • dgrep: Allow to lookup for outgoing queries.

    References: pull request 260

  • Show versions in UI.

    References: pull request 258

  • dnspbgen: Add real-time flag to throttle message generation.

    References: pull request 253

  • dgrep: Add timestamp range specifier option.

    References: pull request 246

1.3.0-alpha4

Released: May 29, 2019

Note: versions 1.3.0-alpha1, -alpha2 and -alpha3 are internal only.

  • Reporting API

  • Event Aggregator

  • Make user grouping interval configurable for scan_malware.

1.2.7

Released: February 8, 2019

DEPLOYMENT NOTES

Django Prometheus Metrics

To monitor database query metrics, you must replace the ENGINE property of your database, replacing django.db.backends with django_prometheus.db.backends:

DATABASES = {
    'default': {
        'ENGINE': 'django_prometheus.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    },
}

This has already been applied to the default sqlite database.

Add API-key Option to Setup API Authentication

This PR adds support for egateway API authentication.

This is done by configuring egateway with the desired secret (api-key option). The secret must be provided through the HTTP x-api-key header.

To match that need, dstore-web EGATEWAY_URLS has been updated to support URIs like https://egateway.local:1234#s3cr3t.

Changelog

  • egateway: limit the number of results scanned by egateway

  • ui: malware scan ‘no results found’ message

  • tcpdistro: prevent FD leak when opening a corrupted file

  • egateway: fix a nullptr deref on a query w/ only additional filters

  • egateway: add api-key option to setup api authentication

  • egateway: allow multiple search terms in query

  • dstore-web: django prometheus metrics

  • malware_scan cmd fixes

  • enable hardening measures (SSP, PIE, full RELRO, fortify)

  • crow: properly stop metrics webserver

  • ui: adjust search results fields and add tcp field

  • dgrep: add support for customer and device query params

  • prometheus: only display metric header for distinct metrics

  • just as for the new dcat utility below, this adds --raw modifier to dgrep output. Combined with --quiet, you get a stream of raw protobuf messages instead of json output

  • this adds a dcat utility that pretty-prints a dstore data file

  • tcpdistro: add compression-level parameter

1.2.6

Released: January 14, 2019

  • tcpdistro: Move bucket list to a LRU list with regular cleaning

1.2.5

Released: November 14, 2018

NOTE: DStore 1.2.5 needs to be reinstalled but after that upgrading will work again.

  • Update dnsmessage.proto to sync with PDNS

  • dstore-web: Fix accidental removal of /usr/share/dstore-web when upgrading dstore

  • egateway: Add an ‘ecs-override-requestor’ option, export more values via the API

    • Clean up the protobuf bytes to ComboAddress conversions
    • Export ‘ecs’, ‘serverId’, a non-overridden ‘from’ and ‘tcp’ via the API
    • Add an ‘ecs-override-requestor’ option to be able to configure whether an ECS value should override the ‘from’ value when present (default, existing behaviour)

1.2.4

Released: October 5, 2018

  • Grafana dashboard for dstore

  • dstore-web: change default search range from 12 months to 1 hour

  • dstore-web: fix error on packages upgrade

  • pbscanner: fix the error message when reading from a corrupted file

  • dirwalker: explictly remove copy constructor

1.2.3

Released: September 27, 2018

  • Define systemd SyslogIdentifier

  • metrics: Add Prometheus support to tcpdistro and egateway

  • dgrep: Add username and device ID to output

  • pbscanner: Close the directory file descriptor in DirWalker

1.2.2

Released: June 1, 2018

  • Sharding

  • Support for outgoing protobuf logs

  • program:`dnspbgen

  • Stability improvements

  • Accept all 2xx HTTP codes as positive

1.2.1

Released: June 1, 2018

  • Support New Notification Centre API

1.2.0

Released: May 28, 2018

  • The dstore-web packages are now built against Python 3.6.

  • Drop requirement for Protobuf 3.

  • Add on-disk compression based on zstd. This is enabled by default.

  • Add dstore-dist, a service that can duplicate and shard data over multiple dstore nodes.

  • Add support for storing, searching and extracting per-device information.