Changelog ========= .. changelog:: :version: 2.0.0 :released: Dec 5th, 2023 .. change:: :tags: Improvement :pullreq: 389 packaging: Enterprise Linux 9 Support and many python fixes packaging: Production image for dstore-dist and top-n-reporter uploaded to OX registry in CI .. change:: :tags: Improvement :pullreq: 311 tests: Add and update dstore-dist tests in CI .. change:: :tags: Refactoring :pullreq: 318 cleanup: Remove legacy dstore code leaving only dstore-dist and related tools .. change:: :tags: Improvement :pullreq: 313 security: Add features outstanding from Nixu code-audit .. change:: :tags: Improvement :pullreq: 316 logging: Add structured logging to dstore-dist .. change:: :tags: Improvement :pullreq: 319 api: Add history API to dstore-dist for debugging .. changelog:: :version: 1.5.0 :released: May 3rd, 2023 .. change:: :tags: Improvement :pullreq: 384 python: update ``requirements.txt`` to eliminate setuptools failures .. change:: :tags: Improvement :pullreq: 380, 382, 383 docs: upload docs to docs.powerdns.com as part of build .. change:: :tags: New features :pullreq: 378 dstore-dist: compression support for Kafka .. change:: :tags: Improvement :pullreq: 378 go: Update Go to v1.20.2 .. change:: :tags: New features :pullreq: 381 dstore-dist: connect and write timeouts for PDNS destinations .. changelog:: :version: 1.4.4 :released: October 20nd, 2022 .. change:: :tags: Improvement :pullreq: 371 dnspb2json: * Include the dnspb2json tool as dstore-dist-dnspb2json in the dstore-dist RPM package. * -b option to record raw protobuf streams instead of outputting JSON * -L option to output one JSON object per line for easy grep and processing .. change:: :tags: Improvement :pullreq: 374 dstore-dist: null Kafka key for better load balancing .. change:: :tags: Improvement :pullreq: 370 ev_aggregator: fix elasticsearch webhook username/password config flag parsing .. change:: :tags: Improvement :pullreq: 373 dstore-dist: add filters and JSON output for new protobuf fields New filters include: * `tag_prefix` * `socket_protocol` * `is_newly_observed_domain` * `policy_type` * `policy_kind` * `validation_state` and `is_validation_state_bogus` * `trace_event` and `trace_match` (see event-trace-enabled recursor setting) * `meta_key`, `meta_key_string` and `meta_key_int` The example config has been updated with examples for the new filters. dnspb2json now also supports a -d option that will output JSON in the same format as dstore-dist uses to write to Kafka queues. This format is different from the default format, e.g. it uses snake_case instead of camelCase. .. changelog:: :version: 1.4.3 :released: April 22nd, 2022 .. changelog:: :version: 1.4.3-rc1 :released: March 18th, 2022 .. change:: :tags: Improvement :pullreq: 366 dstore-dist: Support appending arbitrary tags to messages (route specific) .. change:: :tags: Improvement :pullreq: 367 dstore-dist: Build and test on Oracle Linux 8 instead of Centos 8 .. change:: :tags: Improvement :pullreq: 365 dstore-dist: Add TLS support for both inbound and outbound traffic topn: Add TLS support for inbount traffic .. change:: :tags: Improvement :pullreq: 363 report: Support reporting not just on a per-user basis, but also per-device. .. change:: :tags: Improvement :pullreq: 361 dstore-ev-aggregator: add support for (now mandatory) "cat:" and "rule:" prefixes while reading tag list .. change:: :tags: Improvement, Dev :pullreq: 364, 369 dstore-dist, topn: Add them configured to the development enviroment .. change:: :tags: Bug fixes, Dev :pullreq: 362 Fix and update regression tests .. changelog:: :version: 1.4.2 :released: October 15, 2021 .. changelog:: :version: 1.4.2-rc1 :released: October 1, 2021 .. change:: :tags: Improvement :pullreq: 359 top-n: support reporting by source IP .. change:: :tags: Improvement :pullreq: 349 top-n: improve templates and provide example configuration files for dstore-dist, topn-reporter and kibana .. change:: :tags: Improvement :pullreq: 343 ev_aggregator: performance improvements .. change:: :tags: Improvement :pullreq: 357 dstore-dist: add is_incoming_response and is_outgoing_query filters .. change:: :tags: Improvement :pullreq: 348 dstore-dist: exit with non-zero code upon error at startup .. change:: :tags: Improvement :pullreq: 353 golang: upgrade protobuf dependency to a more recent version .. change:: :tags: Documentation :pullreq: 352 egateway: document HTTP API .. change:: :tags: Documentation :pullreq: 350 Add a documentation target to the CI .. changelog:: :version: 1.4.1 :released: March 12, 2021 .. changelog:: :version: 1.4.1-beta2 :released: February 23, 2021 .. changelog:: :version: 1.4.1-beta1 :released: February 23, 2021 .. change:: :tags: New features :pullreq: 346 dstore-dist: add file and syslog destinations support for logging .. change:: :tags: Improvements :pullreq: 345 dstore-dist: improve unit tests for protobuf/kafka integration .. changelog:: :version: 1.4.0 :released: January 14, 2021 .. changelog:: :version: 1.4.0-beta3 :released: December 18, 2020 .. change:: :tags: Bug Fixes :pullreq: 344 build: update alpine linux base images and centos fixes .. changelog:: :version: 1.4.0-beta2 :released: December 16, 2020 .. change:: :tags: New features :pullreq: 342 dstore-dist: add support and tooling to measure and report Top N domains .. changelog:: :version: 1.4.0-beta1 :released: November 27, 2020 .. change:: :tags: New features :pullreq: 339 dstore-dist: support writing dns messages as JSON for Kafka destinations .. change:: :tags: Improvements :pullreq: 340 all: support new Device ID format while keeping backward compatibility .. changelog:: :version: 1.4.0-alpha2 :released: November 13, 2020 .. change:: :tags: New Features :pullreq: 332 dstore-dist: add TLS support for Kafka destination .. change:: :tags: New Features :pullreq: 329 dstore-dist: add support for blackhole destination .. change:: :tags: Improvements :pullreq: 337 doc: fix pdf generation .. change:: :tags: Bug Fixes :pullreq: 327 python-dist: upgrade Twisted package .. change:: :tags: Improvements :pullreq: 333 dstore-dist: kafka: allow multiple dnsmessage per kafka message .. changelog:: :version: 1.4.0-alpha1 :released: October 27, 2020 .. change:: :tags: New Features :pullreq: dstore-dist: add Kafka support as destination for outgoing messages .. change:: :tags: New Features :pullreq: 314 dstore-dist: add sampling and rate limiting support .. change:: :tags: Improvements :pullreq: 323 dstore-dist: allow filtering for qname and subdomains of qname .. changelog:: :version: 1.3.3 :released: July 27, 2020 .. change:: :tags: Improvements :pullreq: 324 Use unique build IDs for debug files. .. changelog:: :version: 1.3.2 :released: July 22, 2020 .. changelog:: :version: 1.3.2-beta2 :released: July 17, 2020 .. change:: :tags: Improvements Also build release packages for CentOS 8. .. changelog:: :version: 1.3.2-beta1 :released: July 17, 2020 .. change:: :tags: Improvements :pullreq: 322 Add support for centos 8 .. changelog:: :version: 1.3.2-alpha1 :released: May 14, 2020 .. change:: :tags: Bug Fixes :pullreq: 310, 311 Fix compilation issues with recent versions of the build chain .. change:: :tags: Improvements :pullreq: 316 Add override file to the dstore-report-api gunicorn config .. change:: :tags: Bug Fixes :pullreq: 318 Update Event Aggregator to handle protobuf for non-filtered DNS queries .. changelog:: :version: 1.3.1 :released: February 11, 2020 .. change:: :tags: Improvements :pullreq: 306 Build and use our own python distribution. .. change:: :tags: Improvements :pullreq: 305 Sharding support in dstore-dist and other improvements. .. change:: :tags: Improvements :pullreq: 307 Aggregation feature for ``ev_aggregator``. .. changelog:: :version: 1.3.0 :released: November 19, 2019 .. changelog:: :version: 1.3.0-beta3 :released: November 6, 2019 .. change:: :tags: Bug Fixes :pullreq: 302 report-api: Make sure a default value PROM_STATS_DIR is defined. .. change:: :tags: Bug Fixes :pullreq: 300 report-api: Fix dstore install on RHEL 7. .. changelog:: :version: 1.3.0-beta2 :released: October 23, 2019 .. change:: :tags: Improvements :pullreq: 278 event-aggregator: Add support to Prometheus metrics. .. change:: :tags: Improvements :pullreq: 279 report-api: Add support to Prometheus metrics. .. changelog:: :version: 1.3.0-beta1 :released: July 30, 2019 .. change:: :tags: New Features :pullreq: 288 egateway: Allow searching by device name. .. change:: :tags: Improvements :pullreq: 264 text2tcp: Close the connection gracefully in order to avoid issues on the server end. .. change:: :tags: Bug Fixes :pullreq: 289 Fix several issues that came up deploying the dstore-1.3.0 alphas: - dstore-ev-aggregator: fix an issue preventing Redis authentication to work correctly when the password is specified in the config file instead of command line, - dstore-report-api: handle API queries correctly when usernames (and possibly other fields) can be tokenised, - dstore-ev-aggregator: add a retry mechanism to gracefully handle situations where Redis connections are broken. .. change:: :tags: Bug Fixes :pullreq: 287 dstore-report-api: Fix OpenAPI spec error where ``user_id`` was specified instead of ``username``. .. changelog:: :version: 1.3.0-alpha6 :released: June 21, 2019 .. change:: :tags: Improvements :pullreq: 285 Fix a couple of nits for event aggregator and dstore_alert .. change:: :tags: Improvements :pullreq: 284 UI: remove `queries without response` stat .. change:: :tags: Improvements :pullreq: 265 Improves code readability .. change:: :tags: Improvements :pullreq: 250 egateway: upgrade HTTP handling to libh2o .. changelog:: :version: 1.3.0-alpha5 :released: June 14, 2019 .. change:: :tags: Bug Fixes :pullreq: 266 Fix in protobuf split function and batch pool handling. .. change:: :tags: New Features :pullreq: 262 Web UI: export results as CSV .. change:: :tags: Improvements :pullreq: 260 dgrep: Allow to lookup for outgoing queries. .. change:: :tags: Improvements :pullreq: 258 Show versions in UI. .. change:: :tags: Improvements :pullreq: 253 dnspbgen: Add real-time flag to throttle message generation. .. change:: :tags: Improvements :pullreq: 246 dgrep: Add timestamp range specifier option. .. changelog:: :version: 1.3.0-alpha4 :released: May 29, 2019 *Note: versions 1.3.0-alpha1, -alpha2 and -alpha3 are internal only.* .. change:: :tags: New Features Reporting API .. change:: :tags: New Features Event Aggregator .. change:: :tags: Improvements Make user grouping interval configurable for `scan_malware`. .. changelog:: :version: 1.2.7 :released: February 8, 2019 **DEPLOYMENT NOTES** **Django Prometheus Metrics** To monitor database query metrics, you must replace the ``ENGINE`` property of your database, replacing ``django.db.backends`` with ``django_prometheus.db.backends``: :: DATABASES = { 'default': { 'ENGINE': 'django_prometheus.db.backends.sqlite3', 'NAME': os.path.join(BASE_DIR, 'db.sqlite3'), }, } This has already been applied to the default sqlite database. **Add API-key Option to Setup API Authentication** This PR adds support for ``egateway`` API authentication. This is done by configuring ``egateway`` with the desired secret (``api-key`` option). The secret must be provided through the HTTP ``x-api-key`` header. To match that need, ``dstore-web`` ``EGATEWAY_URLS`` has been updated to support URIs like ``https://egateway.local:1234#s3cr3t``. **Changelog** .. change:: :tags: Improvements egateway: limit the number of results scanned by egateway .. change:: :tags: Improvements ui: malware scan 'no results found' message .. change:: :tags: Bug Fixes tcpdistro: prevent FD leak when opening a corrupted file .. change:: :tags: Bug Fixes egateway: fix a nullptr deref on a query w/ only additional filters .. change:: :tags: New Features egateway: add api-key option to setup api authentication .. change:: :tags: Improvements egateway: allow multiple search terms in query .. change:: :tags: New Features dstore-web: django prometheus metrics .. change:: :tags: Bug Fixes, Improvements ``malware_scan`` cmd fixes .. change:: :tags: Improvements enable hardening measures (SSP, PIE, full RELRO, fortify) .. change:: :tags: Bug Fixes crow: properly stop metrics webserver .. change:: :tags: Improvements ui: adjust search results fields and add ``tcp`` field .. change:: :tags: Improvements dgrep: add support for customer and device query params .. change:: :tags: Bug Fixes prometheus: only display metric header for distinct metrics .. change:: :tags: Improvements just as for the new dcat utility below, this adds ``--raw`` modifier to dgrep output. Combined with ``--quiet``, you get a stream of raw protobuf messages instead of json output .. change:: :tags: New Features this adds a dcat utility that pretty-prints a dstore data file .. change:: :tags: New Features tcpdistro: add compression-level parameter .. changelog:: :version: 1.2.6 :released: January 14, 2019 .. change:: :tags: Improvements tcpdistro: Move bucket list to a LRU list with regular cleaning .. changelog:: :version: 1.2.5 :released: November 14, 2018 **NOTE**: DStore 1.2.5 needs to be reinstalled but after that upgrading will work again. .. change:: :tags: Improvements Update dnsmessage.proto to sync with PDNS .. change:: :tags: Bug Fixes dstore-web: Fix accidental removal of /usr/share/dstore-web when upgrading dstore .. change:: :tags: Bug Fixes egateway: Add an 'ecs-override-requestor' option, export more values via the API - Clean up the protobuf bytes to ComboAddress conversions - Export 'ecs', 'serverId', a non-overridden 'from' and 'tcp' via the API - Add an 'ecs-override-requestor' option to be able to configure whether an ECS value should override the 'from' value when present (default, existing behaviour) .. changelog:: :version: 1.2.4 :released: October 5, 2018 .. change:: :tags: New Features Grafana dashboard for dstore .. change:: :tags: Improvements dstore-web: change default search range from 12 months to 1 hour .. change:: :tags: Bug Fixes dstore-web: fix error on packages upgrade .. change:: :tags: Bug Fixes pbscanner: fix the error message when reading from a corrupted file .. change:: :tags: Bug Fixes dirwalker: explictly remove copy constructor .. changelog:: :version: 1.2.3 :released: September 27, 2018 .. change:: :tags: New Features Define systemd SyslogIdentifier .. change:: :tags: New Features metrics: Add Prometheus support to tcpdistro and egateway .. change:: :tags: Improvements dgrep: Add username and device ID to output .. change:: :tags: Bug Fixes pbscanner: Close the directory file descriptor in DirWalker .. changelog:: :version: 1.2.2 :released: June 1, 2018 .. change:: :tags: New Features Sharding .. change:: :tags: New Features Support for outgoing protobuf logs .. change:: :tags: New Features `program:`dnspbgen` .. change:: :tags: New Features `program:`dnspb2json` .. change:: :tags: Bug Fixes Stability improvements .. change:: :tags: Bug Fixes Accept all 2xx HTTP codes as positive .. changelog:: :version: 1.2.1 :released: June 1, 2018 .. change:: :tags: New Features, API Support New Notification Centre API .. changelog:: :version: 1.2.0 :released: May 28, 2018 .. change:: :tags: Web The :program:`dstore-web` packages are now built against Python 3.6. .. change:: :tags: Internals Drop requirement for Protobuf 3. .. change:: :tags: New Features Add on-disk compression based on `zstd `__. This is enabled by default. .. change:: :tags: New Features Add :program:`dstore-dist`, a service that can duplicate and shard data over multiple :program:`dstore` nodes. .. change:: :tags: New Features, Web Add support for storing, searching and extracting per-device information.