Installation and Configuration¶
The supported method of installing the dstore software components is using the Ansible role provided by Open-Xchange. This chapter describes the packages, their contents and configuration files for completeness.
Repository¶
All packages required for dstore are provided in a software repository maintained by Open-Xchange. If you do not have access to this repository, please contact Open-Xchange support.
This repository contains the packages built for the deployment’s operating system version and architecture.
Directories and Files¶
dstore store its data in /var/lib/dstore by default.
Furthermore, unless otherwise specified, all program configuration is in /etc/dstore.
In that configuration directory, there is one configuration file for each service and one global file.
This global configuration is called dstore.conf and it contains several settings.
| storage-dir: | The path to the directory where the data is stored. This can be changed if /var/lib/dstore is not the preferred directory. |
|---|---|
| hash-split: | The number of buckets the data is split into. |
Todo
Write about disk-layout and mounting multiple disks under /var/lib/dstore for performance reasons.
tcpdistro¶
The tcpdistro service accepts a stream of protobuf-encoded messages and stores them to disk.
On start-up, it reads dstore.conf and tcpdistro.conf.
The latter file contains these settings:
| listen-address: | The IP and port the service listens on |
|---|---|
| answer-limit: | The maximum number of answer-RRs to store per answer packet |
When dstore-alert is used the following settings are relevant:
| enable-alerts: | Whether or not to enable sending alerts to dstore-alert |
|---|---|
| alert-address: | The address where dstore-alert can receive messages. |
| alert-tags: | Incoming packets with these tags will be sent to dstore-alert |
dstore_alert¶
The dstore_alert service accepts a stream of
protobuf-encoded messages and send them to an HTTP endpoint using the
OX Protect Notification API.
On startup it reads dstore_alert.conf, which contains the
following settings:
| listen_address: | The IP and port the service listens on for PBDNSMessages |
|---|---|
| alert-listen-address: | |
| The IP and port the service listens on for PBAlertMessages. | |
| webhook-url: | The URL of the Protect Notification Center API Endpoint. |
| webhook-conns: | The maximum number of HTTP connections per webhook thread. |
| webhook-threads: | |
| The number of threads to use for webhooks. | |
| webhook-secret: | The secret to use when generating the X-Wforce-Signature header |
| webhook-api-key: | |
| The value to place in the X-API-Key header. | |
| worker-threads: | The number of worker threads to create. Each received message will be passed to a pool of worker theads and processed. |
| platform-url: | The URL of the PDNS Platform Admin API. This is used to download the list of category names and titles. |
| malware-tags: | The tags that indicate that a PBDNSMessage is related to malware. |
| blacklist-tags: | The tags that indicate that a PBDNSMessage is related to blacklisting. |
ev_aggregator¶
See the separate documentation.
egateway¶
The egateway provides HTTP-based access to the data contained in dstore.
It reads both dstore.conf and egateway.conf.
The latter file has these options:
| listen-address: | Specifies the address/port to which egateway should bind to. |
|---|
dstore-dist¶
See the separate documentation.
dstore-web¶
dstore-web is a Django application written in Python 3.6.
As Python 3.6 is relatively new with regards to operating system release cycles, dstore-web depends on the rh-python-36 software collection on Red Hat Enterprise Linux based operating systems (including CentOS and Oracle Linux).
If these collections are not available in the target environment, Open-Xchange can provide the required packages in the software repository as well.
When installing dstore-web, a system user and group called dstore-web are created, an sqlite3 database for users and audit-logs is created in /var/lib/dstore-web.
The dstore-web service runs as this unprivileged user.
Two programs are installed as part of the dstore-web package:
dstore-web: | This program is a tool to manage the installation, add and remove users and apply database migrations. |
|---|---|
dstore-web-server: | |
| This program launches the webserver on the configured port. It is invoked by the init-system. | |
The configuration files are installed into /etc/dstore-web.
gunicorn_config.py¶
This file configures the dstore-web-server. The defaults included with the installation work well for most installations. However, some tuning or a different listen port might be in order.
The file itself has comments, but some options are described in more detail below.
| bind: | This option has a python list with all IP addresses and ports where the webserver should listen. It is recommended to listen on the local loopback interface and have a webserver like nginx or Apache in front of dstore-web-server. |
|---|---|
| workers: | This option sets the number of workers that handle connections. By default this is set to twice the number of CPUs in the system. |
| threads: | The number of threads per worker. |
settings.py¶
A Django settings file. This configures the dstore-web application itself. This file contains ample comments that describe the options that can be set.
The most important setting is the EGATEWAY_URL, this should contain the URL to the egateway.
dstore-report-api¶
dstore-report-api is an application written in Python 3.6, which allows global and per-user information about filtering events to be retrieved using a REST API.
When installing dstore-report-api, a system user and group called dstore-report-api are created. The dstore-report-api service runs as this unprivileged user.
One program is installed as part of the dstore-report-api package:
dstore-report-api-webserver: | |
|---|---|
| This program launches the webserver on the configured port. It is invoked by the init-system. | |
The configuration files are installed into /etc/dstore-report-api.
Prometheus metrics are available on the /metrics endpoint; these metrics are stored/retrieved in the directory specified by the RUNTIME_DIRECTORY environment variable, which is set in the dstore-report-api.service file using the RuntimeDirectory directive. This directory is required to be set because dstore-report-api runs under gunicorn, and is multi-process, which means that prometheus stats must be gathered from all processes not just one; the runtime directory is used to store the stats from each process. If this directory is not set or doesn’t exist, the dstore-report-api process will fail to start.
dstore-report-api-web.conf¶
This file configures the dstore-report-api webserver. The defaults included with the installation work well for most installations. However, some tuning or a different listen port might be in order.
The file itself has comments, but some options are described in more detail below.
| bind: | This option has a python list with all IP addresses and ports where the webserver should listen. It is recommended to listen on the local loopback interface and have a webserver like nginx or Apache in front of dstore-report-api. |
|---|---|
| workers: | This option sets the number of workers that handle connections. By default this is set to twice the number of CPUs in the system. |
| threads: | The number of threads per worker. |
dstore-report-api-instance.conf¶
A settings file for the dstore-report-api. This configures the dstore-report-api application itself.
By default the file contains the following settings:
ELASTICSEARCH_URL = ["http://127.0.0.1:9200"]
ELASTICSEARCH_TIMEOUT = 1
ELASTICSEARCH_INDEX = "logstash-*"
#ELASTICSEARCH_USER = "elastic"
#ELASTICSEARCH_PASSWORD = "changeme"
AUTH_TOKEN = "secret"
LOG_LEVEL = logging.INFO
LOG_FORMAT = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
These are explained as follows:
| ELASTICSEARCH_URL: | |
|---|---|
| This option sets the address of the elasticsearch server where the filtering events are stored. In most cases this and AUTH_PASSWORD are the only two parameters that will need to be changed. | |
| ELASTICSEARCH_TIMEOUT: | |
| This option controls the timeout between the webs application and the elasticsearch server. | |
| ELASTICSEARCH_INDEX: | |
| This option controls which index pattern will be used for elasticsearch searches. If the filtering events are stored using a more specific index pattern than the default, then using that will increase the speed of searches and reduce the load on elasticsearch. | |
| ELASTICSEARCH_USER: | |
| If this option and ELASTICSEARCH_PASSWORD are both set, then they will be used to perform basic authentication against the Elasticsearch server. Both must be set for basic authentication to be performed. | |
| ELASTICSEARCH_PASSWORD: | |
| The password to use for basic authentication to elasticsearch. The ELASTICSEARCH_USER option must also be set for basic authentication to be performed. | |
| AUTH_TOKEN: | This option controls the token used for authorization to the web application REST API, and if set it must be present in the X-API-Key header for all requests. |
| LOG_LEVEL: | This option controls the syslog log level. |
| LOG_FORMAT: | This option controls the log format used for logging. |