Installation and Configuration
==============================
The supported method of installing the :program:`dstore` software components is using the *Ansible* role provided by Open-Xchange.
This chapter describes the packages, their contents and configuration files for completeness.
Repository
----------
All packages required for :program:`dstore` are provided in a software repository maintained by Open-Xchange.
If you do not have access to this repository, please contact Open-Xchange support.
This repository contains the packages built for the deployment's operating system version and architecture.
Directories and Files
---------------------
:program:`dstore` store its data in ``/var/lib/dstore`` by default.
Furthermore, unless otherwise specified, all program configuration is in ``/etc/dstore``.
In that configuration directory, there is one configuration file for each service and one global file.
This global configuration is called ``dstore.conf`` and it contains several settings.
:storage-dir: The path to the directory where the data is stored. This can be changed if ``/var/lib/dstore`` is not the preferred directory.
:hash-split: The number of buckets the data is split into.
.. todo::
Write about disk-layout and mounting multiple disks under ``/var/lib/dstore`` for performance reasons.
``tcpdistro``
-------------
The :program:`tcpdistro` service accepts a stream of protobuf-encoded messages and stores them to disk.
On start-up, it reads ``dstore.conf`` and ``tcpdistro.conf``.
The latter file contains these settings:
:listen-address: The IP and port the service listens on
:answer-limit: The maximum number of answer-RRs to store per answer packet
When :program:`dstore-alert` is used the following settings are relevant:
:enable-alerts: Whether or not to enable sending alerts to :program:`dstore-alert`
:alert-address: The address where :program:`dstore-alert` can receive messages.
:alert-tags: Incoming packets with these tags will be sent to :program:`dstore-alert`
``dstore_alert``
----------------
The :program:`dstore_alert` service accepts a stream of
protobuf-encoded messages and send them to an HTTP endpoint using the
OX Protect Notification API.
On startup it reads ``dstore_alert.conf``, which contains the
following settings:
:listen_address: The IP and port the service listens on for
PBDNSMessages
:alert-listen-address: The IP and port the service listens on for
PBAlertMessages.
:webhook-url: The URL of the Protect Notification Center API
Endpoint.
:webhook-conns: The maximum number of HTTP connections per webhook
thread.
:webhook-threads: The number of threads to use for webhooks.
:webhook-secret: The secret to use when generating the
X-Wforce-Signature header
:webhook-api-key: The value to place in the X-API-Key header.
:worker-threads: The number of worker threads to create. Each received
message will be passed to a pool of worker theads and
processed.
:platform-url: The URL of the PDNS Platform Admin API. This is used to
download the list of category names and titles.
:malware-tags: The tags that indicate that a PBDNSMessage is related
to malware.
:blacklist-tags: The tags that indicate that a PBDNSMessage is related
to blacklisting.
``ev_aggregator``
-----------------
See the separate documentation.
``egateway``
------------
The :program:`egateway` provides HTTP-based access to the data contained in :program:`dstore`.
It reads both ``dstore.conf`` and ``egateway.conf``.
The latter file has these options:
:listen-address: Specifies the address/port to which :program:`egateway` should bind to.
``dstore-dist``
---------------
See the separate documentation.
``dstore-web``
--------------
:program:`dstore-web` is a `Django `__ application written in Python 3.6.
As Python 3.6 is relatively new with regards to operating system release cycles, :program:`dstore-web` depends on the ``rh-python-36`` `software collection `__ on Red Hat Enterprise Linux based operating systems (including CentOS and Oracle Linux).
If these collections are not available in the target environment, Open-Xchange can provide the required packages in the software repository as well.
When installing :program:`dstore-web`, a system user and group called *dstore-web* are created, an sqlite3 database for users and audit-logs is created in ``/var/lib/dstore-web``.
The :program:`dstore-web` service runs as this unprivileged user.
Two programs are installed as part of the :program:`dstore-web` package:
:``dstore-web``: This program is a tool to manage the installation, add and remove users and apply database migrations.
:``dstore-web-server``: This program launches the webserver on the configured port. It is invoked by the init-system.
The configuration files are installed into ``/etc/dstore-web``.
``gunicorn_config.py``
~~~~~~~~~~~~~~~~~~~~~~
This file configures the :program:`dstore-web-server`.
The defaults included with the installation work well for most installations.
However, some tuning or a different listen port might be in order.
The file itself has comments, but some options are described in more detail below.
:bind: This option has a python list with all IP addresses and ports where the webserver should listen. It is recommended to listen on the local loopback interface and have a webserver like nginx or Apache in front of :program:`dstore-web-server`.
:workers: This option sets the number of workers that handle connections. By default this is set to twice the number of CPUs in the system.
:threads: The number of threads per worker.
``settings.py``
~~~~~~~~~~~~~~~
A `Django settings `__ file.
This configures the :program:`dstore-web` application itself.
This file contains ample comments that describe the options that can be set.
The most important setting is the ``EGATEWAY_URL``, this should contain the URL to the :program:`egateway`.
``dstore-report-api``
---------------------
:program:`dstore-report-api` is an application written in Python 3.6, which allows global and per-user information about filtering events to be retrieved using a REST API.
When installing :program:`dstore-report-api`, a system user and group
called *dstore-report-api* are created. The :program:`dstore-report-api` service runs as this unprivileged user.
One program is installed as part of the :program:`dstore-report-api` package:
:``dstore-report-api-webserver``: This program launches the webserver on the configured port. It is invoked by the init-system.
The configuration files are installed into ``/etc/dstore-report-api``.
Prometheus metrics are available on the /metrics endpoint; these
metrics are stored/retrieved in the directory specified by the RUNTIME_DIRECTORY
environment variable, which is set in the dstore-report-api.service
file using the RuntimeDirectory directive. This directory is required
to be set because dstore-report-api runs under gunicorn, and is
multi-process, which means that prometheus stats must be gathered from
all processes not just one; the runtime directory is used to store the
stats from each process. If this directory is not set or doesn't
exist, the dstore-report-api process will fail to start.
``dstore-report-api-web.conf``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This file configures the :program:`dstore-report-api` webserver.
The defaults included with the installation work well for most installations.
However, some tuning or a different listen port might be in order.
The file itself has comments, but some options are described in more detail below.
:bind: This option has a python list with all IP addresses and ports where the webserver should listen. It is recommended to listen on the local loopback interface and have a webserver like nginx or Apache in front of :program:`dstore-report-api`.
:workers: This option sets the number of workers that handle connections. By default this is set to twice the number of CPUs in the system.
:threads: The number of threads per worker.
``dstore-report-api-instance.conf``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A settings file for the :program:`dstore-report-api`.
This configures the :program:`dstore-report-api` application itself.
By default the file contains the following settings::
ELASTICSEARCH_URL = ["http://127.0.0.1:9200"]
ELASTICSEARCH_TIMEOUT = 1
ELASTICSEARCH_INDEX = "logstash-*"
#ELASTICSEARCH_USER = "elastic"
#ELASTICSEARCH_PASSWORD = "changeme"
AUTH_TOKEN = "secret"
LOG_LEVEL = logging.INFO
LOG_FORMAT = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
These are explained as follows:
:ELASTICSEARCH_URL: This option sets the address of the elasticsearch
server where the filtering events are stored. In
most cases this and AUTH_PASSWORD are the only two
parameters that will need to be changed.
:ELASTICSEARCH_TIMEOUT: This option controls the timeout between the
webs application and the elasticsearch
server.
:ELASTICSEARCH_INDEX: This option controls which index pattern will be
used for elasticsearch searches. If the
filtering events are stored using a more
specific index pattern than the default, then
using that will increase the speed of searches
and reduce the load on elasticsearch.
:ELASTICSEARCH_USER: If this option and ELASTICSEARCH_PASSWORD are
both set, then they will be used to perform
basic authentication against the Elasticsearch
server. Both must be set for basic
authentication to be performed.
:ELASTICSEARCH_PASSWORD: The password to use for basic authentication
to elasticsearch. The ELASTICSEARCH_USER
option must also be set for basic
authentication to be performed.
:AUTH_TOKEN: This option controls the token used for authorization
to the web application REST API, and if set it must be
present in the X-API-Key header for all requests.
:LOG_LEVEL: This option controls the syslog log level.
:LOG_FORMAT: This option controls the log format used for logging.