Installation and Configuration ============================== The supported method of installing the :program:`dstore` software components is using the *Ansible* role provided by Open-Xchange. This chapter describes the packages, their contents and configuration files for completeness. Repository ---------- All packages required for :program:`dstore` are provided in a software repository maintained by Open-Xchange. If you do not have access to this repository, please contact Open-Xchange support. This repository contains the packages built for the deployment's operating system version and architecture. Directories and Files --------------------- :program:`dstore` store its data in ``/var/lib/dstore`` by default. Furthermore, unless otherwise specified, all program configuration is in ``/etc/dstore``. In that configuration directory, there is one configuration file for each service and one global file. This global configuration is called ``dstore.conf`` and it contains several settings. :storage-dir: The path to the directory where the data is stored. This can be changed if ``/var/lib/dstore`` is not the preferred directory. :hash-split: The number of buckets the data is split into. .. todo:: Write about disk-layout and mounting multiple disks under ``/var/lib/dstore`` for performance reasons. ``tcpdistro`` ------------- The :program:`tcpdistro` service accepts a stream of protobuf-encoded messages and stores them to disk. On start-up, it reads ``dstore.conf`` and ``tcpdistro.conf``. The latter file contains these settings: :listen-address: The IP and port the service listens on :answer-limit: The maximum number of answer-RRs to store per answer packet When :program:`dstore-alert` is used the following settings are relevant: :enable-alerts: Whether or not to enable sending alerts to :program:`dstore-alert` :alert-address: The address where :program:`dstore-alert` can receive messages. :alert-tags: Incoming packets with these tags will be sent to :program:`dstore-alert` ``dstore_alert`` ---------------- The :program:`dstore_alert` service accepts a stream of protobuf-encoded messages and send them to an HTTP endpoint using the OX Protect Notification API. On startup it reads ``dstore_alert.conf``, which contains the following settings: :listen_address: The IP and port the service listens on for PBDNSMessages :alert-listen-address: The IP and port the service listens on for PBAlertMessages. :webhook-url: The URL of the Protect Notification Center API Endpoint. :webhook-conns: The maximum number of HTTP connections per webhook thread. :webhook-threads: The number of threads to use for webhooks. :webhook-secret: The secret to use when generating the X-Wforce-Signature header :webhook-api-key: The value to place in the X-API-Key header. :worker-threads: The number of worker threads to create. Each received message will be passed to a pool of worker theads and processed. :platform-url: The URL of the PDNS Platform Admin API. This is used to download the list of category names and titles. :malware-tags: The tags that indicate that a PBDNSMessage is related to malware. :blacklist-tags: The tags that indicate that a PBDNSMessage is related to blacklisting. ``ev_aggregator`` ----------------- See the separate documentation. ``egateway`` ------------ The :program:`egateway` provides HTTP-based access to the data contained in :program:`dstore`. It reads both ``dstore.conf`` and ``egateway.conf``. The latter file has these options: :listen-address: Specifies the address/port to which :program:`egateway` should bind to. ``dstore-dist`` --------------- See the separate documentation. ``dstore-web`` -------------- :program:`dstore-web` is a `Django `__ application written in Python 3.6. As Python 3.6 is relatively new with regards to operating system release cycles, :program:`dstore-web` depends on the ``rh-python-36`` `software collection `__ on Red Hat Enterprise Linux based operating systems (including CentOS and Oracle Linux). If these collections are not available in the target environment, Open-Xchange can provide the required packages in the software repository as well. When installing :program:`dstore-web`, a system user and group called *dstore-web* are created, an sqlite3 database for users and audit-logs is created in ``/var/lib/dstore-web``. The :program:`dstore-web` service runs as this unprivileged user. Two programs are installed as part of the :program:`dstore-web` package: :``dstore-web``: This program is a tool to manage the installation, add and remove users and apply database migrations. :``dstore-web-server``: This program launches the webserver on the configured port. It is invoked by the init-system. The configuration files are installed into ``/etc/dstore-web``. ``gunicorn_config.py`` ~~~~~~~~~~~~~~~~~~~~~~ This file configures the :program:`dstore-web-server`. The defaults included with the installation work well for most installations. However, some tuning or a different listen port might be in order. The file itself has comments, but some options are described in more detail below. :bind: This option has a python list with all IP addresses and ports where the webserver should listen. It is recommended to listen on the local loopback interface and have a webserver like nginx or Apache in front of :program:`dstore-web-server`. :workers: This option sets the number of workers that handle connections. By default this is set to twice the number of CPUs in the system. :threads: The number of threads per worker. ``settings.py`` ~~~~~~~~~~~~~~~ A `Django settings `__ file. This configures the :program:`dstore-web` application itself. This file contains ample comments that describe the options that can be set. The most important setting is the ``EGATEWAY_URL``, this should contain the URL to the :program:`egateway`. ``dstore-report-api`` --------------------- :program:`dstore-report-api` is an application written in Python 3.6, which allows global and per-user information about filtering events to be retrieved using a REST API. When installing :program:`dstore-report-api`, a system user and group called *dstore-report-api* are created. The :program:`dstore-report-api` service runs as this unprivileged user. One program is installed as part of the :program:`dstore-report-api` package: :``dstore-report-api-webserver``: This program launches the webserver on the configured port. It is invoked by the init-system. The configuration files are installed into ``/etc/dstore-report-api``. Prometheus metrics are available on the /metrics endpoint; these metrics are stored/retrieved in the directory specified by the RUNTIME_DIRECTORY environment variable, which is set in the dstore-report-api.service file using the RuntimeDirectory directive. This directory is required to be set because dstore-report-api runs under gunicorn, and is multi-process, which means that prometheus stats must be gathered from all processes not just one; the runtime directory is used to store the stats from each process. If this directory is not set or doesn't exist, the dstore-report-api process will fail to start. ``dstore-report-api-web.conf`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This file configures the :program:`dstore-report-api` webserver. The defaults included with the installation work well for most installations. However, some tuning or a different listen port might be in order. The file itself has comments, but some options are described in more detail below. :bind: This option has a python list with all IP addresses and ports where the webserver should listen. It is recommended to listen on the local loopback interface and have a webserver like nginx or Apache in front of :program:`dstore-report-api`. :workers: This option sets the number of workers that handle connections. By default this is set to twice the number of CPUs in the system. :threads: The number of threads per worker. ``dstore-report-api-instance.conf`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A settings file for the :program:`dstore-report-api`. This configures the :program:`dstore-report-api` application itself. By default the file contains the following settings:: ELASTICSEARCH_URL = ["http://127.0.0.1:9200"] ELASTICSEARCH_TIMEOUT = 1 ELASTICSEARCH_INDEX = "logstash-*" #ELASTICSEARCH_USER = "elastic" #ELASTICSEARCH_PASSWORD = "changeme" AUTH_TOKEN = "secret" LOG_LEVEL = logging.INFO LOG_FORMAT = '%(asctime)s - %(name)s - %(levelname)s - %(message)s' These are explained as follows: :ELASTICSEARCH_URL: This option sets the address of the elasticsearch server where the filtering events are stored. In most cases this and AUTH_PASSWORD are the only two parameters that will need to be changed. :ELASTICSEARCH_TIMEOUT: This option controls the timeout between the webs application and the elasticsearch server. :ELASTICSEARCH_INDEX: This option controls which index pattern will be used for elasticsearch searches. If the filtering events are stored using a more specific index pattern than the default, then using that will increase the speed of searches and reduce the load on elasticsearch. :ELASTICSEARCH_USER: If this option and ELASTICSEARCH_PASSWORD are both set, then they will be used to perform basic authentication against the Elasticsearch server. Both must be set for basic authentication to be performed. :ELASTICSEARCH_PASSWORD: The password to use for basic authentication to elasticsearch. The ELASTICSEARCH_USER option must also be set for basic authentication to be performed. :AUTH_TOKEN: This option controls the token used for authorization to the web application REST API, and if set it must be present in the X-API-Key header for all requests. :LOG_LEVEL: This option controls the syslog log level. :LOG_FORMAT: This option controls the log format used for logging.