egateway ======== ``egateway`` allows you to search programmatically in the data stored by ``tcpdistro``. This can be done thanks to an http API. /get-endpoint-data ------------------ Main entrypoint of the API. Search for DNS messages. By default, will look into customer traffic (queries and responses), which can be tuned with ``outgoing`` and ``only_responses`` parameters. Parameters : - ``q`` search terms - ``start`` unix timestamp for the beginning of the timeframe you are looking for - ``end`` unix timestamp for the end of the timeframe you are looking for - ``limit`` limit the number of results - ``preasons`` search terms, separated by commas, within applied policies - ``preasons_tags`` search terms, separated by commas, within tags, which are not the applied policy - ``has_preason`` has been applied any policy - ``only_responses`` only search for responses (either incoming or direct) - ``outgoing`` only search for outgoing queries and incoming responses (recursive traffic) Search terms is a list of keywords, separated by spaces with additional prefixes. - ``~address`` search by answer address. example: ``~1.2.3.4`` - ``=message-id`` search by message id. example: ``=123e4567-e89b-12d3-a456-426614174000`` - ``c:customer`` search by requestor identifier. example: ``c:powerdns`` - ``d:device`` search by device id. example: ``d:msisdn:07747012345`` - ``dn:device-name`` search by device name field. example: ``dn:johns-computer`` - ``rcode:[!]code`` search by dns response code. can search for all code except if ``!`` is added. example: ``rcode:3`` search for NXDomain responses, ``rcode:!0`` search for all response code except NoError. See https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml#dns-parameters-6 - the rest is considered to be either an IP address or a domain name. example: ``1.2.3.4``, ``foo.example.com.`` Examples:: $ http "http://127.0.0.1:8081/get-endpoint-data?q=%3D79d173da-9d83-4c89-ab6f-577a1a6eb59f" HTTP/1.1 200 OK Connection: keep-alive Date: Fri, 28 May 2021 09:57:42 GMT Server: h2o/2.2.6 content-type: application/json; charset=utf-8 transfer-encoding: chunked { "items": [ { "A": "2001:db8::1", "B": "192.0.2.42", "deviceId": "4005ffeeeeddaadddd", "ecs": "2001:db8::1", "from": "192.0.2.1", "latencyUsec": -850884, "msgType": 2, "preason": "nxdomain", "qid": "79d173da-9d83-4c89-ab6f-577a1a6eb59f", "qr": true, "queryTimeSec": 1605178842, "queryTimeUsec": 850884, "question": "23foobar.com", "rcode": 1234, "requestorId": "ncook", "response": "0.0.0.0", "serverId": "456", "tags": [ "porn", "gambling", "OXP-platform-facebook" ], "tcp": false, "timeSec": 1605178842, "timeUsec": 0, "type": "" } ], "luaresult": {}, "msec": 0.306, "number": 1, "timegraph": [] } $ http 'http://127.0.0.1:8081/get-endpoint-data?q=d:mac:000036000001%20rcode%3A!0' HTTP/1.1 200 OK Connection: keep-alive Date: Fri, 28 May 2021 10:06:17 GMT Server: h2o/2.2.6 content-type: application/json; charset=utf-8 transfer-encoding: chunked { "items": [ { "A": "2001:db8::1", "B": "192.0.2.42", "deviceId": "mac:000036000001", "ecs": "2001:db8::1", "from": "192.0.2.1", "latencyUsec": -608116, "msgType": 2, "preason": "nxdomain", "qid": "2e026b97-6b92-4be2-a7c4-97d6919b33ee", "qr": true, "queryTimeSec": 1605178887, "queryTimeUsec": 608116, "question": "29foobar.com", "rcode": 1234, "requestorId": "ncook", "response": "0.0.0.0", "serverId": "456", "tags": [ "porn", "gambling", "OXP-platform-facebook" ], "tcp": false, "timeSec": 1605178887, "timeUsec": 0, "type": "" }, [...] ], "luaresult": {}, "msec": 0.366, "number": 3, "timegraph": [] } /num-db ------- Some useful statistics on indexed entries and software version. Example:: $ http 127.0.0.1:8081/num-db HTTP/1.1 200 OK Connection: keep-alive Date: Thu, 27 May 2021 09:29:19 GMT Server: h2o/2.2.6 content-type: application/json; charset=utf-8 transfer-encoding: chunked { "bytes": 5481, "entries": 54, "indexBytes": 0, "version": "git" } /metrics -------- Prometheus metrics exporter. Example:: $ http 127.0.0.1:8081/metrics HTTP/1.1 200 OK Connection: keep-alive Date: Thu, 27 May 2021 09:25:07 GMT Server: h2o/2.2.6 content-type: text/plain; charset=utf-8 transfer-encoding: chunked # HELP dstore_egateway_queries_total Number of queries received # TYPE dstore_egateway_queries_total counter dstore_egateway_queries_total 0 # HELP dstore_egateway_queries_options_total Number of queries received by option # TYPE dstore_egateway_queries_options_total counter dstore_egateway_queries_options_total{option="timegraph"} 0 dstore_egateway_queries_options_total{option="lua"} 0 dstore_egateway_queries_options_total{option="response_only"} 0 # HELP dstore_egateway_queries_type_total Total number of searches by query type # TYPE dstore_egateway_queries_type_total counter dstore_egateway_queries_type_total{type="requestor_id"} 0 dstore_egateway_queries_type_total{type="device_id"} 0 dstore_egateway_queries_type_total{type="source"} 0 dstore_egateway_queries_type_total{type="domain"} 0 dstore_egateway_queries_type_total{type="qid"} 0 dstore_egateway_queries_type_total{type="content"} 0 dstore_egateway_queries_type_total{type="preason"} 0 # HELP dstore_egateway_records_matched_total Total number of records matched by queries # TYPE dstore_egateway_records_matched_total counter dstore_egateway_records_matched_total 0 # HELP dstore_egateway_records_returned_total Total number of records returned # TYPE dstore_egateway_records_returned_total counter dstore_egateway_records_returned_total 0 # HELP dstore_egateway_buckets_scanned_total Total number of buckets scanned # TYPE dstore_egateway_buckets_scanned_total counter dstore_egateway_buckets_scanned_total 0 # HELP dstore_egateway_bytes_scanned_total Total number of bytes scanned # TYPE dstore_egateway_bytes_scanned_total counter dstore_egateway_bytes_scanned_total 0 # HELP dstore_egateway_messages_scanned_total Total number of messages scanned # TYPE dstore_egateway_messages_scanned_total counter dstore_egateway_messages_scanned_total 0 # HELP dstore_egateway_queries_latency_seconds Histogram of queries latency. # TYPE dstore_egateway_queries_latency_seconds histogram dstore_egateway_queries_latency_seconds_bucket{le="0.01"} 0 dstore_egateway_queries_latency_seconds_bucket{le="0.03"} 0 dstore_egateway_queries_latency_seconds_bucket{le="0.05"} 0 dstore_egateway_queries_latency_seconds_bucket{le="0.1"} 0 dstore_egateway_queries_latency_seconds_bucket{le="0.3"} 0 dstore_egateway_queries_latency_seconds_bucket{le="1"} 0 dstore_egateway_queries_latency_seconds_bucket{le="3"} 0 dstore_egateway_queries_latency_seconds_bucket{le="5"} 0 dstore_egateway_queries_latency_seconds_bucket{le="10"} 0 dstore_egateway_queries_latency_seconds_bucket{le="20"} 0 dstore_egateway_queries_latency_seconds_bucket{le="30"} 0 dstore_egateway_queries_latency_seconds_bucket{le="60"} 0 dstore_egateway_queries_latency_seconds_bucket{le="120"} 0 dstore_egateway_queries_latency_seconds_bucket{le="180"} 0 dstore_egateway_queries_latency_seconds_bucket{le="+Inf"} 0 dstore_egateway_queries_latency_seconds_count 0 dstore_egateway_queries_latency_seconds_sum 0.000000 # HELP dstore_egateway_info Software version and configuration # TYPE dstore_egateway_info gauge dstore_egateway_info{version="git",hash_split="1024",storage_dir="./storage",question="1",message_id="1",requestor_id="1",policy="1",tags="1",store_queries="1"} 1 up 1 # HELP process_start_time_seconds Start time of the process since unix epoch in seconds. # TYPE process_start_time_seconds gauge process_start_time_seconds 1622107494 # HELP process_resident_memory_bytes Resident memory size in bytes. # TYPE process_resident_memory_bytes gauge process_resident_memory_bytes 737280 # HELP process_cpu_user_seconds_total Total user CPU time spent in seconds. # TYPE process_cpu_user_seconds_total counter process_cpu_user_seconds_total 0.003000 # HELP process_cpu_system_seconds_total Total system CPU time spent in seconds. # TYPE process_cpu_system_seconds_total counter process_cpu_system_seconds_total 0.006000 # HELP process_open_fds Number of open file descriptors. # TYPE process_open_fds gauge process_open_fds 9 Authentication -------------- If ``egateway`` has been set up with an API key, you have to provide it through the ``x-api-key`` http header. Example:: $ http 127.0.0.1:8081/num-db HTTP/1.1 403 Forbidden Connection: keep-alive Content-Length: 9 Date: Fri, 28 May 2021 10:35:26 GMT Server: h2o/2.2.6 content-type: text/plain; charset=utf-8 forbidden $ http -v 127.0.0.1:8081/num-db x-api-key:0xpassw0rd GET /num-db HTTP/1.1 Accept: */* Accept-Encoding: gzip, deflate Connection: keep-alive Host: 127.0.0.1:8081 User-Agent: HTTPie/2.4.0 x-api-key: 0xpassw0rd HTTP/1.1 200 OK Connection: keep-alive Date: Fri, 28 May 2021 10:34:09 GMT Server: h2o/2.2.6 content-type: application/json; charset=utf-8 transfer-encoding: chunked { "bytes": 5481, "entries": 54, "indexBytes": 0, "version": "git" }