Streaming Infrastructure Monitoring with Grafana (DNS)

In this short serie we are going to have a look at how monitoring a streaming infrastructure can be achieved with Grafana. In this article we will look at how we can collect statistics from a DNS server (Bind 9 software) and present on a dashboard in Grafana.

Before we begin let us look at the reference infrastructure we are using in this example.

In this infrastructure we have an Nginx based origin server where the media segments are placed. In our distribution network we have a number of Nginx based reverse proxy servers with the purpose to cache the media segments and requests. And the third component in this architecture is the DNS servers which are responsible for directing the client to the, for the moment, best Nginx cache.

We want to start by monitoring the health and statistics of the DNS servers which this article will cover. To visualize the data we will use Grafana — an open source software for time series analytics. And to be able to visualize the data we need a way to collect it and store it. To store the data we use Influx DB — an open source time series database designed to handle high write and query loads. And to collect data we use CollectD — an open source system statistics collection daemon in combination with the statistics channel that can be enabled in Bind 9 (DNS software).

For our monitoring setup we will need a Bind 9 collector, an Influx DB and Grafana server. To prepare the monitoring setup for running in a private or public cloud I am running these components as Docker Containers.

First, we need to enable the statistics channel in Bind 9 so we add the following lines to the global configuration (named.conf)

statistics-channels {
  inet 0.0.0.0 port 8053;
};

You can test this by using curl (HTTP GET) on port 8053 of your DNS server and you will receive an XML with all the statistics. This is the data that we want to parse and store in our Influx DB. With CollectD comes a number of plugins and one of them is a plugin to parse Bind statistics. I found out that I need a relatively new version of CollectD for it to work with the latest version of Bind 9, and as the available public Docker Images only contained older versions of CollectD I’ve created one. It is available on Eyevinn Technology’s public Docker Repository.

eyevinntechnology/docker-collectd:5.6-1.0

You are most welcome to use it.

The next step is to configure the CollectD to collect data from Bind 9 (over the above channel) and push to our Influx DB. The configuration file for collecting data we name source-bind.conf and place it in the conf.d directory of CollectD.

LoadPlugin bind
<Plugin "bind">
        URL "http://bind:8053/"
        OpCodes true
        QTypes true
        ServerStats true
        ZoneMaintStats true
        ResolverStats false
        MemoryStats true
</Plugin>

And the second configuration file we name target-influx.conf:

LoadPlugin network
<Plugin network>
        Server "influxdb" "25826"
</Plugin>

The conf.d directory where we put these configuration files are mounted in the Docker Container with the following Docker Compose configuration.

version: '2'
services:
  dns:
    restart: always
    image: eyevinntechnology/docker-collectd:5.6-1.0
    volumes:
      - /private/var/docker/dns-collect/conf.d:/etc/collectd/collectd.conf.d

The next step is to create the collectd database:

curl -X POST http://influxdb:8086/query --data-urlencode "q=CREATE DATABASE collectd"

And then configure Influx with the following configuration.

[collectd]
enabled = true
bind-address = ":25826"
database = "collectd"
typesdb = "/usr/share/collectd/types.db"

With Influx DB running as a Docker Container the Docker Compose file would then look like this.

version: '2'
services:
  influxdb:
    restart: always
    image: influxdb:latest
    environment:
      - INFLUXDB_COLLECTD_ENABLED=true
      - INFLUXDB_COLLECTD_BIND_ADDRESS=:25826
      - INFLUXDB_COLLECTD_DATABASE=collectd
      - INFLUXDB_COLLECTD_TYPESDB=/usr/local/share/collectd/types.db
    volumes:
      - /private/var/docker/influxdb:/var/lib/influxdb
      - /private/var/docker/influxdb/collectd:/usr/local/share/collectd

Where the types.db file is placed in the /private/var/docker/influxdb/collectd directory that is being mounted in the Influx DB Container.

The last step is to get this data visualized with Grafana. The Docker Compose setup for Grafana is quite simple and I basically used the default one.

After configured the Influx DB datasource in Grafana and to use the collectd database you can write and visualize queries for example like this:

"SELECT derivative("value", 1m) FROM "bind_value" WHERE ("type" = 'dns_rcode' AND "type_instance" = 'tx-FORMERR' AND "host" =~ /^631b34308072$/) AND time >= now() - 1h;"

Hope you found this article useful and in the next article we are going to look at monitoring the Nginx caches. Thank you for reading!

Jonas Rydholm Birmé is a solution architect at Eyevinn Technology. Eyevinn Technology is the leading independent consultant firm specializing in video technology and media distribution. Also the proud organizer of Streaming Tech Sweden — a yearly tech conference in Stockholm for the streaming tech community.