Envoy statistics with StatsD

For an overview of other options for gathering statistics on Ambassador Edge Stack, see the Statistics and Monitoring overview.

At the core of Ambassador Edge Stack is Envoy Proxy, which has built-in support for exporting a multitude of statistics about its own operations to StatsD (or to the modified DogStatsD used by Datadog).

If enabled, then Ambassador Edge Stack has Envoy expose this information via the ubiquitous and well-tested StatsD protocol. To enable this, you will simply need to set the environment variable STATSD_ENABLED=true in Ambassador Edge Stack's deployment YAML:

When this variable is set, Ambassador Edge Stack by default sends statistics to a Kubernetes service named statsd-sink on UDP port 8125 (the usual port of the StatsD protocol). You may instead tell Ambassador Edge Stack to send the statistics to a different StatsD server by setting the STATSD_HOST environment variable. This can be useful if you have an existing StatsD sink available in your cluster.

We have included a few example configurations in the statsd-sink/ directory to help you get started. Clone the repository to get local, editable copies.

Using Graphite as the StatsD sink

Graphite is a web-based real-time graphing system. Spin up an example Graphite setup:

This sets up the statsd-sink service and a deployment that contains Graphite and its related infrastructure. Graphite's web interface is available at http://statsd-sink/ from within the cluster. Use port forwarding to access the interface from your local machine:

This sets up Graphite access at http://localhost:8080/.

Using Prometheus StatsD Exporter as the StatsD sink

Prometheus is an open-source monitoring and alerting system. Prometheus does not natively understand the StatsD protocol, but you can deploy the Prometheus StatsD Exporter to act as the StatsD sink, and it will translate from StatsD to the exposition format that Prometheus requires. An example of how deploying Prometheus StatsD Exporter is available in prom-statsd-sink.yaml.

To finally get the statistics to Prometheus, you then configure a Prometheus target to read from statsd-sink on port 9102.

You could instead also add the statsd-sink service and Prometheus StatsD Exporter as a sidecar on the Ambassador Edge Stack pod. If you do this, make sure to set STATSD_HOST=localhost so that UDP packets are routed to the sidecar.

Configuring how Prometheus StatsD Exporter translates from StatsD to the Prometheus format

It may be desirable to change how metrics produced by the statsd-sink are named, labeled and grouped when they finally make it to Prometheus.

For example, by default, each service that Ambassador Edge Stack serves will create a new metric using its name. For the service called usersvc you will see this metric envoy.cluster.usersvc_cluster.upstream_rq_total. This may lead to problems if you are trying to create a single aggregate that is the sum of all similar metrics from different services. In this case, it is common to differentiate the metrics for an individual service with a label. This can be done by configuring a Prometheus StatsD Exporter "mapping" (not to be confused with an Ambassador Edge Stack Mapping). See Metric Mapping and Configuration in the Prometheus StatsD Exporter documentation to learn how to modify its mappings.

Configuring Prometheus StatsD Exporter with Helm

If you deploy Prometheus using Helm the value that you should change in order to add a mapping is prometheusExporter.configuration. Set it to something like this:

Configuring Prometheus StatsD Exporter in prom-statsd-sink.yaml

If you are using the prom-statsd-sink.yaml example, edit its ambassador-config ConfigMap with your Prometheus mappings, for example:

Using the Prometheus Operator to configure Prometheus for use with the Prometheus StatsD Exporter

If you don't already have a Prometheus setup, the Prometheus Operator is a powerful way to create and deploy Prometheus instances. Use the following YAML to quickly configure the Prometheus Operator with Ambassador Edge Stack:

• statsd-sink.yaml Creates the Prometheus Stats Exporter deployment and statsd-sink service that receives the statistics from Ambassador Edge Stack and translates them to Prometheus metrics. It also creates a ServiceMonitor resource that tells the Prometheus Operator to configure Prometheus to fetch those metrics from the StatsD Exporter.
• prometheus.yaml Deploys the Prometheus Operator and creates Prometheus resource that tells the Prometheus Operator to create the actual Prometheus deployment.

Make sure that the ServiceMonitor is in the same namespace as Ambassador Edge Stack. A walk-through of the basics of configuring the Prometheus Operator with Ambassador Edge Stack is available here.

Ensure STATSD_ENABLED is set to "true" and apply the YAML with kubectl:

Wait for a minute after the pods spin up and then access the Prometheus dashboard by port-forwarding the Prometheus pod and going to http://localhost:9090/ on a web-browser.

Using Grafana to visualize statistics gathered by Prometheus

If you're using Grafana, Alex Gervais has written a template Ambassador Edge Stack dashboard for Grafana that works with either the metrics exposed by the Prometheus StatsD Exporter, or by [the :8877/metrics endpoint].

Using Datadog DogStatsD as the StatsD sink

If you are a user of the Datadog monitoring system, pulling in the Envoy statistics from Ambassador Edge Stack is very easy.

Because the DogStatsD protocol is slightly different than the normal StatsD protocol, in addition to setting Ambassador Edge Stack's STATSD_ENABLED=true environment variable, you also need to set the DOGSTATSD=true environment variable:

Then, you will need to deploy the DogStatsD agent in to your cluster to act as the StatsD sink. To do this, replace the sample API key in our sample YAML file with your own, then apply that YAML:

This sets up the statsd-sink service and a deployment of the DogStatsD agent that forwards the Ambassador Edge Stack statistics to your Datadog account.

Additionally, Ambassador Edge Stack supports setting the dd.internal.entity_id statitics tag using the DD_ENTITY_ID environment variable. If this value is set, statistics will be tagged with the value of the environment variable. Otherwise, this statistics tag will be omitted (the default).