DocsEdge Stack1.14Debugging
Debugging
If you’re experiencing issues with the Ambassador Edge Stack and cannot diagnose the issue through the "Diagnostics" tab from the Edge Policy Console, this document covers various approaches and advanced use cases for debugging Ambassador Edge Stack issues.
We assume that you already have a running Ambassador Edge Stack installation in the following sections.
Check Ambassador Edge Stack status
First, check to see if the Edge Policy Console is reachable. If it is successful, try to diagnose your original issue with the Console.
If it is not successful, complete the following to see if Ambassador Edge Stack is running:
Get a list of Pods in the
ambassador
namespace withkubectl get pods -n ambassador
.The terminal should print something similar to the following:
Then, check the Ambassador Edge Stack Deployment with the following:
kubectl get -n ambassador deployments
After a brief period, the terminal will print something similar to the following:
Check that the “desired” number of Pods equals the “current” and “available” number of Pods. If they are not equal, check the status of the associated Pods with the following command:
kubectl get pods -n ambassador
.Use the following command for details about the history of the Deployment:
kubectl describe -n ambassador deployment ambassador
Look for data in the “Replicas” field near the top of the output. For example:
Replicas: 3 desired | 3 updated | 3 total | 3 available | 0 unavailable
Look for data in the “Events” log field near the bottom of the output, which often displays data such as a fail image pull, RBAC issues, or a lack of cluster resources. For example:
Additionally, use the following command to “describe” the individual Pods:
kubectl describe pods -n ambassador <ambassador-pod-name>
Look for data in the “Status” field near the top of the output. For example,
Status: Running
Look for data in the “Events” field near the bottom of the output, as it will often show issues such as image pull failures, volume mount issues, and container crash loops. For example:
In both the Deployment Pod and the individual Pods, take the necessary action to address any discovered issues.
Review Ambassador Edge Stack logs
Ambassador Edge Stack logging can provide information on anything that might be abnormal or malfunctioning. While there may be a large amount of data to sort through, look for key errors such as the Ambassador Edge Stack process restarting unexpectedly, or a malformed Envoy configuration.
Log levels
Ambassador Edge Stack has two switches that will control different log levels.
Envoy debug logs
Envoy debug logging shows verbose information on the actions Envoy is taking on every request. It can be useful for understanding why connections are being closed or if Envoy or the upstream service is the source of the error.
You can turn on Debug mode in the Edge Policy Console.
Ambassador Edge Stack debug logging
Ambassador Edge Stack is built on top of Emissary-ingress and runs an additional process for authentication, rate limiting, the developer portal, ACME, etc. Debug logging for this process will give more information on why you may see errors with these functions.
You can adjust the Ambassador Edge Stack log level by setting the
AES_LOG_LEVEL
environment variable.
Viewing logs
You can turn on Debug mode in the Edge Policy Console, which generates verbose logging data that can be useful when trying to find a subtle error or bug.
Use the following command to target an individual Ambassador Edge Stack Pod:
kubectl get pods -n ambassador
The terminal will print something similar to the following:
Then, run the following:
kubectl logs -n ambassador <ambassador-pod-name>
The terminal will print something similar to the following:
Examine Pod and container contents
You can examine the contents of the Ambassador Edge Stack Pod for issues, such as if volume mounts are correct and TLS certificates are present in the required directory, to determine if the Pod has the latest Ambassador Edge Stack configuration, or if the generated Envoy configuration is correct or as expected. In these instructions, we will look for problems related to the Envoy configuration.
To look into an Ambassador Edge Stack Pod, use the container shell with the
kube-exec
and the/bin/sh
commands. For example,kubectl exec -it -n ambassador <ambassador-pod-name> -- /bin/sh
Determine the latest configuration. If you haven't overridden the configuration directory, the latest configuration will be in
/ambassador/snapshots
. If you have overridden it, Ambassador Edge Stack saves configurations in$AMBASSADOR_CONFIG_BASE_DIR/snapshots
.In the snapshots directory:
snapshot.yaml
contains the full input configuration that Ambassador Edge Stack has found;aconf.json
contains the Ambassador Edge Stack configuration extracted from the snapshot;ir.json
contains the IR constructed from the Ambassador Edge Stack configuration; andeconf.json
contains the Envoy configuration generated from the IR.
The Envoy configuration is then split into
$AMBASSADOR_CONFIG_BASE_DIR/bootstrap-ads.json
and$AMBASSADOR_CONFIG_BASE_DIR/envoy/envoy.json
, which are the actual input files handed to Envoy.In the snapshots directory, the current configuration will be stored in files with no digit suffix, and older configurations have increasing numbers. For example,
ir.json
is current,ir-1.json
is the next oldest, thenir-2.json
, etc.If something is wrong with
snapshot
oraconf
, there is an issue with your configuration. If something is wrong withir
oreconf
, you should open an issue on Github.To find the main configuration for Envoy, run:
$AMBASSADOR_CONFIG_BASE_DIR/envoy/envoy.json
.For the bootstrap configuration, which has details about Envoy statistics, logging, and auth, run:
$AMBASSADOR_CONFIG_BASE_DIR/bootstrap-ads.json
.For further details, you can print the Envoy configuration that is generated during the Ambassador Edge Stack configuration. The file will be titled
envoy-N.json
where N matches the number of theambassador-config-N
directory number. Run the following command:# cat envoy-2.json
The terminal will print something similar to the following:
The contents of the Envoy configuration files can be very useful when looking for subtle mapping issues or bugs.