Ambassador Edge Stack Release Notes

Support for the Envoy V2 API is deprecated in Ambassador Edge Stack v2.1, and will be removed in Ambassador Edge Stack v2.2.0. The AMBASSADOR_ENVOY_API_VERSION environment variable will be removed at the same time. Only the Envoy V3 API will be supported (this has been the default since Ambassador Edge Stack v1.14.0).

Docker BuildKit is enabled for all Emissary builds. Additionally, the Go build cache is fully enabled when building images, speeding up repeated builds.

In Ambassador Edge Stack 2.1.0 and 2.1.1, an OAuth2 Filter with clientAuthentication.method=jwtAssertion would not function correctly as it would fail to select the signing-method-appropriate function to parse the private key.

In Ambassador Edge Stack 2.1.0 and 2.1.1, an ifRequestHeader selector (in a FilterPolicy, OAuth2 Filter useSessionCookies, or OAuth2 Filter insteadOfRedirect) without a value or valueRegex would erroneously behave as if valueRegex='^$', rather than performing a simple presence check.

Ambassador Edge Stack 2.1.1 generated invalid Envoy configuration for getambassador.io/v2 Mappings that set spec.cors.origins to a string rather than a list of strings; this has been fixed, and these Mappings should once again function correctly.

Changes to the weight of Mapping in a canary group will now always be correctly managed during reconfiguration; such changes could have been missed in earlier releases.

A Mapping that is not part of a canary group, but that has a weight less than 100, will be correctly configured to receive all traffic as if the weight were 100.

Using rewrite: "" in a Mapping is correctly handled to mean "do not rewrite the path at all".

Any Mapping that uses the host_redirect field is now properly discovered and used. Thanks to Gabriel Féron for contributing this bugfix!

Mappings with DNS wildcard hostname will now be correctly matched with Hosts. Previously, the case where both the Host and the Mapping use DNS wildcards for their hostnames could sometimes not correctly match when they should have.

If the ambassador Module sets a global default for add_request_headers, add_response_headers, remove_request_headers, or remove_response_headers, it is often desirable to be able to turn off that setting locally for a specific Mapping. For several releases this has not been possible for Mappings that are native Kubernetes resources (as opposed to annotations), as an empty value ("mask the global default") was erroneously considered to be equivalent to unset ("inherit the global default"). This is now fixed.

It is now possible to set a Mapping spec.error_response_overrides body.text_format to an empty string or body.json_format to an empty dict. Previously, this was possible for annotations but not for native Kubernetes resources.

Resources that exist as getambassador.io/config annotations rather than as native Kubernetes resources are now validated and internally converted to v3alpha1 and, the same as native Kubernetes resources.

Resource validation errors are now reported more consistently; it was the case that in some situations a validation error would not be reported.

Support for the Envoy V2 API is deprecated in Ambassador Edge Stack v2.1, and will be removed in Ambassador Edge Stack v2.2.0. The AMBASSADOR_ENVOY_API_VERSION environment variable will be removed at the same time. Only the Envoy V3 API will be supported (this has been the default since Ambassador Edge Stack v1.14.0).

In Edge Stack 2.1.0, it erroneously ignored Filters, FilterPolicies, and RateLimits that were created as v3alpha1 (but correctly paid attention to them if they were created as v2 or older). This is fixed; it will now correctly pay attention to both API versions.

Support for the Envoy V2 API is deprecated in Ambassador Edge Stack v2.1, and will be removed in Ambassador Edge Stack v2.2.0. The AMBASSADOR_ENVOY_API_VERSION environment variable will be removed at the same time. Only the Envoy V3 API will be supported (this has been the default since Ambassador Edge Stack v1.14.0).

Ambassador Edge Stack supports getambassador.io/v2 CRDs, to simplify migration from Ambassador Edge Stack 1.X. Note: it is important to read the migration documentation before starting migration.

The Ambassador Edge Stack CRDs (Filter, FilterPolicy, and RateLimit) will now be validated for correct syntax by Kubernetes itself. This means that kubectl apply will reject invalid CRDs before they are actually applied, preventing them from causing errors.

The incremental reconfiguration cache could miss some updates when multiple Mappings had the same prefix ("canary"ing multiple Mappings together). This has been corrected, so that all such updates correctly take effect.

When using Kubernetes Secrets to store ACME private keys (as the Edge Stack ACME client does), an error would always be logged about the Secret not being present, even though it was present, and everything was working correctly. This error is no longer logged.

When using gzip compression, upstream services will no longer receive compressed data. This bug was introduced in 1.14.0. The fix restores the default behavior of not sending compressed data to upstream services.

Update to busybox 1.34.1 to resolve CVE-2021-28831, CVE-2021-42378, CVE-2021-42379, CVE-2021-42380, CVE-2021-42381, CVE-2021-42382, CVE-2021-42383, CVE-2021-42384, CVE-2021-42385, and CVE-2021-42386.

Update Python dependencies to resolve CVE-2020-28493 (jinja2), CVE-2021-28363 (urllib3), and CVE-2021-33503 (urllib3).

Previous built images included some Python packages used only for test. These have now been removed, resolving CVE-2020-29651.

When Ambassador Edge Stack makes a cacheable internal request (such as fetching the JWKS endpoint for a JWT Filter), if a cache-miss occurs but a request for that resource is already in-flight, then instead of performing a second request in parallel, it will now wait for the first request to finish and (if the response is cacheable) use that response. If the response turns out to be non-cacheable, then it will proceed to make the second request. This avoids the situation where if a cache entry expires during a moment with high number of concurrent requests, then Edge Stack creates a deluge of concurrent requests to the resource when one aught to have sufficed; this allows the result to be returned more quickly while putting less load on the remote resource. However, if the response turns out to be non-cacheable, then this does effectively serialize requests, increasing the latency for concurrent requests.

It is now possible to set the circuit_breakers for AuthServices, exactly the same as for Mappings and TCPMappings. This makes it possible to configure your AuthService to be able to handle more than 1024 concurrent requests.

When Ambassador Edge Stack completes an internal request (such as fetching the JWKS endpoint for a JWT Filter) it logs (at the info log level) how long the request took. Previously, the duration logged was how long it took to receive the response header, and did not count the time it takes to receive the entire response body; now it properly times the entire thing. Additionally, it now separately logs the "total duration" and the "networking duration", in order to make it possible to identify when a request was delayed waiting for other requests to finish.

Any token delimited by '%' is now validated agains a whitelist of valid Envoy command operators. Any mapping containing an error_response_overrides section with invalid command operators will be discarded.

The Host CRD now correctly supports the mappingSelector element, as documented. As a transition aid, selector is a synonym for mappingSelector; a future version of Ambassador Edge Stack will remove the selector element.

We're pleased to introduce Ambassador Edge Stack 2.0.4 for general availability! The 2.X family introduces a number of changes to allow Ambassador Edge Stack to more gracefully handle larger installations, reduce global configuration to better handle multitenant or multiorganizational installations, reduce memory footprint, and improve performance. We welcome feedback!! Join us on Slack and let us know what you think.

The x.getambassador.io/v3alpha1 API version has become the getambassador.io/v3alpha1 API version. The Ambassador- prefixes from x.getambassador.io/v3alpha1 resources have been removed for ease of migration. Note that getambassador.io/v3alpha1 is the only supported API version for 2.0.4 — full support for getambassador.io/v2 will arrive soon in a later 2.X version.

The getambassador.io/v3alpha1 API version and the published chart and manifests have been updated to support Kubernetes 1.22. Thanks to Mohit Sharma for contributions to this feature!

You can now set dns_type between strict_dns and logical_dns in a Mapping to configure the Service Discovery Type.

You can now set respect_dns_ttl to true to force the DNS refresh rate for a Mapping to be set to the record's TTL obtained from DNS resolution.

You can now set buffer_limit_bytes in the ambassador Module to to change the size of the upstream read and write buffers. The default is 1MiB.

The release now shows its actual released version number, rather than the internal development version number.

Large configurations no longer cause Ambassador Edge Stack to be unable to communicate with Ambassador Cloud.

The l7Depth element of the Listener CRD is properly supported.

We're pleased to introduce Ambassador Edge Stack 2.0.3 as a developer preview. The 2.X family introduces a number of changes to allow Ambassador Edge Stack to more gracefully handle larger installations, reduce global configuration to better handle multitenant or multiorganizational installations, reduce memory footprint, and improve performance. We welcome feedback!! Join us on Slack and let us know what you think.

The environment variable AES_LOG_LEVEL now also sets the log level for the diagd logger.

You can now set dns_type in the AmbassadorMapping to configure how Envoy will use the DNS for the service.

It is no longer necessary to set DOCKER_BUILDKIT=0 when building Emissary. A future change will fully support BuildKit.

We're pleased to introduce Ambassador Edge Stack 2.0.2 as a developer preview. The 2.X family introduces a number of changes to allow Ambassador Edge Stack to more gracefully handle larger installations, reduce global configuration to better handle multitenant or multiorganizational installations, reduce memory footprint, and improve performance. We welcome feedback!! Join us on Slack and let us know what you think.

Upgraded envoy to 1.17.4 to address security vulnerabilities CVE-2021-32777, CVE-2021-32778, CVE-2021-32779, and CVE-2021-32781.

You can now set allow_chunked_length in the Ambassador Module to configure the same value in Envoy.

Envoy-configuration snapshots get saved (as ambex-#.json) in /ambassador/snapshots. The number of snapshots is controlled by the AMBASSADOR_AMBEX_SNAPSHOT_COUNT environment variable; set it to 0 to disable. The default is 30.

We're pleased to introduce Ambassador Edge Stack 2.0.1 as a developer preview. The 2.X family introduces a number of changes to allow Ambassador Edge Stack to more gracefully handle larger installations, reduce global configuration to better handle multitenant or multiorganizational installations, reduce memory footprint, and improve performance. We welcome feedback!! Join us on Slack and let us know what you think.

Ambassador Agent reports sidecar process information and AmbassadorMapping OpenAPI documentation to Ambassador Cloud to provide more visibility into services and clusters.

The optional stats_prefix element of the AmbassadorListener CRD now determines the prefix of HTTP statistics emitted for a specific AmbassadorListener.

The optional stats_name element of AmbassadorMapping, AmbassadorTCPMapping, AuthService, LogService, RateLimitService, and TracingService now sets the name under which cluster statistics will be logged. The default is the service, with non-alphanumeric characters replaced by underscores.

The AmbassadorMapping resource can now specify docs.timeout_ms to set the timeout when the Dev Portal is fetching API specifications.

The Dev Portal will now strip HTML tags when displaying search results, showing just the actual content of the search result.

We have updated to k8s.io/klog/v2 to track upstream and to quiet unnecessary log output.

Logs now include subsecond time resolutions, rather than just seconds.

Set AMBASSADOR_AMBEX_NO_RATELIMIT to true to completely disable ratelimiting Envoy reconfiguration under memory pressure. This can help performance with the endpoint or Consul resolvers, but could make OOMkills more likely with large configurations. The default is false, meaning that the rate limiter is active.

Consul certificate-rotation logging now includes the fingerprints and validity timestamps of certificates being rotated.

The maxStale field is now supported in in the JWT Filter to configure how long $productname$ should cache OIDC responses for similar to the existing maxStale field in the OAuth2 Filter.

We're pleased to introduce Ambassador Edge Stack 2.0.0 as a developer preview. The 2.X family introduces a number of changes to allow Ambassador Edge Stack to more gracefully handle larger installations, reduce global configuration to better handle multitenant or multiorganizational installations, reduce memory footprint, and improve performance. We welcome feedback!! Join us on Slack and let us know what you think.

Ambassador Edge Stack 2.0.0 introduces API version x.getambassador.io/v3alpha1 for configuration changes that are not backwards compatible with the 1.X family. API versions getambassador.io/v0, getambassador.io/v1, and getambassador.io/v2 are deprecated. Further details are available in the Major Changes in 2.X document.

The new AmbassadorListener CRD defines where and how to listen for requests from the network, and which AmbassadorHost definitions should be used to process those requests. Note that the AmbassadorListener CRD is mandatory and consolidates all port configuration; see the AmbassadorListener documentation for more details.

Where AmbassadorMapping's host field is either an exact match or (with host_regex set) a regex, the new hostname element is always a DNS glob. Use hostname instead of host for best results.

The behavior of the Ambassador module prune_unreachable_routes field is now automatic, which should reduce Envoy memory requirements for installations with many AmbassadorHosts

Each AmbassadorHost can specify its requestPolicy.insecure.action independently of any other AmbassadorHost, allowing for HTTP routing as flexible as HTTPS routing.

Ambassador Edge Stack 2.0.0 fixes a regression in detecting the Ambassador Kubernetes service that could cause the wrong IP or hostname to be used in Ingress statuses -- thanks, Noah Fontes!

Ambassador Edge Stack 2.0.0 fixes a bug where mTLS could use the wrong configuration when SNI and the :authority header didn't match

The TLSContext redirect_cleartext_from and AmbassadorHost requestPolicy.insecure.additionalPort elements are no longer supported. Use a AmbassadorListener for this functionality instead.

The use_proxy_protocol element of the Ambassador Module is no longer supported, as it is now part of the AmbassadorListener resource (and can be set per-AmbassadorListener rather than globally).

An AmbassadorMapping will only be matched with an AmbassadorHost if the AmbassadorMapping's host or the AmbassadorHost's selector (or both) are explicitly set, and match. This change can significantly improve Ambassador Edge Stack's memory footprint when many AmbassadorHosts are involved. Further details are available in the 2.0.0 Changes document.

An AmbassadorHost or Ingress resource is now required when terminating TLS -- simply creating a TLSContext is not sufficient. Further details are available in the AmbassadorHost CRD documentation.

By default, Ambassador Edge Stack will configure Envoy using the V3 Envoy API. This change is mostly transparent to users, but note that Envoy V3 does not support unsafe regular expressions or, e.g., Zipkin's V1 collector protocol. Further details are available in the Major Changes in 2.X document.

The tls module and the tls field in the Ambassador module are no longer supported. Please use TLSContext resources instead.

The environment variable AMBASSADOR_FAST_RECONFIGURE is now set by default, enabling the higher-performance implementation of the code that Ambassador Edge Stack uses to generate and validate Envoy configurations.

Service Preview and the AGENT_SERVICE environment variable are no longer supported. The Telepresence product replaces this functionality.

The edgectl CLI tool has been deprecated; please use the emissary-ingress helm chart instead.

Upgraded Envoy to address security vulnerabilities CVE-2021-43824, CVE-2021-43825, CVE-2021-43826, CVE-2022-21654, and CVE-2022-21655.

You can now set respect_dns_ttl in Ambassador Mappings. When true it configures that upstream's refresh rate to be set to resource record’s TTL

You can now set dns_type in Ambassador Mappings to use Envoy's logical_dns resolution instead of the default strict_dns.

You can now set buffer_limit_bytes in the ambassador Module to to change the size of the upstream read and write buffers. The default is 1MiB.

You can now set preserve_servers in Ambassador Edge Stack's DevPortal resource to configure the DevPortal to use server definitions from the OpenAPI document when displaying connection information for services in the DevPortal.

Upgraded Envoy to 1.17.4 to address security vulnerabilities CVE-2021-32777, CVE-2021-32778, CVE-2021-32779, and CVE-2021-32781.

Update from Envoy 1.15 to 1.17.3

You can now set allow_chunked_length in the Ambassador Module to configure the same value in Envoy.

AMBASSADOR_ENVOY_API_VERSION now defaults to V3

Logs now include subsecond time resolutions, rather than just seconds.

Fixed a regression when specifying a comma separated string for cors.origins on the Mapping resource. ([#3609](https://github.com/emissary-ingress/emissary/issues/3609))

Envoy-configuration snapshots get saved (as ambex-#.json) in /ambassador/snapshots. The number of snapshots is controlled by the AMBASSADOR_AMBEX_SNAPSHOT_COUNT environment variable; set it to 0 to disable. The default is 30.

Set AMBASSADOR_AMBEX_NO_RATELIMIT to true to completely disable ratelimiting Envoy reconfiguration under memory pressure. This can help performance with the endpoint or Consul resolvers, but could make OOMkills more likely with large configurations. The default is false, meaning that the rate limiter is active.

The Mapping resource can now specify docs.timeout_ms to set the timeout when the Dev Portal is fetching API specifications.

The Dev Portal will now strip HTML tags when displaying search results, showing just the actual content of the search result.

Consul certificate-rotation logging now includes the fingerprints and validity timestamps of certificates being rotated.

Configuring multiple TCPMappings with the same ports (but different hosts) no longer generates invalid Envoy configuration.

Ambassador Agent now accurately reports up-to-date Endpoint information to Ambassador Cloud

Ambassador Agent reports ConfigMaps and Deployments to Ambassador Cloud to provide a better Argo Rollouts experience. See [Argo+Ambassador documentation](https://www.getambassador.io/docs/argo) for more info.

Add AMBASSADOR_JSON_LOGGING to enable JSON for most of the Ambassador control plane. Some (but few) logs from gunicorn and the Kubernetes client-go package still log text.

Fixed a bug where the Consul resolver would not actually use Consul endpoints with TCPMappings.

Ambassador now calculates its own memory usage in a way that is more similar to how the kernel OOMKiller tracks memory.

Fixed a regression where Ambassador snapshot data was logged at the INFO label when using AMBASSADOR_LEGACY_MODE=true.

Fix a regression from 1.8.0 that prevented ambassador Module config keys proper_case and preserve_external_request_id from working correctly.

Fixed a regression in detecting the Ambassador Kubernetes service that could cause the wrong IP or hostname to be used in Ingress statuses (thanks, [Noah Fontes](https://github.com/impl)!

Incorporate the Envoy 1.15.5 security update by adding the reject_requests_with_escaped_slashes option to the Ambassador module.