Upgrade Ambassador Edge Stack 1.14.X to Ambassador Edge Stack 2.3.2 (YAML)

We're pleased to introduce Ambassador Edge Stack 2.3.2! The 2.X family introduces a number of changes to allow Ambassador Edge Stack to more gracefully handle larger installations (including multitenant or multiorganizational installations), reduce memory footprint, and improve performance. In keeping with SemVer, Ambassador Edge Stack 2.X introduces some changes that aren't backward-compatible with 1.X. These changes are detailed in Major Changes in Ambassador Edge Stack 2.X.

Migration Overview

The recommended strategy for migration is to run Ambassador Edge Stack 1.14 and Ambassador Edge Stack 2.3.2 side-by-side in the same cluster. This gives Ambassador Edge Stack 2.3.2 and Ambassador Edge Stack 1.14 access to all the same configuration resources, with some important caveats:

1. Ambassador Edge Stack 1.14 will not see any getambassador.io/v3alpha1 resources.

   This is intentional; it provides a way to apply configuration only to Ambassador Edge Stack 2.3.2, while not interfering with the operation of your Ambassador Edge Stack 1.14 installation.

2. If needed, you can use labels to further isolate configurations.

   If you need to prevent your Ambassador Edge Stack 2.3.2 installation from seeing a particular bit of Ambassador Edge Stack 1.14 configuration, you can apply a Kubernetes label to the configuration resources that should be seen by your Ambassador Edge Stack 2.3.2 installation, then set its AMBASSADOR_LABEL_SELECTOR environment variable to restrict its configuration to only the labelled resources.

   For example, you could apply a version-two: true label to all resources that should be visible to Ambassador Edge Stack 2.3.2, then set AMBASSADOR_LABEL_SELECTOR=version-two=true in its Deployment.

3. Ambassador Edge Stack 1.14 must remain in control of ACME while both installations are running.

   The processes that handle ACME challenges cannot be managed by both Ambassador Edge Stack 1.X and Ambassador Edge Stack 2.3.2 at the same time. The instructions below disable ACME in Ambassador Edge Stack 2.3.2, allowing Ambassador Edge Stack 1.14 to continue managing it.

   This implies that any new Hosts used for Ambassador Edge Stack 1.14 should be created using getambassador.io/v2 so that Ambassador Edge Stack 1.14 can see them.

4. Check AuthService and RateLimitService resources, if any.

   If you have an AuthService or RateLimitService installed, make sure that they are using the namespace-qualified DNS name. If they are not, the initial migration tests may fail.

   Additionally, when installing with Helm, you must make sure that Ambassador Edge Stack 2.3.2 does not attempt to create duplicate AuthService and RateLimitService entries. Add

   and

   on the Helm command line to prevent duplicating these resources.

5. Be careful to only have one Ambassador Edge Stack Agent running at a time.

   The Ambassador Edge Stack Agent is responsible for communications between Ambassador Edge Stack and Ambassador Cloud. If multiple versions of the Agent are running simultaneously, Ambassador Cloud could see conflicting information about your cluster.

   The migration YAML used below to install Ambassador Edge Stack 2.3.2 will not install a duplicate agent. If you are building your own YAML, make sure not to include a duplicate agent.

6. If you use ACME for multiple Hosts, add a wildcard Host too.

   This is required to manage a known issue. This issue will be resolved in a future Ambassador Edge Stack release.

7. Be careful about label selectors on Kubernetes Services!

   If you have services in Ambassador Edge Stack 1.14 that use selectors that will match Pods from Ambassador Edge Stack 2.3.2, traffic will be erroneously split between Ambassador Edge Stack 1.14 and Ambassador Edge Stack 2.3.2. The labels used by Ambassador Edge Stack 2.3.2 include:

You can also migrate by installing Ambassador Edge Stack 2.3.2 in a separate cluster. This permits absolute certainty that your Ambassador Edge Stack 1.14 configuration will not be affected by changes meant for Ambassador Edge Stack 2.3.2, and it eliminates concerns about ACME, but it is more effort.

Side-by-Side Migration Steps

Migration is an eight-step process:

1. Make sure that older configuration resources are not present.

   Ambassador Edge Stack 2.X does not support getambassador.io/v0 or getambassador.io/v1 resources, and Kubernetes will not permit removing support for CRD versions that are still in use for stored resources. To verify that no resources older than getambassador.io/v2 are active, run

   If v1 is present in the output, do not begin migration. The old resources must be converted to getambassador.io/v2 and the storedVersion information in the cluster must be updated. If necessary, contact Ambassador Labs on Slack for more information.

2. Install new CRDs.

   Before installing Ambassador Edge Stack 2.3.2 itself, you must configure your Kubernetes cluster to support its new getambassador.io/v3alpha1 configuration resources. Note that getambassador.io/v2 resources are still supported, but you must install support for getambassador.io/v3alpha1 to run Ambassador Edge Stack 2.3.2, even if you intend to continue using only getambassador.io/v2 resources for some time.

   Ambassador Edge Stack 2.3.2 includes a Deployment in the `emissary-system` namespace called emissary-apiext. This is the APIserver extension that supports converting Ambassador Edge Stack CRDs between getambassador.io/v2and getambassador.io/v3alpha1. This Deployment needs to be running at all times.
   If the emissary-apiext Deployment's Pods all stop running, you will not be able to use getambassador.io/v3alpha1 CRDs until restarting the emissary-apiext Deployment.
   There is a known issue with the emissary-apiext service that impacts all Ambassador Edge Stack 2.x and 3.x users. Specifically, the TLS certificate used by apiext expires one year after creation and does not auto-renew. All users who are running Ambassador Edge Stack/Emissary-ingress 2.x or 3.x with the apiext service should proactively renew their certificate as soon as practical by running kubectl delete --all secrets --namespace=emissary-system to delete the existing certificate, and then restart the emissary-apiext deployment with kubectl rollout restart deploy/emissary-apiext -n emissary-system. This will create a new certificate with a one year expiration. We will issue a software patch to address this issue well before the one year expiration. Note that certificate renewal will not cause any downtime.

3. Install Ambassador Edge Stack 2.3.2.

   After installing the new CRDs, you need to install Ambassador Edge Stack 2.3.2 itself in the same namespace as your existing Ambassador Edge Stack 1.14 installation. It's important to use the same namespace so that the two installations can see the same secrets, etc.

   We publish three manifests for different namespaces. Use only the one that matches the namespace into which you installed Ambassador Edge Stack 1.14:

   • aes-emissaryns-migration.yaml for the emissary namespace;
   • aes-defaultns-migration.yaml for the default namespace; and
   • aes-ambassadorns-migration.yaml for the ambassador namespace.

   All three files are set up as follows:

   • They set the AES_ACME_LEADER_DISABLE environment variable to prevent Ambassador Edge Stack 2.3.2 from trying to manage ACME (leaving Ambassador Edge Stack 1.14 to do it instead).
   • They do NOT set AMBASSADOR_LABEL_SELECTOR.
   • They do NOT install the Ambassador Agent.
   • They do NOT create an AuthService or a RateLimitService. It is very important that Ambassador Edge Stack 2.3.2 not attempt to create these resources, as they are already provided for your Ambassador Edge Stack 1.14 installation.

   If any of these do not match your situation, download aes-ambassadorns-migration.yaml and edit it as needed.

   Assuming you're using the ambassador namespace, as was typical for Ambassador Edge Stack 1.14:

   Make sure that at most one installation of Ambassador Edge Stack is running without setting the AES_ACME_LEADER_DISABLE flag. This prevents collisions in trying to manage ACME.

4. Install Listeners and Hosts as needed.

   An important difference between Ambassador Edge Stack 1.14 and Ambassador Edge Stack 2.3.2 is the new mandatory Listener CRD. Also, when running both installations side by side, you will need to make sure that a Host is present for the new Ambassador Edge Stack 2.3.2 Service. For example:

   Make sure that any Hosts you create use API version getambassador.io/v2, so that they can be managed by Ambassador Edge Stack 1.14 as long as both installations are running.

   This example requires that you know the hostname for the Ambassador Edge Stack Service ($EMISSARY_HOSTNAME) and that you have created a TLS Secret for it in $EMISSARY_TLS_SECRET.

5. Test!

   Your Ambassador Edge Stack 2.3.2 installation can support the getambassador.io/v2 configuration resources used by Ambassador Edge Stack 1.14, but you may need to make some changes to the configuration, as detailed in the documentation on configuring Ambassador Edge Stack Communications and updating CRDs to getambassador.io/v3alpha1.

   Kubernetes will not allow you to have a getambassador.io/v3alpha1 resource with the same name as a getambassador.io/v2 resource or vice versa: only one version can be stored at a time.
   
   If you find that your Ambassador Edge Stack 2.3.2 installation and your Ambassador Edge Stack 1.14 installation absolutely must have resources that are only seen by one version or the other way, see overview section 2, "If needed, you can use labels to further isolate configurations".

   If you find that you need to roll back, just reinstall your 1.14 CRDs, delete your installation of Ambassador Edge Stack 2.3.2, and delete the emissary-system namespace.

6. When ready, switch over to Ambassador Edge Stack 2.3.2.

   You can run Ambassador Edge Stack 1.14 and Ambassador Edge Stack 2.3.2 side-by-side as long as you care to. However, taking full advantage of Ambassador Edge Stack 2.X's capabilities requires updating your configuration to use getambassador.io/v3alpha1 configuration resources, since some useful features in Ambassador Edge Stack 2.3.2 are only available using getambassador.io/v3alpha1 resources.

   When you're ready to have Ambassador Edge Stack 2.3.2 handle traffic on its own, switch your original Ambassador Edge Stack 1.14 Service to point to Ambassador Edge Stack 2.3.2. Use kubectl edit -n ambassador service ambassador and change the selectors to:

   Repeat using kubectl edit -n ambassador service ambassador-admin for the ambassador-admin Service.

7. Install the Ambassador Edge Stack 2.3.2 Ambassador Agent.

   First, scale the 1.14 agent to 0:

   Once that's done, install the new Agent:

8. Finally, enable ACME in Ambassador Edge Stack 2.3.2.

   First, scale the 1.14 Ambassador to 0:

   Once that's done, enable ACME in Ambassador Edge Stack 2.3.2:

Congratulations! At this point, Ambassador Edge Stack 2.3.2 is fully running, and it's safe to remove the ambassador and ambassador-agent Deployments:

Once Ambassador Edge Stack 1.14 is no longer running, you may convert any remaining getambassador.io/v2 resources to getambassador.io/v3alpha1. You may also want to redirect DNS to the edge-stack Service and remove the ambassador Service.