Upgrade Emissary-ingress 1.14.X (Helm)

We're pleased to introduce Emissary-ingress 2.5.1! The 2.X family introduces a number of changes to allow Emissary-ingress to more gracefully handle larger installations (including multitenant or multiorganizational installations), reduce memory footprint, and improve performance. In keeping with SemVer, Emissary-ingress 2.X introduces some changes that aren't backward-compatible with 1.X. These changes are detailed in Major Changes in Emissary-ingress 2.X.

Migration Overview

The recommended strategy for migration is to run Emissary-ingress 1.14 and Emissary-ingress 2.5.1 side-by-side in the same cluster. This gives Emissary-ingress 2.5.1 and Emissary-ingress 1.14 access to all the same configuration resources, with some important caveats:

1. Emissary-ingress 1.14 will not see any getambassador.io/v3alpha1 resources.

   This is intentional; it provides a way to apply configuration only to Emissary-ingress 2.5.1, while not interfering with the operation of your Emissary-ingress 1.14 installation.

2. If needed, you can use labels to further isolate configurations.

   If you need to prevent your Emissary-ingress 2.5.1 installation from seeing a particular bit of Emissary-ingress 1.14 configuration, you can apply a Kubernetes label to the configuration resources that should be seen by your Emissary-ingress 2.5.1 installation, then set its AMBASSADOR_LABEL_SELECTOR environment variable to restrict its configuration to only the labelled resources.

   For example, you could apply a version-two: true label to all resources that should be visible to Emissary-ingress 2.5.1, then set AMBASSADOR_LABEL_SELECTOR=version-two=true in its Deployment.

3. Be careful about label selectors on Kubernetes Services!

   If you have services in Emissary-ingress 1.14 that use selectors that will match Pods from Emissary-ingress 2.5.1, traffic will be erroneously split between Emissary-ingress 1.14 and Emissary-ingress 2.5.1. The labels used by Emissary-ingress 2.5.1 include:

4. Be careful to only have one Emissary-ingress Agent running at a time.

   The Emissary-ingress Agent is responsible for communications between Emissary-ingress and Ambassador Cloud. If multiple versions of the Agent are running simultaneously, Ambassador Cloud could see conflicting information about your cluster.

   The best way to avoid multiple agents when installing with Helm is to use --set agent.enabled=false to tell Helm not to install a new Agent with Emissary-ingress 2.5.1. Once testing is done, you can switch Agents safely.

You can also migrate by installing Emissary-ingress 2.5.1 in a separate cluster. This permits absolute certainty that your Emissary-ingress 1.14 configuration will not be affected by changes meant for Emissary-ingress 2.5.1, and it eliminates concerns about ACME, but it is more effort.

Side-by-Side Migration Steps

Migration is a seven-step process:

1. Make sure that older configuration resources are not present.

   Emissary-ingress 2.X does not support getambassador.io/v0 or getambassador.io/v1 resources, and Kubernetes will not permit removing support for CRD versions that are still in use for stored resources. To verify that no resources older than getambassador.io/v2 are active, run

   If v1 is present in the output, do not begin migration. The old resources must be converted to getambassador.io/v2 and the storedVersion information in the cluster must be updated. If necessary, contact Ambassador Labs on Slack for more information.

2. Install new CRDs.

   Before installing Emissary-ingress 2.5.1 itself, you must configure your Kubernetes cluster to support its new getambassador.io/v3alpha1 configuration resources. Note that getambassador.io/v2 resources are still supported, but you must install support for getambassador.io/v3alpha1 to run Emissary-ingress 2.5.1, even if you intend to continue using only getambassador.io/v2 resources for some time.

   Emissary-ingress 2.5.1 includes a Deployment in the `emissary-system` namespace called emissary-apiext. This is the APIserver extension that supports converting Emissary-ingress CRDs between getambassador.io/v2and getambassador.io/v3alpha1. This Deployment needs to be running at all times.
   If the emissary-apiext Deployment's Pods all stop running, you will not be able to use getambassador.io/v3alpha1 CRDs until restarting the emissary-apiext Deployment.
   There is a known issue with the emissary-apiext service that impacts all Emissary-ingress 2.x and 3.x users. Specifically, the TLS certificate used by apiext expires one year after creation and does not auto-renew. All users who are running Emissary-ingress/Ambassador Edge Stack 2.x or 3.x with the apiext service should proactively renew their certificate as soon as practical by running kubectl delete --all secrets --namespace=emissary-system to delete the existing certificate, and then restart the emissary-apiext deployment with kubectl rollout restart deploy/emissary-apiext -n emissary-system. This will create a new certificate with a one year expiration. We will issue a software patch to address this issue well before the one year expiration. Note that certificate renewal will not cause any downtime.

3. Install Emissary-ingress 2.5.1.

   After installing the new CRDs, you need to install Emissary-ingress 2.5.1 itself in the same namespace as your existing Emissary-ingress 1.14 installation. It's important to use the same namespace so that the two installations can see the same secrets, etc.

   Start by making sure that your emissary Helm repo is set correctly:

   Typically, Emissary-ingress 1.14 was installed in the ambassador namespace. If you installed Emissary-ingress 1.14 in a different namespace, change the namespace in the commands below.

   • If you do not need to set AMBASSADOR_LABEL_SELECTOR:

   • If you do need to set AMBASSADOR_LABEL_SELECTOR, use --set, for example:

   You must use the emissary-ingress Helm chart for Emissary-ingress 2.X. Do not use the ambassador Helm chart.
   Emissary-ingress 2.5.1 includes a Deployment in the emissary namespace called emissary-apiext. This is the APIserver extension that supports converting Emissary-ingress CRDs between getambassador.io/v2and getambassador.io/v3alpha1. This Deployment needs to be running at all times.
   If the emissary-apiext Deployment's Pods all stop running, you will not be able to use getambassador.io/v3alpha1 CRDs until restarting the emissary-apiext Deployment.

4. Install Listeners and Hosts as needed.

   An important difference between Emissary-ingress 1.14 and Emissary-ingress 2.5.1 is the new mandatory Listener CRD. Also, when running both installations side by side, you will need to make sure that a Host is present for the new Emissary-ingress 2.5.1 Service. For example:

   This example requires that you know the hostname for the Emissary-ingress Service ($EMISSARY_HOSTNAME) and that you have created a TLS Secret for it in $EMISSARY_TLS_SECRET.

5. Test!

   Your Emissary-ingress 2.5.1 installation can support the getambassador.io/v2 configuration resources used by Emissary-ingress 1.14, but you may need to make some changes to the configuration, as detailed in the documentation on configuring Emissary-ingress Communications and updating CRDs to getambassador.io/v3alpha1.

   Kubernetes will not allow you to have a getambassador.io/v3alpha1 resource with the same name as a getambassador.io/v2 resource or vice versa: only one version can be stored at a time.
   
   If you find that your Emissary-ingress 2.5.1 installation and your Emissary-ingress 1.14 installation absolutely must have resources that are only seen by one version or the other way, see overview section 2, "If needed, you can use labels to further isolate configurations".

   If you find that you need to roll back, just reinstall your 1.14 CRDs and delete your installation of Emissary-ingress 2.5.1.

6. When ready, switch over to Emissary-ingress 2.5.1.

   You can run Emissary-ingress 1.14 and Emissary-ingress 2.5.1 side-by-side as long as you care to. However, taking full advantage of Emissary-ingress 2.X's capabilities requires updating your configuration to use getambassador.io/v3alpha1 configuration resources, since some useful features in Emissary-ingress 2.5.1 are only available using getambassador.io/v3alpha1 resources.

   When you're ready to have Emissary-ingress 2.5.1 handle traffic on its own, switch your original Emissary-ingress 1.14 Service to point to Emissary-ingress 2.5.1. Use kubectl edit service ambassador and change the selectors to:

   Repeat using kubectl edit service ambassador-admin for the ambassador-admin Service.

7. Finally, install the Emissary-ingress 2.5.1 Ambassador Agent.

   First, scale the 1.14 agent to 0:

   Once that's done, install the new Agent. Note that if you needed to set AMBASSADOR_LABEL_SELECTOR, you must add that to this helm upgrade command.

Congratulations! At this point, Emissary-ingress 2.5.1 is fully running and it's safe to remove the ambassador and ambassador-agent Deployments:

Once Emissary-ingress 1.14 is no longer running, you may convert any remaining getambassador.io/v2 resources to getambassador.io/v3alpha1. You may also want to redirect DNS to the edge-stack Service and remove the ambassador Service.