Upgrade Ambassador Edge Stack 3.9.X to Ambassador Edge Stack 3.12.1 (Helm)

Since Ambassador Edge Stack's configuration is entirely stored in Kubernetes resources, upgrading between versions is straightforward.

Ambassador Edge Stack 3 is functionally compatible with Ambassador Edge Stack 2.x, but with any major upgrade there are some changes to consider. Such as, Envoy removing support for V2 Transport Protocol features. Below we will outline some of these changes and things to consider when upgrading.

Resources to check before migrating to 3.12.1.

Ambassador Edge Stack 3.X has been upgraded from Envoy 1.17.X to Envoy 1.22 which removed support for the Envoy V2 Transport Protocol. This means all AuthService, RatelimitService, LogServices, and External Filters must be updated to use the V3 Protocol. Additionally support for some of the runtime bootstrap flags has been removed.

You can refer to the Major changes in Ambassador Edge Stack 3.x guide for an overview of the changes.

1. Ambassador Edge Stack 3.2 fixed a bug with Host.spec.selector\mappingSelector and Listener.spec.selector not being properly enforced. In previous versions, if only a single label from the selector was present on the resource then they would be associated. Additionally, when associating Hosts with Mappings, if the Mapping configured a hostname that matched the hostname of the Host then they would be associated regardless of the configuration of the selector\mappingSelector on the Host.

   Before upgrading, review your Ambassador resources, and if you make use of the selectors, ensure that every other resource you want it to be associated with contains all the required labels.

   The environment variable DISABLE_STRICT_LABEL_SELECTORS can be set to "true" on the Ambassador Edge Stack deployment to revert to the old incorrect behavior to help prevent any configuration issues after upgrading in the event that not all manifests making use of the selectors have been corrected yet.

   For more information on DISABLE_STRICT_LABEL_SELECTORS see the Environment Variables page.

2. Check Transport Protocol usage on all resources before migrating.

   The AuthService, RatelimitService, LogServices, and External Filters that use the grpc protocol will now need to explicilty set protocol_version: "v3". If not set or set to v2 then an error will be posted and a static response will be returned.

   protocol_version should be updated to v3 for all of the above resources while still running Ambassador Edge Stack 2.5.1. As of version 2.3.z+, support for protocol_version v2 and v3 is supported in order to allow migration from protocol_version v2 to v3 before upgrading to Ambassador Edge Stack 3.12.1 where support for v2 is removed.

   Upgrading any application code for your own implementations of these services is very straightforward.

   The following imports simply need to be updated to switch from Envoy's Transport Protocol v2 to v3, and then the configuration for these resources can be updated to add the protocl_version: "v3" when the updated service is deployed.

   v2 Imports:

   v3 Imports:

3. Check removed runtime changes

4. Support for LightStep tracing driver removed

Ambassador Edge Stack 3.4 is based on Envoy 1.24.1 which removed support for the LightStep tracing driver. The team at LightStep and the maintainers of Envoy-Proxy recommend that users instead leverage the OpenTelemetry Collector to send tracing information to LightStep. We have written a guide which can be found here Distributed Tracing with OpenTelemetry and Lightstep that outlines how to set this up. It is important that you follow this upgrade path prior to upgrading or you will drop tracing data.

Migration Steps

1. Install new CRDs.

   After reviewing the changes in 3.x and confirming that you are ready to upgrade, the process is the same as upgrading minor versions in previous version of Ambassador Edge Stack and does not require the complex migration steps that the migration from 1.x to 2.x required.

   Before installing Ambassador Edge Stack 3.12.1 itself, you need to update the CRDs in your cluster; Helm will not do this for you. This is mandatory during any upgrade of Ambassador Edge Stack.

   Ambassador Edge Stack 3.12.1 includes a Deployment in the `emissary-system` namespace called emissary-apiext. This is the APIserver extension that supports converting Ambassador Edge Stack CRDs between getambassador.io/v2and getambassador.io/v3alpha1. This Deployment needs to be running at all times.
   If the emissary-apiext Deployment's Pods all stop running, you will not be able to use getambassador.io/v3alpha1 CRDs until restarting the emissary-apiext Deployment.
   There is a known issue with the emissary-apiext service that impacts all Ambassador Edge Stack 2.x and 3.x users. Specifically, the TLS certificate used by apiext expires one year after creation and does not auto-renew. All users who are running Ambassador Edge Stack/Emissary-ingress 2.x or 3.x with the apiext service should proactively renew their certificate as soon as practical by running kubectl delete --all secrets --namespace=emissary-system to delete the existing certificate, and then restart the emissary-apiext deployment with kubectl rollout restart deploy/emissary-apiext -n emissary-system. This will create a new certificate with a one year expiration. We will issue a software patch to address this issue well before the one year expiration. Note that certificate renewal will not cause any downtime.

2. Create an Ambassador Cloud Connect Token

   Ambassador Edge Stack requires a license to operate. Licenses are obtained and refreshed automatically using an API Token. If your installation of Ambassador Edge Stack is already connected to the cloud then you can skip this step. Otherwise, you can create a Cloud Connect Token in Ambassador Cloud and apply it using the command below. For more information on licensing please refer to the license page.

3. Install Ambassador Edge Stack 3.12.1.

   After installing the new CRDs, use Helm to install Ambassador Edge Stack 3.12.1. Start by making sure that your datawire Helm repo is set correctly:

   Then, update your Ambassador Edge Stack installation in the ambassador namespace. If necessary for your installation (e.g. if you were running with AMBASSADOR_SINGLE_NAMESPACE set), you can choose a different namespace.

   You must use the edge-stack Helm chart to install Ambassador Edge Stack 3.X.