Upgrade NGINX Ingress Controller
This document describes how to upgrade F5 NGINX Ingress Controller when a new version releases.
It covers the necessary steps for minor versions as well as major versions (Such as 3.x to 4.x).
Many of the nuances in upgrade paths relate to how custom resource definitions (CRDs) are managed.
If you are running NGINX Ingress Controller v3.x, you should read Upgrade from NGINX Ingress Controller v3.x to v4.0.0 before continuing.
To upgrade the CRDs, pull the Helm chart source, then use kubectl apply:
helm pull oci://ghcr.io/nginx/charts/nginx-ingress --untar --version 2.9.0
kubectl apply -f crds/Alternatively, CRDs can be upgraded without pulling the chart by running:
kubectl apply -f https://raw.githubusercontent.com/nginx/kubernetes-ingress/v5.8.0/deploy/crds.yamlIn the above command, v5.8.0 represents the version of the NGINX Ingress Controller release rather than the Helm chart version.
The following warning is expected and can be ignored:
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply.Check the release notes for a new release for any special upgrade procedures.
Once the CRDs have been upgraded, you can then upgrade the release chart.
When upgrading to version 5.6.0 or later, if you configured NGINX Service Mesh previously, remove any related values from the Helm chart before upgrading. These settings only applied to previous NGINX Service Mesh deployments. Users who never deployed service mesh are unaffected.
The command depends on if you installed the chart using the registry or from source.
To upgrade a release named my-release, use the following command:
helm upgrade my-release oci://ghcr.io/nginx/charts/nginx-ingress --version 2.9.0helm upgrade my-release .ExternalDNS v0.21.0 and later reads DNSEndpoint resources only from the upstream externaldns.k8s.io/v1alpha1 API group. By default, NGINX Ingress Controller writes DNSEndpoint resources to its own externaldns.nginx.org/v1 group. To use ExternalDNS v0.21.0 or later, switch NGINX Ingress Controller to the upstream group.
Use one of these settings to choose the group:
- Helm:
controller.externalDNSGroupVersion. For more information, see Helm chart parameters. - Command-line argument:
-external-dns-group-version.
NGINX Ingress Controller doesn’t ship the upstream DNSEndpoint CRD. Install it yourself, either with the ExternalDNS Helm chart or from the ExternalDNS repository:
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/external-dns/v0.21.0/config/crd/standard/dnsendpoints.externaldns.k8s.io.yamlIf NGINX Ingress Controller doesn’t manage any DNS records yet, install the CRD and set the upstream group. If you have existing records, follow the steps in Migrate in stages to keep every DNS record in place.
The role-based access control (RBAC) permissions depend on how you installed NGINX Ingress Controller:
- The Helm chart grants access only to the group you select.
- The manifest
deployments/rbac/rbac.yamlgrants access to both groups. You can change the argument without editing RBAC.
NGINX Ingress Controller doesn’t convert or delete existing DNSEndpoint resources. You control the cut-over, and the order of the steps protects your DNS records.
Install the upstream CRD and verify the new DNSEndpoint before ExternalDNS reads itIf you run ExternalDNS with
--policy=sync, don’t let ExternalDNS read theexternaldns.k8s.iogroup before NGINX Ingress Controller populates it. When the CRD exists but has no matching objects, ExternalDNS treats the empty result as valid. It then deletes every DNS record it owns.With
--policy=upsert-only, ExternalDNS doesn’t delete records, so the records stay in place.A missing CRD or missing RBAC is safer: ExternalDNS fails to start and doesn’t change any records. A
DNSEndpointthat exists but is empty or out of date is as dangerous as no object at all.
If the cluster doesn’t serve the group you select, NGINX Ingress Controller exits at startup with this error:
Cannot enable external-dns: API group/version "externaldns.k8s.io/v1alpha1" is not served by the cluster; install the DNSEndpoint CRD for itThe new pods go into CrashLoopBackOff. Your DNS records stay in place, because the old object remains and ExternalDNS still reads it. The effect on traffic depends on your rollout strategy:
- Deployment with the default
RollingUpdatestrategy: the old NGINX Ingress Controller pods keep serving traffic. - DaemonSet, or a Deployment with the
Recreatestrategy: the replaced pods are gone. The affected nodes have no running NGINX Ingress Controller until you install the CRD.
After you install the CRD, the new pods start without further action. NGINX Ingress Controller checks for the CRD only at startup. If someone removes the CRD later, NGINX Ingress Controller doesn’t detect it.
Migrate before you upgrade ExternalDNS to v0.21.0 or later. ExternalDNS v0.20.x and earlier reads either group, which makes a staged migration possible. Complete the check for each step before you start the next one.
| Step | Action | Check before you continue |
|---|---|---|
| 0 | Install the upstream DNSEndpoint CRD. An NGINX Ingress Controller upgrade doesn’t install it. |
kubectl get crd dnsendpoints.externaldns.k8s.io succeeds. |
| 1 | Upgrade NGINX Ingress Controller. Keep the group at the default, externaldns.nginx.org/v1. |
DNS records are unchanged. The externaldns.nginx.org/v1 object still exists and is current. |
| 2 | Set the Helm value controller.externalDNSGroupVersion: externaldns.k8s.io/v1alpha1, or the argument -external-dns-group-version=externaldns.k8s.io/v1alpha1. |
Don’t continue until the externaldns.k8s.io/v1alpha1 object exists. Its .spec.endpoints must match the targets and time-to-live (TTL) of the old object. The new NGINX Ingress Controller pods must be Ready. |
| 3 | Remove --crd-source-apiversion from your ExternalDNS v0.20.x configuration. The default of this argument is externaldns.k8s.io/v1alpha1, so ExternalDNS now reads the upstream group. |
status.observedGeneration matches metadata.generation on the new object. After one full sync interval, ExternalDNS logs All records are already up to date. |
| 4 | Upgrade ExternalDNS to v0.21.0 or later. | The same check as step 3. |
| 5 | Delete the old externaldns.nginx.org/v1 object. |
DNS records are unchanged. |
Steps 3 through 5 don’t change any DNS records. The old and new objects have the same name and namespace. ExternalDNS builds its ownership label from these values (crd/<NAMESPACE>/<NAME>). As a result, the TXT registry record stays the same, and ExternalDNS finds no changes to make.
Keep steps 2 and 3 short and separate:
- Do steps 2 and 3 close together. After step 2, NGINX Ingress Controller stops updating the old object, but ExternalDNS still reads it. ExternalDNS doesn’t publish VirtualServer changes that you make between the two steps.
- Don’t combine step 3 with step 4. ExternalDNS v0.21.0 and later ignores
--crd-source-apiversion, so step 4 also switches the group. If you make both changes at once and your records change, you can’t tell which change caused it.
To confirm that ExternalDNS read the new object, compare its observed generation with its generation:
kubectl get dnsendpoints.externaldns.k8s.io <VIRTUALSERVER_NAME> -o jsonpath='{.status.observedGeneration}/{.metadata.generation}'Replace <VIRTUALSERVER_NAME> with the name of your VirtualServer. The DNSEndpoint has the same name.
When the two values match, ExternalDNS read the resource. ExternalDNS sets status.observedGeneration when it reads the resource, before it sends records to your DNS provider. A match confirms that ExternalDNS read the resource, not that the DNS record is live.
Until you delete the old object in step 5, you can return to the starting state. Revert both settings together:
- Set
controller.externalDNSGroupVersion(or-external-dns-group-version) back toexternaldns.nginx.org/v1. - Add
--crd-source-apiversion=externaldns.nginx.org/v1back to your ExternalDNS configuration. If you already upgraded ExternalDNS, roll it back to v0.20.x. ExternalDNS v0.21.0 and later can’t read theexternaldns.nginx.orggroup.
If you revert only one setting, ExternalDNS reads a group that NGINX Ingress Controller doesn’t write to. This causes the same empty result that the warning in Understand the migration risks describes.
If you skip step 5, the old object isn’t left behind forever. NGINX Ingress Controller sets an owner reference from each DNSEndpoint to its VirtualServer. When you delete the VirtualServer, Kubernetes deletes the old object too.
NGINX Ingress Controller manages only the group you select. It never deletes objects in the other group. To delete the old object in step 5, use kubectl delete with your own credentials. With a Helm installation, the NGINX Ingress Controller service account loses its permissions on externaldns.nginx.org after step 2.
This upgrade path is intended for 3.x to 4.0.0 onlyThe instructions in this section are intended only for users upgrading from NGINX Ingress Controller 3.x to 4.0.0. Internal changes meant that backwards compability was not possible, requiring extra steps to upgrade.
This section provides step-by-step instructions for upgrading NGINX Ingress Controller from version v3.x to v4.0.0.
There are two necessary steps required
- Update the
apiVersionvalue of custom resources - Configure structured logging.
If you want to use NGINX Plus, you will also need to follow the Create a license Secret topic.
If you’re using Helm chart version v2.x, update your GlobalConfiguration, Policy, and TransportServer resources from apiVersion: k8s.nginx.org/v1alpha1 to apiVersion: k8s.nginx.org/v1 before upgrading to NGINX Ingress Controller 4.0.0.
If the Helm chart you have been using is v1.0.2 or earlier (NGINX Ingress Controller v3.3.2), upgrade to Helm chart v1.4.2 (NGINX Ingress Controller v3.7.2) before updating your GlobalConfiguration, Policy, and TransportServer resources.
The example below shows the change for a Policy resource: you must do the same for all GlobalConfiguration and TransportServer resources.
apiVersion: k8s.nginx.org/v1alpha1
kind: Policy
metadata:
name: rate-limit-policy
spec:
rateLimit:
rate: 1r/s
key: ${binary_remote_addr}
zoneSize: 10MapiVersion: k8s.nginx.org/v1
kind: Policy
metadata:
name: rate-limit-policy
spec:
rateLimit:
rate: 1r/s
key: ${binary_remote_addr}
zoneSize: 10MWarningIf a GlobalConfiguration, Policy or TransportServer resource is deployed withapiVersion: k8s.nginx.org/v1alpha1, it will be deleted during the upgrade process.
After you move the custom resources to v1, run the following kubectl commands before upgrading to v4.0.0 Custom Resource Definitions (CRDs) to avoid webhook errors caused by leftover v1alpha1 resources. For details, see GitHub issue #7010.
kubectl patch customresourcedefinitions transportservers.k8s.nginx.org --subresource='status' --type='merge' -p '{"status":{"storedVersions": ["v1"]}}'kubectl patch customresourcedefinitions globalconfigurations.k8s.nginx.org --subresource='status' --type='merge' -p '{"status":{"storedVersions": ["v1"]}}'To configure structured logging, you must update your log deployment arguments from an integer to a string. You can also choose different formats for the log output.
These options apply to NGINX Ingress Controller logs, and do not affect NGINX logs.
| Level arguments | Format arguments |
|---|---|
trace |
json |
debug |
text |
info |
glog |
warning |
|
error |
|
fatal |
The Helm value controller.logLevel is now a string instead of an integer.
To change the rendering of the log format, use the controller.logFormat key.
controller:
logLevel: info
logFormat: jsonThe command line argument -v has been replaced with -log-level, and takes a string instead of an integer. The argument -logtostderr has also been deprecated.
To change the rendering of the log format, use the -log-format argument.
args:
- -log-level=info
- -log-format=jsonIf you’re using NGINX Plus with NGINX Ingress Controller, you should read the Create a license Secret topic to set up your NGINX Plus license.
The topic also contains guidance for sending reports to NGINX Instance Manager, which is necessary for air-gapped environments.
Earlier versions required usage reporting through the cluster connector. This is no longer needed because it’s now built into NGINX Plus.
Starting in version 3.1.0, NGINX Ingress Controller uses updated Helm resource names, labels, and annotations to follow Helm best practices. See the changes.
When you upgrade with Helm from a version earlier than 3.1.0, some resources such as Deployment, DaemonSet, and Service are recreated. This causes downtime.
To reduce downtime, update all resources to use the new naming convention. The following steps help you do that.
The following steps apply to both 2.x and 3.0.x releases.
The steps you should follow depend on your Helm release name:
Use kubectl describe on deployment/daemonset to get the Selector value:
kubectl describe deployments -n <NAMESPACE>Replace <NAMESPACE> with the Kubernetes namespace where NGINX Ingress Controller is installed.
Copy the key=value under Selector, such as:
Selector: app=nginx-ingress-nginx-ingressCheck out the latest available tag using git checkout v5.8.0
Go to /kubernetes-ingress/charts/nginx-ingress
Update the selectorLabels: {} field in the values.yaml file located at /kubernetes-ingress/charts/nginx-ingress with the copied Selector value.
selectorLabels: {app: nginx-ingress-nginx-ingress}Run helm upgrade with following arguments set:
--set serviceNameOverride="nginx-ingress-nginx-ingress"
--set controller.name=""
--set fullnameOverride="nginx-ingress-nginx-ingress"It might look like this:
helm upgrade nginx-ingress oci://ghcr.io/nginx/charts/nginx-ingress --version 0.19.0 --set controller.kind=deployment/daemonset --set controller.nginxplus=false/true --set controller.image.pullPolicy=Always --set serviceNameOverride="nginx-ingress-nginx-ingress" --set controller.name="" --set fullnameOverride="nginx-ingress-nginx-ingress" -f values.yamlOnce the upgrade process has finished, use kubectl describe on the deployment to verify the change by reviewing its events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 9m11s deployment-controller Scaled up replica set nginx-ingress-nginx-ingress-<OLD_VERSION> to 1
Normal ScalingReplicaSet 101s deployment-controller Scaled up replica set nginx-ingress-nginx-ingress-<NEW_VERSION> to 1
Normal ScalingReplicaSet 98s deployment-controller Scaled down replica set nginx-ingress-nginx-ingress-<OLD_VERSION> to 0 from 1Replace <OLD_VERSION> with the version you’re upgrading from and <NEW_VERSION> with the version you’re upgrading to.
If you use custom NGINX configuration templates, review and apply any upstream template changes before upgrading. For more information, see Custom templates.
Upgrading to version 5.5 or laterIn version 5.5, the default server block was removed frommain-templateand is now generated byingress-template. You must update both templates before upgrading. If you had customizations inside the default server block in yourmain-template, the patch will remove them. Migrate those changes to youringress-templatemanually.
-
Clone the repository:
git clone https://github.com/nginx/kubernetes-ingress.git -
Generate a diff between the version you are upgrading from and the version you are upgrading to. For
main-templateandingress-template, useversion1. Forvirtualserver-templateandtransportserver-template, useversion2:shell git diff v<CURRENT_VERSION>..v<TARGET_VERSION> -- internal/configs/version1/<TEMPLATE_FILE> > upstream.patch git diff v<CURRENT_VERSION>..v<TARGET_VERSION> -- internal/configs/version2/<TEMPLATE_FILE> > upstream.patchReplace the placeholders as follows:
<CURRENT_VERSION>: the version you’re upgrading from<TARGET_VERSION>: the version you’re upgrading to<TEMPLATE_FILE>: the name of the template file
-
Extract your custom template from the ConfigMap to a local file. The valid key names are
main-template,ingress-template,virtualserver-template, andtransportserver-template:kubectl get configmap <NAME> -n nginx-ingress -o jsonpath='{.data.<TEMPLATE_KEY>}' > my-template.tmplReplace
<NAME>with the name of the resource and<TEMPLATE_KEY>with the key of the template in your ConfigMap. -
Apply the patch to your custom template:
patch my-template.tmpl < upstream.patchIf the patch fails to apply some changes, review the rejected changes in
my-template.tmpl.rejand apply them by manually. -
Update the template key with the patched content. If you manage your ConfigMap directly, update the relevant key in your manifest and run
kubectl apply -f <YOUR_CONFIGMAP.YAML>. If you use Helm, set the key undercontroller.config.entriesin your values file and runhelm upgrade.