Upgrade Guide

This guide explains how to upgrade the NGINX Management Suite modules, NGINX Agent, and NGINX Plus.

Overview

NGINX Management Suite is a family of management plane solutions that enable governance of the NGINX data plane to easily scale, secure, and monitor applications and APIs.

At the core of the NGINX Management Suite is Instance Manager. This core module lets you track, secure, and configure your NGINX OSS and NGINX Plus instances.

Instance Manager is available as a standalone module. Additionally, because Instance Manager is at the heart of NGINX Management Suite, it’s automatically installed whenever you install other modules, such as API Connectivity Manager.


Before You Begin

Before upgrading, it’s useful to have an upgrade plan. An upgrade plan helps determine what tasks you need to complete before and after installing new versions of the NGINX Management Suite modules and the NGINX Agent.

Review Required Documentation

Read the release notes for information about new features and changes, resolved issues, known issues, and supported upgrade paths.

Confirm the recommended system requirements and settings for the NGINX Management Suite module you’re upgrading:

Verify the Upgrade Path

Verify that your current version of Instance Manager can be upgraded to the target version. You may be able to upgrade directly to the target version, or you may need to upgrade to an intermediate version first.

You can find the supported upgrade paths in the release notes:

To see which version of an NGINX Management Suite module is installed, run the following commands:

  • Look up the installed version of Instance Manager:

    yum info nms-instance-manager
    
  • Look up the installed version of API Connectivity Manager:

    yum info nms-api-connectivity-manager
    
  • Look up the installed version of the Security Monitoring module:

    yum info nms-sm
    

  • Look up the installed version of Instance Manager:

    dpkg -s nms-instance-manager
    
  • Look up the installed version of API Connectivity Manager:

    dpkg -s nms-api-connectivity-manager
    
  • Look up the installed version of the Security Monitoring module:

    dpkg -s nms-sm
    

Schedule a Maintenance Window

Schedule a time in which you will perform the upgrade. Let users know when the upgrade will occur and how they might be affected.

Upgrading doesn’t take long, just under a minute to upgrade the Instance Manager server and as long to upgrade the NGINX Agent. How long the entire upgrade process takes depends on how many Agents you need to upgrade. Make sure to include time to verify and test the upgrade as well.

During an upgrade, you’ll need to stop the NGINX Agent. When the Agent is stopped, the following conditions occur:

  • you won’t be able to publish configuration changes,
  • metrics reporting will be queued and picked up once the Agent is running again,
  • client connections to Instance Manager may be disconnected, and
  • users may be logged out and need to reauthenticate after the upgrade.

Upgrade a Test Server

Upgrade Instance Manager in a test environment before upgrading your production server. Upgrading a test server can help identify issues that may slow down or block the upgrade of your production system.


Upgrade Instance Manager

Select the upgrade procedure that’s appropriate for your environment:

Upgrade Instance Manager from the NGINX Management Suite repo

Upgrade Instance Manager from NGINX Management Suite Repo

This section explains how to upgrade Instance Manager using a Linux package manager – Yum or Apt – to retrieve packages from a public repository. You’ll need to have Internet access to complete these steps.

If you don’t have access to the Internet, refer to the Upgrade in an Offline Environment section.

  1. To upgrade to the latest version of the Instance Manger, run the following command:

    sudo yum update -y nms-instance-manager
    

  1. To upgrade to the latest version of the Instance Manager, run the following command:

    sudo apt-get update
    sudo apt-get upgrade -y nms-instance-manager
    
  1. To enable the NGINX Management Suite services, take the following steps:

    1. Enable the NGINX Management Suite services:

      sudo systemctl enable nms
      sudo systemctl enable nms-core
      sudo systemctl enable nms-dpm
      sudo systemctl enable nms-ingestion
      sudo systemctl enable nms-integrations
      
    2. Start the NGINX Management Suite services:

      sudo systemctl start nms
      sudo systemctl start nms-core
      sudo systemctl start nms-dpm
      sudo systemctl start nms-ingestion
      sudo systemctl start nms-integrations
      

      To verify the NGINX Management Services are running, run the following command:

      ps aufx | grep nms
      

      NGINX Management Suite components started this way run by default as the non-root nms user inside the nms group, both of which are created during installation.

    3. To verify the NGINX Management Suite services are running, run the following command:

      ps aufx | grep nms
      
    Overview: NGINX Management Suite services

    The following table lists the services that are installed with NGINX Management Suite. The “NMS Platform” services are installed with Instance Manager, which is the foundation for the NGINX Management Suite platform.

    Service Module Description
    nms NMS Platform A pseudo service used to start the other nms-* services.
    nms-core NMS Platform The core service hosts the APIs for setting up and configuring the control plane and analyzing analytics information (metrics, events, and alerts).
    nms-dpm NMS Platform The data plane manager (DPM) service hosts the APIs for managing and configuring NGINX instances on the data plane. The DPM also monitors the state of data plane resources and generates reports and event messages.
    nms-ingestion NMS Platform The ingestion service collects metrics, security violations, and events from NGINX Agents that aren’t sent to the data plane manager. These metrics can be forwarded to external datastores.
    nms‑integrations NMS Platform Note: This service is for future integrations; no user management or configuration is needed at this time.
    nms-acm API Connectivity Manager The API Connectivity Manager service.

  2. Restart the NGINX web server:

    sudo systemctl restart nginx
    

After you’ve successfully upgraded Instance Manager, take the following steps to restart the services and optionally reapply SELinux Policies:

  1. Reload NGINX:

    sudo nginx -s reload
    
  2. (Optional) Reapply the SELinux policy if it’s enabled:

    sudo semodule -n -i /usr/share/selinux/packages/nms.pp
    sudo /usr/sbin/load_policy
    sudo restorecon -F -R /usr/bin/nms-core
    sudo restorecon -F -R /usr/bin/nms-dpm
    sudo restorecon -F -R /usr/bin/nms-ingestion
    sudo restorecon -F -R /usr/lib/systemd/system/nms.service
    sudo restorecon -F -R /usr/lib/systemd/system/nms-core.service
    sudo restorecon -F -R /usr/lib/systemd/system/nms-dpm.service
    sudo restorecon -F -R /usr/lib/systemd/system/nms-ingestion.service
    sudo restorecon -F -R /var/lib/nms/modules/manager.json
    sudo restorecon -F -R /var/lib/nms/modules.json
    sudo restorecon -F -R /var/lib/nms/streaming
    sudo restorecon -F -R /var/lib/nms
    sudo restorecon -F -R /var/lib/nms/dqlite
    sudo restorecon -F -R /var/run/nms
    sudo restorecon -F -R /var/lib/nms/modules
    sudo restorecon -F -R /var/log/nms
    
  3. Restart the Instance Manager services:

    sudo systemctl restart nms
    
  4. Make sure to upgrade the NGINX Agent as well.

Upgrade Instance Manager in an offline environment

Upgrade Instance Manager Offline

To upgrade Instance Manager in an offline environment, take the following steps:

  1. Log in to the MyF5 Customer Portal and download the Instance Manager package files, or use the package provided by your NGINX Sales Team.

  2. Upgrade the Instance Manager package:

    sudo yum -y --nogpgcheck update /home/user/nms-instance-manager_<version>.x86_64.rpm
    

  1. Log in to the MyF5 Customer Portal and download the Instance Manager package files, or use the package provided by your NGINX Sales Team.

  2. Upgrade the Instance Manager package:

    sudo apt-get -y install -f /home/user/nms-instance-manager_<version>_amd64.deb
    
  1. To enable the NGINX Management Suite services, take the following steps:

    1. Enable the NGINX Management Suite services:

      sudo systemctl enable nms
      sudo systemctl enable nms-core
      sudo systemctl enable nms-dpm
      sudo systemctl enable nms-ingestion
      sudo systemctl enable nms-integrations
      
    2. Start the NGINX Management Suite services:

      sudo systemctl start nms
      sudo systemctl start nms-core
      sudo systemctl start nms-dpm
      sudo systemctl start nms-ingestion
      sudo systemctl start nms-integrations
      

      To verify the NGINX Management Services are running, run the following command:

      ps aufx | grep nms
      

      NGINX Management Suite components started this way run by default as the non-root nms user inside the nms group, both of which are created during installation.

    3. To verify the NGINX Management Suite services are running, run the following command:

      ps aufx | grep nms
      
    Overview: NGINX Management Suite services

    The following table lists the services that are installed with NGINX Management Suite. The “NMS Platform” services are installed with Instance Manager, which is the foundation for the NGINX Management Suite platform.

    Service Module Description
    nms NMS Platform A pseudo service used to start the other nms-* services.
    nms-core NMS Platform The core service hosts the APIs for setting up and configuring the control plane and analyzing analytics information (metrics, events, and alerts).
    nms-dpm NMS Platform The data plane manager (DPM) service hosts the APIs for managing and configuring NGINX instances on the data plane. The DPM also monitors the state of data plane resources and generates reports and event messages.
    nms-ingestion NMS Platform The ingestion service collects metrics, security violations, and events from NGINX Agents that aren’t sent to the data plane manager. These metrics can be forwarded to external datastores.
    nms‑integrations NMS Platform Note: This service is for future integrations; no user management or configuration is needed at this time.
    nms-acm API Connectivity Manager The API Connectivity Manager service.

  2. Restart the NGINX web server:

    sudo systemctl restart nginx
    

After you’ve successfully upgraded Instance Manager, take the following steps to restart the services and optionally reapply SELinux Policies:

  1. Reload NGINX:

    sudo nginx -s reload
    
  2. (Optional) Reapply the SELinux policy if it’s enabled:

    sudo semodule -n -i /usr/share/selinux/packages/nms.pp
    sudo /usr/sbin/load_policy
    sudo restorecon -F -R /usr/bin/nms-core
    sudo restorecon -F -R /usr/bin/nms-dpm
    sudo restorecon -F -R /usr/bin/nms-ingestion
    sudo restorecon -F -R /usr/lib/systemd/system/nms.service
    sudo restorecon -F -R /usr/lib/systemd/system/nms-core.service
    sudo restorecon -F -R /usr/lib/systemd/system/nms-dpm.service
    sudo restorecon -F -R /usr/lib/systemd/system/nms-ingestion.service
    sudo restorecon -F -R /var/lib/nms/modules/manager.json
    sudo restorecon -F -R /var/lib/nms/modules.json
    sudo restorecon -F -R /var/lib/nms/streaming
    sudo restorecon -F -R /var/lib/nms
    sudo restorecon -F -R /var/lib/nms/dqlite
    sudo restorecon -F -R /var/run/nms
    sudo restorecon -F -R /var/lib/nms/modules
    sudo restorecon -F -R /var/log/nms
    
  3. Restart the Instance Manager services:

    sudo systemctl restart nms
    
  4. Make sure to upgrade the NGINX Agent as well.

Upgrade Instance Manager from a helm chart

Complete the following steps to upgrade Instance Manager from a helm chart.

Helm Chart Prerequisites

To install or upgrade Instance Manager using a helm chart, you need the following:

  • Kubernetes 1.21.3+ (linux/amd64) with client access to the Kubernetes API server

  • An externally-accessible private Docker registry to push the container images to

  • The following binaries:

    • Docker 20.10+ (linux/amd64)
    • helm v3.8.0+
    • kubectl v1.21.3+
    • openssl v1.1.1+
    • tar v1.20+

Download Upgradable Helm Bundle

Important:
Before you begin, make sure to verify the supported upgrade paths for the version of Instance Manager you’re upgrading to.

Take the following steps to download the helm chart bundle to your virtual machine:

  1. On the MyF5 website, select Resources > NGINX Downloads.

  2. In the NGINX products list, select Instance Manager.

  3. Select the following download options:

    • Product version:
    • Linux distribution
    • Architecture
  4. Download the nms-helm-<version>.tar.gz files.

Extract Helm Bundle

The nms-helm-<version>.tar.gz file from Step 1: Download Helm Bundle includes several Docker container images and the helm package tarball. To extract these files, take the following steps:

  1. Make a directory to extract the package into:

    mkdir -p <directory name>
    
  2. Extract the package into the target directory:

    tar -xzf nms-helm-<version>.tar.gz -C <directory name>
    

    The extracted files include the following Docker images and helm package:

    Docker images

    • nms-apigw-img.tar.gz
    • nms-core-img.tar.gz
    • nms-dpm-img.tar.gz
    • nms-ingestion-img.tar.gz
    • nms-integrations-img.tar.gz

    Helm package

    • nms-hybrid-<version>.tgz

Load Docker Images

Take the following steps to extract and load the Docker container images:

  1. Change to the directory where you extracted the Docker images:

    cd <directory name>
    
  2. Load the Docker images:

    docker load -i nms-apigw-img.tar.gz
    docker load -i nms-core-img.tar.gz
    docker load -i nms-ingestion-img.tar.gz
    docker load -i nms-dpm-img.tar.gz
    docker load -i nms-integrations-img.tar.gz
    

    The output looks similar to the following:

    $ docker load -i nms-apigw-img.tar.gz
    1b5933fe4b5: Loading layer [==================================================>]  5.796MB/5.796MB
    fbe0fc9bcf95: Loading layer [==================================================>]  17.86MB/17.86MB
    ...
    112ae1f604e0: Loading layer [==================================================>]   67.8MB/67.8MB
    4b6a693b90f4: Loading layer [==================================================>]  3.072kB/3.072kB
    Loaded image: nms-apigw:2.1.0
    
  3. For the output of each docker load command, note the loaded image’s name and tag. For example, in the output directly above, nms-apigw is the image name and 2.1.0 is the tag. You’ll need to reference these images and tags in the next section when pushing the images to your private registry. The tag 2.1.0 could be different depending on the product version you downloaded from MyF5.

Push Images to Private Registry

Before you begin: To complete this step, you need an externally-accessible private Docker registry to push the container images to.

After loading the Docker images, you can now tag and push the images to your private Docker registry. Replace <my-docker-registry> in the examples below with the path to your private Docker registry.

  1. Log in to your private registry:

    docker login <my-docker-registry>
    
  2. Tag the images with the values you noted in Step 3: Load Docker Images above. In this example, the images are tagged with version 2.1.0.

    docker tag nms-apigw:2.1.0 <my-docker-registry>/nms-apigw:2.1.0
    docker tag nms-core:2.1.0 <my-docker-registry>/nms-core:2.1.0
    docker tag nms-dpm:2.1.0 <my-docker-registry>/nms-dpm:2.1.0
    docker tag nms-ingestion:2.1.0 <my-docker-registry>/nms-ingestion:2.1.0
    docker tag nms-integrations:2.1.0 <my-docker-registry>/nms-integrations:2.1.0
    
  3. Push the images to your private registry:

    docker push <my-docker-registry>/nms-apigw:2.1.0
    docker push <my-docker-registry>/nms-core:2.1.0
    docker push <my-docker-registry>/nms-dpm:2.1.0
    docker push <my-docker-registry>/nms-ingestion:2.1.0
    docker push <my-docker-registry>/nms-integrations:2.1.0
    

Extract Helm Chart

Run the following command to extract the helm chart:

tar -xzf nms-hybrid-<version>.tgz

The package contents are extracted to a directory called nms-hybrid.

Configure Helm Chart

Important:
By default, a helm upgrade overwrites the previous installation’s configuration.

After you extract the helm chart, you can edit the bundle to customize the deployment settings for your environment. To configure the helm chart, see the following sections:

Image Repositories and Tags

Configure the helm chart to pull from your private Docker registry:

  1. Open values.yaml file for editing.
  2. Locate imagePullSecrets and add the credentials for your private Docker registry. For instructions on creating a secret, see the Kubernetes documentation Pull an Image from a Private Registry.
  3. Update the image:repository and image:tag values for the apigw, core, dpm, ingestion, and integrations services with the values you used when completing the Push Images to Private Registry steps.
  4. Save and close the values.yaml file.

Alternatively, you can use the helm upgrade --set command to set values from the command line.

Config Files Customizations

If you made changes to content in the /files or /generated folder in your current release, merge those changes into the newly extracted helm bundle.

For example, if you edited /generated/nginx.conf to enable OIDC in your current release, copy this and/or any referenced file to same location in the newly extracted helm bundle to continue using those changes.

Annotate Current Release Kubernetes Secrets

Get the helm release name for the Instance Manager app that’s installed on your Kubernetes.

For example, in the following command output, the release name for chart nms-hybrid-2.5.0 is nim, and the namespace is nms. These values will be used to apply annotations.

> helm list -A

NAME	NAMESPACE	REVISION	UPDATED                             	STATUS  	CHART           	APP VERSION
nim 	nms      	1       	2022-11-09 12:14:01.774871 -0800 PST	deployed	nms-hybrid-2.5.0	2.5.0

Apply following annotations with your current release and namespace values (as shown above),

KIND=secret
NAME=nms-internal-certs
RELEASE=nim
NAMESPACE=nms
kubectl annotate --overwrite $KIND $NAME meta.helm.sh/release-name=$RELEASE -n nms
kubectl annotate --overwrite $KIND $NAME meta.helm.sh/release-namespace=$NAMESPACE -n nms
kubectl label $KIND $NAME app.kubernetes.io/managed-by=Helm -n nms

Set Kubernetes Update Strategy To Recreate

As of version 2.6.0 and prior, the helm chart is configured to use the “Rollover” Kubernetes upgrade strategy by default.

Run the following command to patch deployments to use the “Recreate” upgrade strategy:

kubectl -n nms patch deployment core -p '{"spec": {"strategy": {"type": "Recreate", "rollingUpdate": null}}}'
kubectl -n nms patch deployment dpm -p '{"spec": {"strategy": {"type": "Recreate", "rollingUpdate": null}}}'
kubectl -n nms patch deployment integrations -p '{"spec": {"strategy": {"type": "Recreate", "rollingUpdate": null}}}'

Run Upgrade

Run the helm upgrade command to upgrade your current NMS Instance Manager on kubernetes to a new version.

The following example upgrades NMS Instance Manager release “nim” in the namespace nms to new extracted helm chart:

helm upgrade -n nms nim ./nms-hybrid

Validate Upgrade

To check the status of the release, run the following command. In this example, nms is the namespace and nim is the application name specified during installation.

helm -n nms status nim

If the deployment was successful, the status should be STATUS: deployed.


Upgrade API Connectivity Manager

Select the upgrade procedure that’s appropriate for your environment:

Upgrade API Connectivity Manager from the NGINX Management Suite repo

Upgrade API Connectivity Manager from NGINX Management Suite Repo

Upgrade API Connectivity Manager from the NGINX Management Suite Repo:

This section explains how to upgrade API Connectivity Manager using a Linux package manager – Yum or Apt – to retrieve packages from a public repository. You’ll need to have Internet access to complete these steps.

  1. To upgrade to the latest version of API Connectivity Manager, run the following command:

    sudo yum update -y nms-api-connectivity-manager
    

  1. To upgrade to the latest version of API Connectivity Manager, run the following commands:

    sudo apt-get update
    sudo apt-get upgrade -y nms-api-connectivity-manager
    
  1. To enable and start the NGINX Management Suite services for API Connectivity Manager, take the following steps:

    1. Enable the NGINX Management Suite services:

      sudo systemctl enable nms
      sudo systemctl enable nms-core
      sudo systemctl enable nms-dpm
      sudo systemctl enable nms-ingestion
      sudo systemctl enable nms-integrations
      sudo systemctl enable nms-acm
      
    2. Start the NGINX Management Suite services:

      sudo systemctl start nms
      

      NGINX Management Suite components started this way run by default as the non-root nms user inside the nms group, both of which are created during installation.

    3. To verify the NGINX Management Suite services are running, run the following command:

      ps aufx | grep nms
      
  2. Restart the NGINX web server:

    sudo systemctl restart nginx
    
Upgrade Developer Portal from the NGINX Management Suite repo

Upgrade the Developer Portal from NGINX Management Suite Repo

This section explains how to upgrade the Developer Portal using a Linux package manager – Yum or Apt – to retrieve packages from a public repository. You’ll need to have Internet access to complete these steps.

To upgrade the Developer Portal, take the following steps:

  1. To install the latest version of the Developer Portal, run the following command:

    sudo yum update -y nginx-devportal nginx-devportal-ui
    
  2. Enable the following Developer Portal service:

    sudo systemctl enable nginx-devportal.service
    
  3. Restart the Developer Portal service:

    sudo systemctl restart nginx-devportal.service
    

  1. To install the latest version of the Developer Portal, run the following commands:

    sudo apt-get update
    sudo apt-get upgrade -y nginx-devportal nginx-devportal-ui
    
  2. Enable the following Developer Portal service:

    sudo systemctl enable nginx-devportal.service
    
  3. Restart the Developer Portal service:

    sudo systemctl restart nginx-devportal.service
    
Upgrade API Connectivity Manager in an offline environment

Upgrade API Connectivity Manager Offline

To upgrade API Connectivity Manager in an offline environment, take the following steps:

  1. Log in to the MyF5 Customer Portal and download the API Connectivity Manager package files, or use the package provided by your NGINX Sales Team.

  2. Upgrade the API Connectivity Manager package:

    sudo yum -y --nogpgcheck update /home/user/nms-api-connectivity-manager_<version>.x86_64.rpm
    

  1. Log in to the MyF5 Customer Portal and download the API Connectivity Manager package files, or use the package provided by your NGINX Sales Team.

  2. Upgrade the API Connectivity Manager package:

    sudo apt-get -y install -f /home/user/nms-api-connectivity-manager_<version>_amd64.deb
    
  1. To enable and start the NGINX Management Suite services for API Connectivity Manager, take the following steps:

    1. Enable the NGINX Management Suite services:

      sudo systemctl enable nms
      sudo systemctl enable nms-core
      sudo systemctl enable nms-dpm
      sudo systemctl enable nms-ingestion
      sudo systemctl enable nms-integrations
      sudo systemctl enable nms-acm
      
    2. Start the NGINX Management Suite services:

      sudo systemctl start nms
      

      NGINX Management Suite components started this way run by default as the non-root nms user inside the nms group, both of which are created during installation.

    3. To verify the NGINX Management Suite services are running, run the following command:

      ps aufx | grep nms
      
  2. Restart the NGINX web server:

    sudo systemctl restart nginx  
    
Upgrade DevPortal in an offline environment

Upgrade the Developer Portal Offline

To upgrade the Developer Portal in an offline environment, take the following steps:

  1. Log in to the MyF5 Customer Portal and download the Developer Portal package files, or use the package provided by your NGINX Sales Team.

  2. Upgrade the Developer Portal packages:

    sudo yum -y --nogpgcheck update /home/user/nginx-devportal_<version>.x86_64.rpm
    sudo yum -y --nogpgcheck update /home/user/nginx-devportal-ui_<version>.x86_64.rpm
    
  3. Enable the following Developer Portal service:

    sudo systemctl enable nginx-devportal.service
    
  4. Restart the Developer Portal service:

    sudo systemctl restart nginx-devportal.service
    

  1. Log in to the MyF5 Customer Portal and download the Developer Portal package files, or use the package provided by your NGINX Sales Team.

  2. Upgrade the Developer Portal packages:

    sudo apt-get -y install -f /home/user/nginx-devportal_<version>_amd64.deb
    sudo apt-get -y install -f /home/user/nginx-devportal-ui_<version>_amd64.deb
    
  3. Enable the following Developer Portal service:

    sudo systemctl enable nginx-devportal.service
    
  4. Restart the Developer Portal service:

    sudo systemctl restart nginx-devportal.service
    

Upgrade the NGINX Agent

Important:
You should upgrade the NGINX Agent whenever you upgrade Instance Manager to keep the versions in sync. A failure to do so might cause unexpected issues.

To upgrade the NGINX Agent, take the following steps:

  1. Open an SSH connection to the server where you’ve installed the NGINX Agent and log in.

  2. Make a backup copy of the following locations to ensure that you can successfully recover if the upgrade has issues:

    • /etc/nginx-agent
    • config_dirs values for any configuration specified in /etc/nginx-agent/nginx-agent.conf
  3. Stop the NGINX Agent:

    sudo systemctl stop nginx-agent
    
  4. Install the updated version of the NGINX Agent:

    curl https://<NGINX-INSTANCE-MANAGER-FQDN>/install/nginx-agent | sudo su -
    
  5. Start the NGINX Agent:

    sudo systemctl start nginx-agent
    

Upgrade NGINX Plus

We encourage you to upgrade NGINX Plus to take advantage of the latest improvements and features.

  1. Follow the steps to upgrade NGINX Plus for your distribution.

  2. After upgrading NGINX Plus, restart the NGINX Agent:

    sudo systemctl restart nginx-agent
    

Troubleshooting

If you experience problems with the upgrade, follow the steps to create a support package. The support package script packages system and service information into a tar archive for troubleshooting and debugging purposes.

If you need help with troubleshooting, contact NGINX Customer Support.

You can search for Instance Manager troubleshooting guides and knowledge base articles on the AskF5 site.