End of Sale Notice:

Commercial support for NGINX Service Mesh is available to customers who currently have active NGINX Microservices Bundle subscriptions. F5 NGINX announced the End of Sale (EoS) for the NGINX Microservices Bundles as of July 1, 2023.

See our End of Sale announcement for more details.

Deploy with NGINX Ingress Controller

This topic describes how to install and use F5 NGINX Ingress Controller with NGINX Service Mesh

Overview

You can deploy NGINX Ingress Controller for Kubernetes with NGINX Service Mesh to control both ingress and egress traffic.

Important:

NGINX Ingress Controller can be used for free with NGINX Open Source. Paying customers have access to NGINX Ingress Controller with NGINX Plus. To deploy NGINX Ingress Controller with NGINX Service Mesh, you must use either:

  • Open Source NGINX Ingress Controller version 3.0+
  • NGINX Plus version of NGINX Ingress Controller

Visit the NGINX Ingress Controller product page for more information.

Supported Versions

The supported NGINX Plus Ingress Controller versions for each release are listed in the technical specifications doc.

The documentation for the latest stable release of NGINX Ingress Controller is available at docs.nginx.com/nginx-ingress-controller. For version specific documentation, deployment configs, and configuration examples, select the tag corresponding to your desired version in GitHub.

Secure Communication Between NGINX Ingress Controller and NGINX Service Mesh

The NGINX Ingress Controller can participate in the mTLS cert exchange with services in the mesh without being injected with the sidecar proxy. The SPIRE server - the certificate authority of the mesh - issues certs and keys for NGINX Ingress Controller and pushes them to the SPIRE agents running on each node in the cluster. NGINX Ingress Controller fetches these certs and keys from the SPIRE agent via a unix socket and uses them to communicate with services in the mesh.

Cert Rotation with NGINX Ingress Controller

The ttl of the SVID certificates issued by SPIRE is set to 1hr by default. You can change this when deploying the mesh; refer to the nginx-meshctl documentation for more information.

When using NGINX Ingress Controller with mTLS enabled, it is best practice to keep the ttl at 1 hour or greater.

Install NGINX Ingress Controller with mTLS enabled

To configure NGINX Ingress Controller to communicate with mesh workloads over mTLS you need to make a modification to the Ingress Controller’s Pod spec. This section describes each modification that is required, but if you’d like to jump to installation, go to the Install with Manifests or Install with Helm sections.

  1. Add NGINX Service Mesh label

    One or both of the following labels must be added to the Ingress Controller’s Pod spec, depending on your use case:

    labels:
  nsm.nginx.com/enable-ingress: "true"
  ...

    labels:
  nsm.nginx.com/enable-egress: "true"
  ...

    These labels tell NGINX Service Mesh to mutate the Ingress Controller Pod with the proper configuration in order to properly integrate with the mesh.

Note:

All communication between NGINX Ingress Controller and the upstream Services occurs over mTLS, using the certificates and keys generated by the SPIRE server. Therefore, NGINX Ingress Controller can only route traffic to Services in the mesh that have an mtls-mode of permissive or strict. In cases where you need to route traffic to both mTLS and non-mTLS Services, you may need another Ingress Controller that does not participate in the mTLS fabric.

Refer to the NGINX Ingress Controller’s Running Multiple Ingress Controllers guide for instructions on how to configure multiple Ingress Controllers.

If you would like to enable egress traffic, refer to the Enable Egress section of this guide.

Install with Manifests

Before installing NGINX Ingress Controller, you must install NGINX Service Mesh with an mTLS mode of permissive, or strict. NGINX Ingress Controller will try to fetch certs from the SPIRE agent on startup. If it cannot reach the SPIRE agent, startup will fail, and NGINX Ingress Controller will go into CrashLoopBackoff state. The state will resolve once NGINX Ingress Controller connects to the SPIRE agent. For instructions on how to install NGINX Service Mesh, see the Installation guide.

Note:
Before continuing, check the NGINX Ingress Controller supported versions section and make sure you are working off the correct release tag for all NGINX Ingress Controller instructions.

NGINX OSS Ingress Controller

  1. Build or Pull the NGINX OSS Ingress Controller image:
  2. Set up Kubernetes Resources for NGINX Ingress Controller using Kubernetes manifests:
  3. Create the NGINX Ingress Controller as a Deployment or DaemonSet in Kubernetes using one of the following example manifests:

NGINX Plus Ingress Controller

  1. Build or Pull the NGINX Plus Ingress Controller image:
  2. Set up Kubernetes Resources for NGINX Plus Ingress Controller using Kubernetes manifests:
  3. Create the NGINX Plus Ingress Controller as a Deployment or DaemonSet in Kubernetes using one of the following example manifests:

Install with Helm

Before installing NGINX Ingress Controller, you must install NGINX Service Mesh with an mTLS mode of permissive, or strict. NGINX Ingress Controller will try to fetch certs from the SPIRE agent on startup. If it cannot reach the SPIRE agent, startup will fail, and NGINX Ingress Controller will go into CrashLoopBackoff state. The state will resolve once NGINX Ingress Controller connects to the SPIRE agent. For instructions on how to install NGINX Service Mesh, see the Installation guide.

Note:
NGINX Plus Ingress Controller v2.2+ or NGINX OSS Ingress Controller v3.0+ is required to deploy via Helm and integrate with NGINX Service Mesh.

Follow the instructions to install the NGINX Ingress Controller with Helm. Set the nginxServiceMesh.enable parameter to true.

Note:
This will configure NGINX Ingress Controller to route ingress traffic to NGINX Service Mesh workloads. If you would like to enable egress traffic, refer to the Enable Egress section of this guide.

The values-nsm.yaml file contains all the configuration parameters that are relevant for integration with NGINX Service Mesh. You can use this file if you are installing NGINX Ingress Controller via chart sources.

Expose your applications

With mTLS enabled, you can use Kubernetes Ingress, VirtualServer, and VirtualServerRoutes resources to configure load balancing for HTTP and gRPC applications. TCP load balancing via TransportServer resources is not supported.

Note:
The NGINX Ingress Controller’s custom resource TransportServer and the SMI Spec’s custom resource TrafficSplit share the same Kubernetes short name ts. To avoid conflicts, use the full names transportserver(s) and trafficsplit(s) when managing these resources with kubectl.

To learn how to expose your applications using NGINX Ingress Controller, refer to the Expose an Application with NGINX Ingress Controller tutorial.

Enable Egress

You can configure NGINX Ingress Controller to act as the egress endpoint of the mesh, enabling your meshed services to communicate securely with external, non-meshed services.

Note:
Multiple endpoints for a single egress deployment are supported, but multiple egress deployments are not supported.

Enable with Manifests

If you are installing NGINX Ingress Controller with manifests follow the Install with Manifests instructions and make the following change to the NGINX Ingress Controller Pod spec:

  • Add the following label to the NGINX Ingress Controller Pod spec:

    labels:
  nsm.nginx.com/enable-egress: "true"
  ...

    This label prevents automatic injection of the sidecar proxy and configures the NGINX Ingress Controller as the egress endpoint of the mesh.

    This will create a virtual server block in NGINX Ingress Controller that terminates TLS connections using the SPIFFE certs fetched from the SPIRE agent.

Enable with Helm

Note:
NGINX Plus Ingress Controller v2.2+ or NGINX OSS Ingress Controller v3.0+ is required to deploy via Helm and integrate with NGINX Service Mesh.

If you are installing NGINX Ingress Controller with Helm, follow the Install with Helm instructions and set nginxServiceMesh.enableEgress to true.

Allow Pods to route egress traffic through NGINX Ingress Controller

If egress is enabled you can configure Pods to route all egress traffic - requests to non-meshed services - through NGINX Ingress Controller. This feature can be enabled by adding the following annotation to the Pod spec of an application Pod:

config.nsm.nginx.com/default-egress-allowed: "true"

This annotation can be removed or changed after deployment and the egress behavior of the Pod will be updated accordingly.

Create internal routes for non-meshed services

Internal routes represent a route from NGINX Ingress Controller to a non-meshed service. This route is called “internal” because it is only accessible from a Pod in the mesh and is not accessible from the public internet.

Caution:
If you deploy NGINX Ingress Controller without mTLS enabled, the internal routes could be accessible from the public internet. We do not recommend using the egress feature with a plaintext deployment of NGINX Ingress Controller.

To generate an internal route, create an Ingress resource or VirtualServer resource using the information for your non-meshed service.

Add the following configuration, depending on the type of resource you created:

  • For an Ingress resource, add the following annotation to the resource definition:
nsm.nginx.com/internal-route: "true"
  • For a VirtualServer resource, add the following field to the custom resource definition:
spec:
 internalRoute: true

If your non-meshed service is external to Kubernetes, follow the ExternalName services example.

Note:
The nsm.nginx.com/internal-route: "true" Ingress annotation or internalRoute: true VirtualServer field is still required for routing to external endpoints.

The NGINX Ingress Controller egress tutorial provides instructions for creating internal routes for non-meshed services.

Enable Ingress and Egress Traffic

There are a couple ways to enable both ingress and egress traffic using the NGINX Ingress Controller. You can either allow both ingress and egress traffic through the same NGINX Ingress Controller, or deploy two NGINX Ingress Controllers: one for handling ingress traffic only and the other for handling egress traffic.

For the single deployment option, follow the installation instructions and the instructions to Enable Egress. If you would like to configure two Ingress Controllers to keep ingress and egress traffic separate you can leverage Ingress Classes.

Enable UDP Traffic

By default, NGINX Ingress Controller only routes TCP traffic. You can configure it to route UDP traffic by making the following changes to the NGINX Ingress Controller before deploying:

  • Enable GlobalConfiguration resources for NGINX Ingress Controller by following the setup defined in the GlobalConfiguration Resource documentation.

    This allows you to define global configuration parameters for the NGINX Ingress Controller, and create a UDP listener to route ingress UDP traffic to your backend applications.

Important:
mTLS does not affect UDP communication, as mTLS in NGINX Service Mesh applies only to TCP traffic at this time.

Create a GlobalConfiguration Resource

To allow UDP traffic to be routed to your Kubernetes applications, create a UDP listener in NGINX Ingress Controller. This can be done via a GlobalConfiguration Resource.

To create a GlobalConfiguration resource, see the NGINX Ingress Controller documentation to create a listener with protocol UDP.

Ingress UDP Traffic

You can pass and load balance UDP traffic by using a TransportServer resource. This will link the UDP listener defined in the Create a GlobalConfiguration Resource step with an upstream associated with your designated backend UDP application.

To create a TransportServer resource, follow the steps outlined in the TransportServer NGINX Ingress Controller guide and link the UDP listener with the name and port of your backend service.

To learn how to expose a UDP application using NGINX Ingress Controller, see the Expose a UDP Application with NGINX Ingress Controller tutorial.

Plaintext configuration

Deploy NGINX Service Mesh with mtls-mode set to off and follow the instructions to deploy NGINX Ingress Controller.

Add the enable-ingress and/or the enable-egress label shown below to the NGINX Ingress Controller Pod spec:

nsm.nginx.com/enable-ingress: "true"
nsm.nginx.com/enable-egress: "true"
Caution:
All communication between NGINX Ingress Controller and the services in the mesh will be over plaintext! We do not recommend using the egress feature with a plaintext deployment of NGINX Ingress Controller, it is possible that internal routes could be accessible from the public internet. We highly recommend installing NGINX Ingress Controller with mTLS enabled.

NGINX Ingress Controller Metrics

To enable metrics collection for the NGINX Ingress Controller, take the following steps:

  1. Run the NGINX Ingress Controller with the -enable-prometheus-metrics command line argument. The NGINX Ingress Controller exposes NGINX metrics in Prometheus format via the /metrics path on port 9113. This port is customizable via the -prometheus-metrics-listen-port command-line argument; consult the Command Line Arguments section of the NGINX Ingress Controller docs for more information on available command line arguments.
Note:
If using the NGINX Plus Ingress Controller, add this additional flag to enable latency metrics: -enable-latency-metrics

  1. Add the following Prometheus annotations NGINX Ingress Controller Pod spec:

    prometheus.io/scrape: "true"
prometheus.io/port: "<prometheus-metrics-listen-port>"

  2. Add the resource name as a label to the NGINX Ingress Controller Pod spec:

    • For Deployment:

      nsm.nginx.com/deployment: <name of NGINX Ingress Controller Deployment>

    • For DaemonSet:

      nsm.nginx.com/daemonset: <name of NGINX Ingress Controller DaemonSet>

    This allows metrics scraped from NGINX Ingress Controller Pods to be associated with the resource that created the Pods.

View the metrics in Prometheus

The NGINX Service Mesh uses the Pod’s container name setting to identify the NGINX Ingress Controller metrics that should be consumed by the Prometheus server.

Add the applicable nginx-ingress scrape config to your Prometheus configuration and consult Monitoring and Tracing for installation instructions.

Available metrics

For a list of the NGINX Ingress Controller metrics, consult the Available Metrics section of the NGINX Ingress Controller docs.

Note:
The NGINX Plus metrics exported by the NGINX Plus Ingress Controller are renamed from nginx_ingress_controller_<metric-name> to nginxplus_<metric-name> to be consistent with the metrics exported by NGINX Service Mesh sidecars. For example, nginx_ingress_controller_upstream_server_response_latency_ms_count is renamed to nginxplus_upstream_server_response_latency_ms_count. The Ingress Controller specific metrics, such as nginx_ingress_controller_nginx_reloads_total, are not renamed.

For more information on metrics, a list of Prometheus labels, and examples of querying and filtering, see the Prometheus Metrics doc.

To view the metrics, use port-forwarding:

kubectl port-forward -n nginx-mesh svc/prometheus 9090

Monitor your application in Grafana

NGINX Service Mesh provides a custom dashboard that you can import into your Grafana deployment to monitor your application. To import and view the dashboard, port-forward your Grafana service:

kubectl port-forward -n <grafana-namespace> svc/grafana 3000

Then you can navigate your browser to localhost:3000 to view Grafana.

Here is a view of the provided “NGINX Mesh Top” dashboard:

How NGINX Ingress Controller Integrates with NGINX Service Mesh

Mutating Webhook

NGINX Service Mesh version v1.7+ provides a mutating webhook that detects and configures instances of NGINX Ingress Controller.

Pod Spec Changes

  1. NGINX Service Mesh mounts and configures the SPIRE agent socket based on environment

    The SPIRE agent socket needs to be mounted to the Ingress Controller Pod so the Ingress Controller can fetch its certificates and keys from the SPIRE agent. This allows the Ingress Controller to authenticate with workloads in the mesh. For more information on how SPIRE distributes certificates see the SPIRE section in the architecture doc.

  • Kubernetes

    The following `hostPath` is added as a volume to the Ingress Controller's Pod spec:

  ```yaml
  volumes:
  - hostPath:
      path: /run/spire/sockets
      type: DirectoryOrCreate
    name: spire-agent-socket
  ```

NGINX Service Mesh also mounts the socket to the Ingress Controller's container spec:

  ```yaml
  volumeMounts:
  - mountPath: /run/spire/sockets
    name: spire-agent-socket
  ```

  • OpenShift

    To mount the SPIRE agent socket in OpenShift, NGINX Service Mesh adds the following csi driver to the Ingress Controller’s Pod spec:

    volumes:
- csi:
  driver: csi.spiffe.io
  readOnly: true
name: spire-agent-socket

    and mount the socket to the Ingress Controller’s container spec:

    volumeMounts:
- mountPath: /run/spire/sockets
  name: spire-agent-socket

    For more information as to why a CSI Driver is needed for loading the agent socket in OpenShift, see Introduction in the OpenShift Considerations doc.

  1. NGINX Service Mesh adds a command line argument

    The following argument is added to the Ingress Controller’s container args:

    args:
  - -spire-agent-address=/run/spire/sockets/agent.sock
  ...
-
    • The spire-agent-address passes the address of the SPIRE agent /run/spire/sockets/agent.sock to the Ingress Controller.

    If egress is enabled NGINX Service Mesh also adds the following argument to the Ingress Controller’s container args:

    args:
  - -enable-internal-routes
  ...
-

  2. NGINX Service Mesh adds a SPIFFE label

    labels:
  spiffe.io/spiffeid: "true"
  ...

    These labels tell NGINX Service Mesh to mutate the Ingress Controller Pod with the proper configuration in order to properly integrate with the mesh.