NGINX Ingress Controller and Open Service Mesh

This document outlines how to integrate F5 NGINX Ingress Controller with Open Service Mesh (OSM)

Open Service Mesh will work with both versions of F5 NGINX Ingress controller: the free as well as the NGINX Plus versions.

Below is a link to the official F5 NGINX Ingress Controller documentation. F5 NGINX Ingress controller

Integrating NGINX Ingress Controller with Open Service Mesh

There are two ways to integrate NGINX Ingress Controller with Open Service Mesh (OSM):

  1. Injecting an envoy sidecar directly with NGINX Ingress Controller.
  2. Using the Open Service Mesh ingressBackend “proxy” feature.

NGINX Ingress controller and OSM with sidecar proxy injected

Install OSM in the cluster

osm install --mesh-name osm-nginx --osm-namespace osm-system

Mark the F5 NGINX Ingress controller namespace for sidecar injection

NOTE: Depending on how you install NGINX Ingress controller, you might need to create the namespace. For example, if you are using manifests to install NGINX Ingress controller, you can complete all of the steps on our documentation page, EXCEPT, actually deploying NGINX Ingress controller. This is because, when using the sidecar approach, OSM needs to “manage” the namespace so it knows what namespaces it needs to inject sidecars into.

Next thing we need to do is install OSM into the NGINX Ingress controller namespace so that the envoy sidecar will be injected into NGINX Ingress controller. First, create the nginx-ingress namespace:

kubectl create ns nginx-ingress

Then “mark” the nginx-ingress namespace for OSM to deploy a sidecar.

osm namespace add nginx-ingress --mesh-name osm-nginx

The above command will use the mark the nginx-ingress namespace, where OSM will be installed (sidecar)

Install F5 NGINX Ingress controller

Links to the complete install guides:

Using Helm to install NGINX Ingress Using Manifests to install NGINX Ingress

When using the sidecar method, ensure that you add the correct annotations listed below. This ensures proper integration of NGINX Ingress Controller with the envoy sidecar proxy.

Helm installs

If using helm, add the following annotation to your values.yaml file:

Under controller.pod.annotations:

pod:
  annotations: {
    openservicemesh.io/inbound-port-exclusion-list: "80, 443"
    }

You can also use the set command available with helm to set these at install time.

helm install nic01 nginx-stable/nginx-ingress -n nginx-ingress --create-namespace --set controller.pod.annotations.'openservicemesh\.io/inbound\-port\-exclusion\-list=\{ "80"\, "443"\ }'

Change your release accordingly to match your environment.

Manifest installs

For your manifest deployments, add the following annotation.

annotations:
  openservicemesh.io/inbound-port-exclusion-list: "80,443"

Sample deployment file with required annotation 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-ingress
  namespace: nginx-ingress
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-ingress
  template:
    metadata:
      labels:
        app: nginx-ingress
      annotations:
        openservicemesh.io/inbound-port-exclusion-list: "80,443"

This annotation is required when injecting envoy sidecar into NGINX Ingress controller. InboundPortExclusionList defines a global list of ports to exclude from inbound traffic interception by the sidecar proxy.

Install a Test Application

To test the integration, we will use the httpbin sample application from the Ingress With Kubernetes NGINX Ingress Controller guide.

The following three commands will create the namespace for the application, add the namespace to OSM for monitoring, then install the application.

kubectl create ns httpbin
osm namespace add httpbin --mesh-name osm-nginx
kubectl apply -f https://raw.githubusercontent.com/openservicemesh/osm-docs/release-v1.2/manifests/samples/httpbin/httpbin.yaml -n httpbin

Verify that the envoy sidecar has been injected into NGINX Ingress Controller 

kubectl get pods -n nginx-ingress
NAME                             READY   STATUS    RESTARTS       AGE
nginx-ingress-7b9557ddc6-zw7l5   2/2     Running   1 (5m8s ago)   5m19s

2/2 shows we have two containers in the NGINX Ingress controller pod: NGINX Ingress and Envoy

Configure your NGINX VirtualServer yaml to similar below

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: httpbin
  namespace: httpbin
spec:
  host: httpbin.example.com
  tls:
    secret: secret01
  upstreams:
  - name: httpbin
    service: httpbin
    port: 14001
    use-cluster-ip: true
  routes:
  - path: /
    action:
      proxy:
        upstream: httpbin
        requestHeaders:
          set:
          - name: Host
            value: httpbin.httpbin.svc.cluster.local

Test your configuration:

 curl  http://httpbin.example.com/get -v
*   Trying 172.19.0.2:80...
* TCP_NODELAY set
* Connected to httpbin.example.com (172.19.0.2) port 80 (#0)
> GET /get HTTP/1.1
> Host: httpbin.example.com
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.23.3
< Date: Sun, 19 Feb 2023 19:06:47 GMT
< Content-Type: application/json
< Content-Length: 454
< Connection: keep-alive
< access-control-allow-origin: *
< access-control-allow-credentials: true
< x-envoy-upstream-service-time: 2
<
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.httpbin.svc.cluster.local",
    "Osm-Stats-Kind": "Deployment",
    "Osm-Stats-Name": "httpbin",
    "Osm-Stats-Namespace": "httpbin",
    "Osm-Stats-Pod": "httpbin-78555f5c4b-t6qln",
    "User-Agent": "curl/7.68.0",
    "X-Envoy-Internal": "true",
    "X-Forwarded-Host": "httpbin.example.com"
  },
  "origin": "172.19.0.1",
  "url": "http://httpbin.example.com/get"
}
* Connection #0 to host httpbin.example.com left intact

Using The Open Service Mesh ingressBackend “proxy” Feature

Install OSM into the cluster. By running the following command, you will install OSM into the cluster with the mesh name osm-nginx using the osm-system namespace.

osm install --mesh-name osm-nginx --osm-namespace osm-system

Once OSM has been installed, this next command will mark NGINX Ingress Controller as part of the OSM mesh, while also disabling sidecar injection. NOTE: The nginx-ingress name can be created as part of the NGINX Ingress install process, or manually. If you are creating it manually, the namespace must created before you “add” the namespace to Open Service Mesh.

osm namespace add nginx-ingress --mesh-name osm-nginx --disable-sidecar-injection

Install F5 NGINX Ingress controller

Links to the complete install guides:

Using Helm to install NGINX Ingress Using Manifests to install NGINX Ingress

NOTE: This method does NOT require annotations added to the deployment, compared to the sidecar install method.

Install a Test Application

To test the integration, we will use the httpbin sample application from the Ingress With Kubernetes NGINX Ingress Controller guide.

The following three commands will create the namespace for the application, add the namespace to OSM for monitoring, then install the application.

kubectl create ns httpbin
osm namespace add httpbin --mesh-name osm-nginx
kubectl apply -f https://raw.githubusercontent.com/openservicemesh/osm-docs/release-v1.2/manifests/samples/httpbin/httpbin.yaml -n httpbin

mTLS Setup

To enable mTLS for NGINX Ingress Controller and OSM, you need to configure the IngressBackend API to use https as the backend protocol as and trigger OSM to issue a certificate. NGINX will use this certificate to proxy HTTPS connections to the TLS backends. The client certificate and certificate authority (CA) certificate will be stored in a Kubernetes secret that NGINX will use for authentication."

To begin, edit the osm-mesh-config resource:

kubectl edit meshconfig osm-mesh-config -n osm-system

You will need to update under certificate to look like this:

spec:
  certificate:
    ingressGateway:
      secret:
        name: osm-nginx-client-cert
        namespace: osm-system
      subjectAltNames:
      - nginx-ingress.nginx-ingress.cluster.local
      validityDuration: 24h

This will generate a new client certificate (osm-nginx-client-cert) that NGINX Ingress controller will use for mTLS. The SAN, subjectAltNames, is the following form:

<service_account>.<namespace>.cluster.local

With the above OSM mesh config changed, that secret will be created in the osm-system namespace. There will also be the osm-ca-bundle secret as well, which is autogenerated by OSM.

kubectl get secrets -n osm-system
NAME                              TYPE                 DATA   AGE
osm-ca-bundle                     Opaque               2      37m
osm-nginx-client-cert             kubernetes.io/tls    3      17m

Now, we need to “export” out these certificates in order to use them with NGINX Ingress Controller.

kubectl get secret osm-ca-bundle -n osm-system -o yaml > osm-ca-bundle-secret.yaml
kubectl get secret osm-nginx-client-cert -n osm-system -o yaml > osm-nginx-client-cert.yaml

We need to edit the two exported out .yaml files and change a few parts.

Edit osm-ca-bundle-secret.yaml Remove the private.key section under data. Change the namespace field to your nginx-ingress location Change the type to type: nginx.org/ca

Updated file should look like the following.

apiVersion: v1
kind: Secret
metadata:
  name: osm-ca-bundle
  namespace: nginx-ingress
type: nginx.org/ca
data:
  ca.crt: <ca_cert_data>

Edit osm-nginx-client-cert.yaml Remove the ca.crt in the data section Change the namespace to the nginx-ingress namespace.

Updated file should look like the following.

apiVersion: v1
kind: Secret
metadata:
  name: osm-nginx-client-cert
  namespace: nginx-ingress
type: kubernetes.io/tls
data:
  tls.crt: <tls_crt_data>
  tls.key: <tls_key_data>

Then apply these two secrets to the cluster.

kubectl apply -f osm-ca-bundle-secret.yaml
kubectl apply -f osm-nginx-client-cert.yaml

Ensure the secrets exist in the nginx-ingress namespace:

kubectl get secrets -n nginx-ingress
NAME                    TYPE                DATA   AGE
osm-nginx-client-cert   kubernetes.io/tls   2      23m
osm-ca-bundle           nginx.org/ca        1      23m

We now need to create our CRDs (virtualServer and policy). Here is the policy resource that holds the mTLS information. Make sure you apply the policy or the mTLS connection will not work. (required for virtualServer)

apiVersion: k8s.nginx.org/v1
kind: Policy
metadata:
  name: osm-mtls
  namespace: nginx-ingress
spec:
  egressMTLS:
    tlsSecret: osm-nginx-client-cert
    trustedCertSecret: osm-ca-bundle
    verifyDepth: 2
    verifyServer: on
    sslName: httpbin.httpbin.cluster.local

Here is an example virtualServer resource as well as the ingressBackend.

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: httpbin
  namespace: httpbin
spec:
  policies:
    - name: osm-mtls
      namespace: nginx-ingress
  host: httpbin.example.com
  tls:
    secret: secret01
  upstreams:
  - name: httpbin
    service: httpbin
    port: 14001
    tls:
      enable: true
  routes:
  - path: /
    action:
      pass: httpbin
---
kind: IngressBackend
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: https
    tls:
      skipClientCertValidation: false
  sources:
  - kind: Service
    namespace: nginx-ingress
    name: nginx-ingress
  - kind: AuthenticatedPrincipal
    name: nginx-ingress.nginx-ingress.cluster.local

Once these are applied, verify they are valid (virtualServer) and committed (ingressBackend):

kubectl get vs,ingressbackend -A
NAMESPACE   NAME                                  STATE   HOST                  IP    PORTS   AGE
httpbin     virtualserver.k8s.nginx.org/httpbin   Valid   httpbin.example.com                 26m

NAMESPACE   NAME                                               STATUS
httpbin     ingressbackend.policy.openservicemesh.io/httpbin   committed

You can now send traffic through NGINX Ingress Controller with open service mesh.

curl http://httpbin.example.com/get -v
*   Trying 172.18.0.2:80...
* TCP_NODELAY set
* Connected to httpbin.example.com (172.18.0.2) port 80 (#0)
> GET /get HTTP/1.1
> Host: httpbin.example.com
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.23.3
< Date: Sat, 18 Feb 2023 22:41:27 GMT
< Content-Type: application/json
< Content-Length: 280
< Connection: keep-alive
< access-control-allow-origin: *
< access-control-allow-credentials: true
< x-envoy-upstream-service-time: 1
<
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.example.com",
    "User-Agent": "curl/7.68.0",
    "X-Envoy-Internal": "true",
    "X-Forwarded-Host": "httpbin.example.com"
  },
  "origin": "172.18.0.1",
  "url": "http://httpbin.example.com/get"
}
* Connection #0 to host httpbin.example.com left intact