Sidecar Proxy Injection

Learn about the configuration options for NGINX Service Mesh sidecar proxy injection.

Overview

NGINX Service Mesh works by injecting a sidecar proxy into Kubernetes resources. You can choose to inject the sidecar proxy into the YAML or JSON definitions for your Kubernetes resources in the following ways:

• Automatic Injection
• Manual Injection

Note:

When you inject the sidecar proxy into a Kubernetes resource, the injected config uses the global mTLS setting. You can define the global setting when you deploy NGINX Service Mesh, or use the default setting.

Refer to Secure Mesh Traffic using mTLS for more information.

Important:

The sidecar proxy will not be injected into Pods that define multiple container ports with the same port number or for container ports with the SCTP protocol.

UDP and TCP is an exception to this, and may be specified on the same port.

The mesh supports the following Kubernetes resources and API versions for injection:

Automatic Proxy Injection

NGINX Service Mesh uses automatic injection by default. This means that any time a user creates a Kubernetes Pod resource, the NGINX Service Mesh automatically injects the sidecar proxy into the Pod. Automatic injection applies to all namespaces in your Kubernetes cluster.

Enable or Disable Automatic Proxy Injection by Namespace

By default, NGINX Service Mesh can access resources in all Kubernetes namespaces.

To disable this setting, deploy the mesh using the --disable-auto-inject flag:

nginx-meshctl deploy ... --disable-auto-inject

To enable injection for a specific namespace, add the injector.nsm.nginx.com/auto-inject=enabled label. This will only work if the mesh is deployed with global automatic sidecar injection disabled.

Note:
If you add this label to a namespace where Pods already exist you will need to restart those Pods for the sidecar to be injected. By the same token if you remove this label from a namespace where Pods exist and have the sidecar injected, you will need to restart them to remove the sidecar.

You can also enable injection by adding the --enabled-namespaces flag to your deploy command.

For example, to disable automatic injection in all namespaces and enable it only in the namespaces “prod” and “staging”, you would run the following command:

nginx-meshctl deploy ... --disable-auto-inject --enabled-namespaces="prod,staging"

Important:
In an upcoming version, NGINX Service Mesh will be removing the --disabled-namespaces flag.

Similarly, you can deploy NGINX Service Mesh with automatic injection enabled and specify a list of the namespaces that you want to exclude. To do so, use the --disabled-namespaces flag when you deploy.

The following deploy command enables automatic injection in all namespaces except “test”:

nginx-meshctl deploy ... --disabled-namespaces="test"

Note:
If you need to modify the auto injection settings after you’ve deployed NGINX Service Mesh, you can do so by using the REST API.

Enable or Disable Automatic Proxy Injection on a Resource

For more granular control, you can override the global automatic injection setting on a per-resource basis. To do so, add the following label to the resource’s PodTemplateSpec:

injector.nsm.nginx.com/auto-inject: "enabled|disabled"

Injecting Sidecar into Existing Resources

To inject the sidecar into existing resources you must re-roll those resources after installing NGINX Service Mesh.

kubectl rollout restart <resource type>/<resource name>

For example:

kubectl rollout restart deployment/frontend

Note:
Refer to NGINX Service Mesh Labels and Annotations for more information.

See Also:
Refer to the Kubernetes kubectl Cheat Sheet documentation for more information about rolling resources.

Manual Proxy Injection

Before running the inject command, make sure your Kubernetes user has the permission to create the resource inject in APIGroup nsm.nginx.com. Below is an example of a ClusterRole with this permission:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: nsm-inject
rules:
- apiGroups:
  - nsm.nginx.com
  resources:
  - inject
  verbs:
  - create

Note:
If your Kubernetes user account has the ClusterAdmin role, then no additional permissions are necessary to run the inject command.

To inject the sidecar proxy into a resource manually, use the nginx-meshctl inject command. Provide the path to the resource definition file and your desired output filename.

nginx-meshctl inject < <resource-file> > <new-resource-file>

For example, the following command will write the updated config for “resource.yaml” to a new file, “resource-injected.yaml”:

nginx-meshctl inject < resource.yaml > resource-injected.yaml

Depending on the network connection and the size of the file you’re injecting, timeouts may occur while running the inject command. If this happens, you can use the --timeout flag to increase the timeout. The default timeout is 5 seconds.

nginx-meshctl --timeout 10s inject < resource.yaml > resource-injected.yaml

Ignore Specific Ports

You can set the proxy to ignore ports for either incoming or outgoing traffic. The NGINX Service Mesh applies the configurations at injection time.

• For automatic injection, add the following annotations to the PodTemplateSpec in your resource definition:

  config.nsm.nginx.com/ignore-incoming-ports: "port1, port2, ..., portN"
  config.nsm.nginx.com/ignore-outgoing-ports: "port1, port2, ..., portN"
  

• For manual injection, you can use the annotations above or specify the ports when running the nginx-meshctl inject command.

  nginx-meshctl inject --ignore-incoming-ports "port1,port2,...,portN", --ignore-outgoing-ports "port1,port2,...,portN" < resource.yaml > resource-injected.yaml
  

Note:
Refer to NGINX Service Mesh Annotations for more information around annotations.

Default Ignored Ports

By default, the following ports are ignored by the proxy:

• 53 (DNS)

HTTPGet Health Probe Rewrite

If mTLS mode is set to strict, then readiness, liveness, and startup probes using HTTP GET do not work. This is due to kubelet not having the correct client certificates. To remedy this, application HTTP/S health probes are rewritten at injection time. The new probes point to an endpoint on the sidecar proxy, which redirects the health check to the original destination on the application. This allows the health check to bypass the SSL verification, while still sending the health check to the intended destination.

UDP MTU Sizing

NGINX Service Mesh automatically detects and adjusts the eth0 interface to support the 32 bytes of space required for PROXY Protocol V2. See the UDP and eBPF architecture section for more information.