Upgrading EOL NGINX Ingress controller with integrated Wallarm modules¶
These instructions describe the steps to upgrade deployed end‑of‑life Wallarm Ingress Controller (version 3.6 and lower) to the new version with Wallarm node 5.0.
Wallarm nodes 3.6 and lower are not supported
You are recommended to upgrade the Wallarm nodes 3.6 and lower since these versions are not supported, they are end-of-life.
Node configuration and traffic filtration have been significantly simplified in the Wallarm node of the latest versions. Before upgrading the modules, please carefully review the list of changes and general recommendations. Please note that some settings of the latest node are incompatible with the nodes 3.6 and lower.
The upgraded version of Community Ingress NGINX Controller
If you upgrade the node from version 3.4 or lower, please note that the version of Community Ingress NGINX Controller the Wallarm Ingress controller is based on has been upgraded from 0.26.2 to 1.9.5.
Since the operation of Community Ingress NGINX Controller 1.9.5 has been significantly changed, its configuration has to be adjusted to these changes during the Wallarm Ingress controller upgrade.
These instructions contain the list of Community Ingress NGINX Controller settings you probably have to change. Nevertheless, please draw up and individual plan for the configuration migration based on the Community Ingress NGINX Controller release notes.
Requirements¶
-
Kubernetes platform version 1.24-1.27
-
Helm package manager
-
Compatibility of your services with the Community Ingress NGINX Controller version 1.9.5
-
Access to the account with the Administrator role and two‑factor authentication disabled in Wallarm Console for the US Cloud or EU Cloud
-
Access to
https://us1.api.wallarm.com
for working with US Wallarm Cloud or tohttps://api.wallarm.com
for working with EU Wallarm Cloud -
Access to
https://charts.wallarm.com
to add the Wallarm Helm charts. Ensure the access is not blocked by a firewall -
Access to the Wallarm repositories on Docker Hub
https://hub.docker.com/r/wallarm
. Make sure the access is not blocked by a firewall -
Access to the IP addresses below for downloading updates to attack detection rules and API specifications, as well as retrieving precise IPs for your allowlisted, denylisted, or graylisted countries, regions, or data centers
Step 1: Inform Wallarm technical support that you are upgrading filtering node modules (only if upgrading node 2.18 or lower)¶
If upgrading node 2.18 or lower, inform Wallarm technical support that you are updating filtering node modules up to 5.0 and ask to enable new IP lists logic for your Wallarm account.
When new IP lists logic is enabled, please open Wallarm Console and ensure that the section IP lists is available.
Step 2: Disable the Active threat verification module (only if upgrading node 2.16 or lower)¶
If upgrading Wallarm node 2.16 or lower, please disable the Active threat verification module in Wallarm Console → Vulnerabilities → Configure.
The module operation can cause false positives during the upgrade process. Disabling the module minimizes this risk.
Step 3: Update API port¶
Starting with version 4.0, the filtering node uploads data to the Cloud using the us1.api.wallarm.com:443
(US Cloud) and api.wallarm.com:443
(EU Cloud) API endpoints instead of us1.api.wallarm.com:444
and api.wallarm.com:444
.
If you upgrade the node from the version 3.x or lower and your server with the deployed node has a limited access to the external resources and the access is granted to each resource separately, then after upgrade the synchronization between the filtering node and the Cloud will stop.
To restore the synchronization, in your configuration, change port 444
to 443
for API endpoint for each resource.
Step 4: Update the Wallarm Helm chart repository¶
Add the Wallarm Helm repository containing all chart versions by using the command below. Please use the Helm repository for further work with the Wallarm Ingress controller.
Step 5: Update the values.yaml
configuration¶
To migrate to Wallarm Ingress controller 5.0, update the following configuration specified in the values.yaml
file:
-
Standard configuration of Community Ingress NGINX Controller
-
Wallarm module configuration
Standard configuration of Community Ingress NGINX Controller¶
-
Check out the release notes on Community Ingress NGINX Controller 0.27.0 and higher and define the settings to be changed in the
values.yaml
file. -
Update the defined settings in the
values.yaml
file.
There are the following setting probably to be changed:
-
Proper reporting of end user public IP address if requests are passed through a load balancer before being sent to the Wallarm Ingress controller.
-
IngressClasses configuration. The version of used Kubernetes API has been upgraded in the new Ingress controller that requires IngressClasses to be configured via the
.controller.ingressClass
,.controller.ingressClassResource
and.controller.watchIngressWithoutClass
parameters. -
Validation of Ingress syntax via "admission webhook" is now enabled by default.
Disabling the Ingress syntax validation
It is recommended to disable the Ingress syntax validation only if it destabilizes the operation of Ingress objects.
-
Label format. If the
values.yaml
file sets pod affinity rules, change the label format in these rules, e.g.:controller: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchExpressions: - - key: app + - key: app.kubernetes.io/name operator: In values: - waf-ingress - - key: component + - key: app.kubernetes.io/component operator: In values: - - waf-ingress + - controller - - key: release + - key: app.kubernetes.io/instance operator: In values: - waf-ingress-ingress topologyKey: kubernetes.io/hostname weight: 100
Wallarm module configuration¶
Change the Wallarm module configuration set in the values.yaml
file as follows:
-
If upgrading from version 2.18 or lower, migrate the IP list configuration. There are the following parameters potentially to be deleted from
values.yaml
:Since IP list core logic has been significantly changed in Wallarm node 3.x, it is required to adjust IP list configuration appropriately.
-
Generate an API token for the Deploy role and pass its value in the
controller.wallarm.token
parameter. -
Ensure that the expected behavior of settings listed below corresponds to the changed logic of the
off
andmonitoring
filtration modes:- Directive
wallarm_mode
- General filtration rule configured in Wallarm Console
- Endpoint-targeted filtration rules configured in Wallarm Console
If the expected behavior does not correspond to the changed filtration mode logic, please adjust the Ingress annotations and other settings to released changes.
- Directive
-
Get rid of the explicit monitoring service configuration. In the new Wallarm Ingress controller version, the monitoring service is enabled by default and does not require any additional configuration.
-
If the page
&/usr/share/nginx/html/wallarm_blocked.html
configured via ConfigMap was returned to blocked requests, adjust its configuration to the released changes.In new node version, the Wallarm sample blocking page has the updated UI with no logo and support email specified by default.
-
If you have customized the
overlimit_res
attack detection via thewallarm_process_time_limit
andwallarm_process_time_limit_block
NGINX directives, please transfer this settings to the rule and delete from thevalues.yaml
file.
Step 6: Transfer the overlimit_res
attack detection configuration from directives to the rule¶
Starting from the version 3.6, you can fine-tune the overlimit_res
attack detection using the rule in Wallarm Console.
Earlier, the wallarm_process_time_limit
and wallarm_process_time_limit_block
NGINX directives have been used. The listed directives are considered to be deprecated with the new rule release and will be deleted in future releases.
If the overlimit_res
attack detection settings are customized via the listed directives, it is recommended to transfer them to the rule as follows:
-
Open Wallarm Console → Rules and proceed to the Limit request processing time rule setup.
-
Configure the rule as done via the NGINX directives:
- The rule condition should match the NGINX configuration block with the
wallarm_process_time_limit
andwallarm_process_time_limit_block
directives specified. -
The time limit for the node to process a single request (milliseconds): the value of
wallarm_process_time_limit
.Risk of running out of system memory
The high time limit and/or continuation of request processing after the limit is exceeded can trigger memory exhaustion or out-of-time request processing.
-
The node will either block or pass the
overlimit_res
attack depending on the node filtration mode:- In the monitoring mode, the node forwards the original request to the application address. The application has the risk to be exploited by the attacks included in both processed and unprocessed request parts.
- In the safe blocking mode, the node blocks the request if it is originated from the graylisted IP address. Otherwise, the node forwards the original request to the application address. The application has the risk to be exploited by the attacks included in both processed and unprocessed request parts.
- In the block mode, the node blocks the request.
- The rule condition should match the NGINX configuration block with the
-
Delete the
wallarm_process_time_limit
andwallarm_process_time_limit_block
NGINX directives from thevalues.yaml
configuration file.If the
overlimit_res
attack detection is fine-tuned using both the directives and the rule, the node will process requests as the rule sets.
Step 7: Check out all coming K8s manifest changes¶
To avoid unexpectedly changed Ingress controller behavior, check out all coming K8s manifest changes using Helm Diff Plugin. This plugin outputs the difference between the K8s manifests of the deployed Ingress controller version and of the new one.
To install and run the plugin:
-
Install the plugin:
-
Run the plugin:
helm diff upgrade <RELEASE_NAME> -n <NAMESPACE> wallarm/wallarm-ingress --version 5.0.3 -f <PATH_TO_VALUES>
<RELEASE_NAME>
: the name of the Helm release with the Ingress controller chart<NAMESPACE>
: the namespace the Ingress controller is deployed to<PATH_TO_VALUES>
: the path to thevalues.yaml
file defining the Ingress controller 5.0 settings
-
Make sure no changes can affect the stability of the running services and carefully examine the errors from stdout.
If stdout is empty, ensure the
values.yaml
file is valid.
Please note the changes of the following configuration:
-
Immutable field, e.g. the Deployment and/or StatefulSet selectors.
-
Pod labels. The changes can lead to the NetworkPolicy operation termination, e.g.:
apiVersion: networking.k8s.io/v1 kind: NetworkPolicy spec: egress: - to: - namespaceSelector: matchExpressions: - key: name operator: In values: - kube-system # ${NAMESPACE} podSelector: matchLabels: # RELEASE_NAME=waf-ingress - app: waf-ingress + app.kubernetes.io/component: "controller" + app.kubernetes.io/instance: "waf-ingress" + app.kubernetes.io/name: "waf-ingress" - component: waf-ingress
-
Configuration of Prometheus with new labels, e.g.:
- job_name: 'kubernetes-ingress' kubernetes_sd_configs: - role: pod namespaces: names: - kube-system # ${NAMESPACE} relabel_configs: # RELEASE_NAME=waf-ingress # Selectors - - source_labels: [__meta_kubernetes_pod_label_app] + - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name] action: keep regex: waf-ingress - - source_labels: [__meta_kubernetes_pod_label_release] + - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_instance] action: keep regex: waf-ingress - - source_labels: [__meta_kubernetes_pod_label_component] + - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_component] action: keep - regex: waf-ingress + regex: controller - source_labels: [__meta_kubernetes_pod_container_port_number] action: keep regex: "10254|18080" # Replacers - action: replace target_label: __metrics_path__ regex: /metrics - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: kubernetes_pod_name - source_labels: [__meta_kubernetes_pod_name] regex: (.*) action: replace target_label: instance replacement: "$1"
-
Analyze all other changes.
Step 8: Upgrade the Ingress controller¶
There are three ways of upgrading the Wallarm Ingress controller. Depending on whether there is a load balancer deployed to your environment, select the upgrade method:
-
Deployment of the temporary Ingress controller
-
Regular re‑creation of the Ingress controller release
-
Ingress controller release re‑creation without affecting the load balancer
Using the staging environment or minikube
If the Wallarm Ingress controller is deployed to your staging environment, it is recommended to upgrade it first. With all services operating correctly in the staging environment, you can proceed to the upgrade procedure in the production environment.
Unless it is recommended to deploy the Wallarm Ingress controller 5.0 with the updated configuration using minikube or another service first. Ensure all services operates as expected and then upgrade the Ingress controller in the production environment.
This approach helps to avoid downtime of the services in the production environment.
Method 1: Deployment of the temporary Ingress controller¶
By using this method, you can deploy Ingress Controller 5.0 as an additional entity in your environment and switch the traffic to it gradually. It helps to avoid even temporary downtime of services and ensures safe migration.
-
Copy the IngressClass configuration from the
values.yaml
file of the previous version to thevalues.yaml
file for the Ingress controller 5.0.With this configuration, the Ingress controller will identify the Ingress objects but will not process their traffic.
-
Deploy the Ingress controller 5.0:
helm install <RELEASE_NAME> -n <NAMESPACE> wallarm/wallarm-ingress --version 5.0.3 -f <PATH_TO_VALUES>
<RELEASE_NAME>
: the name for the Helm release of the Ingress controller chart<NAMESPACE>
: the namespace to deploy the Ingress controller to<PATH_TO_VALUES>
: the path to thevalues.yaml
file defining the Ingress controller 5.0 settings
-
Ensure all services operate correctly.
-
Switch the load to the new Ingress controller gradually.
Method 2: Regular re‑creation of the Ingress controller release¶
If the load balancer and Ingress controller are NOT described in the same Helm chart, you can just re‑create the Helm release. It will take several minutes and the Ingress controller will be unavailable for this time.
If Helm chart sets the configuration of a load balancer
If Helm chart sets the configuration of a load balancer along with the Ingress controller, release re‑creation can lead to a long load balancer downtime (depends on the cloud provider). The load balancer IP address can be changed after the upgrade if the constant address was not assigned.
Please analyze all possible risks if using this method.
To re‑create the Ingress controller release:
-
Delete the previous release:
-
<RELEASE_NAME>
: the name of the Helm release with the Ingress controller chart -
<NAMESPACE>
: the namespace the Ingress controller is deployed to
Please do not use the
--wait
option when executing the command since it can increase the upgrade time. -
-
Create a new release with Ingress controller 5.0:
helm install <RELEASE_NAME> -n <NAMESPACE> wallarm/wallarm-ingress --version 5.0.3 -f <PATH_TO_VALUES>
-
<RELEASE_NAME>
: the name for the Helm release of the Ingress controller chart -
<NAMESPACE>
: the namespace to deploy the Ingress controller to -
<PATH_TO_VALUES>
: the path to thevalues.yaml
file defining the Ingress controller 5.0 settings
-
-
Set the
wait = false
option in the Terraform configuration to decrease the upgrade time: -
Delete the previous release:
-
Create the new release with the Ingress controller 5.0:
Method 3: Ingress controller release re‑creation without affecting the load balancer¶
When using the load balancer configured by the cloud provider, it is recommended to upgrade the Ingress controller with this method because it does not affect the load balancer.
Release re‑creation will take several minutes and the Ingress controller will be unavailable for this time.
-
Get objects to be deleted (except for the load balancer):
helm get manifest <RELEASE_NAME> -n <NAMESPACE> | yq -r '. | select(.spec.type != "LoadBalancer") | .kind + "/" + .metadata.name' | tr 'A-Z' 'a-z' > objects-to-remove.txt
To install the utility
yq
, please use the instructions.Objects to be deleted will be output to the
objects-to-remove.txt
file. -
Delete listed objects and re‑create the relese:
cat objects-to-remove.txt | xargs kubectl delete --wait=false -n <NAMESPACE> && \ helm upgrade <RELEASE_NAME> -n <NAMESPACE> wallarm/wallarm-ingress --version 5.0.3 -f `<PATH_TO_VALUES>`
To decrease service downtime, it is NOT recommended to execute commands separately.
-
Ensure all objects are created:
The output should say that all objects already exist.
There are the following parameters passed in the commands:
-
<RELEASE_NAME>
: the name of the Helm release with the Ingress controller chart -
<NAMESPACE>
: the namespace the Ingress controller is deployed to -
<PATH_TO_VALUES>
: the path to thevalues.yaml
file defining the Ingress controller 5.0 settings
Step 9: Test the upgraded Ingress controller¶
-
Check that the version of the Helm chart was updated:
The chart version should correspond to
wallarm-ingress-5.0.2
. -
Get the list of pods specifying the name of the Wallarm Ingress controller in
<INGRESS_CONTROLLER_NAME>
:Each pod status should be STATUS: Running or READY: N/N. For example:
-
Send the request with the test Path Traversal attack to the Wallarm Ingress controller address:
If the filtering node is working in the
block
mode, the code403 Forbidden
will be returned in response to the request and the attack will be displayed in Wallarm Console → Attacks.
Step 10: Adjust the Ingress annotations to released changes¶
Adjust the following Ingress annotations to the changes released in Ingress controller 5.0:
-
If upgrading from version 2.18 or lower, migrate the IP list configuration. Since IP list core logic has been significantly changed in Wallarm node 3.x, it is required to adjust IP list configuration appropriately by changing Ingress annotations (if applied).
-
Ensure that the expected behavior of settings listed below corresponds to the changed logic of the
off
andmonitoring
filtration modes:- Directive
wallarm_mode
- General filtration rule configured in Wallarm Console
- Endpoint-targeted filtration rules configured in Wallarm Console
If the expected behavior does not correspond to the changed filtration mode logic, please adjust the Ingress annotations to released changes.
- Directive
-
If the Ingress is annotated with
nginx.ingress.kubernetes.io/wallarm-instance
, rename this annotation tonginx.ingress.kubernetes.io/wallarm-application
.Only the annotation name has changed, its logic remains the same. The annotation with the former name will be deprecated soon, so you are recommended to rename it before.
-
If the page
&/usr/share/nginx/html/wallarm_blocked.html
configured via Ingress annotations is returned to blocked requests, adjust its configuration to the released changes.In new node versions, the Wallarm blocking page has the updated UI with no logo and support email specified by default.
Step 11: Re-enable the Active threat verification module (only if upgrading node 2.16 or lower)¶
Learn the recommendation on the Active threat verification module setup and re-enable it if required.
After a while, ensure the module operation does not cause false positives. If discovering false positives, please contact the Wallarm technical support.