Schema-Based Testing Setup ¶

This article describes how to enable and configure Wallarm's Schema-Based Testing.

Enable¶

Schema-Based Testing is disabled by default. To enable:

1. Contact sales@wallarm.com to activate the Security Testing subscription plan for your account.

2. Go to the Security Testing → Schema-Based → Test policies tab and create at least one policy.

Prerequisites¶

Token¶

Schema-Based Testing requires a token for authorizing data exchange between running Docker container and Wallarm Cloud. The token can be created in two ways:

• Automatically - Schema-Based Testing will create it automatically and include into the docker run command on first attempt to copy Docker command from any policy. Other policies will re-use already existing token.

• Manually - in Wallarm Console, go to Settings → API Tokens, click New token; on creation, set Token usage to Schema-Based Testing agent. All policies will use this token.

Whitelist¶

When you are planning to run Wallarm's Schema-Based Testing against domains protected by some security tools, it is recommended to add the client IP (IP from where the Docker command is being run) to the allowlist, otherwise majority of requests will be blocked and the vulnerabilities will not be found.

This includes the case when Wallarm itself is used as the protection tool for these domains - see details on Wallarm's allowlist here.

Test policy types¶

You can configure test policy based on OpenAPI specification (OAS) or Postman collection. Note that one test policy is aimed at only one type of testing (either OAS or Postman-based).

• OAS is more focused on input validation, injection, misconfiguration detection

• Postman - on complex business logic and access control vulnerabilities

OAS-based test policies¶

OpenAPI specification (OAS)-based test policy defines persistently:

• Application's OpenAPI specification

• Tests to run

Besides persistent parameters that are the same for any test run, each test policy may optionally include parameters that can be overridden during each next test run (Runtime parameters). Re-defining the runtime parameters can be useful for embedding of Docker into the CI/CD pipelines:

• Application's Target URL

  (although can be redefined during each run, some initial value is required)

• Authentication parameters

To configure test policy:

1. Go to Wallarm Console → Security Testing → Schema-Based → Test policies.

2. Click Add policy, attach OpenAPI specification file.

3. Select test types to run.

4. Set Target URL (can be overridden dynamically during each test run).

5. Optionally, add authentication Runtime parameters.

   

Postman collection-based test policies¶

With Postman-based security testing you can automate security scans alongside your regular API tests, ensuring that each API run is thoroughly tested for vulnerabilities.

Pre-check of Postman collection¶

Before using Postman data for schema-based security testing:

1. Ensure that you collection and environment files contain:

   • Functional tests for API endpoints
   • Location of the target application
   • All environment variables set
   • Necessary credentials to authenticate in the target application

2. (Recommended) Check whether the Postman collection is working. For example, this can be done by running the command:

   newman run my_collection.json -e my_environment.json
   

   This can tell you in advance whether there are any problems, for example, related to the quality of the Postman collection, the availability of the target URL, or that all the necessary variables are specified.

Configuring test policy in Wallarm¶

In Wallarm, Postman collection-based test policy defines:

• Application's Postman collection.

• Postman environment file(s) (optional if all configuration is stored in the main collection file).

  Target URL and authentication

  Application's Target URL and authentication parameters are defined in the Postman collection or environment files.

• Test case selection is not currently supported for security testing based on Postman collections.

To configure test policy:

1. Go to Wallarm Console → Security Testing → Schema-Based → Test policies.

2. Click Add policy, set Source to Postman collection.

3. Attach Postman collection file.

4. Optionally, attach Postman environment file(s). Attaching 2 files allows running testing twice with different variable values (for example, different credentials) and then comparing results.

   

Business logic security testing¶

For business logic testing (OWASP API1:2023 Broken Object Level Authorization (BOLA) and OWASP API5:2023 Broken Function Level Authorization (BFLA)) Wallarm's Schema-Based Security Testing requires API traffic from at least two different authenticated users, preferably with varying privileges (e.g., admin and regular users). This diversity enables more effective business logic security tests and provides a more thorough assessment.

Example 1: Using Postman collections with requests from two users¶

Do the following:

1. Create a Postman collection containing requests from two authenticated users. For example, in the target application, include login and activity requests from User A and User B in the same collection.

   

2. Verify that all requests are executed correctly.

3. Create a test policy with the created Postman collection.

Example 2: Using Postman environments for multiple users¶

Do the following:

1. Create two Postman environment files, each holding the credentials for a different authenticated user, e.g.:

   • env1.json for User A
   • env2.json for User B

2. Create a test policy that uses your Postman collection and both environment files.

   In this setup, the collection is executed twice - once with env1.json (User A) and once with env2.json (User B) - so each request runs under both user contexts.

Editing existing policy¶

You can edit previously created policies: while clicking policy itself opens its Docker command info, you can click the edit button to access the edit dialog:

Docker run¶

Running command¶

As test policy is created, it provides you with the Docker run command which allows you start tests for your application:

1. Go to Wallarm Console → Security Testing → Schema-Based → Test policies.

2. Click you policy. The policy's Docker command will be displayed.

   

3. If necessary, redefine Docker log level and format.

4. Copy command and run it or embed into your CI/CD pipeline. This will run security tests selected in the policy for your application.

   Remember that, for OAS-based run, you can override the policy's Runtime parameters on each run by adding the corresponding -e parameters to the Docker run command, for example:

   -e TARGET_URL="http://dvapi.st.wallarm.tools" 
   -e AUTH_HEADER="Authorization: Bearer <VALUE>"
   

   You can simplify re-defining these parameters by selecting the Rewrite authentication data and/or Rewrite target URL checkboxes that add to the command:

   -e AUTH_HEADER="${AUTH_HEADER}" -e TARGET_URL="${TARGET_URL}"
   

5. View run statistics and test run results on the Test runs tab.

Environment variables¶

When you create and save a test policy in Wallarm, it automatically creates a Docker run command, adds all required -e variables and sets values for them. Generally, variables aim to tell Docker container to which account in Wallarm Cloud to connect and which test policy to take as the basis for testing of your application:

Available flags¶

You can use flags to modify behavior of Wallarm's Schema-Based Testing running as Docker container. To see the full list of available flags, use:

<YOUR DOCKER RUN COMMAND FOR SCHEMA-BASED TESTING> --help

The outup will be something like:

Wallarm Schema-Based Security Testing - API security testing tool

Usage:
  /app/main [flags]

Flags:
GENERAL:
   -v, -version           Show version information
   -fail-severity string  Set fail severity level (critical, high, medium, low, info) (default "high")

PREFLIGHT-CHECKS:
   -skip-target-url-connectivity-check  Skip checking target URL connectivity

REPORTS:
   -r, -report            Enable report generation
   -report-format string  Report format (json) (default "json")
   -report-output string  Report output file path (default "report/results.json")
   -report-pretty         Pretty-print JSON output (default true)
   -no-summary            Disable summary output to stdout

LOGGING:
   -l, -log-level string   Set log level (debug, info, warn, error) (default "info")
   -f, -log-format string  Set log format (json, colored, text) (default "colored")

See the explanation for the -fail-severity flag in the next section.

Test run success criteria¶

If your include Wallarm's Schema-Based test runs into your CI/CD pipeline, it is important to define the criteria of test run success. You can do that with the -fail-severity flag with possible values:

• critical

• high (default)

• medium

• low

• info

For example:

docker run -e WALLARM_API_HOST="us1.api.wallarm.com" -e WALLARM_API_TOKEN="<YOUR_SCHEMA-BASED_TESTING_AGENT-TOKEN>" -e WALLARM_CLIENT_ID="<YOUR_COMPANY_ID>" -e WALLARM_TESTING_POLICY_ID="<YOUR_TEST_POLICY_ID>" --network="host" wallarm/security-testing:0.3.2 -fail-severity medium

...will lead to the following: if testing finds at least 1 medium (or high or critical) security issue, the test run will exit with code 1 ("test failed").

Deleting policies¶

You can delete a test policy. If you do so:

• Information on previous test runs will remain untouched

• You will not be able to run Docker's command based on the deleted policy

• If policy's Docker containers are running, they will continue to do so and the testing will continue

• When policy's Docker containers stop, you will not be able to re-run them

Was this page helpful?

How can we improve this article? (Optional)

Information is unclear or confusing

Insufficient information

Outdated or incorrect information

0/240

Send