Assertions in JMeter

The assertion in JMeter is used to validate the response of the request, that you have sent to the server. The assertion is a process where you verify the expected result with the actual result of the request at run time. If you need to apply assertion on a particular Sampler, then add it as a child of that Sampler.

The Assertion is the process of comparing the expected result with actual result

Considerations Before Setting Assertions in JMeter

The Cost of JMeter Assertions:
  • All assertions come with a cost, in terms of CPU or memory consumption. However, some assertions carry a greater cost than others.
  • According to the JMeter Performance and Tuning Tips guide, the Response Assertion and the Duration Assertion are typically lower-impact choices.
  • Whereas Compare Assertion and other XML-based ones like the XPath Assertion consume more CPU and memory.
The Scope of JMeter Assertions:
  • You must also consider the scope when setting assertions.
  • Assertions can be applied to the main sample and its subsamples, or only to subsamples.
  • Some assertions, like the Response Assertion or the Size Assertion, can also be used against a JMeter Variable.
  • Code-based assertions (such as Beanshell, BSF, and JSR223) don’t have the GUI element that identifies scope. This means you must manually implement all assertion logic – including scope.
Combining Assertions:

You can add more than one assertion to the sampler, controller, thread group, or test plan. Failed assertions will cause all affected samples to fail, so caution is essential.

Introduction to JMeter

Commonly Used Assertions & Their Uses in JMeter

Response Assertions:
  • The most commonly used assertion is the Response Assertion, which checks whether a response text/body/code/message/header contains, matches, or equals a specified pattern.
    The Pattern can be either be:
    
     a “string” for “Equals” or “Substring” clauses
    a “Perl5-style” Regular Expression for “Contains” or “Matches” clause​

Response Entities that can be checked include :

  • Text response: This is for the response that can be displayed in a browser
  • Document (text): This is for anything supported by Apache Tika (it assumes the presence of apache-tika.jar in /lib folder of a JMeter installation).
    This can include PDF, Office, audio, and video formats. Be careful, because this can be memory-intensive for high loads.
  • URL Sampled: This assertion is used against a requested URL to ensure it matches expectations. For example, you may want to check that the redirect URL does not contain an error somewhere in the path.
  • Response Code : This checks to ensure the response code is expected. For 4xx and 5xx response codes, make sure you have checked the Ignore Status box (see below for a full explanation).
  • Response Message : This verifies that the response message appears as expected.
  • Response Headers : This is used against Response Headers to see if a specific HTTP header is present or absent.
  • Ignore Status : JMeter out-of-the-box considers all 4xx and 5xx responses as failures. If your test case is negative and, for example, a 404 error is expected, you will check this box to suppress JMeter’s built-in status code check and substitute it with your own status code assertion.

The Following Example demonstrates how to validate JMeter Tests Pass/Fail Status

  • Launch the JMeter in your system and then
  • Open any recorded files. Here I am opening HTTP(s) Cookie Manager
    http-cookie-manger-open-test-plan-jmeter
  • Once the Recorded file has been opened in the JMeter, Add Thread group into the Test Plan and then Copy and paste the test script into the Thread Group.
  • Right Click on the Test Plan and then Add-->Config Element-->HTTP Cookie Manager, if your test plan does not have a cookie manager
    add-http-cookie-manager-jmeter
  • We have added HTTP Cookie Manager into the Test Plan.
  • Next, add the View Results Tree in the HTTP Request and run the test plan so that some results will be stored in the View Results Tree.
    restore-test-scripts-inside-the-view-result-tree-jmeter
  • Now click on the authenticate-81 script and then click on the Response Data and then scroll down to the sentence as below.
    scroll-down-to-welcome-sentence-jmeter
  • Now copy the Test-inside the h4 and then go to the Authenticate-81 a request under Thread Group, right-click on it and then add Assertion.
    add-response-assertion-to-the-authenticate-jmeter
  • The Response Assertion will look like as below. Select the Click on Text Response in the Field to Test and then click on Contains under Pattern Matching Rules.
    response-assertion-page-jmeter
  • Now Go back to the View Results Tree and Select the Text(Welcome to the Secure are When you are done click logout below) and then go to the Purchase request under the Thread group and then click on the Add From Clipboard.
    add-sentence-inside-the-patterns-to-test-jmeter
  • That means in the Response check if it contains Welcome to the Secure Area. When you are done click logout below. If this text is present then it is Pass or else it is failed.
  • Now add one more Response Assertion to the Authenticate-81 Request under the Thread Group.
  • And then click on the Response code and then check the response code in the View Results Tree and enter the same in the Response Assertion.
  • Enter the Response code 200 as shown below and then click on the Equals.
    add-response-code-to-the-asseritiom-jmeter
  • Now, save the Test As Assertions.jmx and run the file.
    save-the-assertions-as-jmeter
  • We have passed the assertions without any error.
    successfully-passed-assertions-without-any-error-jmeter
  • Now delete the Cookie Manager and Clear all the View Results Tree and then run the Test Plan, you can see that the assertion has failed here.
    delete-the-cookie-manager-jmeter
  • After the execution of the Test Plan, you can see an Assertion error as shown below.
    assertion-error-jmeter
  • If you extend the /Authenticate-81, then you can see that the Response Assertion has been failed.
    assertion-result-jmeter
  • Now add Cookie Manager to the Thread Group and run the test plan again you won't get any error.
    add-cookie-manager-and-then-run-again-jmeter
  • If you want to see the results of assertion, you can add Assertion Result under the Test Plan(Add-->Listerner-->Assertion Results)
    add-assertion-result-under-test-plan-jmeter
  • Once you add this Assertion listener this will show you what are the assertions you have, what are the expectations, and what are the actual logs will be displayed in the Assertion Results.
  • Now run the test plan again the assertion result will clearly show the error.
    assertion-result-jmeter
  • But when you add assertions and the HTTP cookies you will get the output assertions neatly.
    assertion-results-without-any-error-jmeter
  • So in this way you need not validate the assertions manually, the Assertion Result will display the errors.

The Following Example Demonstrates Size Assertion

  • Add Size Assertion to the authenticate-81 request.
    add-size-assertion-to-authenticate-jmeter
  • The size assertion validates how much the size in bytes the response is generated.
  • Click on the View Results Tree and then click on the Authenticate-81 and then open the Sampler Result, you will find the total bytes of that request.
    size-bytes-in-sampler-jmeter
  • In the above image, you will see the size in bytes are 3846, which means the 3846 bytes response has been generated for the authenticate-81 request.
  • The total size in bytes is required because we need to know at a point of time if authenticate-81 can handle 100 requests, that means 100*3846=384600 bytes are written by the server on one single second. this is the throughput parameter.
  • It is like how many processes have been happened from the server-side.
  • Sometimes this number goes down if you have a heavy load on the application and it is not able to load full size of bytes.
  • Now, go to the Size Assertion and then click on it.
    size-assertion-page-jmeter
  • Enter the 3000 as an average and select the Greater Than(>) symbol.
    size-assertion-validation-jmeter
  • Now run the Test Plan, the Test Plan will execute without any error.
  • That means when we are checking Full Response the Size which is return by the response should be less than the total size but it can be greater than the mentioned size.
  • If it is greater than the total size in bytes then the JMeter will throw an error.

The Following example Demonstrates the Duration Assertion

  • Add Duration Assertion to the Test Plan as shown below.
    add-duration-assertion-to-the-test-plan-jmeter
  • The Duration Assertion gives how much time did the Authentication request takes to generate a response.
  • If you open the View Result Tree and go to the sampler, you can see the load time as below.
    load-time-in-sampler-result-jmeter
  • So the Response has taken 1,440/630/1925 milliseconds to generate the response. if we put 35 MS in the Duration Assertion and run the test plan then the result will be failed.
    duration-assertion-result-jmeter
Duration Assertion:
  • Duration assertion is used to verify how long a request takes to complete. The Duration measured in milliseconds, and, if any request lasts longer than the value specified, the sample is marked as failed.
  • When you get a Duration Assertion failure, the output appears like this.
Size Assertion:

Size Assertion checks the response length to see if it’s equal/not equal/greater/less than the expected size in bytes. It can be applied to:

  • A full response (body and headers)
  • Response headers
  • Response body
  • Response code
  • Response message
  • The easiest way to check the response size is through the View Results Tree Listener.
  • Here is an example of the sample output: For the Full Response assertion, it should be testing “=” comparison type and 1591 bytes.
  • For the Response Body - 1270 bytes
    For the Response Headers - 321 bytes
XML Assertion:
  • The XML Assertion checks that the returned response body is XML-compliant. Only the syntax is checked. Any external resources are neither downloaded nor validated.
  • When there is invalid XML code, the reason for failure will be reported in an Assertion Failure message.
    <?xml version="1.0"?>
    
    <note>
    
       <to>Tove</to>
    
       <from>Jani</from>
    
       <heading>Reminder</heading>
    
       <body>Don't forget me this weekend!</body>
    
    /note>​
Beanshell Assertion:
  • The Beanshell Assertion allows you to perform additional checks on a sampler using Beanshell scripting.

Beanshell Assertion offers the following variables :

  • Failure : Boolean (true|false). This indicates whether the sampler is successful.
  • FailureMessage : String. This is a custom message displayed as an assertion failure message.
  • ResponseData : Byte. A byte array representing response data.
  • ResponseCode : String. This represents a response code.
  • ResponseMessage : String. This holds the response message.
  • ResponseHeaders : String. This contains the response headers.
  • SampleResult: org.apache.jmeter.samplers.SampleResult
    This is a JMeter SampleResult class instance that contains results for preceding sampler(s).
    When there are multiple parent samplers this method returns an array of all nested requests.

For example, the following Beanshell Assertion code snippet will return an error if the word Blazemeter does not appear in the URL path:

String path = SampleResult.getURL().getPath();
if (!path.contains("blazemeter")) {
   Failure = true;
   FailureMessage = "URL Path: didn't contain 'blazemeter'" + System.getProperty("line.separator") + "URL Path detected: " + path;
}
MD5Hex Assertion:
  • The MD5Hex assertion checks the MD5 checksum of the actual response against the expected MD5 hash.
  • Content of any length, whether it’s one character or a full HD video file will be represented as a 32-digit hexadecimal number.
  • It is particularly useful for large data-integrity checks.
  • If you need to test the file-download performance and run checks for the content of downloaded files, you can avoid storing megabytes of data in memory by running MD5 hashes only assertion checks.

There are a number of online services and applications for calculating MD5 checksums :

  • WinMD5Free : for Windows
  • md5sum : for Linux and Unix
  • md5 : for MacOSX
HTML Assertion:
  • The HTML Assertion checks that the response HTML syntax is a well-formed HTML/XHTML/XML document. So it is handy if your Web application or site requires zero HTML validation errors.
  • Most modern browsers render even invalid HTML pages, but search engine robots or third-party integrations may not be so tolerant.
  • The official documentation on HTML Assertion is pretty comprehensive.
  • When it comes to Assertion Results Visualization, reports will only display a limited number of warnings and errors.
XPath Assertion:
  • The XPath Assertion allows an XPath evaluation against a Web server response to ensure the specified entity is present or an element attribute value matches expectations.
  • For more information on how to use XPath for correlation, visit XPath Extractor in JMeter.
  • The most appropriate use case for XPath Assertion is testing SOAP Web Services XML responses.
XML Schema Assertion:
  • The XML Schema Assertion checks whether the XML response matches the specific XSD schema provided.
  • When running tests with BlazeMeter, just provide a reference schema file along with the test script.
BSF and JSR223 Assertions:
  • The BSF Assertion and JSR223 Assertion use cases are the same as the Beanshell assertion.
  • The only difference is performance.
  • JSR223, in combination with Groovy language, gives almost the same performance as native Java code and Beanshell, whereas other languages have performance constraints.
  • If your scripting assertion code is heavy, consider using JSR223 Sampler and Groovy.
Compare Assertion:
  • The Compare Assertion checks the response content or confirms that the response time of all samplers under the assertion scope is equal.
  • The requests and the assertion should be on the same level in the test plan.
  • The Compare Assertion checks the response content or confirms that the response time of all samplers under the assertion scope is equal.
  • The requests and the assertion should be on the same level in the test plan.
Assertion parameters:

Compare Content | TRUE|FALSE :

  • If TRUE, the content of all affected samplers will be checked to confirm it is identical. Any difference will cause an assertion failure.
  • If FALSE, the content check will be omitted.

Compare Time | -1 | 0 | number :

  • If -1 - response time check will be omitted.
  • If 0 - response time of all affected samplers will be checked to confirm that are identical. Any difference, even 1 ms, will cause an assertion failure.
  • If >0, the response time of all affected samplers will be checked to be no different than the value provided.
  • If the threshold is exceeded, the assertion will be omitted.
  • If both Content and Time checks are specified, the Time check will take precedence.
  • As already mentioned, avoid using this assertion for high loads, since it consumes a lot of CPU and RAM, which can ruin your test and take up a lot of your valuable time.
SMIME Assertion
  • The SMIME Assertion checks whether a response returned from the Mail Reader Sampler is signed.
  • An alternative clause (message is unsigned) can also be specified, regardless of whether the assertion should verify the signer certificate.
  • This assertion requires third-party libraries, so be sure you have following libraries in your JMeter classpath:

bcmail-*.jar

bcprov-*.jar

bcpkix-*.jar

  • These three jars must be downloaded from the Bouncy Castle download area. Be sure that JDK for Bouncy Castle libraries matches JMeter’s current JDK requirements.
  • If you’re using Blazemeter, just drop the files to the File Upload area along with your .jmx script.

Installation and Configuration of JMeter

Viewing the Results of Your Assertions in JMeter

In this section, we are going to learn how to view the results of our Assertions. We can view the results by using the following three modes such as:

  • Assertion Results Visualization.
  • Command Line Mode.
  • Blazemeter Assertions.
Assertion Results Visualization:
  • In this case, we have a sample and an Assertion to test the response, but we don't know how to see what is wrong with the response.
  • In the GUI Mode, there are two ways that assertions can be inspected.
  • Assertion Result Listener : This reveals the label under which all the assertions were taken.
  • View Results Tree Listener : This reveals all the assertions in the test plan.
Both consume significant amounts of memory, so don’t use them during load tests. They should only be used for debugging or opening a .jtl the file generated by non-GUI run.

For non-GUI mode, the following properties are available :

  • meter.save.saveservice.assertions=true | false
  • jmeter.save.saveservice.assertion_results_failure_message=true | false
  • jmeter.save.saveservice.assertion_results= none | first | all
Response Assertion Report Fields:

Three major fields are recorded :

  • Assertion Error (true|false) : This indicates whether the assertion succeeded or Not
    For example The Assertion Error will be true if there is a problem with the assertion, such as an incorrect Beanshell script in the Beanshell Assertion, or the size in bytes is not provided for the Size Assertion. An assertion error causes the affected sample(s) to fail.
  • Assertion Failure (true|false) : This indicates whether an assertion is successful.
    If the actual assertion result matches the expected result, it will be true, otherwise, it will be false.
    If the Assertion Failure = false, affected sampler(s) will be considered failed.
  • Assertion Failure Message (string) : This is a built-in or custom message that clarifies the details of the assertion failure.

Recording Scripts in JMeter

Comment / Suggestion Section
Point our Mistakes and Post Your Suggestions