Modifying Headlines in HTTP (s) Requests in UI Automation Test – Smashing Magazine

Quick overview ↬

Being able to change headers in a test environment is a great thing to have. It allows control over your application, as you can bypass authentication, set cookies and so on. In this article, Nafees Nehar explores some methods that allow changing headers in an automation test setup.

There are different methods to change headlines. You can change headings by using browser extensions or proxy apps (such as Charles and Proxyman) that intercept the request and allow you to change the answer by including the headings.

But first, let’s start from the beginning.

HTTP (s) headers are key value pairs that can be used by the client or server to pass additional information along with an HTTP (s) request or response. They have additional information about the data being sent. An HTTP (s) header consists of a name that is case sensitive, followed by a colon (:), then by its value. Field names distinguish uppercase and lowercase letters, but field values ​​distinguish uppercase and lowercase letters. A header can have multiple values, which are separated by commas.

I was working on a program that opened the delivered web page and then allowed the user to modify items, add events, add analytics and so on. without the need to code. This was done by loading the page iframe and provides various options to the user on top of that. I tried with lots of sites to see how they behave iframe. I noticed that most of the sites are not working iframe because of x-frame-options and content-security-policy headlines.

Almost all websites have a header X-frame-options set to deny or sameorigin due to which the browser does not allow to load the web page in one iframe or do not when a cross-origin request tries to load it into a iframe. Also it content-security-policy header her frame-ancestors Directive to prevent this.

It was very important to load the page iframe, I wandered around the internet to find a way to load it. It was clear I had to set aside X-frame-options heading to allowall or remove it completely. That was when I fell over Requestly extension, which gave me the function to change X-frame-options header by matching the page URL and thus allowing me to override X-frame-options error correction header.

That was when I first witnessed the power of network headers. They carry data about the data being transferred. The ability to change the headings of traffic passing through your browser is a great tool to have. In addition to overriding X-frame-optionsyou can delete headings to minimize online tracking, override content-security-policy header, test sites in production and so on.

While testing web applications, changing headlines provides a great hack:

  • to test the guest mode of an application;
  • can set cookies using headings;
  • testing certain parts of an application that are disabled by default and can be enabled by sending a custom request header;
  • to test various test cases associated with headings;
  • to bypass the approval flow in your application by passing on the authorization header.

When I was told about the automated testing of webapps, it dawned on me that changing headlines should be a feature there due to its huge usability in webapp testing. Therefore, I decided to write this piece to shed some light on the ways to change headers in automated testing.

More after the jump! Continue reading below ↓

Selenium

Selenium is widely used as a test automation framework for end-to-end testing of web applications. It was developed in 2004. Selenium IDE was originally used, but it only supported firefox, then Selenium RC was developed to enable cross-browser testing. Selenium WebDriver is now used as it supports the mobile experience and dynamic websites. It mimics a real user who interacts with the web page.

Benefits of using Selenium WebDriver

  • Selenium WebDriver is open source;
  • It offers bindings to all major programming languages;
  • Works across multiple OS. Tests written in Windows would easily work on Mac;
  • Keyboard and cursor simulation are supported;
  • Add-ons can be installed.

Limitations of Selenium WebDriver

  • No support for changing headings;
  • No support for adding request parameters;
  • Unable to block a request.

As discussed, the ability to modify headers helps tremendously with testing applications, but Selenium WebDriver does not support it and they do not plan to include it lately.

This article focuses on different approaches to changing headers in a Selenium automation setup.

Procedure 1: Selenium Wire (Python)

Selenium Wire extends Selenium Python bindings to give you access to the underlying requests from the browser. You create your code the same way you do with Selenium, but you get additional APIs to inspect requests and responses and make changes to them right away.

It allows to change requests and responses on the go using interceptors. It can also block requests, mock replies, add request parameters to the URL.

Use

# interceptor function intercepts the network request
# If one arg is provided, requests are intercepted
# and can be modified
def interceptor(request):
    request.headers['New-Header'] = 'Some Value'
# setting the driver's request_interceptor to equal
# the customised interceptor
driver.request_interceptor = interceptor
driver.get(<URL_where_to_modify_the_header>)

# All requests will now contain New-Header

Duplicate header names are allowed in an HTTP request, so before setting up the replacement header, you must first delete the existing header using. del otherwise there will be two headings with the same name.

# A response interceptor takes two args which
# then allows to tinker with the response
def interceptor(request, response):  
    if request.url == 'https://server.com/some/path':
        response.headers['New-Header'] = 'Some Value'
driver.response_interceptor = interceptor
driver.get(<URL_where_to_modify_the_header)

# Responses from https://server.com/some/path will now contain 
# the New-Header

A response interceptor should accept two arguments, one for the initial request and one for the response.

Limitations

  • This is only available as a Python module.
  • It does not support other languages.

There are tools like Requestly, which is a one-stop tool for troubleshooting and modifying network requests. Requestly allows users to change headers, redirect URLs, change hosts, mock API responses, delay network requests, insert custom scripts, etc. It provides ready-to-go npm package, which is an extension wrap that enables use of expansion in Selenium. It supports Chrome, Firefox and Edge.

Use of Requestly In Selenium

Requestly allows you to do many more things than just change headlines. All of these things can be set up in a web interface and the shared list can be loaded into the application, giving the user the flexibility to easily edit the rules of a web application.

To install: npm i @requestly/selenium

Use

A Modify Headers rule can be created on app.requestly.io/rules after installing the extension.

Adding a heading to all requests

Adding a heading to all requests. (Large preview)

Once you have created the URL, click on it Share button to generate a link to the URL.

The URL for the above created sharedList are here.

This URL is a Requestly rule that adds Access-Control-Allow-Origin header for all requests.

This rule can be used in Selenium WebDriver using sharedList URL, described below:

require("chromedriver");
const { Builder } = require("selenium-webdriver");
const chrome = require("selenium-webdriver/chrome");
const { getRequestlyExtension, importRequestlySharedList } = require("@requestly/selenium");

const options = new chrome.Options().addExtensions(getRequestlyExtension("chrome"));
const driver = new Builder()
    .forBrowser("chrome")
    .setChromeOptions(options)
    .build();

// Imports Rules in Selenium using Requestly sharedList feature
// importRequestlySharedList(driver, <sharedList_URL>);

importRequestlySharedList(driver, 'https://app.requestly.io/rules/#sharedList/1626984924247-Adding-Headers-Example');

More information can be found here.

Limitations

  • It offers an npm package that limits the module to JavaScript only.
  • Shared lists must be created manually to use the rules in Selenium, therefore the rules can not be checked through the code written for the Selenium automation test.

Procedure 3: Using Puppeteer

Puppeteer is a sheet music library developed by Google, which provides a high-level API for managing headless Chrome or Chromium over the DevTools protocol. It can also be configured to use full (non-headless) Chrome or Chromium.

When talking about setting up browser automation, Selenium automatically comes to mind, but since the advent of Puppeteer, it has been widely used for web scraping. It gives more control over chrome than Selenium, probably because of Google’s support for it. Also for the same reason it excludes the need for an external driver to run the browser.

Use

const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto('<https://example.com>');
    await page.screenshot({ path: 'example.png' });
    await browser.close();
})();

That page.setExtraHTTPHeaders(headers) method can be used to set headings.

It can also modify and remove existing headings. When requests are intercepted using a combination of page.setRequestInterception(true) and page.on().

await page.setRequestInterception(true);
page.on('request', request => {
// Override headers
    const headers = Object.assign({}, request.headers(), {
    foo: 'bar', // set "foo" header
    origin: undefined, // remove "origin" header
});

request.continue({headers});

});

Code source

Limitations

  • Puppetry is limited to chrome for now. It can not be used for testing across browsers.
  • It has a smaller community compared to selenium. There seems to be more support for Selenium in the community.
  • It only supports JavaScript.

Conclusion

Changing network headers happens to be a very powerful tool in a test environment. There is much more to it than meets the eye in this article. I have tried to cover some of the simplest methods to change headers in UI Automation Testing.

Each method has some advantages based on the use case and the language you use. Here are some of the preferences:

  • If you want to use Selenium in Python, you should prefer Selenium-Wire as it provided a lot of features on top of Selenium-Webdriver.
  • If you only want to test on chrome, you should prefer puppetry because of its google support and out-of-the-box support of all features, including changing headlines.
  • If you work with Selenium and want to perform tests across browsers, you’d prefer to add extensions to mimic the test in Selenium-Webdriver. If your use case requires you to inject scripts or redirect network resources at runtime (in addition to changing headers), then Requestly is an ideal choice.

Hope this article gives you insight into how to change headlines in an automated test of web applications.

Additional resources

Smashing Editorial
(vf, yk, han)

Leave a Comment