Read time: 3 minutes

Launching Requestium: An integration layer between Requests and Selenium for automation of web actions

From time to time at Tryolabs we have the need of simulating user interactions on websites. To tackle this problem, we usually use Requests, the beloved Python HTTP library, for simple sites; and Selenium, the popular browser automation tool, for sites that make heavy use of Javascript. Using Requests generally results in faster and more concise code, while using Selenium makes development faster on Javascript heavy sites.

After writing several of these interactions we found ourselves with the need of writing code that made use of both these approaches at the same time. Ideally, we wanted to be able to write most of the user interaction using Requests, while being able to seamlessly switch to using Selenium for the Javascript intensive parts.

Initially we did this switch manually but quickly found that it resulted in lots of unnecessary boilerplate code that was hard to read. It became clear that we could use a more sophisticated way to integrate these two methods of web automation: hence Requestium was born.

Requestium aims at solving this problem by integrating a Selenium webdriver into a Requests Session, while adding several other improvements such as integrated parsing of websites using Parsel, an excellent HTML parsing tool.


First create a session as you would do on Requests, and optionally add arguments for the web-driver if you plan to use one.

from requestium import Session, Keys

s = Session(webdriver_path='./chromedriver',
            webdriver_options={'arguments': ['headless']})

You don’t need to parse the response, it is done automatically when calling xpath, css or re.

title = s.get('').xpath(
    '//title/text()').extract_first(default='Default Title')

Regex require less boilerplate when compared to Python’s standard re module.

response = s.get('')

# Extracts the first match
identifier = response.re_first(r'ID_\d\w\d', default='ID_1A1')

# Extracts all matches as a list
users ='user_\d\d\d')

The Session object is just a regular Requests’s session object, so you can use all of its methods.'', data={'field1': 'data1'})
s.proxies.update({'http': '', 'https': ''})

And you can switch to using the Selenium webdriver to run any js code.

s.transfer_session_cookies_to_driver()  # You can mantain the session if needed

The driver object is a Selenium webdriver object, so you can use any of the normal selenium methods plus new methods added by Requestium.

    "//input[@class='user_name']").send_keys('James Bond', Keys.ENTER)

# New method which waits for element to load instead of failing
# Useful for single page web apps

Requestium also adds xpath, css, and re methods to the Selenium driver object.

if'ID_\d\w\d some_pattern'):
    print('Found it!')

And finally you can switch back to using Requests.

s.transfer_driver_cookies_to_session()'', data={'key1': 'value1'})

Closing remarks

As you can see, the library is quite simple, but it can simplify writing these sorts of scripts a great deal. If you find this useful, please feel free to explore our Github repository!

Like what you read?

Subscribe to our newsletter and get updates on Deep Learning, NLP, Computer Vision & Python.

No spam, ever. We'll never share your email address and you can opt out at any time.
Comments powered by Disqus