From time to time at Tryolabs we have the need of simulating user interactions on websites. To tackle this problem, we usually use Requests, the beloved Python HTTP library, for simple sites; and Selenium, the popular browser automation tool, for sites that make heavy use of Javascript. Using Requests generally results in faster and more concise code, while using Selenium makes development faster on Javascript heavy sites.

After writing several of these interactions we found ourselves with the need of writing code that made use of both these approaches at the same time. Ideally, we wanted to be able to write most of the user interaction using Requests, while being able to seamlessly switch to using Selenium for the Javascript intensive parts.

Initially we did this switch manually but quickly found that it resulted in lots of unnecessary boilerplate code that was hard to read. It became clear that we could use a more sophisticated way to integrate these two methods of web automation: hence Requestium was born.

Requestium aims at solving this problem by integrating a Selenium webdriver into a Requests Session, while adding several other improvements such as integrated parsing of websites using Parsel, an excellent HTML parsing tool.

Usage

First create a session as you would do on Requests, and optionally add arguments for the web-driver if you plan to use one.

You don't need to parse the response, it is done automatically when calling xpath, css or re.

Regex require less boilerplate when compared to Python's standard re module.

The Session object is just a regular Requests's session object, so you can use all of its methods.

And you can switch to using the Selenium webdriver to run any js code.

The driver object is a Selenium webdriver object, so you can use any of the normal selenium methods plus new methods added by Requestium.

Requestium also adds xpath, css, and re methods to the Selenium driver object.

And finally you can switch back to using Requests.

Closing remarks

As you can see, the library is quite simple, but it can simplify writing these sorts of scripts a great deal. If you find this useful, please feel free to explore our Github repository!