Initially we did this switch manually but quickly found that it resulted in lots of unnecessary boilerplate code that was hard to read. It became clear that we could use a more sophisticated way to integrate these two methods of web automation: hence Requestium was born.
Requestium aims at solving this problem by integrating a Selenium webdriver into a Requests Session, while adding several other improvements such as integrated parsing of websites using Parsel, an excellent HTML parsing tool.
First create a session as you would do on Requests, and optionally add arguments for the web-driver if you plan to use one.
You don’t need to parse the response, it is done automatically when calling xpath, css or re.
Regex require less boilerplate when compared to Python’s standard
The Session object is just a regular Requests’s session object, so you can use all of its methods.
And you can switch to using the Selenium webdriver to run any js code.
The driver object is a Selenium webdriver object, so you can use any of the normal selenium methods plus new methods added by Requestium.
Requestium also adds xpath, css, and re methods to the Selenium driver object.
And finally you can switch back to using Requests.
As you can see, the library is quite simple, but it can simplify writing these sorts of scripts a great deal. If you find this useful, please feel free to explore our Github repository!
Like what you read?
Subscribe to our newsletter and get updates on Deep Learning, NLP, Computer Vision & Python.