How to change WebDriver’s IP address

I have already written several articles on how to use WebDriver for web scraping, but I have never touched on the topic of changing WebDriver’s IP address. Nevertheless, this topic is quite crucial when you come to web scraping, and here I’d like to show you an example of using proxies with WebDriver in Python (and you can easily convert it into your language API).

First of all you need a proxy server. You can use any proxy service you like, but if you don’t have one, I would recommend that you consider BestProxyAndVPN (if you’re ready to pay for proxy services). I have tried many proxy services and ended up using this one as the most stable.

Now let’s look at two pieces of code displaying how to make WebDriver send its requests through a proxy.

Using Proxy with Chrome Driver

If you use Selenium WebDriver with the Chrome browser, you can tune it to use a proxy in the following way:

Using Proxy with Chrome Driver

Python

1 2 3

options = Options ( ) options . add_argument ( '--proxy-server=46.102.106.37:13228' ) browser = webdriver . Chrome ( executable_path = 'ChromeDriverPath' , chrome_options = options )

Don’t forget to change the proxy server address and ChromeDriverPath to your values.

Using Proxy with Firefox WebDriver

If you prefer scraping with the Firefox browser, you can cause it to use proxy with the following code:

Using proxy with Firefox and WebDriver

Python

1 2 3 4 5 6 7

profile = FirefoxProfile ( ) profile . set_preference ( "network.proxy.type" , 1 ) profile . set_preference ( "network.proxy.http" , '46.102.106.59' ) profile . set_preference ( "network.proxy.http_port" , '13228' ) profile . set_preference ( "network.proxy.ssl" , '46.102.106.59' ) profile . set_preference ( "network.proxy.ssl_port" , '13228' ) browser = webdriver . Firefox ( profile )

And again, don’t forget to change the proxy address and port to your values.

EDIT: Here is another way to specify proxy servers for Firefox webdriver (thanks to Curro Pavon):

Using proxy parameter

Python

1 2 3 4 5 6

my_proxy = Proxy ( {‘ proxyType’ : ProxyType . MANUAL , ‘ httpProxy’ : ’ 127.0.0.1 : 8118′ , ‘ sslProxy’ : ’ 127.0.0.1 : 8118′ , ‘ noProxy’ : ‘ www . google - analytics . com , ajax . googleapis . com , apis . google . com’ } ) browser = webdriver . Firefox ( proxy = my_proxy )