Intro

To perform well at Search Engine Optimization (SEO), you need to allow your website to be crawled by the search engine bots. Unfortunately, there are many types of bots that perform similar tasks. Not all of them are related to search engines. Therefore, you can optimize your firewall by whitelisting the IP addresses for the real search engine bots while blocking others.

When your SEO is done well, your webpages will be properly indexed and their contents easily found whenever users search for related keywords online. This translates to a higher number of visitors to your website and subsequently you’ll get a much better sales conversion rate.

Let’s take a look at the IP2Location Robot Whitelist page

Thankfully, you don’t have to look far to get the list of search engine robots’ IP addresses. IP2Location has kindly provided an easy to use webpage where you can select which search engine you want to whitelist and the format of your firewall.

Above you can see the supported formats that the page can output. Major firewalls such as Apache .htaccess, Linux iptables, .NET Web.config and Cisco ACL are there. All you need to do is put them into your firewall configuration and those search engine robots will be whitelisted.

What are the supported search engines?

Don’t worry. Most major search engines are in the list. You can find Baidu, Bing, Blekko, DuckDuckGo, ExaLead, Facebook, Gigablast, Google, LinkedIn, Pinterest, Sogou, Yahoo and Yandex.

Do note that unregistered users can only download the IP addresses for DuckDuckGo, Facebook, LinkedIn, Pinterest and Sogou. If you wish to download the others, you will need to register for free at the page. This is to prevent abuse of the free data provided.

Visit the IP2Location Robot Whitelist page and test out the whitelist for yourself.