How to bypass bot protection using python requests?

Question:

I need to get some information from this page :

But the site takes me to the bot page after several page hits.

I am using the following code:

import requests
url = 'http://www.propertyguru.com.sg/singapore-property-listing/property-for-rent?sort=date&order=desc&limit=20&market=residential'
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
response = requests.get(url, headers=headers)
print(response.content)

Answer:

Try adding more headers to the request that the browser itself sends in order to simulate it better (for example, referer , connection ).

The site may also have a limit on the number of requests for a certain time from one IP address. Try to make requests, for example, no more than one in 5 seconds.

Scroll to Top