javascript - how to bypass robots.txt while crawling - Stack Overflow

admin管理员组

文章数量:1302403

Can anyone please tell me if there is any way to ignore or bypass robots.txt while crawling. Is there any way to modify script in such way that it ignores robots.txt and go on with crawling?

Or is there any other way to achieve the same?

User-agent: *
Disallow: /

User-agent: Googlebot
Disallow:

Can anyone please tell me if there is any way to ignore or bypass robots.txt while crawling. Is there any way to modify script in such way that it ignores robots.txt and go on with crawling?

Or is there any other way to achieve the same?

User-agent: *
Disallow: /

User-agent: Googlebot
Disallow:

• javascript
• jquery

Share Follow asked Jan 21, 2015 at 15:00 PratikPratik 7911 gold badge11 silver badge1111 bronze badges 2

• 7 robots.txt is a suggestion, not a requirement. If you want to ignore it, you just ignore it. – Blazemonger Commented Jan 21, 2015 at 15:01
• Maybe you have problems with Cross-Origin-Requests, not with robots.txt? – Boldewyn Commented Jan 21, 2015 at 15:03

Add a ment  | 

2 Answers 2

Sorted by: Reset to default 4

if you are writing Crawler in mechanize (Python) and have an interface with robot.txt then use the following mand:

import mechanize
br = mechanize.Browser()
br.set_handle_robots(False)

If you are writing a crawler then you have plete control of it. You can make it behave nicely or you can make it behave badly.

If you don't want your crawler to respect robots.txt then just write it so it doesn't. You might be using a library that respects robots.txt automatically, if so then you will have to disable that (which will usually be an option you pass to the library when you call it).

---

There is no way to use client side JavaScript to cause a crawler reading the page embedding the JS to stop respecting robots.txt.

本文标签： javascripthow to bypass robotstxt while crawlingStack Overflow