javascript - Puppeteer loads blank page with 429 when accessing URL - Stack Overflow

admin管理员组

文章数量:1315252

I am working on a project to obtain pricing information from a hotel website, but I cannot perform any searches when loading the website in puppeteer.

Here is a snippet of my JavaScript that opens Chrome.

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({ headless: false, devTools: false });
  const page = await browser.newPage();
  await page.setUserAgent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3683.103 Safari/537.36');

  await page.goto('/');

})();

How are they detecting that I'm using Chrome controlled by Puppeteer, even though it is running a headful browser?

Thanks

I am working on a project to obtain pricing information from a hotel website, but I cannot perform any searches when loading the website in puppeteer.

Here is a snippet of my JavaScript that opens Chrome.

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({ headless: false, devTools: false });
  const page = await browser.newPage();
  await page.setUserAgent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3683.103 Safari/537.36');

  await page.goto('https://www.hyatt./');

})();

How are they detecting that I'm using Chrome controlled by Puppeteer, even though it is running a headful browser?

Thanks

• javascript
• node.js
• web-scraping
• chromium
• puppeteer

Share Improve this question Follow asked Jun 7, 2019 at 20:06 Jeff RainerJeff Rainer 12311 silver badge55 bronze badges

Add a ment  | 

3 Answers 3

Sorted by: Reset to default 3

They're fingerprinting you. It's some javascript that enumerates things like your browser plugins and navigator object and sends that back to the server.

So you'd have to intercept that post and change it to something a normal browser would send.

Try adding the following header which is expected

'accept-language' : 'en-US,en;q=0.9'

works for me in other languages.

From this answer it looks like you need something like:

await page.setExtraHTTPHeaders({
    'Accept-Language': 'en-US,en;q=0.9'
});

The page is using Fingerprintjs2 to analyse your browser before letting you access the page.

On your first request, the page is sending you to a special detection page, which is using Fingerprint2. Quote from the github repository:

Modern & flexible browser fingerprinting library

The code will check all kinds of browser information, like specific attributes, timezone, pixelRatio of your device, etc. You can find the actual source code in this file on github.

You will have to change all kind of browser attributes to make the fingerprinting library actually think you are a "normal user".

本文标签： javascriptPuppeteer loads blank page with 429 when accessing URLStack Overflow