javascript - Node `https.get` returns 403, `axios.get` and `curl` return 200 - Stack Overflow

admin管理员组

文章数量:1350044

I'm trying to perform a GET request using Node's native https module and a 3rd party axios module (which is pretty monly used):

const endpoint = '';

require('axios')
    .get(endpoint)
    .then(response => console.log('axios', response.status));
// prints 200

require('https')
    .get(endpoint, {},
        response => console.log('https', response.statusCode));
// prints 403

The axios request works fine as expected; I can view response.data as well. But the https request fails with a 403.

• Strangely, both used to work last time I checked a few months ago.
• I haven't updated Node on my machine (node v 14.5.0). I've also tried on another machine as well (13.3.0).
• curl from the mand line also works fine.

Since https used to work, my hunch is that the server must have changed something, and that https.get and axios.get must be doing something slightly differently (e.g. sending different headers by default) that the server is no longer indifferent to.

I'm trying to perform a GET request using Node's native https module and a 3rd party axios module (which is pretty monly used):

const endpoint = 'https://www.pathofexile./api/trade/data/stats';

require('axios')
    .get(endpoint)
    .then(response => console.log('axios', response.status));
// prints 200

require('https')
    .get(endpoint, {},
        response => console.log('https', response.statusCode));
// prints 403

The axios request works fine as expected; I can view response.data as well. But the https request fails with a 403.

• Strangely, both used to work last time I checked a few months ago.
• I haven't updated Node on my machine (node v 14.5.0). I've also tried on another machine as well (13.3.0).
• curl from the mand line also works fine.

Since https used to work, my hunch is that the server must have changed something, and that https.get and axios.get must be doing something slightly differently (e.g. sending different headers by default) that the server is no longer indifferent to.

• javascript
• node.js
• axios

Share Improve this question Follow edited Oct 23, 2020 at 15:30 Marco Bonelli 69.7k2121 gold badges127127 silver badges146146 bronze badges asked Oct 23, 2020 at 14:52 junvarjunvar 11.6k22 gold badges3434 silver badges5151 bronze badges 4

• What happens if you reverse the two? I.E. first https then axios? – Marco Bonelli Commented Oct 23, 2020 at 14:53
• same thing. I've actually been trying them independently, I just bined them to make the snippet show that I'm doing the exact same thing otherwise. – junvar Commented Oct 23, 2020 at 14:56
• Ok, interesting. I thought the server could have been rate-limiting your requests or something similar. – Marco Bonelli Commented Oct 23, 2020 at 14:57
• 1 I think your hunch is absolutely correct. – Adam Specker Commented Oct 23, 2020 at 14:59

Add a ment  | 

2 Answers 2

Sorted by: Reset to default 9

After a little bit of testing, I noticed that by default the https module does not include an User-Agent header, while axios and curl both do. The server is rejecting your request because you have no User-Agent set.

To solve this, simply specify a custom User-Agent of your choice:

require('https')
    .get(endpoint, {headers: {'User-Agent': 'whatever'}},
        response => console.log('https', response.statusCode));

After few web results focusing on node https didn't find an answer, I found another stackoverflow post about python gets returning 403 while curl returns 200.

It turns out it was an issue with the request's 'User-Agent' header.

Curl by default sends the user agent 'curl/7.72.0':

> GET /94548/2401383/manifest.js HTTP/1.1
> Host: cdn.flashtalking.
> User-Agent: curl/7.72.0
> Accept: */*

Axios by default sends the user agent 'axios/0.20.0\r\n':

_header: 'GET /api/trade/data/stats HTTP/1.1\r\n' +
      'Accept: application/json, text/plain, */*\r\n' +
      'User-Agent: axios/0.20.0\r\n' +
      'Host: www.pathofexile.\r\n' +
      'Connection: close\r\n' +
      '\r\n',

While node's https by default sends no user agent:

  _header: 'GET /api/trade/data/stats HTTP/1.1\r\n' +
    'Host: www.pathofexile.\r\n' +
    'Connection: close\r\n' +
    '\r\n',

Apparently, the server now checks for a non empty User-Aagent header. Updating the https request to do so gets a 200:

const endpoint = 'https://www.pathofexile./api/trade/data/stats';

require('axios')
    .get(endpoint)
    .then(response => console.log('axios', response.status));
// prints 200

require('https')
    .get(endpoint, {},
        response => console.log('https', response.statusCode));
// prints 403


require('https')
    .get(endpoint, {header: {headers: {'User-Agent': '_'}}}},
        response => console.log('https', response.statusCode));
// prints 200

本文标签： javascriptNode httpsget returns 403axiosget and curl return 200Stack Overflow