javascript - node fetching json from url - Stack Overflow

admin管理员组

文章数量:1405118

I am trying to fetch and parse the JSON returned by the below, in Nodejs. It seems the problem is that the node code is trying to parse the JSON string which contains newline characters, but I am not sure how to avoid this, and why fetch works in chrome devtools. I'm guessing the problem is my simplistic knowledge of https requests, so I would really appreciate it if somebody can explain what is going wrong.

const url = "=&pageNumber=1&contextId=1126&onlyShow=&&&&&&&"

I can successfully run this in the chrome devtools open on the domain. Also, the JSON appears to automatically get parsed to an object, which I didn't expect.

fetch(url).then(res => res.json()).then(json => console.log(json.pageItems))

I have tried the following in node, but none work.

const fetch = require("node-fetch");
await fetch(url).then(res => res.json()); // Unexpected token  in JSON at position 0

const rp = require('request-promise-native');
const json = await rp({uri: url}) // returns JSON string with newline characters
JSON.parse(json) // Unexpected token  in JSON at position 0

I am trying to fetch and parse the JSON returned by the below, in Nodejs. It seems the problem is that the node code is trying to parse the JSON string which contains newline characters, but I am not sure how to avoid this, and why fetch works in chrome devtools. I'm guessing the problem is my simplistic knowledge of https requests, so I would really appreciate it if somebody can explain what is going wrong.

const url = "https://beta.charitymission.gov.uk/umbraco/api/charityApi/getSearchResults?searchText=&pageNumber=1&contextId=1126&onlyShow=&&&&&&&"

I can successfully run this in the chrome devtools open on the domain. Also, the JSON appears to automatically get parsed to an object, which I didn't expect.

fetch(url).then(res => res.json()).then(json => console.log(json.pageItems))

I have tried the following in node, but none work.

const fetch = require("node-fetch");
await fetch(url).then(res => res.json()); // Unexpected token  in JSON at position 0

const rp = require('request-promise-native');
const json = await rp({uri: url}) // returns JSON string with newline characters
JSON.parse(json) // Unexpected token  in JSON at position 0

• javascript
• node.js
• json
• fetch-api

Share Improve this question Follow edited Nov 8, 2019 at 22:24 Max888 asked Nov 8, 2019 at 22:08 Max888Max888 3,85055 gold badges3535 silver badges7676 bronze badges

Add a ment  | 

2 Answers 2

Sorted by: Reset to default 3

The problem is that your JSON file is saved in UTF-8 BOM format.

What does it mean?

Your file begins with a so-called Byte Order Mark character, U+FEFF (Zero Width No-break Space).

That's invisible (has zero width), but still there, and the JSON interpreter fails to parse it.

How to solve?

1. Save the JSON data without BOM, or

2. Use res.text() and then JSON.parse(text.slice(1)) to remove the leading character:

   fetch(url)
   .then(res => res.text())
   .then(text => JSON.parse(text.slice(1)))
   .then(json => console.log(json.pageItems))
   

Why does it work in Chrome?

Chrome automatically removes the BOM character from any fetched/downloaded file, to avoid similar issues.

Try using the default fetch:

const json = await fetch(url).then(res => res.json())

res.json() parses the JSON data and returns an object. Use await to return the data into the variable json.

本文标签： javascriptnode fetching json from urlStack Overflow