javascript - How to fetch data over multiple pages? - Stack Overflow

admin管理员组

文章数量:1201370

My project is based on React, redux, redux-saga, es6 and I try to fetch data from this API:

,2000,2004,2007?&returnFields=CharacteristicLabel,Indicator,IndicatorId,Value&f=json

As you can see, this specific API call shows data with a limit of 100 data per page spread on 40 pages.

According to this answer: .php?t=msg&th=2086&goto=9591&S=Google it says that you can extend the limit to a maximum of 3000data per page.

However, in some cases I would do an API call that exceeds that limit which means I would not receive all my data doing it like this:

export function fetchMetaData(countryCode: string, surveyYears: string) {
return (fetch('/' + countryCode + ',' + surveyYears + '?returnFields=CharacteristicLabel,Indicator,IndicatorId,Value&f=json')
    .then(response => response.json())
    .then(json => json.Data.map(survey => survey)))
} 

So my question is; what is the best way to get all data from this API given that I know the total pages of data. The answer in the forum link suggest to loop through the API. However, I can't find the right syntax usage to do this.

My idea would be doing one api call to get the total number of pages. Then store this in a state using redux+redux-saga. Then do a new request sending the total pages as parameter and fetch this total number of pages times. And by doing this I can't figure out the syntax to store the data for each iteration.

My project is based on React, redux, redux-saga, es6 and I try to fetch data from this API:

http://api.dhsprogram.com/rest/dhs/data/BD,2000,2004,2007?&returnFields=CharacteristicLabel,Indicator,IndicatorId,Value&f=json

As you can see, this specific API call shows data with a limit of 100 data per page spread on 40 pages.

According to this answer: http://userforum.dhsprogram.com/index.php?t=msg&th=2086&goto=9591&S=Google it says that you can extend the limit to a maximum of 3000data per page.

However, in some cases I would do an API call that exceeds that limit which means I would not receive all my data doing it like this:

export function fetchMetaData(countryCode: string, surveyYears: string) {
return (fetch('http://api.dhsprogram.com/rest/dhs/data/' + countryCode + ',' + surveyYears + '?returnFields=CharacteristicLabel,Indicator,IndicatorId,Value&f=json')
    .then(response => response.json())
    .then(json => json.Data.map(survey => survey)))
} 

So my question is; what is the best way to get all data from this API given that I know the total pages of data. The answer in the forum link suggest to loop through the API. However, I can't find the right syntax usage to do this.

My idea would be doing one api call to get the total number of pages. Then store this in a state using redux+redux-saga. Then do a new request sending the total pages as parameter and fetch this total number of pages times. And by doing this I can't figure out the syntax to store the data for each iteration.

• javascript
• reactjs
• redux
• fetch-api
• redux-saga

Share Improve this question Follow edited Mar 16, 2022 at 14:20 VLAZ 29k99 gold badges6262 silver badges8282 bronze badges asked Nov 18, 2016 at 12:58 KikelettoKikeletto 5311 gold badge11 silver badge33 bronze badges 4

• A couple of questions: 1. Why would you need to save the total number of pages in redux store? You probably wan't to encapsulate fetching all data into one action (using several async fetches). 2. You more likely meant "...sending the start page as parameter...", right? 3. What specifically is your problem, what have you tried so far? Iterating over arrays? Aggregating? Handling several async calls or promises? – matthias Commented Nov 18, 2016 at 14:00
• 1. Storing total number of pages in redux store was just a suggestion to my own problem and what you propose sounds more logical. 2. Correct. 3. What I have done so far is using redux-saga to handle async fetches. When my component request data by dispatching an action, my watcher in saga receive the action and request data from the function above. What I receive is data from 1 page which I store in the action body and add it to my reducer. My problem is that I can't get all data from every pages using the function above. Is there a smart way to do this? @matthias – Kikeletto Commented Nov 18, 2016 at 14:58
• Okay, we're getting closer to the point. I never used redux-saga and don't know if this lib offers a dedicated and preferred way to solve this. But I don't think this shouldn't stop you in solving your problem in a typical way using promises (or async/await) and functions that deal with building/mapping arrays and objects. If I got you right, you're looking for a way to execute several asynchronous calls to one API, aggregate the gathered data and then hand it over to the store/reducer for creating new state. May I ask: How familiar are you with JavaScript in general? – matthias Commented Nov 18, 2016 at 15:14
• Answer to your question, imo after doing some projects using javascript I would say I understand Javascript/ES6 basic well. However, I have not touched a lot of ES7like async/await or data fetching. This is the first time in my project. – Kikeletto Commented Nov 18, 2016 at 15:47

Add a comment  | 

2 Answers 2

Sorted by: Reset to default 15

A possible solution -the idea is to get the number of pages first, then make the appropriate number of API calls, pushing the promise from each call into an array. We then wait for all the promises to resolve, and do something with the returned data.

async function fetchMetaData() {

    const response = await fetch('apiUrlToGetPageNumber');

    const responses = await Promise.all(
        Array.from(
            Array(resp.data.pagesRequired),
            (_, i) => fetch(`apiUrlToSpecificPage?page=${i}`)
        )
    );
    
    // do something with processedResponses here

}
            
            

Here is another possible solution using async/await. The beauty of this is that the total_pages count is dynamic, so that if it increases while you're processing your request, it'll make sure you get it all.

async function fetchMetaData() {
  let allData = [];
  let morePagesAvailable = true;
  let currentPage = 0;

  while(morePagesAvailable) {
    currentPage++;
    const response = await fetch(`http://api.dhsprogram.com/rest/dhs/data?page=${currentPage}`)
    let { data, total_pages } = await response.json();
    data.forEach(e => allData.unshift(e));
    morePagesAvailable = currentPage < total_pages;
  }

  return allData;
}

本文标签： javascriptHow to fetch data over multiple pagesStack Overflow