javascript - How to Scrape and Convert F1 Website Data into JSON Using Node.js and ScraperAPI? - Stack Overflow

admin管理员组

文章数量:1320661

const express = require("express");
const request = require("request-promise");

const app = express();
const PORT = process.env.PORT || 5000;

const generateApiKey= (apiKey) => `=${apiKey}&autoparse=true`

app.use(express.json());

app.get("/f1", async (req, res) => {
  const {apiKey}=req.query;
  try {
   
    const response = await request(`${generateApiKey}&url=`);
    
    res.json(JSON.parse(response));
  } catch (error) {
    console.error("Error fetching data:", error);
    res.status(500).send(`${error}`);
  }
});

app.listen(PORT, () => console.log(`Server running on port ${PORT}`));

I am trying to create an API that scrapes data from the Formula 1 website (e.g., team pages) and returns it in JSON format using Node.js and ScraperAPI. My current approach is to use request-promise with ScraperAPI to fetch the page, but I'm encountering issues where the data is not being parsed correctly into JSON for some reason This what I get instead of JSON format with the data Picture with what I get in return

const express = require("express");
const request = require("request-promise");

const app = express();
const PORT = process.env.PORT || 5000;

const generateApiKey= (apiKey) => `http://api.scraperapi?api_key=${apiKey}&autoparse=true`

app.use(express.json());

app.get("/f1", async (req, res) => {
  const {apiKey}=req.query;
  try {
   
    const response = await request(`${generateApiKey}&url=https://www.formula1/en/teams/ferrari`);
    
    res.json(JSON.parse(response));
  } catch (error) {
    console.error("Error fetching data:", error);
    res.status(500).send(`${error}`);
  }
});

app.listen(PORT, () => console.log(`Server running on port ${PORT}`));

I am trying to create an API that scrapes data from the Formula 1 website (e.g., team pages) and returns it in JSON format using Node.js and ScraperAPI. My current approach is to use request-promise with ScraperAPI to fetch the page, but I'm encountering issues where the data is not being parsed correctly into JSON for some reason This what I get instead of JSON format with the data Picture with what I get in return

• javascript
• node.js
• express

Share Improve this question Follow edited Jan 18 at 18:54 VLAZ 29.1k99 gold badges6363 silver badges8484 bronze badges asked Jan 18 at 16:35 TinoOoTinoOo 133 bronze badges 1

• 1 Your code has to await both the call to request() and the call to res.json(). The res.json() call needs no argument and you do not need to call JSON.parse() at all. – Pointy Commented Jan 18 at 16:39

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

The issue is in the URL.

In your generateApiKey function, you're returning a URL string, but in your app.get("/f1", ...) route, you're not dynamically passing the API key into that URL correctly. Additionally, in your request call, you're referencing generateApiKey without actually calling it with the apiKey parameter. Furthermore, while the target URL (https://www.formula1/en/teams/ferrari) appears valid, it’s a good practice to encode URLs to prevent issues with special characters in other cases. It's also important to note that the Formula 1 team page likely returns raw HTML, which cannot be directly parsed into JSON unless the page itself provides a structured JSON API response. If the data is embedded in HTML, you may need to use libraries like cheerio to parse the HTML and extract the relevant structured data.

try this:

const express = require("express");
const request = require("request-promise");

const app = express();
const PORT = process.env.PORT || 5000;

// Function to generate the ScraperAPI URL with API Key
const generateApiKey = (apiKey) => `http://api.scraperapi?api_key=${apiKey}&autoparse=true`;

app.use(express.json());

app.get("/f1", async (req, res) => {
  const { apiKey } = req.query;
  if (!apiKey) {
    return res.status(400).send("API key is required");
  }
  
  const targetUrl = "https://www.formula1/en/teams/ferrari";
  try {
    const response = await request(`${generateApiKey(apiKey)}&url=${encodeURIComponent(targetUrl)}`);
    
    // If the response is raw HTML, you may need to parse it further
    try {
      res.json(JSON.parse(response)); // Attempt to parse JSON
    } catch (parseError) {
      res.send(response); // If not JSON, send raw response (likely HTML)
    }
  } catch (error) {
    console.error("Error fetching data:", error);
    res.status(500).send(`Error fetching data: ${error.message}`);
  }
});

app.listen(PORT, () => console.log(`Server running on port ${PORT}`));

本文标签： javascriptHow to Scrape and Convert F1 Website Data into JSON Using Nodejs and ScraperAPIStack Overflow