python - Why do I get a HTTP Error 403: Forbidden when trying to use the OECD API - Stack Overflow

admin管理员组

文章数量:1356566

I'm just trying to use the OECD API to pull a couple of datasets, nothing dramatic, but when I even use their example, I get an error message with HTTP Error 403: Forbidden.

The documentation is here, and the example code I've taken directly from there is:

import pandas as pd

url = '.SDD.STES,DSD_STES@DF_CLI/.M.LI...AA...H?startPeriod=2023-02&dimensionAtObservation=AllDimensions&format=csvfilewithlabels' 
df = pd.read_csv(url) 

As far as I can tell, there's no API key required or anything, so any insight would be much appreciated

I'm just trying to use the OECD API to pull a couple of datasets, nothing dramatic, but when I even use their example, I get an error message with HTTP Error 403: Forbidden.

The documentation is here, and the example code I've taken directly from there is:

import pandas as pd

url = 'https://sdmx.oecd./public/rest/data/OECD.SDD.STES,DSD_STES@DF_CLI/.M.LI...AA...H?startPeriod=2023-02&dimensionAtObservation=AllDimensions&format=csvfilewithlabels' 
df = pd.read_csv(url) 

As far as I can tell, there's no API key required or anything, so any insight would be much appreciated

• python
• http-status-code-403

Share Improve this question Follow edited Apr 1 at 6:23 VLAZ 29.1k99 gold badges6363 silver badges8484 bronze badges asked Mar 31 at 10:31 Geie JoyGeie Joy 1111 bronze badge New contributor Geie Joy is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct. 2

• That exact URL you got there, requested via my browser address bar, downloads me a CSV file of ca. 155kb in size. Guess it might be an issue of URL encoding, the browser perhaps encodes certain characters automatically, whereas your python code doesn't(?). – C3roe Commented Mar 31 at 10:41
• 1 It uses cloudflare verification. If you use VPNs etc, 403 might happen. – TheMaster Commented Mar 31 at 10:43

Add a comment  | 

2 Answers 2

Sorted by: Reset to default 1

You can modify your request to include a User-Agent header, which makes it look like it’s coming from a browser

import pandas as pd
url = 'https://sdmx.oecd./public/rest/data/OECD.SDD.STES,DSD_STES@DF_CLI/.M.LI...AA...H?startPeriod=2023-02&dimensionAtObservation=AllDimensions&format=csvfilewithlabels'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'}
df = pd.read_csv(url, storage_options=headers)
print(df.head())

            

Or you can use the requests library to fetch the data and then pass it to pandas

import pandas as pd
import requests
from io import StringIO

url = 'https://sdmx.oecd./public/rest/data/OECD.SDD.STES,DSD_STES@DF_CLI/.M.LI...AA...H?startPeriod=2023-02&dimensionAtObservation=AllDimensions&format=csvfilewithlabels'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'}
response = requests.get(url, headers=headers)

if response.status_code == 200:
    df = pd.read_csv(StringIO(response.text))
    print(df.head())
else:
    print(f"Failed with status code: {response.status_code}")

I don't know why pandas is (apparently) having an issue.

You could use requests to download the content to a local file then get pandas to read that file as follows:

import requests
import pandas as pd

FILENAME = "foo.csv"
URL = "https://sdmx.oecd./public/rest/data/OECD.SDD.STES,DSD_STES@DF_CLI/.M.LI...AA...H?startPeriod=2023-02&dimensionAtObservation=AllDimensions&format=csvfilewithlabels"
headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:136.0) Gecko/20100101 Firefox/136.0"
}

with requests.get(URL, headers=headers, stream=True) as response:
    response.raise_for_status()
    with open(FILENAME, "wb") as csv:
        for chunk in response.iter_content(4096):
            csv.write(chunk)
    df = pd.read_csv(FILENAME)
    print(df)

Output (partial):

    STRUCTURE                        STRUCTURE_ID                STRUCTURE_NAME ACTION REF_AREA  ... Unit multiplier DECIMALS Decimals BASE_PER Base period
0    DATAFLOW  OECD.SDD.STES:DSD_STES@DF_CLI(4.1)  Composite leading indicators      I      BRA  ...           Units        2      Two      NaN         NaN
1    DATAFLOW  OECD.SDD.STES:DSD_STES@DF_CLI(4.1)  Composite leading indicators      I      BRA  ...           Units        2      Two      NaN         NaN
2    DATAFLOW  OECD.SDD.STES:DSD_STES@DF_CLI(4.1)  Composite leading indicators      I      AUS  ...           Units        2      Two      NaN         NaN
3    DATAFLOW  OECD.SDD.STES:DSD_STES@DF_CLI(4.1)  Composite leading indicators      I      AUS  ...           Units        2      Two      NaN         NaN
4    DATAFLOW  OECD.SDD.STES:DSD_STES@DF_CLI(4.1)  Composite leading indicators      I      AUS  ...           Units        2      Two      NaN         NaN

本文标签： pythonWhy do I get a HTTP Error 403 Forbidden when trying to use the OECD APIStack Overflow