go - HTTP request fails for Incapsula block - Stack Overflow

admin管理员组

文章数量:1324843

I'm simply trying to make an HTTP request to this product on the Pokemon Center website. If you have cookies cleared and open the URL then the request will succeed (200 status), but without any cookies Pokemon only responds with a Set Cookie (no HTML body) and then the webpage reloads with the received cookies and the GET request succeeds and returns the actual HTML.

I'm having trouble replicating this in Golang: my first request responds with 200 and the set cookies, the second request should have the cookies but responds with the same thing instead of actually returning the HTML.

There's a bunch of requests that take place between the first product page GET request and the second one. You can simply see them by making sure you have cookies cleared and then load that URL with the Dev Network tool open.

It seems like an incapsula block, how can I get past this? The images attached are for the first GET request with NO cookies, and the second is for the GET request WITH cookies.

First request:

Second request:

func FetchPokemonURL(client *http.Client, productUrl string, firstTry bool) (string, bool, string) {
    req, err := http.NewRequest("GET", productUrl, nil)
    if err != nil {
        return "", false, "Failed to create request: " + err.Error()
    }

    req.Header.Set("Host", "www.pokemoncenter")
    req.Header.Set("Sec-Fetch-Dest", "document")
    req.Header.Set("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.0.1 Safari/605.1.15")
    req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
    if firstTry {
        req.Header.Set("Sec-Fetch-Site", "none")
    } else {
        req.Header.Set("Sec-Fetch-Site", "same-origin")
    }
    req.Header.Set("Sec-Fetch-Mode", "navigate")
    if !firstTry {
        req.Header.Set("Referer", productUrl)
        req.Header.Set("Cache-Control", "max-age=0")
    }
    req.Header.Set("Accept-Language", "en-US,en;q=0.9")
    req.Header.Set("Priority", "u=0, i")
    req.Header.Set("Accept-Encoding", "gzip, deflate, br")
    req.Header.Set("Connection", "keep-alive")

    resp, err := client.Do(req)
    if err != nil {
        return "", false, fmt.Sprintf("Error making request: %v\n", err)
    }
    defer resp.Body.Close()

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        return "", false, fmt.Sprintf("Error reading response body: %v\n", err)
    }
    strBody := string(body)

    if firstTry {
        return FetchPokemonURL(client, productUrl, false)
    } else {
        return strBody, false, ""
    }
}

本文标签： goHTTP request fails for Incapsula blockStack Overflow