python - Why do PUT requests timeout when GET requests don't - Stack Overflow

admin管理员组

文章数量:1335824

When using the python requests library, and specifying a timeout, GET requests do not timeout when the total time to process the requests takes longer than the timeout, but PUT requests do.

python version: 3.8.10 requests version: 2.32.3

example code:

start = datetime.now()
requests.get(URL_WITH_1GB_FILE, params=None, headers={"User-Agent": "Mozilla/5.0"}, timeout=(1,1), verify=cert)
print(f"GET: {datetime.now()-start}")

even though the entire request takes over 60s to complete, we don't get a timeout. This I believe is expected behaviour, as individual packets are being received within the 1s timeout.

However, when trying to perform the same operation with a PUT request, the request times out, unless the connection timeout is greater than the time it takes for the entire PUT request to complete.

example code:

with open(PATH_TO_1GB_FILE ,"rb" ) as f:
    dat = f.read()
    timeouts = (100, 1) # connect, read
    start = datetime.now()
    requests.put(f"{local}/puttest", params=None, headers={"User-Agent": "Mozilla/5.0"}, timeout=timeouts, data=dat, verify=cert)
    print(f"PUT: {datetime.now()-start}")

with 100s connect timeout, this succeeds as the upload takes about 60s, however if the connect timeout is set to 10s, then this upload fails.

I am uploading 100MB files on 1mbps connections, which would require a 13min timeout, however I want the timeout to trigger much earlier if the internet connection is dropped.

I know I could achieve this by sending smaller chunks, instead of a full 100MB file, but was curious if there is a way for the timeouts to better handle this situation.

When using the python requests library, and specifying a timeout, GET requests do not timeout when the total time to process the requests takes longer than the timeout, but PUT requests do.

python version: 3.8.10 requests version: 2.32.3

example code:

start = datetime.now()
requests.get(URL_WITH_1GB_FILE, params=None, headers={"User-Agent": "Mozilla/5.0"}, timeout=(1,1), verify=cert)
print(f"GET: {datetime.now()-start}")

even though the entire request takes over 60s to complete, we don't get a timeout. This I believe is expected behaviour, as individual packets are being received within the 1s timeout.

However, when trying to perform the same operation with a PUT request, the request times out, unless the connection timeout is greater than the time it takes for the entire PUT request to complete.

example code:

with open(PATH_TO_1GB_FILE ,"rb" ) as f:
    dat = f.read()
    timeouts = (100, 1) # connect, read
    start = datetime.now()
    requests.put(f"{local}/puttest", params=None, headers={"User-Agent": "Mozilla/5.0"}, timeout=timeouts, data=dat, verify=cert)
    print(f"PUT: {datetime.now()-start}")

with 100s connect timeout, this succeeds as the upload takes about 60s, however if the connect timeout is set to 10s, then this upload fails.

I am uploading 100MB files on 1mbps connections, which would require a 13min timeout, however I want the timeout to trigger much earlier if the internet connection is dropped.

I know I could achieve this by sending smaller chunks, instead of a full 100MB file, but was curious if there is a way for the timeouts to better handle this situation.

• python
• python-requests

Share Improve this question Follow asked Nov 19, 2024 at 20:47 GarethCGarethC 15511 silver badge99 bronze badges 9

• maybe set a higher read timeout? (the second timeout tuple item) – eternal_white Commented Nov 19, 2024 at 20:56
• @eternal_white the read timeout does not seem to be the issue here. For the PUT command, if the connect timeout is less than the total request time I get a timeout, if it is greater, I don't get a timeout. The same behaviour is observed regardless of whether the read timeout is 1 or 100s. – GarethC Commented Nov 19, 2024 at 21:09
• The connect timeout is the max time spent before a connection is established (and essentially before the upload can start). The read timeout is the max time spent before a response can be read from the server, but if the server essentially does not respond before it has all of the chunks of the file, this is effectively the time between starting sending the first chunk and receiving confirmation of the last chunk, which is basically all of the upload time. – Grismar Commented Nov 19, 2024 at 21:20
• @GarethC Isn't the read time here also the same time between uploading chunks of data? connection timeout is understood, but I think the problem relays on logic behind the interaction between the server's response and your chunked data. Doesn't increasing read timeout (which in this case should be the time between sending a chunk and the server's acknowledgement) to a very high number solve the problem? Maybe you could just open session between you and the server instead of a single put request? – eternal_white Commented Nov 19, 2024 at 22:29
• @Grismar that was my understanding as well, but in practice adjusting the read timeout doesn't affect anything, I can leave it at 1 and never timeout. However adjusting the connection timeout is what results in issues. – GarethC Commented Nov 20, 2024 at 14:05

 |  Show 4 more comments

1 Answer 1

Sorted by: Reset to default 0

I seem to have come to a solution to my problem, but it doesn't quite align with my understanding of the timeouts.

Desired goal is that when the internet connection drops, the request should timeout, but I don't want it to timeout if the entire request is taking a long time, because I'm sending a lot of data on a slow connection.

In practice I can achieve this with PUT requests by specifying the connection timeout to be a very long time (or None), and have the read timeout be something reasonable.

with open(PATH_TO_1GB_FILE ,"rb" ) as f:
    dat = f.read()
    timeouts = (None, 60) # connect, read
    start = datetime.now()
    requests.put(f"{local}/puttest", params=None, headers={"User-Agent": "Mozilla/5.0"}, timeout=timeouts, data=dat, verify=cert)
    print(f"PUT: {datetime.now()-start}")

This code allows for the entire request to take an indefinite time, but if the internet connection is lost, will timeout within a minute.

本文标签： pythonWhy do PUT requests timeout when GET requests don39tStack Overflow