python - Handle DNS timeout with call to blob_client.upload_blob - Stack Overflow

admin管理员组

文章数量:1336632

While using the Azure storage SDK in Python, I have been unable to override what appears to be a default 90-second timeout to catch a DNS exception occurring within a call to blob_client.upload_blob(). I am looking for a way to override this with a shorter time interval (i.e. 5 seconds).

The following code illustrates this issue using a fictitious account name which DNS cannot resolve. I am using a timeout argument in the call to upload_blob, and I understand from reviewing documentation this enforces a server-side threshold, not a client-side threshold. I have not been successful in getting a client-side threshold to be enforced.

This issue appears similar to this unanswered question: How to handle timeout for Uploading a blob in Azure Storage using Python SDK?. The one (not accepted) solution suggests using a timeout threshold within the call to upload_blob. As noted above (and shown within the code below), this is not producing the desired effect.

from azure.core.exceptions import AzureError
from azure.storage.blob import BlobServiceClient

# Define Azure Storage Blob connection details
connection_string = "DefaultEndpointsProtocol=https;AccountName=test;AccountKey=removed==;EndpointSuffix=core.windows"
container_name = "containername"
blob_name = "blobname"
local_file_path = "c:/temp/test.txt"
 
 
# Create the BlobServiceClient
blob_service_client = BlobServiceClient.from_connection_string(connection_string)
 
# Function to perform the blob upload
def upload_blob_process():
    try:
        with open(local_file_path, "rb") as data:
            blob_client = blob_service_client.get_blob_client(container_name, blob_name)
            blob_client.upload_blob(data, timeout=5)
        print("Blob uploaded successfully!")
    except AzureError as e:
        print(f"Azure error occurred: {e}")
    except Exception as e:
        print(f"An error occurred: {e}")
 
 
upload_blob_process()

While using the Azure storage SDK in Python, I have been unable to override what appears to be a default 90-second timeout to catch a DNS exception occurring within a call to blob_client.upload_blob(). I am looking for a way to override this with a shorter time interval (i.e. 5 seconds).

The following code illustrates this issue using a fictitious account name which DNS cannot resolve. I am using a timeout argument in the call to upload_blob, and I understand from reviewing documentation this enforces a server-side threshold, not a client-side threshold. I have not been successful in getting a client-side threshold to be enforced.

This issue appears similar to this unanswered question: How to handle timeout for Uploading a blob in Azure Storage using Python SDK?. The one (not accepted) solution suggests using a timeout threshold within the call to upload_blob. As noted above (and shown within the code below), this is not producing the desired effect.

from azure.core.exceptions import AzureError
from azure.storage.blob import BlobServiceClient

# Define Azure Storage Blob connection details
connection_string = "DefaultEndpointsProtocol=https;AccountName=test;AccountKey=removed==;EndpointSuffix=core.windows"
container_name = "containername"
blob_name = "blobname"
local_file_path = "c:/temp/test.txt"
 
 
# Create the BlobServiceClient
blob_service_client = BlobServiceClient.from_connection_string(connection_string)
 
# Function to perform the blob upload
def upload_blob_process():
    try:
        with open(local_file_path, "rb") as data:
            blob_client = blob_service_client.get_blob_client(container_name, blob_name)
            blob_client.upload_blob(data, timeout=5)
        print("Blob uploaded successfully!")
    except AzureError as e:
        print(f"Azure error occurred: {e}")
    except Exception as e:
        print(f"An error occurred: {e}")
 
 
upload_blob_process()

• python
• azure
• http
• azure-blob-storage

Share Improve this question Follow asked Nov 19, 2024 at 16:31 GeeGee 41555 silver badges1616 bronze badges 4

• The timeout parameter in upload_blob() applies to each individual HTTP request to the Azure Blob service and it does not affect DNS resolution or connection establishment timeouts but instead limits the server-side operation duration for each request. – Venkatesan Commented Nov 20, 2024 at 3:43
• @Venkatesan, thanks, I'm aware. Do you know of any way to apply a DNS timeout on top of this? – Gee Commented Nov 20, 2024 at 12:15
• This is a socket level timeout and is not affected by overall data size. Client-side read timeouts will be automatically retried. Defaults to 60 seconds, check this helps: github/Azure/azure-sdk-for-python/tree/main/sdk/storage/… – Venkat V Commented Nov 20, 2024 at 12:43
• @Gee Check the below answer. – Venkatesan Commented Nov 22, 2024 at 6:18

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 2

While using the Azure storage SDK in Python, I have been unable to override what appears to be a default 90-second timeout to catch a DNS exception occurring within a call to blob_client.upload_blob(). I am looking for a way to override this with a shorter time interval (i.e. 5 seconds).

As, I mentioned in comment, The timeout parameter in upload_blob() applies to each individual HTTP request to the Azure Blob service and it does not affect DNS resolution or connection establishment timeouts but instead limits the server-side operation duration for each request.

thanks, I'm aware. Do you know of any way to apply a DNS timeout on top of this?

• Unfortunately, there is no built-in way in the Azure Storage SDK to configure a DNS or connection establishment timeout directly.
• You can use a workaround by using the dns.resolver library to perform DNS resolution before initiating the upload, with a custom timeout for DNS resolution. This will help you achieve the same result.

Code:

import dns.resolver
from azure.storage.blob import BlobServiceClient
import time


connection_string = "DefaultEndpointsProtocol=https;AccountName=venkat326;AccountKey=xxxx=;EndpointSuffix=core.windows"

container_name = "result"
blob_name = "test.csv"
local_file_path = "path to file"

# DNS timeout in seconds
DNS_TIMEOUT = 5


# Function to resolve DNS with timeout
def resolve_dns_with_timeout(connection_string, timeout=DNS_TIMEOUT):
 
    blob_service_client = BlobServiceClient.from_connection_string(connection_string)
    account_url = blob_service_client.url
    hostname = account_url.split("//")[1].split("/")[0]

    resolver = dns.resolver.Resolver()
    resolver.lifetime = timeout
    resolver.timeout = timeout
    try:
        resolver.resolve(hostname, "A")
        print(f"DNS resolution for {hostname} succeeded.")
    except dns.resolver.Timeout:
        raise Exception(f"DNS resolution for {hostname} timed out after {timeout} seconds.")
    except dns.resolver.NXDOMAIN:
        raise Exception(f"The domain {hostname} does not exist.")
    except Exception as e:
        raise Exception(f"DNS resolution failed for {hostname}: {e}")


# Function to upload blob with DNS resolution
def upload_blob_with_dns_check(connection_string, container_name, blob_name, local_file_path):
    start_time = time.time()
    try:
        resolve_dns_with_timeout(connection_string)
        blob_service_client = BlobServiceClient.from_connection_string(connection_string)
        blob_client = blob_service_client.get_blob_client(container_name, blob_name)

        with open(local_file_path, "rb") as data:
            blob_client.upload_blob(data)
        print("Blob uploaded successfully!")
    except Exception as e:
        elapsed_time = time.time() - start_time
        print(f"An error occurred: {e}")
        print(f"Elapsed time before timeout or failure: {elapsed_time:.2f} seconds")


# Run the upload process
upload_blob_with_dns_check(connection_string, container_name, blob_name, local_file_path)

If the DNS is wrong, you'll get output like:.

An error occurred: The domain venkat326.blob.core.windows does not exist.
Elapsed time before timeout or failure: 0.13 seconds

If the DNS is correct, you'll get output like:

DNS resolution for venkat326123.blob.core.windows succeeded.
Blob uploaded successfully!

本文标签： pythonHandle DNS timeout with call to blobclientuploadblobStack Overflow