python - Can I get different boto3 instances to share their connection pool to the same host? - Stack Overflow

admin管理员组

文章数量:1401636

I have code that creates few S3 objects but for many access keys. It does so in parallel, but within the same python process.

Each of the boto3 instances has its own connection pool, which can linger even when it is not used. What happens, on occassion, is that the whole thing crashes with some variant of this error:

[Errno 99] Cannot assign requested address

This is because there is simply too many TCP connections on the system. This limit can be altered, but I am hitting up to 27000 connections.

This is how I instantiate boto3:

my_boto_resource = boto3.resource(service_name='s3',
                               use_ssl=False,
                               verify=False,
                               region_name='us-east-1',
                               aws_access_key_id=access_key,
                               aws_secret_access_key=secret_key,
                               endpoint_url=";)
my_boto_client = self.resource.meta.client

These two then get passed to code that populates some test data on for all S3 access keys. Note that the endpoint URL is always the same, it's a custom S3 server that supports unlimited access keys.

I had faced the exact same problem in JavaScript with the @aws-sdk/client-s3 library. But that library allows you to construct S3 instances with pre-created http(s) agent object, which can then be shared across many instances.

Is there some way to share HTTP(s) connection pool for the same host for boto3 instances, like in @aws-sdk/client-s3?

Note: Reducing how many S3 access keys are being processed in parallel has been tried. However, due to very small number of operations per S3 key (sometime just one), this reduced the speed of the whole script significantly.

本文标签： pythonCan I get different boto3 instances to share their connection pool to the same hostStack Overflow