python - S3UploadFailedError due to MissingContentLength when calling PutObject in MLflow using MinIO - Stack Overflow

admin管理员组

文章数量:1199793

When trying to save / upload a file using mlflow.log_artifact() to MinIO, our MLflow users are suddenly getting this error in previously working code, raised in boto3 package (used by the mlflow package, and with S3-compatible local MinIO server acting as a data lake for our local installation of MLflow):

S3UploadFailedError: Failed to upload ./dict-20231204.2.json to mlflow/24/<run_id>/artifacts/dict-20231204.2.json: An error occurred (MissingContentLength) when calling the PutObject operation: You must provide the Content-Length HTTP header.

It is raised here: [..]/python3.11/site-packages/boto3/s3/transfer.py:378, in S3Transfer.upload_file()

Any solutions or at least workarounds to restore file uploads to MinIO buckets?

When trying to save / upload a file using mlflow.log_artifact() to MinIO, our MLflow users are suddenly getting this error in previously working code, raised in boto3 package (used by the mlflow package, and with S3-compatible local MinIO server acting as a data lake for our local installation of MLflow):

S3UploadFailedError: Failed to upload ./dict-20231204.2.json to mlflow/24/<run_id>/artifacts/dict-20231204.2.json: An error occurred (MissingContentLength) when calling the PutObject operation: You must provide the Content-Length HTTP header.

It is raised here: [..]/python3.11/site-packages/boto3/s3/transfer.py:378, in S3Transfer.upload_file()

Any solutions or at least workarounds to restore file uploads to MinIO buckets?

• python
• amazon-s3
• boto
• minio
• mlflow

Share Improve this question Follow edited Jan 22 at 13:04 mirekphd asked Jan 21 at 20:02 mirekphdmirekphd 6,75333 gold badges5454 silver badges7979 bronze badges 1

• 1 The error is quite descriptive, if you could explicitly set the Content Length, it should resolve your issue. But it sounds like you are using a package? If you want to avoid customizing the package, then usually it is more about reviewing what changed. Since you mentioned it was previously working, there must be a change that triggered this behavior. – Winson Tanputraman Commented Jan 22 at 1:15

Add a comment  | 

2 Answers 2

Sorted by: Reset to default 1

A quick workaround is to downgrade the S3 client package that raises the error, thus:

$ pip install boto3"<1.36.0"

---

I have established that the issue (possibly an incompatibility with our legacy MinIO server) has been introduced in version 1.36.0 of boto3 (used by mlflow) and persists in version 1.36.2.

Accroding to this answer, It fixes this issue:

export AWS_REQUEST_CHECKSUM_CALCULATION=when_required
export AWS_RESPONSE_CHECKSUM_VALIDATION=when_required

本文标签：