python - Ensuring Successful File Uploads to S3 Using Presigned URLs - Stack Overflow

admin管理员组

文章数量:1406007

Previously, I used the following approach to allow users to upload files to our cloud storage service:

1. Server-side (Python): Generate a time-limited presigned upload URL.
2. Client-side (Java): Upload the file using the presigned URL.

Java Code: Uploading a File

private static boolean upload(String urlAsString, File inputFile, String checksum) {
    boolean success = false;

    HttpURLConnection urlConnection = null;
    FileInputStream fileInputStream = null;
    OutputStream outputStream = null;

    try {
        URL url = new URL(urlAsString);
        urlConnection = (HttpURLConnection) url.openConnection();
        urlConnection.setRequestMethod(PUT);
        urlConnection.setConnectTimeout(CONNECT_TIMEOUT);
        urlConnection.setReadTimeout(READ_TIMEOUT);
        // 
        urlConnection.setDoOutput(true);

        //
        // Checksum
        //
        if (checksum != null) {
            urlConnection.setRequestProperty("content-md5", checksum);
            urlConnection.setRequestProperty("x-amz-meta-md5", checksum);
        }
        
        //
        // Do this before writting to output stream.
        //
        final long length = inputFile.length();
        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.KITKAT) {
            urlConnection.setFixedLengthStreamingMode(length);
        }
        urlConnection.setRequestProperty("Content-Length", String.valueOf(length));


        fileInputStream = new FileInputStream(inputFile);
        outputStream = urlConnection.getOutputStream();

        byte[] buffer = new byte[BUFFER_SIZE];
        int bufferLength = 0;

        while ((bufferLength = fileInputStream.read(buffer)) != -1) {
            if (bufferLength > 0) {
                outputStream.write(buffer, 0, bufferLength);
            }
        }

        int responseCode = urlConnection.getResponseCode();

        if (responseCode == HttpURLConnection.HTTP_OK) {
            success = true;
        }
    } catch (MalformedURLException e) {
        Log.e(TAG, "", e);
    } catch (IOException e) {
        Log.e(TAG, "", e);
    } finally {
        close(fileInputStream);

        close(outputStream);

        if (urlConnection != null) {
            urlConnection.disconnect();
        }
    }

Python Code: Generating a Presigned Upload URL

def get_presigned_upload_url(s3_client, customer_id, key, checksum):
    presigned_upload_url = None

    if checksum is None:
        presigned_upload_url = s3_client.generate_presigned_url(
            ClientMethod='put_object',
            Params={
                'Bucket': constants.S3_BUCKET_NAME, 
                'Key': get_user_folder_name(customer_id) + key
            },
            ExpiresIn=constants.EXPIRES_IN
        )
    else:
        presigned_upload_url = s3_client.generate_presigned_url(
            ClientMethod='put_object',
            Params={
                'Bucket': constants.S3_BUCKET_NAME, 
                'Key': get_user_folder_name(customer_id) + key,
                'ContentMD5': checksum,
                'Metadata': {
                    'md5' : checksum
                }
            },
            ExpiresIn=constants.EXPIRES_IN
        )

    return presigned_upload_url

Issue: HTTP 200 OK, but File Not in S3?

I noticed cases where the client-side upload returns HttpURLConnection.HTTP_OK, but the file does not reach S3. (Is this even possible?!)

To verify the upload's correctness, I implemented an additional verification step after uploading.

Java Code: Verifying the Upload

private static boolean verifyUpload(String headUrl, String checksum) {
    boolean success = false;
    
    HttpURLConnection headConnection = null;

    try {
        headConnection = (HttpURLConnection) new URL(headUrl).openConnection();
        headConnection.setRequestMethod("HEAD");

        final int headResponseCode = headConnection.getResponseCode();
        if (headResponseCode == HttpURLConnection.HTTP_OK) {
            final String metaMd5 = headConnection.getHeaderField("x-amz-meta-md5");

            success = checksum.equals(metaMd5);
        }
    } catch (MalformedURLException e) {
        Log.e(TAG, "", e);
    } catch (IOException e) {
        Log.e(TAG, "", e);
    } finally {
        if (headConnection != null) {
            headConnection.disconnect();
        }
    }

    return success;
}

Python Code: Generating a Presigned HEAD URL for Verification

def get_presigned_head_url(s3_client, customer_id, key):
    """
    Generate a pre-signed URL to perform a HEAD request on an S3 object.
    This URL allows the client to check if the upload was successful.
    """
    presigned_head_url = s3_client.generate_presigned_url(
        ClientMethod='head_object',
        Params={
            'Bucket': constants.S3_BUCKET_NAME, 
            'Key': get_user_folder_name(customer_id) + key
        },
        ExpiresIn=constants.EXPIRES_IN
    )

    return presigned_head_url

Does This Approach Make Sense?

Would this method reliably verify uploads, given that HTTP_OK does not always mean a successful upload to S3? Any feedback would be appreciated.

本文标签： pythonEnsuring Successful File Uploads to S3 Using Presigned URLsStack Overflow