amazon web services - AWS S3 SHA-256 hash mismatch when uploading csv files from Excel workbook (VBA based restAPI) - Stack Over

admin管理员组

文章数量:1399947

(Summary at the end. If you want background, read on.)

(I have zero formal training in software development, am not employed as an AWS cloud architect or data analyst, etc. I’m a maintenance manager who was tasked with building a relatively simple tool, which quickly grew legs once I realized what could be accomplished with the resources available to me. I’m a novice at best though. It took a ton of research and learning just to get this far, so more than just having the fix dropped in my lap, I want to learn from this process.)

Problem: The project I’m working on probably would have been better off built in something other than Excel. However, we have a decent sized tech base who will be using the tool (~200+ technicians across the network), and Excel is the common denominator preinstalled on all technician laptops, so it made sense at the time. In a nutshell, the workbook utilizes userforms that emulate the HMIs of the machines we use in our facilities. Techs can follow along with the machine HMI, screen-by-screen, to record current equipment settings into the workbook. These values are stored in a hidden sheet which is then uploaded to an S3 bucket as a csv. Throughout testing, I had dummy data in place for the uploads, so file sizes were around 5kB and everything worked as intended. Once I started adding actual data, the file size grew beyond 6kB and I started seeing the SHA-256 mismatches:

XAmzContentSHA256Mismatch - The provided 'x-amz-content-sha256' header does not match what was computed.

Here’s the Class Module I shamelessly borrowed, slightly modified to work with my code: (I don’t know the original author so can’t properly attribute). The functions generating both the authorization header and SHA-256 checksums have not been modified, and everything works as intended when the file size is less than 6kB. I’ve also verified my SHA-256 hashes using the Get-FileHash command in PowerShell – it does indeed match what the VBA API is generating (including when S3 returns a mismatch).

I also understand that either single-part or multi-part upload can be specified for S3 transfers. I assume it defaults to single part with such a small file though. I know that multi-part uploads become recommended when file sizes reach the 100MB range, and when multi-part uploads are enabled, SHA-256 checksums are calculated differently. I don’t believe this is my problem though as I'm pretty sure multi-part isn't an option for uploads less than 5MB.

So, from my limited knowledge, I think I have two options to resolve this: 1) figure out why AWS S3 is calculating SHA-256 hashes differently on these slightly larger (but still miniscule) csv files and correct the auth header accordingly. Or, 2) request S3 to process the upload as an unsigned payload. I know the latter is an option, but I’m having difficulty figuring out how to construct the header to include the request (.html). I am a little leery of using the unsigned payload route if I'm honest though, even if it does provide a workaround - it won't resolve the actual issue, and I will have learned nothing from it (except bad practice I'm sure...). And of course, maybe neither of these two options are a viable solution.

Summary: Using a custom restAPI implemented in VBA to upload csv files to AWS S3 bucket from Excel. Files more than 6kB in size are failing to upload (“XAmzContentSHA256Mismatch” response from S3). Smaller file sizes upload successfully. On failed uploads, I’ve confirmed the SHA-256 hash generated by the custom API is valid by comparing using the Get-FileHash command in Powershell. AWS S3 is generating something different on these slightly larger csv files.

I unfortunately can’t share the workbook itself due to infosec policies, but I'll provide whatever else I can.

本文标签：