admin管理员组文章数量:1123562
I'm encountering an issue with AzCopy when attempting to transfer data from Amazon S3 to Azure Data Lake Storage Gen2. The error message indicates that I lack PUT permission, even though I’ve verified that I have the necessary permissions in both AWS and ADLS. I created a SAS token with all possible permissions checked, and my AWS IAM policy (shown below) should also be sufficient. I’ve also set the AWS environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) in my command prompt:
set AWS_ACCESS_KEY_ID=123
set AWS_SECRET_ACCESS_KEY=ABC
Despite these measures, the error persists. Any guidance or troubleshooting steps would be greatly appreciated.
IAM policy
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation" ], "Resource": "arn:aws:s3:::s3_bucket" }, { "Effect": "Allow", "Action": [ "s3:GetObject" ], "Resource": "arn:aws:s3:::s3_bucket/dir/profiles/*" } ]}
Azcopy command
C:\Users\[user]>azcopy copy "/" "https://[stg_container_name].blob.core.windows/raw-test?[SASTOKEN]" --recursive=true
INFO: Scanning...
INFO: Failed to create one or more destination container(s). Your transfers may still succeed if the container already exists.
INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support
Job 67f9fe5e-f142-bf40-656c-ae0bf93f5e15 has started
Log file is located at: C:\Users\[user]\.azcopy\67f9fe5e-f142-bf40-656c-ae0bf93f5e15.log
**INFO: Authentication failed, it is either not correct, or expired, or does not have the correct permission PUT https://[stg_container_name].blob.core.windows/raw-test/2024/profiledata_GB_2024_06.csv.gz**
--------------------------------------------------------------------------------
RESPONSE 403: 403 Forbidden
ERROR CODE: CannotVerifyCopySource
--------------------------------------------------------------------------------
<?xml version="1.0" encoding="utf-8"?><Error><Code>CannotVerifyCopySource</Code><Message>Forbidden
RequestId:5ae5e3b8-401e-0063-7b44-63ea97000000
Time:2025-01-10T09:46:44.2555118Z</Message></Error>
--------------------------------------------------------------------------------
panic: close of nil channel
goroutine 176 [running]:
github/Azure/azure-storage-azcopy/v10/ste.(*jobMgr).reportJobPartDoneHandler(0xc0005fe008)
D:/a/1/s/ste/mgr-JobMgr.go:718 +0x327
created by github/Azure/azure-storage-azcopy/v10/ste.NewJobMgr in goroutine 1
D:/a/1/s/ste/mgr-JobMgr.go:203 +0xc88
this is the main error in the log file.
RESPONSE Status: 403 Forbidden
Content-Length: 200
Content-Type: application/xml
Date: Fri, 10 Jan 2025 10:22:33 GMT
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
X-Ms-Client-Request-Id: 57a6f2a5-919c-4084-6883-07a038e8a531
X-Ms-Error-Code: CannotVerifyCopySource
X-Ms-Request-Id: 91cc4b22-801e-0042-5149-63e5cf000000
X-Ms-Version: 2023-08-03
Response Details: CannotVerifyCopySourceForbidden
2025/01/10 10:22:33 ERR: [P#0-T#2] COPYFAILED: .csv.gz : 403 : 403 Forbidden. When Staging block from URL. X-Ms-Request-Id: 91cc4b22-801e-0042-5149-63e5cf000000
I'm encountering an issue with AzCopy when attempting to transfer data from Amazon S3 to Azure Data Lake Storage Gen2. The error message indicates that I lack PUT permission, even though I’ve verified that I have the necessary permissions in both AWS and ADLS. I created a SAS token with all possible permissions checked, and my AWS IAM policy (shown below) should also be sufficient. I’ve also set the AWS environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) in my command prompt:
set AWS_ACCESS_KEY_ID=123
set AWS_SECRET_ACCESS_KEY=ABC
Despite these measures, the error persists. Any guidance or troubleshooting steps would be greatly appreciated.
IAM policy
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation" ], "Resource": "arn:aws:s3:::s3_bucket" }, { "Effect": "Allow", "Action": [ "s3:GetObject" ], "Resource": "arn:aws:s3:::s3_bucket/dir/profiles/*" } ]}
Azcopy command
C:\Users\[user]>azcopy copy "https://s3.eu-west-1.amazonaws.com/s3_bucket/dir/dir/dir/dir/" "https://[stg_container_name].blob.core.windows.net/raw-test?[SASTOKEN]" --recursive=true
INFO: Scanning...
INFO: Failed to create one or more destination container(s). Your transfers may still succeed if the container already exists.
INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support
Job 67f9fe5e-f142-bf40-656c-ae0bf93f5e15 has started
Log file is located at: C:\Users\[user]\.azcopy\67f9fe5e-f142-bf40-656c-ae0bf93f5e15.log
**INFO: Authentication failed, it is either not correct, or expired, or does not have the correct permission PUT https://[stg_container_name].blob.core.windows.net/raw-test/2024/profiledata_GB_2024_06.csv.gz**
--------------------------------------------------------------------------------
RESPONSE 403: 403 Forbidden
ERROR CODE: CannotVerifyCopySource
--------------------------------------------------------------------------------
<?xml version="1.0" encoding="utf-8"?><Error><Code>CannotVerifyCopySource</Code><Message>Forbidden
RequestId:5ae5e3b8-401e-0063-7b44-63ea97000000
Time:2025-01-10T09:46:44.2555118Z</Message></Error>
--------------------------------------------------------------------------------
panic: close of nil channel
goroutine 176 [running]:
github.com/Azure/azure-storage-azcopy/v10/ste.(*jobMgr).reportJobPartDoneHandler(0xc0005fe008)
D:/a/1/s/ste/mgr-JobMgr.go:718 +0x327
created by github.com/Azure/azure-storage-azcopy/v10/ste.NewJobMgr in goroutine 1
D:/a/1/s/ste/mgr-JobMgr.go:203 +0xc88
this is the main error in the log file.
RESPONSE Status: 403 Forbidden
Content-Length: 200
Content-Type: application/xml
Date: Fri, 10 Jan 2025 10:22:33 GMT
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
X-Ms-Client-Request-Id: 57a6f2a5-919c-4084-6883-07a038e8a531
X-Ms-Error-Code: CannotVerifyCopySource
X-Ms-Request-Id: 91cc4b22-801e-0042-5149-63e5cf000000
X-Ms-Version: 2023-08-03
Response Details: CannotVerifyCopySourceForbidden
2025/01/10 10:22:33 ERR: [P#0-T#2] COPYFAILED: https://s3.eu-west-1.amazonaws.com/s3_bucket/dir/dir/dir/2024/profiledata_GB_2024_31.csv.gz : 403 : 403 Forbidden. When Staging block from URL. X-Ms-Request-Id: 91cc4b22-801e-0042-5149-63e5cf000000
Share Improve this question edited 20 hours ago Shoaib Maroof asked 21 hours ago Shoaib MaroofShoaib Maroof 3691 gold badge3 silver badges19 bronze badges 8 | Show 3 more comments1 Answer
Reset to default 0The issue was that the AWS IAM policy didn't have the "kms:Decrypt" permission for the KMS Encryption key associated with the S3 bucket
本文标签：
版权声明：本文标题：amazon web services - AzCopy S3 to ADLS Gen2 Transfer Failing with PUT Permission Error Despite Full Access - Stack Overflow 内容由网友自发贡献，该文观点仅代表作者本人， 转载请联系作者并注明出处：http://www.betaflare.com/web/1736579390a1944928.html， 本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌抄袭侵权/违法违规的内容，一经查实，本站将立刻删除。
SAStoken without signature? – Venkatesan Commented 20 hours ago--log-level=DEBUGflag to produce detailed logs that may provide more insight into the failure. – Venkatesan Commented 20 hours ago