python - snakemake public remote s3 without secret key - Stack Overflow

admin管理员组

文章数量:1394990

I am using snakemake with rule need visit public read-only s3 storage

# test
$ aws s3 ls 1000genomes/phase1/phase1.exome.alignment.index.bas.gz --no-sign-request
2012-05-01 03:58:41     423691 phase1.exome.alignment.index.bas.gz
$ aws s3 cp s3://1000genomes/phase1/phase1.exome.alignment.index.bas.gz ./ --no-sign-request
download: s3://1000genomes/phase1/phase1.exome.alignment.index.bas.gz to ./phase1.exome.alignment.index.bas.gz
$ ls phase1.exome.alignment.index.bas.gz
phase1.exome.alignment.index.bas.gz
# public visit ok

# snakemake
storage:
    provider="s3",
    max_requests_per_second=10

rule download_data:
    input:
        vcf=lambda wildcards: storage.s3("s3://1000genomes/phase1/phase1.exome.alignment.index.bas.gz")
    output:
        'output/data/phase1.exome.alignment.index.bas.gz'
    shell:
        # just a example here, aws s3 cp can instead, but we have more complex process
        "cp {input.vcf} {output}"

snakemake -s /path/to/snakemake.smk
The following required arguments are missing for plugin s3: --storage-s3-access-key (or environment variable SNAKEMAKE_STORAGE_S3_ACCESS_KEY), --storage-s3-secret-key (or environment variable SNAKEMAKE_STORAGE_S3_SECRET_KEY).

I treid to set SNAKEMAKE_STORAGE_S3_ACCESS_KEY and SNAKEMAKE_STORAGE_S3_SECRET_KEY environment to ''.

export SNAKEMAKE_STORAGE_S3_ACCESS_KEY=''
export SNAKEMAKE_STORAGE_S3_SECRET_KEY=''

but its seems not work.

snakemake -s /path/to/snakemake.smk
Failed to check existence of s3://1000genomes/phase1/phase1.exome.alignment.index.bas.gz
RuntimeError: 34 (AWS_ERROR_INVALID_ARGUMENT): An invalid argument was passed to a function.

本文标签： pythonsnakemake public remote s3 without secret keyStack Overflow