Is there a way to create Snowpipe or Ingestion in Snowflake for a public Azure Blob dataset? - Stack Overflow

admin管理员组

文章数量:1375859

I have done Snowpipe before for a azure Blob before, but since it was private I had access to tenant_id and all authentications. Maybe the question seems backward but when a storage is public, Can we create a data pipeline based on it? I've ingested the data to an external storage in Snowflake, and from there tried to create a pipe to copy data from External to raw table. It still ask for credentials.

    CREATE OR REPLACE STAGE <name>_stage
    URL = 'azure://azureopendatastorage.blob.core.windows/<>'
    FILE_FORMAT = (TYPE = PARQUET);

    CREATE OR REPLACE PIPE raw_pipe
    AUTO_INGEST = TRUE
    AS
    COPY INTO nyc_raw
    FROM @<name>_stage
    PATTERN = '.*\.parquet$'
    FILE_FORMAT = (TYPE = 'PARQUET')
    ON_ERROR = 'CONTINUE';

I have done Snowpipe before for a azure Blob before, but since it was private I had access to tenant_id and all authentications. Maybe the question seems backward but when a storage is public, Can we create a data pipeline based on it? I've ingested the data to an external storage in Snowflake, and from there tried to create a pipe to copy data from External to raw table. It still ask for credentials.

    CREATE OR REPLACE STAGE <name>_stage
    URL = 'azure://azureopendatastorage.blob.core.windows/<>'
    FILE_FORMAT = (TYPE = PARQUET);

    CREATE OR REPLACE PIPE raw_pipe
    AUTO_INGEST = TRUE
    AS
    COPY INTO nyc_raw
    FROM @<name>_stage
    PATTERN = '.*\.parquet$'
    FILE_FORMAT = (TYPE = 'PARQUET')
    ON_ERROR = 'CONTINUE';

• azure
• snowflake-cloud-data-platform
• blob
• pipeline
• snowpipe

Share Improve this question Follow asked Mar 27 at 16:25 MinaBMinaB 4111 silver badge33 bronze badges

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

AUTO_INGEST requires a Snowflake notification integration that is linked to an Azure Storage queue. This is why it's asking for credentials.

It's still possible to create a snowpipe but you should use AUTO_INGEST=FALSE, and run refresh commands by a task (or any other scheduler).

  CREATE OR REPLACE PIPE raw_pipe
    AUTO_INGEST = FALSE
    AS
    COPY INTO nyc_raw
    FROM @my_stage
    PATTERN = '.*\.parquet$'
    FILE_FORMAT = (TYPE = 'PARQUET')
    ON_ERROR = 'CONTINUE';

本文标签： Is there a way to create Snowpipe or Ingestion in Snowflake for a public Azure Blob datasetStack Overflow