Dremio MAX DateTime Aggregation pushdown for ClickHouse - Stack Overflow

admin管理员组

文章数量:1278983

I'm trying to build a custom Dremio ARP ClickHouse connector.
What I can't achieve is to force Dremio to pushdown a query with max aggregation on ClickHouse DateTime column. Max aggregation works fine with other types that I defined - varchar or integer.. But not with DateTime..

Here is what I have in arp.yaml file:

data_types:
  mappings:
    - source:
        name: "DateTime"
      dremio:
        name: "timestamp"

    - source:
        name: "timestamp"
      dremio:
        name: "timestamp"

    - source:
        name: "DateTime32"
      dremio:
        name: "timestamp"

    - source:
        name: "DateTime64"
      dremio:
        name: "timestamp"
...
relational_algebra:
  aggregation:
    enable: true
    functions:
      - names:
          - "max"
          - "min"
        signatures:
          - args:
              - "integer"
            return: "integer"
          - args:
              - "double"
            return: "double"
          - args:
              - "varchar"
            return: "varchar"
          - args:
              - "date"
            return: "date"
          - args:
              - "timestamp"
            return: "timestamp"    

So it should work but it doesn't.. Dremio loads a whole dataset and applies the aggregation itself which is very slow in case of a huge dataset.

Can anyone please suggest. Hopefully, from Dremio team.

Thank you

本文标签： Dremio MAX DateTime Aggregation pushdown for ClickHouseStack Overflow