python - Polars out of core sorting and memory usage - Stack Overflow

admin管理员组

文章数量:1345452

From what I understand this is a main use case for Polars: being able to process a dataset that is larger than RAM, using disk space if necessary. Yet I am unable to achieve this in a Kubernetes environment. To replicate locally I tried launching a docker container with a low memory limit:

docker run -it --memory=500m --rm  -v `pwd`:/app python:3.12  /bin/bash
# pip install polars==1.26.0

I checked that it set up the memory limit in cgroups for the current process. Then I ran a script that loads a moderately large dataframe (23M parquet file, 158M uncompressed), using scan_parquet, performs a sort, and outputs the head:

source = "parquet/central_west.df"
df = pl.scan_parquet(source, low_memory=True)
query = df.sort("station_code").head()
print(query.collect(engine="streaming"))

This leads to the process getting killed. It works with a smaller dataframe, or a larger limit. Is polars not reading the limit correctly? Or not able to work with that low of a limit? I understand the "new" streaming engine is still in beta, so I tried the same script with version 1.22.0 of polars, but the result was the same. This seems like a very simple and common use case so I hope I am just missing a configuration trick.

On a hunch and based on a similar question I tried setting POLARS_IDEAL_MORSEL_SIZE=100, but that made no difference, and I feel like I am grasping at straws here.

本文标签： pythonPolars out of core sorting and memory usageStack Overflow