How to split Hive reducers output into multiple files - Stack Overflow

admin管理员组

文章数量:1122846

I inserted data into an Hive table as follow :

SET hive.execution.engine=tez;
SET hive.exec.dynamic.partition=true;
SET hive.exec.dynamic.partition.mode=nonstrict;

INSERT INTO TABLE db.my_table
PARTITION(KEY1,KEY2) 
SELECT * FROM db.my_table_temp  
DISTRIBUTE BY KEY1,KEY2

using the DISTRIBUTE BY KEY1,KEY2 clause prevent Hive from creating multiple small files since data with the same value of (KEY1,KEY2) are all sent to the same reducer before being written on Hive table.

I am now facing the opposite problem : Hive is now creating huge partition files (~8 Gb). Instead I would like each reducer to split its output in order to write multiples files of 500 Mb each.

Is there an hive setting allowing to split reducers output ?

PS : I am using Hive 1.2 on top of Hadoop

本文标签： How to split Hive reducers output into multiple filesStack Overflow