Hive Performance Tuning in Big Data Hadoop

1: Use Tez

set hive.execution.engine=tez;

2: Use ORCFile

CREATE TABLE A_ORC (customerID int, name string, age int, address string) STORED AS ORC tblproperties (“orc.compress" = “SNAPPY”);

3: Use Vectorization

  set hive.vectorized.execution.enabled = true;

  set hive.vectorized.execution.reduce.enabled = true;

4: cost based query optimization

set hive.cbo.enable=true;

set hive.compute.query.using.stats=true;

set hive.stats.fetch.column.stats=true;

set hive.stats.fetch.partition.stats=true;

5: Write good SQL