hive mapreduce reducer 調優
阿新 • • 發佈：2019-02-08
本文針對 reducer 調優，主要涉及一下三個引數：
hive.exec.reducers.bytes.per.reducer
Default Value: 1,000,000,000 prior to Hive 0.14.0; 256 MB (256,000,000) in Hive 0.14.0 and later
Added In: Hive 0.2.0; default changed in 0.14.0 with HIVE-7158 (and HIVE-7917)
Size per reducer. The default in Hive 0.14.0 and earlier is 1 GB, that is, if the input size is 10 GB then 10 reducers will be used. In Hive 0.14.0 and later the default is 256 MB, that is, if the input size is 1 GB then 4 reducers will be used.
說明：每個reducer能夠處理的檔案大小。這個引數控制一個job會有多少個reducer來處理，依據的是輸入檔案的總大小。官方預設值：1G
檢視配置的預設值：
hive> set hive.exec.reducers.bytes.per.reducer;
hive.exec.reducers.bytes.per.reducer=1024000000臨時調參：
hive> set hive.exec.reducers.bytes.per.reducer=15364000000;
hive> set hive.exec.reducers.bytes.per.reducer;
hive.exec mapred.reduce.tasks
- Default Value: -1
- Added In: Hive 0.1.0
The default number of reduce tasks per job. Typically set to a prime close to the number of available hosts. Ignored when mapred.job.tracker is “local”. Hadoop set this to 1 by default, whereas Hive uses -1 as its default value. By setting this property to -1, Hive will automatically figure out what should be the number of reducers.
含義：設定每個job的reduce數。官方預設值為：-1
檢視配置的預設值：
hive> set mapred.reduce.tasks;
mapred.reduce.tasks=-1臨時調參：
hive> set mapred.reduce.tasks=100;
hive> set mapred.reduce.tasks;
mapred.reduce.tasks=100hive.exec.reducers.max
- Default Value: 999 prior to Hive 0.14.0; 1009 in Hive 0.14.0 and later
- Added In: Hive 0.2.0; default changed in 0.14.0 with HIVE-7158 (and HIVE-7917)
Maximum number of reducers that will be used. If the one specified in the configuration property mapred.reduce.tasks is negative, Hive will use this as the maximum number of reducers when automatically determining the number of reducers.
含義：設定最大的reduce數。如果mapreduce計算的reduce數超過設定值的值，則取設定的值。
檢視配置的預設值：
hive> set hive.exec.reducers.max;
hive.exec.reducers.max=1099臨時調參：
hive> set hive.exec.reducers.max=999;
hive> set hive.exec.reducers.max;
hive.exec.reducers.max=999以上所有set的值，都只是臨時調整，不會改變配置檔案中的配置。當你重新開啟hive會話，檢視相應的配置就一目瞭然了。