hive 引數優化之hive.auto.convert.join
Total MapReduce jobs = 1
14/08/24 20:29:11 WARN conf.Configuration: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
14/08/24 20:29:11 WARN conf.Configuration: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
14/08/24 20:29:11 WARN conf.Configuration: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
14/08/24 20:29:11 WARN conf.Configuration: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
14/08/24 20:29:11 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/08/24 20:29:11 WARN conf.Configuration: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
14/08/24 20:29:12 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
Execution log at: /tmp/admin/.log
2014-08-24 08:29:12 Starting to launch local task to process map join; maximum memory = 503447552
2014-08-24 08:29:15 Processing rows: 200000 Hashtable size: 199999 Memory usage: 192705744 percentage: 0.383
2014-08-24 08:29:15 Processing rows: 300000 Hashtable size: 299999 Memory usage: 310744000 percentage: 0.617
2014-08-24 08:29:16 Processing rows: 400000 Hashtable size: 399999 Memory usage: 398019256 percentage: 0.791
2014-08-24 08:29:31 Processing rows: 500000 Hashtable size: 499999 Memory usage: 456595024 percentage: 0.907
Execution failed with exit status: 3
Obtaining error information
Task failed!
Task ID:
Stage-5
Logs:
/export/Logs/ode/admin/hive.log
FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
解決方法：
set hive.mapjoin.localtask.max.memory.usage=0.99
hive.mapjoin.localtask.max.memory.usage
•說明：本地任務可以使用記憶體的百分比 •預設值： 0.90，如果你的localtask mapjoin 表很小可以試試，但徹底解決需要
set hive.auto.convert.join=false;關閉自動mapjoin 但這個引數用的時候一定要注意，
如果你的sql 很長join會常多，關閉mapjoin任務數會成10倍激增，contener滿了任務同樣會非常之慢，set hive.auto.convert.join=false;一定要用在localtask級別這種超輕量及的job上。
--------------------- 本文來自 oaimm 的CSDN 部落格 ，全文地址請點選：https://blog.csdn.net/oaimm/article/details/38846599?utm_source=copy