Spark生態之Alluxio學習25--spark從HDFS和Alluxio讀取時間比較
阿新 • • 發佈:2018-12-25
Spark生態之Alluxio學習 版本:alluxio-1.3.0(tachyon),spark-1.5.2,hadoop-2.6.0
1.解釋
想要分析alluxio加速效果,發現alluxio會出現長尾效應,導致有些task特別耗時,相對於HDFS,並沒有明顯優勢。
2.程式碼:
#~/cloud/alluxio-1.3.0/bin/alluxio fs mkdir -p /xubo/project/SparkSW/input/ for i in 1 2 3 4 5 6 7 8 9 10 do echo 'time:'$i for j in 1 2 3 4 5 6 7 8 9 10 do sh test.sh 10G hdfs://Master:9000/xubo/project/SparkSW/input/'D'$i'Line.fasta' hdfs://Master:9000/xubo/project/alluxio/output/TimeCount #sh test.sh 10G alluxio://Master:19998/xubo/alluxio/uniref/'D'$j'Line.fasta' hdfs://Master:9000/xubo/project/alluxio/output/TimeCount #sh test.sh 10G alluxio://Master:19998/xubo/project/SparkSW/input/'D'$j'Line.fasta' alluxio://Master:19998/xubo/project/alluxio/output/TimeCount done for j in 1 2 3 4 5 6 7 8 9 10 do sh test.sh 10G alluxio://Master:19998/xubo/project/SparkSW/input/'D'$i'Line.fasta' hdfs://Master:9000/xubo/project/alluxio/output/TimeCount done done
3.結果:
[email protected]:~/disk2/xubo/project/alluxio/timeCount$ tail -f runtime201611231839.txt time:1 input:hdfs://Master:9000/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:3.918s input:hdfs://Master:9000/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:3.923s input:hdfs://Master:9000/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:3.848s input:hdfs://Master:9000/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:3.959s input:hdfs://Master:9000/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:3.85s input:hdfs://Master:9000/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:3.816s input:hdfs://Master:9000/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:4.208s input:hdfs://Master:9000/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:3.925s input:hdfs://Master:9000/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:3.001s input:hdfs://Master:9000/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:2.817s input:alluxio://Master:19998/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:3.466s input:alluxio://Master:19998/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:2.83s input:alluxio://Master:19998/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:2.757s input:alluxio://Master:19998/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:2.824s input:alluxio://Master:19998/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:2.813s input:alluxio://Master:19998/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:2.802s input:alluxio://Master:19998/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:2.847s input:alluxio://Master:19998/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:2.88s input:alluxio://Master:19998/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:3.963s input:alluxio://Master:19998/xubo/project/SparkSW/input/D1Line.fasta count:78295 countTime:2.788s time:2 input:hdfs://Master:9000/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.074s input:hdfs://Master:9000/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:2.945s input:hdfs://Master:9000/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.023s input:hdfs://Master:9000/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.999s input:hdfs://Master:9000/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.073s input:hdfs://Master:9000/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:4.206s input:hdfs://Master:9000/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.028s input:hdfs://Master:9000/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:4.049s input:hdfs://Master:9000/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.927s input:hdfs://Master:9000/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.964s input:alluxio://Master:19998/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.875s input:alluxio://Master:19998/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:6.505s input:alluxio://Master:19998/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.001s input:alluxio://Master:19998/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.276s input:alluxio://Master:19998/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.047s input:alluxio://Master:19998/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.081s input:alluxio://Master:19998/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.365s input:alluxio://Master:19998/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.083s input:alluxio://Master:19998/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.141s input:alluxio://Master:19998/xubo/project/SparkSW/input/D2Line.fasta count:156590 countTime:3.027s time:3 input:hdfs://Master:9000/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.572s input:hdfs://Master:9000/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.45s input:hdfs://Master:9000/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.342s input:hdfs://Master:9000/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.336s input:hdfs://Master:9000/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.508s input:hdfs://Master:9000/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.268s input:hdfs://Master:9000/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.558s input:hdfs://Master:9000/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.401s input:hdfs://Master:9000/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.457s input:hdfs://Master:9000/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.513s input:alluxio://Master:19998/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:5.515s input:alluxio://Master:19998/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.846s input:alluxio://Master:19998/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.215s input:alluxio://Master:19998/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.242s input:alluxio://Master:19998/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.181s input:alluxio://Master:19998/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.236s input:alluxio://Master:19998/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:3.047s input:alluxio://Master:19998/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.178s input:alluxio://Master:19998/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.24s input:alluxio://Master:19998/xubo/project/SparkSW/input/D3Line.fasta count:313180 countTime:4.178s time:4 input:hdfs://Master:9000/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:7.852s input:hdfs://Master:9000/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:6.148s input:hdfs://Master:9000/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:7.575s input:hdfs://Master:9000/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:16.612s input:hdfs://Master:9000/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:8.009s input:hdfs://Master:9000/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:8.02s input:hdfs://Master:9000/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:7.937s input:hdfs://Master:9000/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:8.007s input:hdfs://Master:9000/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:16.949s input:hdfs://Master:9000/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:7.795s input:alluxio://Master:19998/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:7.182s input:alluxio://Master:19998/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:4.506s input:alluxio://Master:19998/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:3.463s input:alluxio://Master:19998/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:3.501s input:alluxio://Master:19998/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:3.544s input:alluxio://Master:19998/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:3.747s input:alluxio://Master:19998/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:3.523s input:alluxio://Master:19998/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:3.4s input:alluxio://Master:19998/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:3.325s input:alluxio://Master:19998/xubo/project/SparkSW/input/D4Line.fasta count:626360 countTime:3.519s time:5 input:hdfs://Master:9000/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:3.988s input:hdfs://Master:9000/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:4.004s input:hdfs://Master:9000/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:5.494s input:hdfs://Master:9000/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:4.916s input:hdfs://Master:9000/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:5.479s 16/11/23 18:56:20 WARN QueuedThreadPool: 4 threads could not be stopped input:hdfs://Master:9000/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:3.891s input:hdfs://Master:9000/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:4.73s input:hdfs://Master:9000/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:4.851s input:hdfs://Master:9000/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:4.9s input:hdfs://Master:9000/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:4.966s input:alluxio://Master:19998/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:7.924s input:alluxio://Master:19998/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:6.396s input:alluxio://Master:19998/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:3.487s input:alluxio://Master:19998/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:5.417s input:alluxio://Master:19998/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:4.272s input:alluxio://Master:19998/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:3.465s input:alluxio://Master:19998/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:3.437s input:alluxio://Master:19998/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:4.067s input:alluxio://Master:19998/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:6.194s input:alluxio://Master:19998/xubo/project/SparkSW/input/D5Line.fasta count:1252720 countTime:5.352s time:6 input:hdfs://Master:9000/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:4.883s input:hdfs://Master:9000/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:4.119s 16/11/23 18:59:47 WARN QueuedThreadPool: 4 threads could not be stopped input:hdfs://Master:9000/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:4.172s input:hdfs://Master:9000/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:5.09s input:hdfs://Master:9000/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:7.57s input:hdfs://Master:9000/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:4.578s input:hdfs://Master:9000/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:7.317s input:hdfs://Master:9000/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:4.857s 16/11/23 19:01:03 WARN QueuedThreadPool: 1 threads could not be stopped input:hdfs://Master:9000/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:4.854s 16/11/23 19:01:16 WARN QueuedThreadPool: 1 threads could not be stopped input:hdfs://Master:9000/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:3.994s input:alluxio://Master:19998/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:15.504s input:alluxio://Master:19998/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:6.336s input:alluxio://Master:19998/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:6.456s input:alluxio://Master:19998/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:4.951s 16/11/23 19:02:27 WARN QueuedThreadPool: 2 threads could not be stopped input:alluxio://Master:19998/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:5.525s input:alluxio://Master:19998/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:15.943s input:alluxio://Master:19998/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:4.665s input:alluxio://Master:19998/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:4.814s input:alluxio://Master:19998/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:15.965s input:alluxio://Master:19998/xubo/project/SparkSW/input/D6Line.fasta count:2505440 countTime:4.915s time:7 input:hdfs://Master:9000/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:9.814s input:hdfs://Master:9000/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:9.196s input:hdfs://Master:9000/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:9.434s input:hdfs://Master:9000/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:9.172s input:hdfs://Master:9000/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:20.193s input:hdfs://Master:9000/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:10.454s input:hdfs://Master:9000/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:9.551s input:hdfs://Master:9000/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:9.321s input:hdfs://Master:9000/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:10.313s input:hdfs://Master:9000/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:9.707s input:alluxio://Master:19998/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:52.331s input:alluxio://Master:19998/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:6.583s input:alluxio://Master:19998/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:16.998s input:alluxio://Master:19998/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:16.55s input:alluxio://Master:19998/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:6.496s input:alluxio://Master:19998/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:6.781s input:alluxio://Master:19998/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:7.477s input:alluxio://Master:19998/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:17.51s input:alluxio://Master:19998/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:6.494s input:alluxio://Master:19998/xubo/project/SparkSW/input/D7Line.fasta count:5019006 countTime:17.511s time:8 input:hdfs://Master:9000/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:10.427s input:hdfs://Master:9000/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:9.993s input:hdfs://Master:9000/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:11.242s input:hdfs://Master:9000/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:10.868s input:hdfs://Master:9000/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:7.354s input:hdfs://Master:9000/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:7.015s input:hdfs://Master:9000/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:7.344s input:hdfs://Master:9000/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:7.182s input:hdfs://Master:9000/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:7.262s input:hdfs://Master:9000/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:8.282s input:alluxio://Master:19998/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:42.406s input:alluxio://Master:19998/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:16.769s input:alluxio://Master:19998/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:18.197s input:alluxio://Master:19998/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:7.881s input:alluxio://Master:19998/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:8.913s input:alluxio://Master:19998/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:8.174s input:alluxio://Master:19998/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:20.971s input:alluxio://Master:19998/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:8.434s input:alluxio://Master:19998/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:8.316s input:alluxio://Master:19998/xubo/project/SparkSW/input/D8Line.fasta count:10038012 countTime:16.767s time:9 input:hdfs://Master:9000/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:32.624s input:hdfs://Master:9000/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:19.688s input:hdfs://Master:9000/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:17.345s input:hdfs://Master:9000/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:20.98s input:hdfs://Master:9000/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:13.478s input:hdfs://Master:9000/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:27.69s input:hdfs://Master:9000/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:15.567s input:hdfs://Master:9000/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:14.237s input:hdfs://Master:9000/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:16.413s input:hdfs://Master:9000/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:10.537s input:alluxio://Master:19998/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:99.776s input:alluxio://Master:19998/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:28.482s input:alluxio://Master:19998/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:29.184s input:alluxio://Master:19998/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:17.949s input:alluxio://Master:19998/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:17.29s input:alluxio://Master:19998/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:28.279s input:alluxio://Master:19998/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:30.892s input:alluxio://Master:19998/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:18.895s input:alluxio://Master:19998/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:28.351s input:alluxio://Master:19998/xubo/project/SparkSW/input/D9Line.fasta count:20076024 countTime:20.001s time:10 input:hdfs://Master:9000/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:38.227s input:hdfs://Master:9000/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:25.727s input:hdfs://Master:9000/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:23.691s input:hdfs://Master:9000/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:19.047s input:hdfs://Master:9000/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:19.611s input:hdfs://Master:9000/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:14.391s input:hdfs://Master:9000/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:16.036s input:hdfs://Master:9000/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:16.249s input:hdfs://Master:9000/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:15.324s input:hdfs://Master:9000/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:13.936s input:alluxio://Master:19998/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:151.861s input:alluxio://Master:19998/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:46.322s input:alluxio://Master:19998/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:31.237s input:alluxio://Master:19998/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:31.318s input:alluxio://Master:19998/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:19.097s input:alluxio://Master:19998/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:19.043s input:alluxio://Master:19998/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:30.663s input:alluxio://Master:19998/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:43.809s input:alluxio://Master:19998/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:20.463s input:alluxio://Master:19998/xubo/project/SparkSW/input/D10Line.fasta count:40152048 countTime:31.099s
參考
【1】https://github.com/Alluxio/alluxio
【2】http://www.alluxio.org/
【3】http://spark.apache.org/docs/1.5.2/programming-guide.html
【4】https://github.com/xubo245/SparkLearning