1. 程式人生 > >Windows7作業系統下使用Eclipse來搭建hadoop開發環境

Windows7作業系統下使用Eclipse來搭建hadoop開發環境

[main] WARN org.apache.hadoop.util.NativeCodeLoader- Unable to load native-hadoop library for your platform... using builtin-javaclasses where applicable

[main] INFOorg.apache.hadoop.conf.Configuration.deprecation - session.id is deprecated.Instead, use dfs.metrics.session-id

[main] INFOorg.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics withprocessName=JobTracker, sessionId=

[main] WARNorg.apache.hadoop.mapreduce.JobResourceUploader - No job jar file set.  User classes may not be found. See Job orJob#setJar(String).

[main] INFOorg.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths toprocess : 1

[main] INFOorg.apache.hadoop.mapreduce.JobSubmitter - number of splits:1

[main] INFOorg.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job:job_local1072922152_0001

[main] INFOorg.apache.hadoop.mapreduce.JobSubmitter - Cleaning up the staging areafile:/tmp/hadoop-Cyril/mapred/staging/Cyril1072922152/.staging/job_local1072922152_0001

Exception in thread "main"java.lang.UnsatisfiedLinkError:org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z

      atorg.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)

      atorg.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:609)

      atorg.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:977)

      atorg.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:187)

      atorg.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:174)

      atorg.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:108)

      atorg.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:285)

      atorg.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)

      atorg.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)

      atorg.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)

      atorg.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)

      atorg.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:125)

      atorg.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:163)

      atorg.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731)

      atorg.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240)

      atorg.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)

      atorg.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)

      atjava.security.AccessController.doPrivileged(Native Method)

      atjavax.security.auth.Subject.doAs(Unknown Source)

      atorg.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)

      atorg.apache.hadoop.mapreduce.Job.submit(Job.java:1287)

      atorg.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)

      atWordCount.main(WordCount.java:61)

可能原因:

1、[main] WARNorg.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop libraryfor your platform... using builtin-java classes where applicable,由這條日誌資訊表明缺少hadoop動態庫(Hadoop.dll)

解決辦法:

在網上下載hadoop.dll,將其拷貝到c:\windows\System32\資料夾下即可。

12、程式打包

右鍵單擊工程,點開“Export…”在彈出的對話方塊中選擇“JAR file”,如下圖:

點選“Next”,彈出如下對話方塊。

勾選如圖所示選項,單擊“Browse”新增“JAR file”的輸出路徑,然後單擊“Next”按鈕,彈出如下對話方塊:

單擊“Next”,彈出如下對話方塊:

單擊“Browse”按鈕新增Main class,如下圖所示:
單擊“OK”

單擊“Finish”。

將生成的“WordCount.jar”檔案上傳到叢集的“/srv/ftp”目錄下,然後通過以下命令執行程式:

hadoopjar /srv/ftp/WordCount.jar /input /output