hadoop2.x學習筆記

df -hl 查看磁盤剩余空間
hdfs 數據默認存儲路徑
data/tmp/dfs
data/
└── tmp
├── dfs
│ ├── data
│ ├── name
│ └── namesecondary

文件切分成塊（默認為128M）
默認塊的副本數是3
bin/hdfs dfsadmin
安裝maven環境
tar -zxf apache-maven-3.5.0-bin.tar.gz -C /opt/modules/
tar xzvf apache-maven-3.6.0-bin.tar.gz
設置環境變量
/opt/modules/apache-maven-3.5.0
vim /etc/profile
MAVEN_HOME=/opt/modules/apache-maven-3.5.0
PATH=$PATH:$MAVEN_HOME/bin
保存退出
source /etc/profile
查看maven 版本
mvn -version
設置快捷方式
/usr/share/applications/eclipse.desktop

查看默認啟動字符界面還是圖像化界面

systemctl get-default
設置默認啟動模式：
systemctl set-default graphical.target/multi-user.target

=================================================================================
MapReduce 數據類型
Long -> LongWritable
Int -> INtWritable
=================================================================================
過程
* step 1:
Input
InputFormat
* 讀取數據
* 轉換成<key,value>

讀取文件路徑
sudo bin/hdfs dfs -text /user/hadoop/mapreduce/wordcount/input/wc.input

public class HdfsApp {
/**
* Get FileSystem
* @return
* @throws Exception
*/
public static FileSystem getFileSystem() throws Exception {
// System.out.println( "Hello Hdfs!" );
Configuration conf = new Configuration();
// get filesystem
FileSystem fileSystem = FileSystem.get(conf);
//System.out.println(fileSystem);
return fileSystem;
}

public static void main(String[] args) throws Exception {

FileSystem fileSystem = getFileSystem();
String fileName = "/user/hadoop/mapreduce/wordcount/input/wc.input";
//readPath
Path readPath = new Path(fileName);
//open file
FSDataInputStream inStream = fileSystem.open(readPath);

try{
IOUtils.copyBytes(inStream, System.out, 4096,false);
}catch(Exception e){
e.printStackTrace();
}finally{
IOUtils.closeStream(inStream);
}
}

}

hadoop學習筆記二