linux常用命令 pwd 檢視當前工作目錄的絕對路徑 cat input.txt 檢視input.txt檔案的內容 ls 顯示當前目錄下所有的檔案及子目錄 rm recommender-dm-1.0-SNAPSHOT-lib.jar 刪除當前目錄下recommender-dm-1.0-SNAPSHOT-lib.jar檔案 cp /home/deploy/pctr/recommender-dm_fat.jar ./ 把/home/deploy/pctr/目錄下的recommender-dm_fat.jar複製到當前目錄下 rm -rf 0000* 強行直接刪除（不作任何提示）所有字首為0000的檔案 rm -rf 刪除資料夾 rm -r：向下遞迴，不管有多少級目錄，一併刪除 -f：直接強行刪除，不做任何提示的意思 rz 從客戶端向伺服器上傳檔案（receive：伺服器接收） sz 從伺服器下載檔案（send：伺服器傳送） hadoop hdfs常用命令 hadoop fs -ls /user/deploy/recsys/workspace/ouyangyewei 檢視ouyangyewei目錄檔案 hadoop fs -mkdir /user/deploy/recsys/workspace/ouyangyewei/input 在ouyangyewei目錄下建立input資料夾 hadoop fs -rm /user/deploy/recsys/workspace/ouyangyewei/input/input.txt 刪除input.txt檔案 hadoop fs -rmr /user/deploy/recsys/workspace/ouyangyewei/input 刪除input目錄以及目錄下的所有檔案 hadoop fs -put ./input.txt /home/deploy/recsys/workspace/ouyangyewei/input 把當前目錄下的input.txt檔案複製到input目錄下 hadoop fs -dus /data/share/trackinfo/ds=2014-05-12 檢視檔案 “/data/share/trackinfo/ds=2014-05-12”的大小（以位元組為單位） hadoop jar recommender-dm-1.0-SNAPSHOT-lib.jar com.yhd.ml.statistics.click.WordCount /home/deploy/recsys/workspace/ouyangyewei/input /home/deploy/recsys/workspace/ouyangyewei/output 執行Job，指定的jar包是recommender-dm-1.0-SNAPSHOT-lib.jar，主類是com.yhd.ml.statistics.click.WordCount，輸入目錄是input，輸出目錄是output hadoop job -kill job_201403291618_274044 殺掉hadoop的job hbase常用命令 /usr/local/cloud/hbase/bin/hbase shell 用shell來連線hbase exit 退出hbase shell version 檢視hbase版本 hbase(main):045:0> is_enabled 't1'
true                                                                                                                                                                            
0 row(s) in 0.0020 seconds 測試表t1是否有效 hbase(main):046:0> is_disabled 't1'
false                                                                                                                                                                           
0 row(s) in 0.0010 seconds 測試表t1是否無效 hbase(main):044:0> exists 't1'
Table t1 does exist                                                                                                                                                             
0 row(s) in 0.0270 seconds 測試表t1是否存在 scan 'full_user_profile', {LIMIT=>1} 輸出資料表full_user_profile中的1個RowKey list 列出所有資料表 describe 'full_user_profile' 列出full_user_profile資料表的結構 hbase(main):003:0> disable 'score'
0 row(s) in 2.1080 seconds 使資料表score無效

hbase(main):004:0> drop 'score'
0 row(s) in 10.6740 seconds 刪除資料表score（注意在刪除表之前要使表無效） ------------------------------------------------------------------------- hbase(main):013:0> create 'score', 'name', 'course'
0 row(s) in 5.1050 seconds 建立資料表score，其中name是Row Key，course是列族

hbase(main):014:0> put 'score', 'xiaowen', 'course:China', '95'
0 row(s) in 33.4270 seconds 在列族course下建立列China，值為95

hbase(main):015:0> put 'score', 'xiaowen', 'course:Math', '99'
0 row(s) in 0.0130 seconds 在列族course下建立列Math，值為99
hbase(main):016:0> put 'score', 'xiaowen', 'course:English', '98'
0 row(s) in 0.0040 seconds
在列族course下建立列English，值為98 hbase(main):017:0> scan 'score'
ROW                                           COLUMN+CELL                                                                                                                       
xiaowen                                      column=course:China, timestamp=1400141524101, value=95                                                                            
xiaowen                                      column=course:English, timestamp=1400141591123, value=98                                                                          
xiaowen                                      column=course:Math, timestamp=1400141579107, value=99                                                                             
1 row(s) in 0.0250 seconds 檢視score整張表的資料 hbase(main):018:0> get 'score', 'xiaowen'
COLUMN                                        CELL                                                                                                                              
course:China                                 timestamp=1400141524101, value=95                                                                                                 
course:English                               timestamp=1400141591123, value=98                                                                                                 
course:Math                                  timestamp=1400141579107, value=99                                                                                                 
3 row(s) in 0.0110 seconds
檢視score表的xiaowen行資料 hbase(main):019:0> get 'score', 'xiaowen', 'course:Math'
COLUMN                                        CELL                                                                                                                              
course:Math                                  timestamp=1400141579107, value=99                                                                                                 
1 row(s) in 0.0070 seconds 檢視score表的xiaowen行，course列族上的Math列的資料 hbase(main):008:0> scan 'score'
ROW                                           COLUMN+CELL                                                                                                                       
xiaowen                                      column=course:China, timestamp=1400141524101, value=95                                                                            
xiaowen                                      column=course:English, timestamp=1400141591123, value=98                                                                          
xiaowen                                      column=course:Math, timestamp=1400141579107, value=99                                                                             
xiaoye                                       column=course:China, timestamp=1400143888087, value=85                                                                            
xiaoye                                       column=course:English, timestamp=1400143921395, value=85                                                                          
xiaoye                                       column=course:Math, timestamp=1400143907407, value=85                                                                             
2 row(s) in 0.0240 seconds 檢視score表的所有值 hbase(main):013:0> scan 'score', {COLUMNS=>'course'}
ROW                                           COLUMN+CELL                                                                                                                       
xiaowen                                      column=course:China, timestamp=1400141524101, value=95                                                                            
xiaowen                                      column=course:English, timestamp=1400141591123, value=98                                                                          
xiaowen                                      column=course:Math, timestamp=1400141579107, value=99                                                                             
xiaoye                                       column=course:China, timestamp=1400143888087, value=85                                                                            
xiaoye                                       column=course:English, timestamp=1400143921395, value=85                                                                          
xiaoye                                       column=course:Math, timestamp=1400143907407, value=85                                                                             
2 row(s) in 0.0230 seconds 檢視score表的course列的所有值

hbase(main):014:0> scan 'score', {COLUMNS=>'course:Math'}
ROW                                           COLUMN+CELL                                                                                                                       
xiaowen                                      column=course:Math, timestamp=1400141579107, value=99                                                                             
xiaoye                                       column=course:Math, timestamp=1400143907407, value=85                                                                             
2 row(s) in 0.0270 seconds 檢視score表course:Math列的所有值 hbase(main):021:0> count 'score'
2 row(s) in 0.1880 seconds 統計score表有多少行 -------------------------------------------------------------------- test表的實踐 hbase(main):022:0> create 'test', 'c1', 'c2'
0 row(s) in 1.1260 seconds

hbase(main):023:0> put 'test', 'r1', 'c1:1', 'value1-1/1'
0 row(s) in 0.0360 seconds

hbase(main):024:0> put 'test', 'r1', 'c1:2', 'value1-1/2'
0 row(s) in 0.0210 seconds

hbase(main):025:0> put 'test', 'r1', 'c1:3', 'value1-1/3'
0 row(s) in 0.0170 seconds

hbase(main):026:0> put 'test', 'r1', 'c2:1', 'value1-2/1'
0 row(s) in 0.0100 seconds

hbase(main):027:0> put 'test', 'r1', 'c2:2', 'value1-2/2'
0 row(s) in 0.0060 seconds

hbase(main):028:0> put 'test', 'r2', 'c1:1', 'value2-1/1'
0 row(s) in 0.0110 seconds

hbase(main):029:0> put 'test', 'r2', 'c2:1', 'value2-2/1'
0 row(s) in 0.0080 seconds hbase(main):030:0> scan 'test'
ROW                                           COLUMN+CELL                                                                                                                        
r1                                           column=c1:1, timestamp=1400152716678, value=value1-1/1                                                                             
r1                                           column=c1:2, timestamp=1400152749600, value=value1-1/2                                                                             
r1                                           column=c1:3, timestamp=1400152770555, value=value1-1/3                                                                             
r1                                           column=c2:1, timestamp=1400152793839, value=value1-2/1                                                                             
r1                                           column=c2:2, timestamp=1400152811436, value=value1-2/2                                                                             
r2                                           column=c1:1, timestamp=1400152843148, value=value2-1/1                                                                             
r2                                           column=c2:1, timestamp=1400152858073, value=value2-2/1                                                                             
2 row(s) in 0.0490 seconds hbase(main):031:0> describe 'test'
DESCRIPTION                                                                                                        ENABLED                                                       
{NAME => 'test', FAMILIES => [{NAME => 'c1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SC true                                                          
OPE => '0', VERSIONS => '3', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS                                                                
=> 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true'}, {NAME =>                                                               
  'c2', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSI                                                               
ON => 'NONE', MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_M                                                               
EMORY => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true'}]}                                                                                                             
1 row(s) in 0.2560 seconds 從describe命令可見test表有兩個列族 hive常用命令 show tables; 列出hive裡面所有資料表名 desc userProfile; 顯示資料表userProfile的基本表字段及欄位type desc extended trackinfo; 顯示資料表trackinfo的詳細資訊，包括欄位說明，資料表等 /usr/local/cloud/hive/bin/hive 進入hive資料庫 select attribute_name from pms_attribute where attribute_id=21000 and attribute_value_id=105991; hive的select操作 select user_id, category_id, catgory_pref, attribute_id, attribute_pref, attribute_value_id, attribute_value_pref from userProfile limit 10; hive的select操作，只顯示10行 /usr/local/cloud/hive/bin/hive -e "select category_id, attribute_id, count(user_id) from userProfile group by category_id, attribute_id" >> /home/deploy/recsys/workspace/ouyangyewei/statistics_data/number_attention_of_attribute_for_mobilePhone.csv; 將sql查詢的資料匯出到csv檔案中