Changes between Initial Version and Version 1 of FDC110829/Lab2

Timestamp:

Aug 28, 2011, 11:00:22 PM (13 years ago)

Author:

jazz

Comment:

--

FDC110829/Lab2

@@ +1 @@
+[[PageOutline]]
+
+◢ <[wiki:FDC110829/Lab1 實作一]> | <[wiki:FDC110829 回課程大綱]> ▲ | <[wiki:FDC110829/Lab3 實作三]> ◣
+
+= 實作二 Lab 2 =
+
+{{{
+#!html
+<div style="text-align: center;"><big style="font-weight: bold;"><big>HDFS Shell操作練習<br/>HDFS Shell in practice</big></big></div>
+}}}
+
+{{{
+#!text
+以下練習，請連線至 hadoop.nchc.org.tw 操作。底下的 hXXXX 等於您的用戶名稱。
+}}}
+
+== Content 1: HDFS Shell 基本操作 ==
+== Content 1: Basic HDFS Shell Commands ==
+
+=== 1.1 瀏覽你HDFS目錄 ===
+=== 1.1 Browsing Your HDFS Folder ===
+
+{{{
+~$ hadoop fs -ls
+Found 1 items
+drwxr-xr-x   - hXXXX supergroup          0 2010-01-24 17:23 /user/hXXXX/tmp
+~$ hadoop fs -lsr
+drwxr-xr-x   - hXXXX supergroup          0 2010-01-24 17:23 /user/hXXXX/tmp
+}}}
+
+=== 1.2 上傳資料到 HDFS 目錄 ===
+=== 1.2 Upload Files or Folder to HDFS ===
+
+ * 上傳 Upload
+
+{{{
+~$ hadoop fs -put /etc/hadoop/conf input
+}}} 
+
+ * 檢查 Check
+
+{{{
+~$ hadoop fs -ls
+Found 2 items
+drwxr-xr-x   - hXXXX supergroup          0 2011-04-19 09:16 /user/hXXXX/input
+drwxr-xr-x   - hXXXX supergroup          0 2010-01-24 17:23 /user/hXXXX/tmp
+~$ hadoop fs -ls input
+Found 25 items
+-rw-r--r--   2 hXXXX supergroup        321 2011-04-19 09:16 /user/hXXXX/input/README
+-rw-r--r--   2 hXXXX supergroup       3936 2011-04-19 09:16 /user/hXXXX/input/capacity-scheduler.xml
+-rw-r--r--   2 hXXXX supergroup        196 2011-04-19 09:16 /user/hXXXX/input/commons-logging.properties
+(.... skip ....)
+}}}
+ 
+=== 1.3 下載 HDFS 的資料到本地目錄 ===
+=== 1.3 Download HDFS Files or Folder to Local ===
+
+ * 下載 Download
+
+{{{
+~$ hadoop fs -get input fromHDFS
+}}} 
+
+ * 檢查 Check
+{{{
+~$ ls -al | grep fromHDFS
+drwxr-xr-x    2 hXXXX hXXXX  4096 2011-04-19 09:18 fromHDFS
+~$ ls -al fromHDFS
+總計 160
+drwxr-xr-x 2 hXXXX hXXXX  4096 2011-04-19 09:18 .
+drwx--x--x 3 hXXXX hXXXX  4096 2011-04-19 09:18 ..
+-rw-r--r-- 1 hXXXX hXXXX  3936 2011-04-19 09:18 capacity-scheduler.xml
+-rw-r--r-- 1 hXXXX hXXXX   196 2011-04-19 09:18 commons-logging.properties
+-rw-r--r-- 1 hXXXX hXXXX   535 2011-04-19 09:18 configuration.xsl
+(.... skip ....)
+~$ diff /etc/hadoop/conf fromHDFS/
+}}}  
+
+=== 1.4 刪除檔案 ===
+=== 1.4 Remove Files or Folder ===
+
+{{{
+~$ hadoop fs -ls input/masters
+Found 1 items
+-rw-r--r--   2 hXXXX supergroup         10 2011-04-19 09:16 /user/hXXXX/input/masters
+~$ hadoop fs -rm input/masters
+Deleted hdfs://hadoop.nchc.org.tw/user/hXXXX/input/masters
+}}} 
+
+=== 1.5 直接看檔案 ===
+=== 1.5 Browse Files Directly ===
+
+{{{
+~$ hadoop fs -ls input/slaves
+Found 1 items
+-rw-r--r--   2 hXXXX supergroup         10 2011-04-19 09:16 /user/hXXXX/input/slaves
+~$ hadoop fs -cat input/slaves
+localhost
+}}} 
+
+=== 1.6 更多指令操作 ===
+=== 1.6 More Commands -- Help message ===
+
+{{{
+hXXXX@hadoop:~$ hadoop fs 
+
+Usage: java FsShell
+           [-ls <path>]
+           [-lsr <path>]
+           [-du <path>]
+           [-dus <path>]
+           [-count[-q] <path>]
+           [-mv <src> <dst>]
+           [-cp <src> <dst>]
+           [-rm <path>]
+           [-rmr <path>]
+           [-expunge]
+           [-put <localsrc> ... <dst>]
+           [-copyFromLocal <localsrc> ... <dst>]
+           [-moveFromLocal <localsrc> ... <dst>]
+           [-get [-ignoreCrc] [-crc] <src> <localdst>]
+           [-getmerge <src> <localdst> [addnl]]
+           [-cat <src>]
+           [-text <src>]
+           [-copyToLocal [-ignoreCrc] [-crc] <src> <localdst>]
+           [-moveToLocal [-crc] <src> <localdst>]
+           [-mkdir <path>]
+           [-setrep [-R] [-w] <rep> <path/file>]
+           [-touchz <path>]
+           [-test -[ezd] <path>]
+           [-stat [format] <path>]
+           [-tail [-f] <file>]
+           [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
+           [-chown [-R] [OWNER][:[GROUP]] PATH...]
+           [-chgrp [-R] GROUP PATH...]
+           [-help [cmd]]
+
+Generic options supported are
+-conf <configuration file>     specify an application configuration file
+-D <property=value>            use value for given property
+-fs <local|namenode:port>      specify a namenode
+-jt <local|jobtracker:port>    specify a job tracker
+-files <comma separated list of files>    specify comma separated files to be copied to the map reduce cluster
+-libjars <comma separated list of jars>    specify comma separated jar files to include in the classpath.
+-archives <comma separated list of archives>    specify comma separated archives to be unarchived on the compute machines.
+The general command line syntax is
+hadoop command [genericOptions] [commandOptions]
+}}}  
+ 
+== Content 2: 使用網頁 GUI 瀏覽資訊 ==
+== Content 2: User Web GUI to browse HDFS ==
+ 
+ * [http://localhost:50030 JobTracker Web Interface]
+ * [http://localhost:50070 NameNode Web Interface]
+ 
+== Content 3: 更多 HDFS Shell 的用法 ==
+== Content 3: More about HDFS Shell ==
+ 
+ * hadoop fs <args> ，下面則列出 <args> 的用法[[BR]]Following are the examples of hadoop fs related commands.
+ * 以下操作預設的目錄在 /user/<$username>/ 下[[BR]]By default, your working directory will be at /user/<$username>/.
+{{{
+$ hadoop fs -ls input
+Found 25 items
+-rw-r--r--   2 hXXXX supergroup        321 2011-04-19 09:16 /user/hXXXX/input/README
+-rw-r--r--   2 hXXXX supergroup       3936 2011-04-19 09:16 /user/hXXXX/input/capacity-scheduler.xml
+-rw-r--r--   2 hXXXX supergroup        196 2011-04-19 09:16 /user/hXXXX/input/commons-logging.properties
+(.... skip ....)
+}}}
+ * 完整的路徑則是 '''hdfs://node:port/path''' 如：[[BR]]Or you have to give a __''absolute path''__, such as '''hdfs://node:port/path'''
+{{{
+$ hadoop fs -ls hdfs://hadoop.nchc.org.tw/user/hXXXX/input
+Found 25 items
+-rw-r--r--   2 hXXXX supergroup        321 2011-04-19 09:16 /user/hXXXX/input/README
+-rw-r--r--   2 hXXXX supergroup       3936 2011-04-19 09:16 /user/hXXXX/input/capacity-scheduler.xml
+-rw-r--r--   2 hXXXX supergroup        196 2011-04-19 09:16 /user/hXXXX/input/commons-logging.properties
+(.... skip ....)
+}}}
+
+=== -cat  === 
+
+ * 將路徑指定文件的內容輸出到 STDOUT [[BR]] Print given file content to STDOUT
+{{{
+$ hadoop fs -cat input/hadoop-env.sh
+}}} 
+
+=== -chgrp  ===
+
+ * 改變文件所屬的組 [[BR]] Change '''owner group''' of given file or folder
+{{{
+$ hadoop fs -ls
+Found 2 items
+drwxr-xr-x   - hXXXX supergroup          0 2011-04-19 09:16 /user/hXXXX/input
+drwxr-xr-x   - hXXXX supergroup          0 2010-01-24 17:23 /user/hXXXX/tmp
+$ hadoop fs -chgrp -R ${USER} input
+$ hadoop fs -ls
+Found 2 items
+drwxr-xr-x   - hXXXX hXXXX               0 2011-04-19 09:21 /user/hXXXX/input
+drwxr-xr-x   - hXXXX supergroup          0 2010-01-24 17:23 /user/hXXXX/tmp
+}}}
+
+=== -chmod === 
+
+ * 改變文件的權限 [[BR]] Change '''read and write permission''' of given file or folder
+{{{
+$ hadoop fs -ls
+Found 2 items
+drwxr-xr-x   - hXXXX hXXXX               0 2011-04-19 09:21 /user/hXXXX/input
+drwxr-xr-x   - hXXXX supergroup          0 2010-01-24 17:23 /user/hXXXX/tmp
+$ hadoop fs -chmod -R 755 input
+$ hadoop fs -ls
+Found 2 items
+drwxrwxrwx   - hXXXX hXXXX               0 2011-04-19 09:21 /user/hXXXX/input
+drwxr-xr-x   - hXXXX supergroup          0 2010-01-24 17:23 /user/hXXXX/tmp
+}}}
+
+=== -chown ===
+
+ * 改變文件的擁有者 [[BR]] Change '''owner''' of given file or folder
+{{{
+$ hadoop fs -chown -R ${USER} input
+}}}
+ * 注意：因為在 hadoop.nchc.org.tw 上您沒有管理者權限，因此若要改成其