[[PageOutline]] ◢ <[wiki:FDC110829/Lab1 實作一]> | <[wiki:FDC110829 回課程大綱]> ▲ | <[wiki:FDC110829/Lab3 實作三]> ◣ = 實作二 Lab 2 = {{{ #!html
HDFS Shell操作練習
HDFS Shell in practice
}}} {{{ #!text 以下練習，請連線至 hadoop.nchc.org.tw 操作。底下的 hXXXX 等於您的用戶名稱。 }}} == Content 1: HDFS Shell 基本操作 == == Content 1: Basic HDFS Shell Commands == === 1.1 瀏覽你HDFS目錄 === === 1.1 Browsing Your HDFS Folder === {{{ ~$ hadoop fs -ls Found 1 items drwxr-xr-x - hXXXX supergroup 0 2010-01-24 17:23 /user/hXXXX/tmp ~$ hadoop fs -lsr drwxr-xr-x - hXXXX supergroup 0 2010-01-24 17:23 /user/hXXXX/tmp }}} === 1.2 上傳資料到 HDFS 目錄 === === 1.2 Upload Files or Folder to HDFS === * 上傳 Upload {{{ ~$ hadoop fs -put /etc/hadoop/conf input }}} * 檢查 Check {{{ ~$ hadoop fs -ls Found 2 items drwxr-xr-x - hXXXX supergroup 0 2011-04-19 09:16 /user/hXXXX/input drwxr-xr-x - hXXXX supergroup 0 2010-01-24 17:23 /user/hXXXX/tmp ~$ hadoop fs -ls input Found 25 items -rw-r--r-- 2 hXXXX supergroup 321 2011-04-19 09:16 /user/hXXXX/input/README -rw-r--r-- 2 hXXXX supergroup 3936 2011-04-19 09:16 /user/hXXXX/input/capacity-scheduler.xml -rw-r--r-- 2 hXXXX supergroup 196 2011-04-19 09:16 /user/hXXXX/input/commons-logging.properties (.... skip ....) }}} === 1.3 下載 HDFS 的資料到本地目錄 === === 1.3 Download HDFS Files or Folder to Local === * 下載 Download {{{ ~$ hadoop fs -get input fromHDFS }}} * 檢查 Check {{{ ~$ ls -al | grep fromHDFS drwxr-xr-x 2 hXXXX hXXXX 4096 2011-04-19 09:18 fromHDFS ~$ ls -al fromHDFS 總計 160 drwxr-xr-x 2 hXXXX hXXXX 4096 2011-04-19 09:18 . drwx--x--x 3 hXXXX hXXXX 4096 2011-04-19 09:18 .. -rw-r--r-- 1 hXXXX hXXXX 3936 2011-04-19 09:18 capacity-scheduler.xml -rw-r--r-- 1 hXXXX hXXXX 196 2011-04-19 09:18 commons-logging.properties -rw-r--r-- 1 hXXXX hXXXX 535 2011-04-19 09:18 configuration.xsl (.... skip ....) ~$ diff /etc/hadoop/conf fromHDFS/ }}} === 1.4 刪除檔案 === === 1.4 Remove Files or Folder === {{{ ~$ hadoop fs -ls input/masters Found 1 items -rw-r--r-- 2 hXXXX supergroup 10 2011-04-19 09:16 /user/hXXXX/input/masters ~$ hadoop fs -rm input/masters Deleted hdfs://hadoop.nchc.org.tw/user/hXXXX/input/masters }}} === 1.5 直接看檔案 === === 1.5 Browse Files Directly === {{{ ~$ hadoop fs -ls input/slaves Found 1 items -rw-r--r-- 2 hXXXX supergroup 10 2011-04-19 09:16 /user/hXXXX/input/slaves ~$ hadoop fs -cat input/slaves localhost }}} === 1.6 更多指令操作 === === 1.6 More Commands -- Help message === {{{ hXXXX@hadoop:~$ hadoop fs Usage: java FsShell [-ls ] [-lsr ] [-du ] [-dus ] [-count[-q] ] [-mv ] [-cp ] [-rm ] [-rmr ] [-expunge] [-put ... ] [-copyFromLocal ... ] [-moveFromLocal ... ] [-get [-ignoreCrc] [-crc] ] [-getmerge [addnl]] [-cat ] [-text ] [-copyToLocal [-ignoreCrc] [-crc] ] [-moveToLocal [-crc] ] [-mkdir ] [-setrep [-R] [-w] ] [-touchz ] [-test -[ezd] ] [-stat [format] ] [-tail [-f] ] [-chmod [-R] PATH...] [-chown [-R] [OWNER][:[GROUP]] PATH...] [-chgrp [-R] GROUP PATH...] [-help [cmd]] Generic options supported are -conf specify an application configuration file -D use value for given property -fs specify a namenode -jt specify a job tracker -files specify comma separated files to be copied to the map reduce cluster -libjars specify comma separated jar files to include in the classpath. -archives specify comma separated archives to be unarchived on the compute machines. The general command line syntax is hadoop command [genericOptions] [commandOptions] }}} == Content 2: 使用網頁 GUI 瀏覽資訊 == == Content 2: User Web GUI to browse HDFS == * [http://localhost:50030 JobTracker Web Interface] * [http://localhost:50070 NameNode Web Interface] == Content 3: 更多 HDFS Shell 的用法 == == Content 3: More about HDFS Shell == * hadoop fs ，下面則列出 的用法[[BR]]Following are the examples of hadoop fs related commands. * 以下操作預設的目錄在 /user/<$username>/ 下[[BR]]By default, your working directory will be at /user/<$username>/. {{{ $ hadoop fs -ls input Found 25 items -rw-r--r-- 2 hXXXX supergroup 321 2011-04-19 09:16 /user/hXXXX/input/README -rw-r--r-- 2 hXXXX supergroup 3936 2011-04-19 09:16 /user/hXXXX/input/capacity-scheduler.xml -rw-r--r-- 2 hXXXX supergroup 196 2011-04-19 09:16 /user/hXXXX/input/commons-logging.properties (.... skip ....) }}} * 完整的路徑則是 '''hdfs://node:port/path''' 如：[[BR]]Or you have to give a __''absolute path''__, such as '''hdfs://node:port/path''' {{{ $ hadoop fs -ls hdfs://hadoop.nchc.org.tw/user/hXXXX/input Found 25 items -rw-r--r-- 2 hXXXX supergroup 321 2011-04-19 09:16 /user/hXXXX/input/README -rw-r--r-- 2 hXXXX supergroup 3936 2011-04-19 09:16 /user/hXXXX/input/capacity-scheduler.xml -rw-r--r-- 2 hXXXX supergroup 196 2011-04-19 09:16 /user/hXXXX/input/commons-logging.properties (.... skip ....) }}} === -cat === * 將路徑指定文件的內容輸出到 STDOUT [[BR]] Print given file content to STDOUT {{{ $ hadoop fs -cat input/hadoop-env.sh }}} === -chgrp === * 改變文件所屬的組 [[BR]] Change '''owner group''' of given file or folder {{{ $ hadoop fs -ls Found 2 items drwxr-xr-x - hXXXX supergroup 0 2011-04-19 09:16 /user/hXXXX/input drwxr-xr-x - hXXXX supergroup 0 2010-01-24 17:23 /user/hXXXX/tmp $ hadoop fs -chgrp -R ${USER} input $ hadoop fs -ls Found 2 items drwxr-xr-x - hXXXX hXXXX 0 2011-04-19 09:21 /user/hXXXX/input drwxr-xr-x - hXXXX supergroup 0 2010-01-24 17:23 /user/hXXXX/tmp }}} === -chmod === * 改變文件的權限 [[BR]] Change '''read and write permission''' of given file or folder {{{ $ hadoop fs -ls Found 2 items drwxr-xr-x - hXXXX hXXXX 0 2011-04-19 09:21 /user/hXXXX/input drwxr-xr-x - hXXXX supergroup 0 2010-01-24 17:23 /user/hXXXX/tmp $ hadoop fs -chmod -R 755 input $ hadoop fs -ls Found 2 items drwxrwxrwx - hXXXX hXXXX 0 2011-04-19 09:21 /user/hXXXX/input drwxr-xr-x - hXXXX supergroup 0 2010-01-24 17:23 /user/hXXXX/tmp }}} === -chown === * 改變文件的擁有者 [[BR]] Change '''owner''' of given file or folder {{{ $ hadoop fs -chown -R ${USER} input }}} * 注意：因為在 hadoop.nchc.org.tw 上您沒有管理者權限，因此若要改成其