wiki:FDC110829/Lab2

  1. 實作二 Lab 2
    1. Content 1: HDFS Shell 基本操作
    2. Content 1: Basic HDFS Shell Commands
      1. 1.1 瀏覽你HDFS目錄
      2. 1.1 Browsing Your HDFS Folder
      3. 1.2 上傳資料到 HDFS 目錄
      4. 1.2 Upload Files or Folder to HDFS
      5. 1.3 下載 HDFS 的資料到本地目錄
      6. 1.3 Download HDFS Files or Folder to Local
      7. 1.4 刪除檔案
      8. 1.4 Remove Files or Folder
      9. 1.5 直接看檔案
      10. 1.5 Browse Files Directly
      11. 1.6 更多指令操作
      12. 1.6 More Commands -- Help message
    3. Content 2: 使用網頁 GUI 瀏覽資訊
    4. Content 2: User Web GUI to browse HDFS
    5. Content 3: 更多 HDFS Shell 的用法
    6. Content 3: More about HDFS Shell
      1. -cat
      2. -chgrp
      3. -chmod
      4. -chown

◢ <實作一> | <回課程大綱> ▲ | <實作三> ◣

實作二 Lab 2

HDFS Shell操作練習
HDFS Shell in practice
以下練習，請連線至 hadoop.nchc.org.tw 操作。底下的 hXXXX 等於您的用戶名稱。

Content 1: HDFS Shell 基本操作

Content 1: Basic HDFS Shell Commands

1.1 瀏覽你HDFS目錄

1.1 Browsing Your HDFS Folder

~$ hadoop fs -ls
Found 1 items
drwxr-xr-x   - hXXXX supergroup          0 2010-01-24 17:23 /user/hXXXX/tmp
~$ hadoop fs -lsr
drwxr-xr-x   - hXXXX supergroup          0 2010-01-24 17:23 /user/hXXXX/tmp

1.2 上傳資料到 HDFS 目錄

1.2 Upload Files or Folder to HDFS

  • 上傳 Upload 
~$ hadoop fs -put /etc/hadoop/conf input
  • 檢查 Check 
~$ hadoop fs -ls
Found 2 items
drwxr-xr-x   - hXXXX supergroup          0 2011-04-19 09:16 /user/hXXXX/input
drwxr-xr-x   - hXXXX supergroup          0 2010-01-24 17:23 /user/hXXXX/tmp
~$ hadoop fs -ls input
Found 25 items
-rw-r--r--   2 hXXXX supergroup        321 2011-04-19 09:16 /user/hXXXX/input/README
-rw-r--r--   2 hXXXX supergroup       3936 2011-04-19 09:16 /user/hXXXX/input/capacity-scheduler.xml
-rw-r--r--   2 hXXXX supergroup        196 2011-04-19 09:16 /user/hXXXX/input/commons-logging.properties
(.... skip ....)

1.3 下載 HDFS 的資料到本地目錄

1.3 Download HDFS Files or Folder to Local

  • 下載 Download 
~$ hadoop fs -get input fromHDFS
  • 檢查 Check 
    ~$ ls -al | grep fromHDFS
drwxr-xr-x    2 hXXXX hXXXX  4096 2011-04-19 09:18 fromHDFS
~$ ls -al fromHDFS
總計 160
drwxr-xr-x 2 hXXXX hXXXX  4096 2011-04-19 09:18 .
drwx--x--x 3 hXXXX hXXXX  4096 2011-04-19 09:18 ..
-rw-r--r-- 1 hXXXX hXXXX  3936 2011-04-19 09:18 capacity-scheduler.xml
-rw-r--r-- 1 hXXXX hXXXX   196 2011-04-19 09:18 commons-logging.properties
-rw-r--r-- 1 hXXXX hXXXX   535 2011-04-19 09:18 configuration.xsl
(.... skip ....)
~$ diff /etc/hadoop/conf fromHDFS/

1.4 刪除檔案

1.4 Remove Files or Folder

~$ hadoop fs -ls input/masters
Found 1 items
-rw-r--r--   2 hXXXX supergroup         10 2011-04-19 09:16 /user/hXXXX/input/masters
~$ hadoop fs -rm input/masters
Deleted hdfs://hadoop.nchc.org.tw/user/hXXXX/input/masters

1.5 直接看檔案

1.5 Browse Files Directly

~$ hadoop fs -ls input/slaves
Found 1 items
-rw-r--r--   2 hXXXX supergroup         10 2011-04-19 09:16 /user/hXXXX/input/slaves
~$ hadoop fs -cat input/slaves
localhost

1.6 更多指令操作

1.6 More Commands -- Help message

hXXXX@hadoop:~$ hadoop fs 

Usage: java FsShell
           [-ls <path>]
           [-lsr <path>]
           [-du <path>]
           [-dus <path>]
           [-count[-q] <path>]
           [-mv <src> <dst>]
           [-cp <src> <dst>]
           [-rm <path>]
           [-rmr <path>]
           [-expunge]
           [-put <localsrc> ... <dst>]
           [-copyFromLocal <localsrc> ... <dst>]
           [-moveFromLocal <localsrc> ... <dst>]
           [-get [-ignoreCrc] [-crc] <src> <localdst>]
           [-getmerge <src> <localdst> [addnl]]
           [-cat <src>]
           [-text <src>]
           [-copyToLocal [-ignoreCrc] [-crc] <src> <localdst>]
           [-moveToLocal [-crc] <src> <localdst>]
           [-mkdir <path>]
           [-setrep [-R] [-w] <rep> <path/file>]
           [-touchz <path>]
           [-test -[ezd] <path>]
           [-stat [format] <path>]
           [-tail [-f] <file>]
           [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
           [-chown [-R] [OWNER][:[GROUP]] PATH...]
           [-chgrp [-R] GROUP PATH...]
           [-help [cmd]]

Generic options supported are
-conf <configuration file>     specify an application configuration file
-D <property=value>            use value for given property
-fs <local|namenode:port>      specify a namenode
-jt <local|jobtracker:port>    specify a job tracker
-files <comma separated list of files>    specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars>    specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives>    specify comma separated archives to be unarchived on the compute machines.
The general command line syntax is
hadoop command [genericOptions] [commandOptions]

Content 2: 使用網頁 GUI 瀏覽資訊

Content 2: User Web GUI to browse HDFS

Content 3: 更多 HDFS Shell 的用法

Content 3: More about HDFS Shell

  • hadoop fs <args> ，下面則列出 <args> 的用法
    Following are the examples of hadoop fs related commands.
  • 以下操作預設的目錄在 /user/<$username>/ 下
    By default, your working directory will be at /user/<$username>/. 
    $ hadoop fs -ls input
Found 25 items
-rw-r--r--   2 hXXXX supergroup        321 2011-04-19 09:16 /user/hXXXX/input/README
-rw-r--r--   2 hXXXX supergroup       3936 2011-04-19 09:16 /user/hXXXX/input/capacity-scheduler.xml
-rw-r--r--   2 hXXXX supergroup        196 2011-04-19 09:16 /user/hXXXX/input/commons-logging.properties
(.... skip ....)
  • 完整的路徑則是 hdfs://node:port/path 如：
    Or you have to give a absolute path, such as hdfs://node:port/path 
    $ hadoop fs -ls hdfs://hadoop.nchc.org.tw/user/hXXXX/input
Found 25 items
-rw-r--r--   2 hXXXX supergroup        321 2011-04-19 09:16 /user/hXXXX/input/README
-rw-r--r--   2 hXXXX supergroup       3936 2011-04-19 09:16 /user/hXXXX/input/capacity-scheduler.xml
-rw-r--r--   2 hXXXX supergroup        196 2011-04-19 09:16 /user/hXXXX/input/commons-logging.properties
(.... skip ....)

-cat

  • 將路徑指定文件的內容輸出到 STDOUT
    Print given file content to STDOUT 
    $ hadoop fs -cat input/hadoop-env.sh

-chgrp

  • 改變文件所屬的組
    Change owner group of given file or folder 
    $ hadoop fs -ls
Found 2 items
drwxr-xr-x   - hXXXX supergroup          0 2011-04-19 09:16 /user/hXXXX/input
drwxr-xr-x   - hXXXX supergroup          0 2010-01-24 17:23 /user/hXXXX/tmp
$ hadoop fs -chgrp -R ${USER} input
$ hadoop fs -ls
Found 2 items
drwxr-xr-x   - hXXXX hXXXX               0 2011-04-19 09:21 /user/hXXXX/input
drwxr-xr-x   - hXXXX supergroup          0 2010-01-24 17:23 /user/hXXXX/tmp

-chmod

  • 改變文件的權限
    Change read and write permission of given file or folder 
    $ hadoop fs -ls
Found 2 items
drwxr-xr-x   - hXXXX hXXXX               0 2011-04-19 09:21 /user/hXXXX/input
drwxr-xr-x   - hXXXX supergroup          0 2010-01-24 17:23 /user/hXXXX/tmp
$ hadoop fs -chmod -R 755 input
$ hadoop fs -ls
Found 2 items
drwxrwxrwx   - hXXXX hXXXX               0 2011-04-19 09:21 /user/hXXXX/input
drwxr-xr-x   - hXXXX supergroup          0 2010-01-24 17:23 /user/hXXXX/tmp

-chown

  • 改變文件的擁有者
    Change owner of given file or folder 
    $ hadoop fs -chown -R ${USER} input
  • 注意：因為在 hadoop.nchc.org.tw 上您沒有管理者權限，因此若要改成其
Last modified 13 years ago Last modified on Aug 28, 2011, 11:00:22 PM