wiki:jazz/Hadoop_Lab2

Version 11 (modified by waue, 15 years ago) (diff)

--

實作二: HDFS 指令操作練習

前言

  • 此部份接續實做一

Content 1. 基本操作

1.1 瀏覽你HDFS目錄

1.2 上傳資料到HDFS目錄

1.3 下載HDFS的資料到本地目錄

1.4 更多指令操作

Content 2. Hadoop 運算命令

2.1 Hadoop運算命令 grep

2.2 Hadoop運算命令 WordCount

2.3 更多運算命令

可執行的指令一覽表:

aggregatewordcount An Aggregate based map/reduce program that counts the words in the input files.
aggregatewordhist An Aggregate based map/reduce program that computes the histogram of the words in the input files.
grep A map/reduce program that counts the matches of a regex in the input.
join A job that effects a join over sorted, equally partitioned datasets
multifilewc A job that counts words from several files.
pentomino A map/reduce tile laying program to find solutions to pentomino problems.
pi A map/reduce program that estimates Pi using monte-carlo method.
randomtextwriter A map/reduce program that writes 10GB of random textual data per node.
randomwriter A map/reduce program that writes 10GB of random data per node.
sleep A job that sleeps at each map and reduce task.
sort A map/reduce program that sorts the data written by the random writer.
sudoku A sudoku solver.
wordcount A map/reduce program that counts the words in the input files.

請參考 org.apache.hadoop.examples

Class Summary
AggregateWordCount This is an example Aggregated Hadoop Map/Reduce application. It reads the text input files, breaks each line into words and counts them. The output is a locally sorted list of words and the count of how often they occurred. To run: bin/hadoop jar hadoop-*-examples.jar aggregatewordcount in-dir out-dir numOfReducers textinputformat
AggregateWordHistogram This is an example Aggregated Hadoop Map/Reduce application. Computes the histogram of the words in the input texts. To run: bin/hadoop jar hadoop-*-examples.jar aggregatewordhist in-dir out-dir numOfReducers textinputformat
ExampleDriver A description of an example program based on its class and a human-readable description.
Grep  
Join This is the trivial map/reduce program that does absolutely nothing other than use the framework to fragment and sort the input values. To run: bin/hadoop jar build/hadoop-examples.jar join [-m maps] [-r reduces] [-inFormat input format class] [-outFormat output format class] [-outKey output key class] [-outValue output value class] [-joinOp
RandomTextWriter This program uses map/reduce to just run a distributed job where there is no interaction between the tasks and each task writes a large unsorted random sequence of words.To run: bin/hadoop jar hadoop-${version}-examples.jar randomtextwriter [-outFormat output format class] output
RandomWriter This program uses map/reduce to just run a distributed job where there is no interaction between the tasks and each task write a large unsorted random binary sequence file of BytesWritable.To run: bin/hadoop jar hadoop-${version}-examples.jar randomwriter [-outFormat output format class] output
Sort<K,V> This is the trivial map/reduce program that does absolutely nothing other than use the framework to fragment and sort the input values.To run: bin/hadoop jar build/hadoop-examples.jar sort [-m maps] [-r reduces] [-inFormat input format class] [-outFormat output format class] [-outKey output key class] [-outValue output value class] [-totalOrder pcnt num samples max splits] in-dir out-dir
WordCount This is an example Hadoop Map/Reduce application.

Content 3. 使用網頁Gui瀏覽訊息

練習

Attachments (1)

Download all attachments as: .zip