Changes between Version 14 and Version 15 of MR_manual

Timestamp:

Jun 13, 2008, 5:05:18 PM (16 years ago)

Author:

waue

Comment:

--

MR_manual

@@ -179 +179 @@
  = 四、效能測試 = 
  = 五、開發程式 = 
+ == 5.1 install IBM mapReduce tool == 
+ 1. Download the IBM MapReduce Tools zip file and extract to /tmp/. [[br]]
+ 2. Make sure Eclipse is closed and ... [[br]]
+{{{
+ $ cd /tmp/
+ $ unzip mapreduce_tools.zip
+ $ mv plugins/com.ibm.hipods.mapreduce* /usr/lib/eclipse/plugins/
+}}}
+ 3. Restart Eclipse  [[br]]
+ Check IBM MapReduce Tools plugin installing well [[br]]
+{{{
+Eclipse 
+ File > New > Project 
+see MapReduce category
+}}}
+ == 5.2 Eclipse configure == 
+{{{
+Eclipse
+Window > Preferences > java> compiler
+set compiler compliance level to 5.0 
+}}}
+ * Some eclipse-plugin may exhaust much resource, you may happen to “out of 
+
+memory error”. We suggest to execute eclipse with some parameters as that : 
+{{{
+ $  eclipse -vmargs -Xmx 512m
+}}}
+
+ == 5.3. Run on Eclipse ==
+ === 5.3.1 map-reduce sample code === 
+{{{
+Eclipse
+ File > new > project >  map-reduce project > next > 
+project name : sample
+use default location : V
+use default Hadoop : V
+ > Finish
+}}}
+ * at “Project explorer”, you will see “sample” tree. Now, you should 
+
+create a sample code.
+{{{
+Eclipse
+right click sample > new > file > 
+file name :  WordCount.java
+}}}
+ * the sample code is here 
+
+[http://trac.nchc.org.tw/cloud/attachment/wiki/hadoop-sample-
+
+code/WordCount.java]
+
+ * paste the contents to your new adding file “WordCount.java”
+ === 5.3.2. Connect to Hadoop File System === 
+
+ * Enable the MapReduce servers window 
+{{{
+Eclipse
+ Window > Show View > Other... > MapReduce Tools > MapReduce Servers 
+}}}
+
+ * At the bottom of your window, you should have a "MapReduce Servers" tab. 
+
+If not, see second bullet above. Switch to that tab.
+
+ * At the top right edge of the tab, you should see a little blue elephant 
+
+icons. 
+{{{
+Eclipse
+Click blue elephant to add a new MapReduce server location. 
+Server name : any_you_want 
+Hostname : localhost 
+Installation directory: /home/waue/workspace/nutch/ 
+Username : waue
+}}}
+ * If any password prompt, please input the password which you login to 
+
+local
+
+ * It should show up under a little elephant icon in the Project Explorer 
+
+(on the left side of Eclipse). 
+
+ * ps  :  Pleast make sure your Hadoop is working on local system. If not, 
+
+please refer “session 2 Hadoop Setup” for debuging, or you can not pass 
+
+through.
+
+ $ cd /home/waue/workspace/hadoop/ [[br]] 
+ $ wget http://www.gutenberg.org/etext/132/132.txt   [[br]] 
+ $ bin/hadoop dfs -mkdir input [[br]] 
+ $ bin/hadoop dfs -ls [[br]] 
+{{{
+Found 1 items
+/user/waue/input        <dir>           2008-05-23 15:15        rwxr-xr-x    
+
+  waue     supergroup
+}}}
+ $ bin/hadoop dfs -put 132.txt input [[br]] 
+
+ === 5.3.3 Run === 
+{{{
+Eclipse
+sample >  right click WordCount.java > run as ... > run on Hadoop > choose 
+
+an existing server from the list below > finish
+}}}
+ * A “console” tag will show beside “MapReduce Server” tag. 
+
+ * While Map Reduce is running, you can visit http://localhost:50030/ to 
+
+view that Hadoop is dispatching jobs by Map Reduce.
+
+ * After finish, you can go to http://localhost:50060/ to see the result.
+
+
  = 六、範例教學 = 
  = Reference =