Changes between Version 1 and Version 2 of waue/2009/0506


Ignore:
Timestamp:
May 6, 2009, 2:58:14 PM (15 years ago)
Author:
waue
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • waue/2009/0506

    v1 v2  
    55}}}
    66[[PageOutline]]
     7= 簡介 =
     8 * Hive 是由facebook所捐贈給hadoop的項目
     9 * 功能類似前一天測試的cloudbase
     10 * 更簡便的一點是,Hive已經整合在Hadoop 0.19.1的版本內(但 0.20.0 卻又不見了),幾乎不用特別座設定,並且也有提供自己的交互查詢模式,就不用特別在安裝其他東西了
     11 * 感覺上操作起來有點像hbase !!
     12
     13= 安裝 =
     14
     15
     16
     17= 測試 =
     18[hadoop@gp1 bin]$ export HADOOP=/home/hadoop/hadoop-0.19.1
     19[hadoop@gp1 bin]$ export HIVE_HOME=/home/hadoop/hadoop-0.19.1/contrib/hive/
     20
     21[hadoop@gp1 bin]$ cd $HIVE_HOME
     22[hadoop@gp1 hive]$ bin/hive
     23
     24hive> CREATE TABLE pokes (foo INT, bar STRING);
     25OK
     26Time taken: 0.251 seconds
     27hive> CREATE TABLE invites (foo INT, bar STRING) PARTITIONED BY (ds STRING);
     28OK
     29Time taken: 0.106 seconds
     30hive> SHOW TABLES;
     31OK
     32invites pokes
     33Time taken: 0.107 seconds
     34hive> DESCRIBE invites;
     35OK
     36foo     int
     37bar     string
     38ds      string
     39Time taken: 0.151 seconds
     40hive> ALTER TABLE pokes ADD COLUMNS (new_col INT);
     41OK
     42Time taken: 0.117 seconds
     43hive> ALTER TABLE invites ADD COLUMNS (new_col2 INT COMMENT 『a comment');
     44OK
     45Time taken: 0.152 seconds
     46hive> LOAD DATA LOCAL INPATH 『./examples/files/kv1.txt' OVERWRITE INTO TABLE pokes;
     47Copying data from file:/home/hadoop/hadoop-0.19.1/contrib/hive/examples/files/kv1.txt
     48Loading data to table pokes
     49OK
     50Time taken: 0.288 seconds
     51hive> LOAD DATA LOCAL INPATH 『./examples/files/kv2.txt' OVERWRITE INTO TABLE invites PARTITION (ds=』2008-08-15′);
     52Copying data from file:/home/hadoop/hadoop-0.19.1/contrib/hive/examples/files/kv2.txt
     53Loading data to table invites partition {ds=2008-08-15}
     54OK
     55Time taken: 0.524 seconds
     56hive> LOAD DATA LOCAL INPATH 『./examples/files/kv3.txt' OVERWRITE INTO TABLE invites PARTITION (ds=』2008-08-08′);
     57Copying data from file:/home/hadoop/hadoop-0.19.1/contrib/hive/examples/files/kv3.txt
     58Loading data to table invites partition {ds=2008-08-08}
     59OK
     60Time taken: 0.406 seconds
     61
     62hive> INSERT OVERWRITE DIRECTORY 『/tmp/hdfs_out' SELECT a.* FROM invites a;
     63Total MapReduce jobs = 1
     64Starting Job = job_200902261245_0002, Tracking URL = http://gp1:50030/jobdetails.jsp?jobid=job_200902261245_0002
     65Kill Command = /home/hadoop/hadoop-0.19.1/bin/hadoop job  -Dmapred.job.tracker=gp1:9001 -kill job_200902261245_0002
     66map = 0%,  reduce =0%
     67map = 50%,  reduce =0%
     68map = 100%,  reduce =0%
     69Ended Job = job_200902261245_0002
     70Moving data to: /tmp/hdfs_out
     71OK
     72Time taken: 18.551 seconds
     73
     74hive> select count(1) from pokes;
     75Total MapReduce jobs = 2
     76Number of reducers = 1
     77In order to change numer of reducers use:
     78set mapred.reduce.tasks = <number>
     79Starting Job = job_200902261245_0003, Tracking URL = http://gp1:50030/jobdetails.jsp?jobid=job_200902261245_0003
     80Kill Command = /home/hadoop/hadoop-0.19.1/bin/hadoop job  -Dmapred.job.tracker=gp1:9001 -kill job_200902261245_0003
     81map = 0%,  reduce =0%
     82map = 50%,  reduce =0%
     83map = 100%,  reduce =0%
     84map = 100%,  reduce =17%
     85map = 100%,  reduce =100%
     86Ended Job = job_200902261245_0003
     87Starting Job = job_200902261245_0004, Tracking URL = http://gp1:50030/jobdetails.jsp?jobid=job_200902261245_0004
     88Kill Command = /home/hadoop/hadoop-0.19.1/bin/hadoop job  -Dmapred.job.tracker=gp1:9001 -kill job_200902261245_0004
     89map = 0%,  reduce =0%
     90map = 50%,  reduce =0%
     91map = 100%,  reduce =0%
     92map = 100%,  reduce =100%
     93Ended Job = job_200902261245_0004
     94OK
     95500
     96Time taken: 57.285 seconds
     97
     98hive> INSERT OVERWRITE DIRECTORY 『/tmp/hdfs_out' SELECT a.* FROM invites a;
     99Total MapReduce jobs = 1
     100Starting Job = job_200902261245_0005, Tracking URL = http://gp1:50030/jobdetails.jsp?jobid=job_200902261245_0005
     101Kill Command = /home/hadoop/hadoop-0.19.1/bin/hadoop job  -Dmapred.job.tracker=gp1:9001 -kill job_200902261245_0005
     102map = 0%,  reduce =0%
     103map = 50%,  reduce =0%
     104map = 100%,  reduce =0%
     105Ended Job = job_200902261245_0005
     106Moving data to: /tmp/hdfs_out
     107OK
     108Time taken: 18.349 seconds
     109
     110hive>  INSERT OVERWRITE DIRECTORY 『/tmp/reg_5′ SELECT COUNT(1) FROM invites a;
     111Total MapReduce jobs = 2
     112Number of reducers = 1
     113In order to change numer of reducers use:
     114set mapred.reduce.tasks = <number>
     115Starting Job = job_200902261245_0006, Tracking URL = http://gp1:50030/jobdetails.jsp?jobid=job_200902261245_0006
     116Kill Command = /home/hadoop/hadoop-0.19.1/bin/hadoop job  -Dmapred.job.tracker=gp1:9001 -kill job_200902261245_0006
     117map = 0%,  reduce =0%
     118map = 50%,  reduce =0%
     119map = 100%,  reduce =0%
     120map = 100%,  reduce =17%
     121map = 100%,  reduce =100%
     122Ended Job = job_200902261245_0006
     123Starting Job = job_200902261245_0007, Tracking URL = http://gp1:50030/jobdetails.jsp?jobid=job_200902261245_0007
     124Kill Command = /home/hadoop/hadoop-0.19.1/bin/hadoop job  -Dmapred.job.tracker=gp1:9001 -kill job_200902261245_0007
     125map = 0%,  reduce =0%
     126map = 50%,  reduce =0%
     127map = 100%,  reduce =0%
     128map = 100%,  reduce =17%
     129map = 100%,  reduce =100%
     130Ended Job = job_200902261245_0007
     131Moving data to: /tmp/reg_5
     132OK
     133Time taken: 70.956 seconds