2009-07-02

• [狀況] Namenode 一直起不來
• [解法] 從 namesecondary 拷貝 edits 到 name，然後重新啟動 Namenode

  jazz@hadoop:~$ sudo cp /var/lib/hadoop/cache/hadoop/dfs/namesecondary/current/edits /var/lib/hadoop/cache/hadoop/dfs/name/current/edits
  jazz@hadoop:~$ sudo /etc/init.d/hadoop-namenode restart
  

• [錯誤追蹤] 很可能是因為系統碟沒有空間，但是我主要是追 /var/log/hadoop/hadoop-hadoop-namenode-hadoop.out 看到錯誤訊息是　hadoop045 帳號在跑東西

  ERROR dfs.LeaseManager: /user/hadoop045/trend/output/countries/_logs/history
  

• 接著在 /var/lib/hadoop/cache/hadoop/dfs/namesecondary/current/edits 也看到一堆 hadoop045 的檔案，因此懷疑 edits 檔是紀錄哪些讀寫是未完成。既然 Namenode 都起不來了，那些沒跑完的也跑不完，就乾脆把 edits 歸零了。原始　namenode 錯誤訊息如下：

  09/07/02 09:18:06 INFO dfs.NameNode: STARTUP_MSG: 
  /************************************************************
  STARTUP_MSG: Starting NameNode
  STARTUP_MSG:   host = hadoop/140.110.X.X
  STARTUP_MSG:   args = []
  STARTUP_MSG:   version = 0.18.3-4cloudera0.3.0
  STARTUP_MSG:   build =  -r ; compiled by 'root' on Fri May 29 23:29:49 UTC 2009
  ************************************************************/
  09/07/02 09:18:06 INFO metrics.RpcMetrics: Initializing RPC Metrics with hostName=NameNode, port=8020
  09/07/02 09:18:06 INFO dfs.NameNode: Namenode up at: hadoop.nchc.org.tw/140.110.X.X:8020
  09/07/02 09:18:06 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=NameNode, sessionId=null
  09/07/02 09:18:06 INFO dfs.NameNodeMetrics: Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NullContext
  09/07/02 09:18:06 INFO fs.FSNamesystem: fsOwner=hadoop,hadoop
  09/07/02 09:18:06 INFO fs.FSNamesystem: supergroup=supergroup
  09/07/02 09:18:06 INFO fs.FSNamesystem: isPermissionEnabled=true
  09/07/02 09:18:06 INFO dfs.FSNamesystemMetrics: Initializing FSNamesystemMeterics using context object:org.apache.hadoop.metrics.spi.NullContext
  09/07/02 09:18:06 INFO fs.FSNamesystem: Registered FSNamesystemStatusMBean
  09/07/02 09:18:06 INFO dfs.Storage: Number of files = 3723
  09/07/02 09:18:06 INFO dfs.Storage: Number of files under construction = 6
  09/07/02 09:18:06 INFO dfs.Storage: Image file of size 776834 loaded in 0 seconds.
  09/07/02 09:18:06 ERROR dfs.LeaseManager: /user/hadoop045/trend/output/countries/_logs/history/hadoop.nchc.org.tw_1244039499361_job_200906031031_0393_conf.xml not found in lease.paths (=[/user/hadoop045/trend/output/bayesian_input/_logs/history/hadoop.nchc.org.tw_1244039499361_job_200906031031_0391_hadoop045_word+count, /user/hadoop045/trend/output/countries/_logs/history/hadoop.nchc.org.tw_1244039499361_job_200906031031_0393_hadoop045_linecountC])
  09/07/02 09:18:06 ERROR fs.FSNamesystem: FSNamesystem initialization failed.
  java.io.EOFException
  	at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:323)
  	at org.apache.hadoop.io.UTF8.readFields(UTF8.java:103)
  	at org.apache.hadoop.dfs.FSImage.readString(FSImage.java:1368)
  	at org.apache.hadoop.dfs.FSEditLog.readLong(FSEditLog.java:1119)
  	at org.apache.hadoop.dfs.FSEditLog.loadFSEdits(FSEditLog.java:449)
  	at org.apache.hadoop.dfs.FSImage.loadFSEdits(FSImage.java:846)
  	at org.apache.hadoop.dfs.FSImage.loadFSImage(FSImage.java:675)
  	at org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:289)
  	at org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:80)
  	at org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:294)
  	at org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java:273)
  	at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:148)
  	at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:193)
  	at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:179)
  	at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:830)
  	at org.apache.hadoop.dfs.NameNode.main(NameNode.java:839)
  09/07/02 09:18:06 INFO ipc.Server: Stopping server on 8020
  09/07/02 09:18:06 ERROR dfs.NameNode: java.io.EOFException
  	at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:323)
  	at org.apache.hadoop.io.UTF8.readFields(UTF8.java:103)
  	at org.apache.hadoop.dfs.FSImage.readString(FSImage.java:1368)
  	at org.apache.hadoop.dfs.FSEditLog.readLong(FSEditLog.java:1119)
  	at org.apache.hadoop.dfs.FSEditLog.loadFSEdits(FSEditLog.java:449)
  	at org.apache.hadoop.dfs.FSImage.loadFSEdits(FSImage.java:846)
  	at org.apache.hadoop.dfs.FSImage.loadFSImage(FSImage.java:675)
  	at org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:289)
  	at org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:80)
  	at org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:294)
  	at org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java:273)
  	at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:148)
  	at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:193)
  	at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:179)
  	at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:830)
  	at org.apache.hadoop.dfs.NameNode.main(NameNode.java:839)
  
  09/07/02 09:18:06 INFO dfs.NameNode: SHUTDOWN_MSG: 
  /************************************************************
  SHUTDOWN_MSG: Shutting down NameNode at hadoop/140.110.X.X
  ************************************************************/