Friday, April 13, 2012

LADY GAGA HADOOP NODE ISSUE

Hadoop User group in San Francisco is always a great place to learn about new technologies and meet super interesting people. The last meet up was held at Twitter office. Alex, Dave and Egor joined me this time. Twitter break room is very impressive. It feels like an upscale grocery store with organic soda and healthy snacks. Even though I am a "foodie", I have not seen some of the soda brands they had there. Yay! Go Twitter!

When sessions started, I joined the group about Mahout and it is implication to product recommendations.  Cloudera and WibiData is looking into mahout implementation. In general,  this is not the first time I have heard interest from customers about Mahout.  Mahout definitely  gets a lot of attention.

The second session was about new release of the HBase from Richard from Salesforce. They have implemented quite a few improvements for performance and overall deployment. Another big project to look into.

Alex was telling me about his session about Twitter "reach" session. The basic problem twitter is "Lady Gaga". Every time she tweets some of the twitter servers goes down because she had billion followers!
Guys call it "Lady Gaga problem". I laughed for 20 minutes !!!!




We had a little celebration afterwards!!!



Wednesday, April 4, 2012

Common Hadoop Troubleshooting Tips


1. One of the common problems with new installation is connecting datanode to a different namenode

2012-04-04 18:50:38,863 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /var/lib/hadoop-0.20/cache/hdfs/dfs/data: namenode namespaceID = 1635219806; datanode namespaceID = 976537351
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:238)
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:153)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:410)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:305)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1627)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1567)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1585)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1711)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1728)

2012-04-04 18:50:38,864 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:


Solution:
Delete old data from datanode and connect to namenode again.