hadoop集群警告No groups available for user dr.who

分手后的思念是犯贱 2022-04-12 06:22 246阅读 0赞

刚装完的hadoop集群，在查看namenode的log时，散仙发现有如下的警告信息：

2014-10-30 16:49:18,340 INFO org.apache.hadoop.hdfs.StateChange: STATE* Leaving safe mode after 2 secs
    2014-10-30 16:49:18,340 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 0 racks and 0 datanodes
    2014-10-30 16:49:18,340 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks has 0 blocks
    2014-10-30 16:49:18,485 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode RPC up at: fsedump00yw/192.168.223.163:8020
    2014-10-30 16:49:18,485 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services required for active state
    2014-10-30 16:49:18,452 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
    2014-10-30 16:49:18,454 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8020: starting
    2014-10-30 16:49:25,377 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* registerDatanode: from DatanodeRegistration(192.168.223.168, storageID=DS-1779348018-192.168.223.168-50010-1414658969808, infoPort=50075, ipcPort=50020, storageInfo=lv=-47;cid=CID-0f002fcf-1b19-4004-a76e-a736b0b98df1;nsid=1726826048;c=0) storage DS-1779348018-192.168.223.168-50010-1414658969808
    2014-10-30 16:49:25,392 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.223.168:50010
    2014-10-30 16:49:25,983 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: BLOCK* processReport: Received first block report from 192.168.223.168:50010 after starting up or becoming active. Its block contents are no longer considered stale
    2014-10-30 16:49:25,984 INFO BlockStateChange: BLOCK* processReport: from DatanodeRegistration(192.168.223.168, storageID=DS-1779348018-192.168.223.168-50010-1414658969808, infoPort=50075, ipcPort=50020, storageInfo=lv=-47;cid=CID-0f002fcf-1b19-4004-a76e-a736b0b98df1;nsid=1726826048;c=0), blocks: 0, processing time: 15 msecs
    2014-10-30 16:49:25,998 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* registerDatanode: from DatanodeRegistration(192.168.223.163, storageID=DS-335820978-192.168.223.163-50010-1414658964818, infoPort=50075, ipcPort=50020, storageInfo=lv=-47;cid=CID-0f002fcf-1b19-4004-a76e-a736b0b98df1;nsid=1726826048;c=0) storage DS-335820978-192.168.223.163-50010-1414658964818
    2014-10-30 16:49:26,002 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.223.163:50010
    2014-10-30 16:49:26,409 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: BLOCK* processReport: Received first block report from 192.168.223.163:50010 after starting up or becoming active. Its block contents are no longer considered stale
    2014-10-30 16:49:26,409 INFO BlockStateChange: BLOCK* processReport: from DatanodeRegistration(192.168.223.163, storageID=DS-335820978-192.168.223.163-50010-1414658964818, infoPort=50075, ipcPort=50020, storageInfo=lv=-47;cid=CID-0f002fcf-1b19-4004-a76e-a736b0b98df1;nsid=1726826048;c=0), blocks: 0, processing time: 0 msecs
    2014-10-30 16:49:31,077 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* registerDatanode: from DatanodeRegistration(192.168.223.167, storageID=DS-563266444-192.168.223.167-50010-1414682451971, infoPort=50075, ipcPort=50020, storageInfo=lv=-47;cid=CID-0f002fcf-1b19-4004-a76e-a736b0b98df1;nsid=1726826048;c=0) storage DS-563266444-192.168.223.167-50010-1414682451971
    2014-10-30 16:49:31,078 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.223.167:50010
    2014-10-30 16:49:31,174 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: BLOCK* processReport: Received first block report from 192.168.223.167:50010 after starting up or becoming active. Its block contents are no longer considered stale
    2014-10-30 16:49:31,174 INFO BlockStateChange: BLOCK* processReport: from DatanodeRegistration(192.168.223.167, storageID=DS-563266444-192.168.223.167-50010-1414682451971, infoPort=50075, ipcPort=50020, storageInfo=lv=-47;cid=CID-0f002fcf-1b19-4004-a76e-a736b0b98df1;nsid=1726826048;c=0), blocks: 0, processing time: 0 msecs
    2014-10-30 16:50:31,872 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.223.163
    2014-10-30 16:50:31,872 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs
    2014-10-30 16:50:31,872 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 1
    2014-10-30 16:50:31,872 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Total time for transactions(ms): 1 Number of transactions batched in Syncs: 0 Number of syncs: 2 SyncTimes(ms): 30 
    2014-10-30 16:50:31,873 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Total time for transactions(ms): 1 Number of transactions batched in Syncs: 0 Number of syncs: 3 SyncTimes(ms): 30 
    2014-10-30 16:50:31,876 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits file /data/hadoop-nd/current/edits_inprogress_0000000000000000001 -> /data/hadoop-nd/current/edits_0000000000000000001-0000000000000000002
    2014-10-30 16:50:31,876 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 3
    2014-10-30 16:50:33,246 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Opening connection to http://fsedump00yw:8021/getimage?getimage=1&txid=2&storageInfo=-47:1726826048:0:CID-0f002fcf-1b19-4004-a76e-a736b0b98df1
    2014-10-30 16:50:33,876 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer took 0.63s at 0.00 KB/s
    2014-10-30 16:50:33,876 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000002 size 198 bytes.
    2014-10-30 16:50:33,888 INFO org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Going to retain 2 images with txid >= 0
    2014-10-30 16:53:22,379 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: trying to get DT with no secret manager running
    2014-10-30 16:53:23,180 WARN org.apache.hadoop.security.UserGroupInformation: No groups available for user dr.who
    2014-10-30 16:53:23,253 WARN org.apache.hadoop.security.UserGroupInformation: No groups available for user dr.who

注意上述代码的最后几行发出的警告信息，意思是说对于dr.who这个用户，没有有效的组归属，在hadoop的权限认证中，如果默认没有用户组声明，就会把当前的这个hadoop用户，归属于dr.who用户，当然这个用户在我们的用户组里面是不存在的，所以log里面就记录了这个warn信息。  
  
解决方法如下：  
  
方法（1），在core-site.xml里面添加如下配置：

<property>
        <name>hadoop.http.staticuser.user</name>
        <value>hadoop</value>
    </property>

hadoop是散仙的用户名  
  
方法（2），在当前的用户组里面添加一个dr.who的用户  
gpasswd -a dr.who hadoop  
将dr.who用户添加到hadoop用户组里面  
  
  
如此就能通过linux的权限验证，改完之后，重启hadoop集群之后，就不会出现这样的警告了