hadoop 修复hdfs中文件块错误和文件修复 Error: java.io.IOException: Cannot obtain block length for LocatedBlock

向右看齐 2024-01-20 08:14 25阅读 0赞

### 一：问题 Error: java.io.IOException: Cannot obtain block length for LocatedBlock ###

Error: java.io.IOException: Cannot obtain block length for LocatedBlock{BP-1241298414-127.0.0.1-1523601065028:blk_1073766720_25914; getBlockSize()=1077; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[172.16.144.58:50010,DS-69e6b461-59df-4e19-90ff-e766c0e3535c,DISK], DatanodeInfoWithStorage[172.16.112.83:50010,DS-3cef3847-e522-4b28-a84e-ff5ec9f2b02a,DISK]]}
            at org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:444)
            at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:351)
            at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:282)
            at org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:272)
            at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1132)
            at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:329)
            at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:326)
            at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
            at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:326)
            at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:914)
            at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85)
            at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:554)
            at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:792)
            at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
            at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:177)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:422)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1886)
            at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:171)

### 二：解决办法： ###

**1、找出出问题的文件**

输入下面命令：

hdfs fsck /logs/product/desktop-ecommerce/2018-05-31 -openforwrite

**结果：**

Connecting to namenode via http://master:11070/fsck?ugi=root&openforwrite=1&path=%2Flogs%2Fproduct%2Fdesktop-ecommerce%2F2018-05-31
        FSCK started by root (auth:SIMPLE) from /172.16.199.39 for path /logs/product/desktop-ecommerce/2018-05-31 at Fri Jun 01 09:33:35 CST 2018
        ../logs/product/desktop-ecommerce/2018-05-31/FlumeData.1527714183563 1077 bytes, 1 block(s), OPENFORWRITE: .Status: HEALTHY
         Total size:    5336068 B
         Total dirs:    1
         Total files:   4
         Total symlinks:        0
         Total blocks (validated):  4 (avg. block size 1334017 B)
         Minimally replicated blocks:   3 (75.0 %)
         Over-replicated blocks:    0 (0.0 %)
         Under-replicated blocks:   0 (0.0 %)
         Mis-replicated blocks:     0 (0.0 %)
         Default replication factor:    2
         Average block replication: 1.5
         Corrupt blocks:        0
         Missing replicas:      0 (0.0 %)
         Number of data-nodes:      2
         Number of racks:       1
        FSCK ended at Fri Jun 01 09:33:35 CST 2018 in 1 milliseconds

**找到了问题文件：/logs/product/desktop-ecommerce/2018-05-31/FlumeData.1527714183563**

### 三：对文件执行修复命令 ###

hdfs debug recoverLease -path /logs/product/desktop-ecommerce/2018-05-31/FlumeData.1527714183563 -retries 3

### 四：再次执行mapreduce程序 没有报错 完美解决问题！ ###