2.2 MapReduce之Split原理

Myth丶恋晨 2023-02-28 01:49 37阅读 0赞

## InputFormat.class ##

#### getSplits ####

注意是package org.apache.hadoop.mapreduce;包

Logically split the set of input files for the job.  逻辑上的分片

List<InputSplit> getSplits

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70]

#### RecordReader ####

/** 
 * Logically split the set of input files for the job. 
 * 
 * Each {@link InputSplit} is then assigned to an individual {@link Mapper}
 * for processing.
 *
 * Note: The split is a logical split of the inputs and the
 * input files are not physically split into chunks. For e.g. a split could
 * be <input-file-path, start, offset> tuple. The InputFormat
 * also creates the {@link RecordReader} to read the {@link InputSplit}.
 * 
 * @param context job configuration.
 * @return an array of {@link InputSplit}s for the job.
 */

InputFormat
 	List<InputSplit> getSplits
 	---->InputSplit 交给一个Map来处理
 	Logically split the set of input files for the job
 		逻辑：InputSplit不是真正物理上对应的东西 
 		物理：Block 真实存在
 	RecordReader

针对一个InputSplit创建一个RecordReader

/**
       * Called once at initialization.
       * @param split the split that defines the range of records to read
       * @param context the information about the task
       */
      public abstract void initialize(InputSplit split,
                                      TaskAttemptContext context
                                      )

### TextInputFormat.class ###

#### getSplits ####

/** 
 * Generate the list of files and make them into FileSplits.
 * @param job the job context
 * @throws IOException
 */
 public List<InputSplit> getSplits(JobContext job)

输入数据 ===》 inputFormat
    			---> 有几个InputSplit
    				--->MapTask
    			---> RecordReader
    				--->LineRecordReader

**当文件很小时**  
1个.txt 就只有一个Splits。。。两个文件会产生两个splits  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 1]

在Driver中不设置Inputformat 的话 ， 默认走的的是TextInputFormat

Job ------> JobSubmit -----> InputFormat ------>

本地模式的blockSize为32M  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 2]  
**TestInputFormat中没有getSplits方法，所以走的还是父类（abstract class FileInputFormat）的方法**

SplitSize 默认 blockSzie一样大  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 3]

#### bytesRemaing ####

long bytesRemaining = length;
              while (((double) bytesRemaining)/splitSize > SPLIT_SLOP) {
        
                int blkIndex = getBlockIndex(blkLocations, length-bytesRemaining);
                splits.add(makeSplit(path, length-bytesRemaining, splitSize,
                            blkLocations[blkIndex].getHosts(),
                            blkLocations[blkIndex].getCachedHosts()));
                bytesRemaining -= splitSize;
              }

**SPLIT\_SLOP**

private static final double SPLIT_SLOP = 1.1;   // 10% slop

有1.1的富裕  
也就是说即使 129M的文件(默认blockSize 128M)，SplitSize也是1

[fileSize & blockSize][fileSize _ blockSize]

### NLineInputFormat.class ###

##### default #####

public List<InputSplit> getSplits(JobContext job)
 throws IOException {
 
 List<InputSplit> splits = new ArrayList<InputSplit>();
 int numLinesPerSplit = getNumLinesPerSplit(job);
 for (FileStatus status : listStatus(job)) {
 
 splits.addAll(getSplitsForFile(status,
 job.getConfiguration(), numLinesPerSplit));
 }
 return splits;
 }

如果不设置Splits 个数的话，默认numLinesPerSplit = 1;![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 4]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 5]  
一端设置进去，另一端取出  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 6]

##### setNumLinesPerSplit #####

//设置N line
            NLineInputFormat.setNumLinesPerSplit(job,5);
    
            job.setInputFormatClass(NLineInputFormat.class);
            //job.setInputFormatClass(TextInputFormat.class);

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 7]  
**对于Splits的数量**  
可以看到是一个一个增加的 对数量没有定义  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 8]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70]: /images/20230209/19e5971e2a0b4128a28a303a33ec4d90.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 1]: /images/20230209/973eb3c983b74b59bf284119015e245a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 2]: /images/20230209/66bb3de4891e4352aceab554c511748c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 3]: /images/20230209/af58b660b7d64619884670938d631b81.png
[fileSize _ blockSize]: https://blog.csdn.net/samhacker/article/details/23089157
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 4]: /images/20230209/332a5b1cd126491b90906fb37c5d0600.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 5]: /images/20230209/e694a562b3a94e80857abb3dfc770a8c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 6]: /images/20230209/5f6162446b2e4e4c80a64ed1183d4a8b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 7]: /images/20230209/79380e1d7a354012a4f99bd755c05bb1.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70 8]: /images/20230209/9323ec5cb96d4a268b163a69617b428c.png