Hadoop-4.MapReduce的应用

「爱情、让人受尽委屈。」 2022-03-26 09:39 186阅读 0赞

**目录：**

1.  MapReduce简介
2.  MapReduce应用场景
3.  MapReduce环境搭建
4.  Java操作MapReduce
5.  MapReduce程序运行
6.  序列化
7.  RPC协议
8.  MapReduce输入处理类
9.  MapReduce执行过程：[MapReduce执行过程简概][MapReduce]

## ##

### 一.MapReduce简介 ###

**1.概念**  
![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70][]  
**2.概述**  
![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 1][]

## ##

### 二.MapReduce应用场景   ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 2][]   ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 3][] ###

## ##

### 三.MapReduce环境搭建 ###

**伪分布式环境搭建：在hdfs环境搭建好的前提下，官网文档：**[http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html][http_hadoop.apache.org_docs_stable_hadoop-project-dist_hadoop-common_SingleCluster.html]

1.  配置  
    第一步：先修改文件名，因为hadoop目录下只有mapred-site.xml.template文件：
    
        mv mapred-site.xml.template mapred-site.xml
    
    第二步：修改hadoop安装目录下etc/hadoop/mapred-site.xml:
    
        <configuration>
            <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
            </property>
        </configuration>
    
    第三步：修改hadoop安装目录下etc/hadoop/yarn-site.xml:
    
        <configuration>
            <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
            </property>
        </configuration>
2.  启动
    
    1.  先启动hdfs：start-dfs.sh  
        验证：  
        ![20190123142437716.png][]
    2.  启动yarn，mapReduce建立在yarn框架之上，启动yam就启动了mapReduce：
        
            在hadoop安装目录下的sbin目录下 start-yarn.sh
            验证：
3.  测试：  
    例如：http://localhost:8088/  
    说明：为什么是http，而不是https，https比http协议更安全，有转密技术，https需要资金购买，所以用https作为测试不现实  
    ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 4][]

## ##

### 四.Java操作MapReduce ###

### ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 5][]   ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 6][]   ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 7][] ###

**需求：**  
![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 8][]

**开发步骤：**

1.  将 bbb.txt 上传到hdfs下，对它进行操作
    
        [root@kd01 test]# vi bbb.txt
        [root@kd01 test]# hdfs dfs -put bbb.txt /user/yiguang/
2.  创建一个maven项目，在pom.xml文件中添加依赖，是在hdfs的基础上，所以hdfs的依赖包也需要![20190314004639611.png][]  
      
    pom.xml
    
        <dependencies>
        	<dependency>
        		<groupId>junit</groupId>
        		<artifactId>junit</artifactId>
        		<version>3.8.1</version>
        		<scope>test</scope>
        	</dependency>
        	
        	
        	<dependency>
        		<groupId>org.apache.hadoop</groupId>
        		<artifactId>hadoop-hdfs</artifactId>
        		<version>2.7.3</version>
        	</dependency>
        	<dependency>
        		<groupId>org.apache.hadoop</groupId>
        	 	<artifactId>hadoop-common</artifactId>
        	 	<version>2.7.3</version>
        	</dependency>
        	<dependency>
        		<groupId>org.apache.hadoop</groupId>
        		<artifactId>hadoop-client</artifactId>
        		<version>2.7.3</version>
        	</dependency>
        	
        	
        	<dependency>
        	 	<groupId>org.apache.hadoop</groupId>
        	 	<artifactId>hadoop-mapreduce-client-core</artifactId>
        	 	<version>2.7.3</version>
        	</dependency>
        	<dependency>
        	 	<groupId>org.apache.hadoop</groupId>
        	 	<artifactId>hadoop-mapreduce-client-common</artifactId>
        	 	<version>2.7.3</version>
        	</dependency>
        </dependencies>
3.  编写map函数  
    ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 9][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 10][]
    
        package com.yiguang.test.mapreduceTest;
        
        import java.io.IOException;
        
        import org.apache.hadoop.io.IntWritable;
        import org.apache.hadoop.io.LongWritable;
        import org.apache.hadoop.io.Text;
        import org.apache.hadoop.mapreduce.Mapper;
        
        public class WordMap extends Mapper<LongWritable, Text, Text, LongWritable>{
        	@Override
        	protected void map(LongWritable key,Text value,Context context) throws IOException, InterruptedException {
                        /* 
                         * key: 输入的key 
                         * value: 数据
                         * context: Map上下文 
                         */
        		//得到一行的值
        		//将拿到的一行数据通过空格分割为一个一个的元素
        		String[] str=value.toString().split(" ");
        		for(String str1:str) {
        			//将每一个元素发送到reduce
        			context.write(new Text(str1),new LongWritable(1));
        		}
        	}
        }
4.  编写reduce函数
    
        package com.yiguang.test.mapreduceTest;
        
        import java.io.IOException;
        
        import org.apache.hadoop.io.IntWritable;
        import org.apache.hadoop.io.LongWritable;
        import org.apache.hadoop.io.Text;
        import org.apache.hadoop.mapreduce.Reducer;
        
        
        public class WordReduce extends Reducer<Text,LongWritable,Text,LongWritable>{
        	@Override
        	protected void reduce(Text key,Iterable<LongWritable> values,Context context) throws IOException, InterruptedException {
        		int sum=0;
        		for(LongWritable iw:values) {
        			sum+=iw.get();
        		}
        		//输出
        		context.write(key, new LongWritable(sum));
        	}
        
        }
5.  编写main函数，注意：引入类时不要引入了其他的类，否则会出错，例如：[Hadoop执行job出错：java.lang.ClassCastException][Hadoop_job_java.lang.ClassCastException]
    
        package com.yiguang.test.mapreduceTest;
        
        import java.io.IOException;
        
        import org.apache.hadoop.conf.Configuration;
        import org.apache.hadoop.fs.Path;
        import org.apache.hadoop.io.LongWritable;
        import org.apache.hadoop.io.Text;
        import org.apache.hadoop.mapreduce.Job;
        import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
        import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
        
        
        
        public class WordJob {
        	public static void main(String[] args) throws Exception{
        		//创建一个job=map+reduce
        		Job job=Job.getInstance(new Configuration());
        		job.setJobName("kd01");
        		//指定任务的入口
        		job.setJarByClass(WordJob.class);
        		
        		//指定job的mapper
        		job.setMapperClass(WordMap.class);
        		job.setMapOutputKeyClass(Text.class);
        		job.setMapOutputValueClass(LongWritable.class);
        		
        		//指定job的reducer
        		job.setReducerClass(WordReduce.class);
        		job.setOutputKeyClass(Text.class);
        		job.setOutputValueClass(LongWritable.class);
        		
        		//指定任务的输入:原始数据的存放位置
        		FileInputFormat.addInputPath(job, new Path("/user/yiguang/bbb.txt"));
        		//指定任务的输出：指定一个空目录或者一个不存在的目录
        		FileOutputFormat.setOutputPath(job, new Path("/kd01_out"));
        		
        		//提交任务
        		job.waitForCompletion(true);	
        
        	}
        }
6.  运行：第五个知识点

## ##

### 五.MapReduce程序运行 ###

### **![20190123143740687.png][]** ###

**注意：**在项目编译的时候jdk的版本与在linux中发布hadoop时，hadoop关联的jdk的版本，两者要一致，如果两者不同，可以修改项目编译的版本，在pom.xml中通过插件的方式，修改maven项目jdk编译的版本。[指定maven项目编译JDK版本][maven_JDK]

**发布：**

[root@kd01 home]# hadoop jar mapreduceTest-0.0.1-SNAPSHOT.jar com.yiguang.test.mapreduceTest.WordJob

**查看：**![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 11][]  
![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 12][]  
![20190314000457426.png][]

## ##

### 六.序列化   ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 13][]   ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 14][]   ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 15][]   ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 16][] ###

## ##

### 七.RPC协议 ###

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 17][]  
例如：zookeeper+dubbo，发布接口，供使用者调用  
介绍及使用：[https://www.jb51.net/article/127852.htm][https_www.jb51.net_article_127852.htm]  
[https://blog.csdn.net/zhaowen25/article/details/45443951][https_blog.csdn.net_zhaowen25_article_details_45443951]

### 八.MapReduce输入处理类 ###

类结构

![format_png][]

**InputFormat：**

*  决定读取数据的格式，可以是文件或数据库等
 *  作用**：**
    
    1.  验证作业输入的正确性，如格式等
    2.  将输入文件切割成逻辑分片(InputSplit)，一个InputSplit将会被分配给一个独立的Map任务
    3.  提供RecordReader实现，读取InputSplit中的"K-V对"供Mapper使用
 *  方法：
    
    1.  **List getSplits():** 获取由输入文件计算出输入分片(InputSplit)，解决数据或文件分割成片问题
    2.  **RecordReader <k,v>createRecordReader():</k,v>** 创建RecordReader，从InputSplit中读取数据，解决读取分片中数据问题

**FileInputFormat：**

*  FileInputFormat是所有以文件为数据源的InputFormat实现的基类，FileInputFormat保存作为job输入的所有文件，并实现了对输入文件计算splits的方法。至于获得记录的方法是有不同的子类 ------TextInputFormat进行实现的。
 *  InputForamt负责处理MR的输入部分。

**TextInputFormat **

*  TextInputFormat 是默认的处理类，处理普通文本文件
 *  输入文件中的每一行就是一个记录，Key是这一行的byte offset（起始偏移量），而value是这一行的内容
 *  默认以\\n 或回车键作为一行记录
 *  TextInputFormat继承了FileInputFormat

**InputSplit（重点）**

*  代表一个个逻辑分片，并没有真正存储数据，只是提供了一个如何将数据分片的方法
 *  Split内有Location信息，利于数据局部化
 *  一个InputSplit给一个单独的Map处理
 *  在执行mapreduce 之前，原始数据被分割成若干个split，每个split作为一个map任务的输入，在map执行过程中split被分解为一个个记录（key- value对），map会依次处理每一个记录。  
    FileInputFormat只划分比HDFS block大的文件，所以FileInputFormat 划分的结果是这个文件或这个文件的一部分。  
    如果一个文件的大小比block小，将不会被划分，**这也是Hadoop处理大文件的效率要比处理很多小小文件效率高的原因。**  
    当Hadoop处理很多小文件（文件大小小于hdfs block大小）的时候，由于FileInputFormat 不会对小文件进行划分，所以每一个小文件都会被当做一个split 并分配一个map任务，导致效率低下。

**RecordReader**：将InputSplit拆分成一个个<key,value>对给Map处理，也是实际的文件读取分隔对象</key,value>

[MapReduce]: http://www.cnblogs.com/hipercomer/p/4516581.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70]: /images/20220326/3fb5a322d74a46d0976b8bba5b051554.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 1]: /images/20220326/2a88d2b575a8465094ce582ceb75b726.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20190123124700932.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 3]: /images/20220326/d68891c1da4043988e1d24bbfb4e3db3.png
[http_hadoop.apache.org_docs_stable_hadoop-project-dist_hadoop-common_SingleCluster.html]: http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html
[20190123142437716.png]: /images/20220326/d07fa4b18a5547dfb536580a78f15f8c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 4]: /images/20220326/db24bba01bcd41f6b200bdae8574dd8d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20190123125122976.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20190123143837955.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 7]: /images/20220326/a62d7c0366804ee1b6399fda0575513d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 8]: /images/20220326/ac31faccb4c8483bb8a5394f1534c727.png
[20190314004639611.png]: /images/20220326/8b625abb4c6946e69fb85de6c7cc4656.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/2019012318264463.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 10]: /images/20220326/ad010317a09f41dca87d3edf96d26728.png
[Hadoop_job_java.lang.ClassCastException]: https://shidan66.iteye.com/blog/2001108
[20190123143740687.png]: /images/20220326/869ad59274004e6db513eb4422945795.png
[maven_JDK]: https://blog.csdn.net/shengtianbanzi_/article/details/80557697
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 11]: /images/20220326/eda017ebc0f74b0fabdbc718f6d66150.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 12]: /images/20220326/7b8115a127d9412c9be27a5b6e1358d7.png
[20190314000457426.png]: /images/20220326/7da0572e85894daabd4599e15d8fba61.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 13]: https://img-blog.csdnimg.cn/20190123144017625.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 14]: https://img-blog.csdnimg.cn/20190123143945697.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 15]: https://img-blog.csdnimg.cn/2019012321330424.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 16]: /images/20220326/b6ccfb2c203b43fcab2a9d966ec6e9c4.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lpZ3VhbmdfODIw_size_16_color_FFFFFF_t_70 17]: /images/20220326/5085fe703fed43c99d8836ad7e6e320b.png
[https_www.jb51.net_article_127852.htm]: https://www.jb51.net/article/127852.htm
[https_blog.csdn.net_zhaowen25_article_details_45443951]: https://blog.csdn.net/zhaowen25/article/details/45443951
[format_png]: /images/20220326/a7f4116140314d0c8d8da02ba07a9ee9.png