MapReduce Java API实例-排序

╰+哭是因爲堅強的太久メ 2022-09-03 05:15 221阅读 0赞

# 场景 #

MapReduce Java API实例-统计单词出现频率：

[https://blog.csdn.net/BADAO\_LIUMANG\_QIZHI/article/details/119410169][https_blog.csdn.net_BADAO_LIUMANG_QIZHI_article_details_119410169]

上面进行项目环境搭建的基础上。

怎样实现对下面这组数据进行排序

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70][]

注：

博客：  
[https://blog.csdn.net/badao\_liumang\_qizhi][https_blog.csdn.net_badao_liumang_qizhi]  
关注公众号  
霸道的程序猿  
获取编程相关电子书、教程推送与免费下载。

# 实现 #

输入数据格式为每行有一数值，通过MapReduce实现数据的排序功能。

利用Map阶段的Sort功能将要排序的数值作为map函数的key输出，

并在reduce函数设置一个计数器。

1、Map代码实现

package com.badao.sort;
    
    
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Mapper;
    
    import java.io.IOException;
    import java.util.StringTokenizer;
    
    public class SortMapper extends Mapper<Object,Text,IntWritable,IntWritable> {
    
    
        public static IntWritable data = new IntWritable();
    
        //map将输入中value化成IntWritable类型，作为输出的key
        @Override
        public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
    
            String line = value.toString();
            data.set(Integer.parseInt(line));
            //通过write函数写入到本地文件
            context.write(data,new IntWritable(1));
    
        }
    }

2、Reduce代码实现

package com.badao.sort;
    
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Reducer;
    
    import java.io.IOException;
    
    
    public class SortReducer extends Reducer<IntWritable, IntWritable,IntWritable,IntWritable> {
    
    
        public static IntWritable linenum = new IntWritable(1);
        public static int i =1;
    
        @Override
        public void reduce(IntWritable key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
    
            context.write(new IntWritable(i),key);
            ++i;
        }
    }

3、Job实现

package com.badao.sort;
    
    
    import org.apache.hadoop.conf.Configuration;
    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Job;
    import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
    import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
    import org.apache.hadoop.mapreduce.lib.reduce.IntSumReducer;
    
    import java.io.IOException;
    
    public class SortJob {
        public static void main(String[] args) throws InterruptedException, IOException, ClassNotFoundException {
            jobLocal();
        }
    
        public static void jobLocal()throws IOException, ClassNotFoundException, InterruptedException
        {
            Configuration conf = new Configuration();
            //实例化一个作业，word count是作业的名字
            Job job = Job.getInstance(conf, "jobsort");
            //指定通过哪个类找到对应的jar包
            job.setJarByClass(SortJob.class);
    
            //为job设置Mapper类
            job.setMapperClass(SortMapper.class);
            //为job设置reduce类
            job.setReducerClass(SortReducer.class);
    
            //为job的输出数据设置key类
            job.setOutputKeyClass(IntWritable.class);
            //为job输出设置value类
            job.setOutputValueClass(IntWritable.class);
    
            //为job设置输入路径，输入路径是存在的文件夹/文件
            FileInputFormat.addInputPath(job,new Path("D:\\sortData\\sort.txt"));
            //为job设置输出路径
            FileOutputFormat.setOutputPath(job,new Path("D:\\sortdataout"));
            job.waitForCompletion(true);
        }
    
    }

运行后查看输出文件结果

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70 1][]

[https_blog.csdn.net_BADAO_LIUMANG_QIZHI_article_details_119410169]: https://blog.csdn.net/BADAO_LIUMANG_QIZHI/article/details/119410169
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70]: /images/20220829/51b31b2b003b419f82b2a688986c81e2.png
[https_blog.csdn.net_badao_liumang_qizhi]: https://blog.csdn.net/badao_liumang_qizhi
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JBREFPX0xJVU1BTkdfUUlaSEk_size_16_color_FFFFFF_t_70 1]: /images/20220829/2f28ab39a38549c18a543dc7156c0e42.png