MapReduce - 词频统计

布满荆棘的人生 2024-04-17 23:15 223阅读 0赞

统计一个文本的词频

  1. package Test01;
  3. import org.apache.hadoop.conf.Configuration;
  4. import org.apache.hadoop.fs.Path;
  5. import org.apache.hadoop.io.IntWritable;
  6. import org.apache.hadoop.io.LongWritable;
  7. import org.apache.hadoop.io.Text;
  8. import org.apache.hadoop.mapreduce.Job;
  9. import org.apache.hadoop.mapreduce.Mapper;
  10. import org.apache.hadoop.mapreduce.Reducer;
  11. import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
  12. import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
  15. public class WordCount {
  16.     public static void main(String[] args){
  17.         try{
  18.             //获取配置对象
  19.             Configuration conf = new Configuration();
  20.             //对conf设置
  21.             //获取job
  22.             Job job = Job.getInstance(conf, "WordCount02");
  23.             //对job设置运行主类
  24.             job.setJarByClass(WordCount.class);
  25.             //对job的map端属性设置
  26.             job.setMapperClass(MyMapper.class);
  27.             job.setMapOutputKeyClass(Text.class);
  28.             job.setMapOutputValueClass(LongWritable.class);
  29.             //对job的reduce端属性设置
  30.             job.setReducerClass(MyReducer.class);
  31.             job.setOutputKeyClass(Text.class);
  32.             job.setOutputValueClass(IntWritable.class);
  33.             //设置job的输入路径和输出路径
  34.             FileInputFormat.addInputPath(job, new Path(args[0]));
  35.             FileOutputFormat.setOutputPath(job, new Path(args[1]));
  36.             //提交作业
  37.             int success = job.waitForCompletion(true) ? 0: 1;
  38.             //退出
  39.             System.exit(success);
  40.         }
  41.         catch (Exception e){
  42.             e.printStackTrace();
  43.         }
  44.     }
  45.     //自定义Mapper类
  46.     public static class MyMapper extends Mapper<Object, Text, Text, LongWritable> {
  47.         public Text k =  new Text();
  48.         public LongWritable v = new LongWritable(1L);
  49.         //map函数，map阶段的核心业务逻辑
  50.         @Override
  51.         protected void map(Object key, Text value, Context context){
  52.             try{
  53.                 //获取行值
  54.                 String row = value.toString();//map中每次读取一行
  55.                 //拆分行值
  56.                 String[] words = row.split(" ");
  57.                 for(String st: words){
  58.                     String st2 = "";
  59.                     char[] ct = st.toCharArray();
  60.                     if ((ct[st.length()-1] >= 'a' && ct[st.length()-1] <= 'z') || (ct[st.length()-1] >= 'A' && ct[st.length()-1] <= 'Z')){
  61.                         st2 = String.valueOf(ct);
  62.                     }
  63.                     else {
  64.                         for(int i = 0; i < st.length()-1; i++){
  65.                             st2 += String.valueOf(ct[i]);
  66.                         }
  67.                     }
  68.                     k.set(st2);
  69.                     context.write(k, v);
  70.                 }
  71.             }
  72.             catch (Exception e){
  73.                 e.printStackTrace();
  74.             }
  75.         }
  76.     }
  77.     //自定义reducer类
  78.     public static class MyReducer extends Reducer<Text, LongWritable, Text, IntWritable> {
  79.         public IntWritable v = new IntWritable();
  80.         //reduce函数，reduce阶段的核心业务逻辑
  81.         @Override
  82.         protected void reduce(Text key, Iterable<LongWritable> value, Context context){
  83.             try{
  84.                 //定义计数器
  85.                 int cnt = 0;
  86.                 for(LongWritable i: value){
  87.                     cnt += i.get();
  88.                 }
  89.                 v.set(cnt);
  90.                 //reduce的输出
  91.                 context.write(key, v);
  92.             }
  93.             catch(Exception e){
  94.                 e.printStackTrace();
  95.             }
  96.         }
  97.     }
  98. }

（1）点开IDEA右边的maven，依次点击 Lifecycle - > clean（右键）-> Run Maven Build
watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JsZXNzaW5nWFJZ_size_16_color_FFFFFF_t_70
（2）依次点击 install（右键）-> Run Maven Build
watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JsZXNzaW5nWFJZ_size_16_color_FFFFFF_t_70 1
（3）将target下的jar包发送到Hadoop服务器
watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JsZXNzaW5nWFJZ_size_16_color_FFFFFF_t_70 2
（4）输入命令运行

  1. yarn jar MapReducePractice-1.0-SNAPSHOT.jar Test01.WordCount /test/input/t1.txt /test/output/01

（5）查看运行结果

[hadoop@hadoop105 ~]$ hdfs dfs -cat /test/output/01/part-r-00000
Dream 1
Hello 2
I 1
Love 1
World 1
You 2
are 1
hope 1
my 1

发表评论

表情：
评论列表 （有 0 条评论，223人围观）

还没有评论，来说两句吧...

相关阅读