linux下jar命令和eclipse两种方式生成wordcount.jar包和hadoop下wordcount实例的运行

Dear 丶 2022-08-08 06:15 190阅读 0赞

wordcount项目在包solution 下由三个文件组成，一个是包含主类的WordCount.java，一个是mapper文件WordMapper.java，一个是reducer文件SumReducer.java。

WordCount.java文件源码如下：

package solution;
    
    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
    import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
    import org.apache.hadoop.mapreduce.Job;
    
    public class WordCount {
    
      public static void main(String[] args) throws Exception {
    
        if (args.length != 2) {
          System.out.printf(
              "Usage: WordCount <input dir> <output dir>\n");
          System.exit(-1);
        }
    
        Job job = new Job();
    
        job.setJarByClass(WordCount.class);
    
        job.setJobName("Word Count");
    
        FileInputFormat.setInputPaths(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
    
        job.setMapperClass(WordMapper.class);
        job.setReducerClass(SumReducer.class);
    
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
    
        boolean success = job.waitForCompletion(true);
        System.exit(success ? 0 : 1);
      }
    }

WordMapper.java的源码如下：

package solution;
    
    import java.io.IOException;
    
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.io.LongWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Mapper;
    
    public class WordMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
    
      @Override
      public void map(LongWritable key, Text value, Context context)
          throws IOException, InterruptedException {
    
        String line = value.toString();
    
        for (String word : line.split("\\W+")) {
          if (word.length() > 0) {
    
            context.write(new Text(word), new IntWritable(1));
          }
        }
      }
    }

SumReducer.java的源码如下：

package solution;
    
    import java.io.IOException;
    
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Reducer;
      
    public class SumReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
    
      @Override
    	public void reduce(Text key, Iterable<IntWritable> values, Context context)
    			throws IOException, InterruptedException {
    		int wordCount = 0;
    		for (IntWritable value : values) {
    			wordCount += value.get();
    		}
    		context.write(key, new IntWritable(wordCount));
    	}
    }

一、生成wordcount.jar

1、jar命令生成wordcount.jar

（1）首先进入solution文件夹所在目录，然后编译该项目，生成字节码文件

执行命令为

javac -classpath `hadoop classpath` solution/*.java

注意hadoop classpath 是用后单引号括起来的，表示用该命令执行的结果来填充改位置。

这样在solution下就多出了三个.class文件。

（2）将solution打包成jar

执行命令

jar cvf wordcount.jar solution/*.class

这样就在当前目录下生成了wordcount.jar。

查看wordcount.jar中的内容可以看到solution文件夹和包含清单文件的META-INF文件夹。