IntelliJ IDEA + Maven环境编写第一个hadoop程序

迷南。 2022-05-19 09:19 246阅读 0赞

**1. 新建IntelliJ下的maven项目**

点击File->New->Project，在弹出的对话框中选择Maven，JDK选择你自己安装的版本，点击Next

**2. 填写Maven的`GroupId`和`ArtifactId`**

你可以根据自己的项目随便填，点击`Next`

这样就新建好了一个空的项目

这里程序名填写WordCount,我们的程序是一个通用的网上的范例,用来计算文件中单词出现的次数

**3. 设置程序的编译版本**

打开Intellij的Preference偏好设置，定位到Build, Execution, Deployment->Compiler->Java Compiler，

将WordCount的Target bytecode version修改为你的jdk版本(我的是1.8)

**4. 配置依赖**

编辑pom.xml进行配置

1) 添加apache源

在`project`内尾部添加

<repositories>
        <repository>
            <id>apache</id>
            <url>http://maven.apache.org</url>
        </repository>
    </repositories>

2) 添加hadoop依赖

这里只需要用到基础依赖hadoop-core和hadoop-common；如果需要读写HDFS，  
则还需要依赖hadoop-hdfs和hadoop-client；如果需要读写HBase，则还需要依赖hbase-client

在`project`内尾部添加

![复制代码][copycode.gif]

<dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-core</artifactId>
            <version>1.2.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>2.7.2</version>
        </dependency>
    </dependencies>

![复制代码][copycode.gif]

修改pom.xml完成后，Intellij右上角会提示Maven projects need to be Imported，点击Import Changes以更新依赖,或者点击Enable Auto Import

最后,我的完整的pom.xml如下:

![复制代码][copycode.gif]

<?xml version="1.0" encoding="UTF-8"?>
    <project xmlns="http://maven.apache.org/POM/4.0.0"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
        <modelVersion>4.0.0</modelVersion>
    
        <groupId>com.fun</groupId>
        <artifactId>hadoop</artifactId>
        <version>1.0-SNAPSHOT</version>
    
        <repositories>
            <repository>
                <id>apache</id>
                <url>http://maven.apache.org</url>
            </repository>
        </repositories>
    
        <dependencies>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-core</artifactId>
                <version>1.2.1</version>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-common</artifactId>
                <version>2.7.2</version>
            </dependency>
        </dependencies>
    
        <build>
            <plugins>
                <plugin>
                    <artifactId>maven-dependency-plugin</artifactId>
                    <configuration>
                        <excludeTransitive>false</excludeTransitive>
                        <stripVersion>true</stripVersion>
                        <outputDirectory>./lib</outputDirectory>
                    </configuration>
    
                </plugin>
            </plugins>
        </build>
    </project>

![复制代码][copycode.gif]

**5. 编写主程序**

WordCount.java

![复制代码][copycode.gif]

/**
     * Created by jinshilin on 16/12/7.
     */
    import java.io.IOException;
    import java.util.StringTokenizer;
    
    import org.apache.hadoop.conf.Configuration;
    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Job;
    import org.apache.hadoop.mapreduce.Mapper;
    import org.apache.hadoop.mapreduce.Reducer;
    import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
    import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
    
    public class WordCount {
    
        public static class TokenizerMapper
                extends Mapper<Object, Text, Text, IntWritable> {
    
            private final static IntWritable one = new IntWritable(1);
            private Text word = new Text();
    
            public void map(Object key, Text value, Context context
            ) throws IOException, InterruptedException {
                StringTokenizer itr = new StringTokenizer(value.toString());
                while (itr.hasMoreTokens()) {
                    word.set(itr.nextToken());
                    context.write(word, one);
                }
            }
        }
    
        public static class IntSumReducer
                extends Reducer<Text, IntWritable, Text, IntWritable> {
            private IntWritable result = new IntWritable();
    
            public void reduce(Text key, Iterable<IntWritable> values,
                               Context context
            ) throws IOException, InterruptedException {
                int sum = 0;
                for (IntWritable val : values) {
                    sum += val.get();
                }
                result.set(sum);
                context.write(key, result);
            }
        }
    
        public static void main(String[] args) throws Exception {
            Configuration conf = new Configuration();
            Job job = Job.getInstance(conf, "word count");
            job.setJarByClass(WordCount.class);
            job.setMapperClass(TokenizerMapper.class);
            job.setCombinerClass(IntSumReducer.class);
            job.setReducerClass(IntSumReducer.class);
            job.setOutputKeyClass(Text.class);
            job.setOutputValueClass(IntWritable.class);
            FileInputFormat.addInputPath(job, new Path(args[0]));
            FileOutputFormat.setOutputPath(job, new Path(args[1]));
            System.exit(job.waitForCompletion(true) ? 0 : 1);
        }
    }

![复制代码][copycode.gif]

**6. 配置输入和输出结果文件夹**

1) 添加和src目录同级的input文件夹到项目中

在input文件夹中放置一个或多个输入文件源

我的输入文件源如下:

test.segmented:

dfdfadgdgag  
aadads  
fudflcl  
cckcer  
fadf  
dfdfadgdgag  
fudflcl  
fuck  
fuck  
fuckfuck  
haha  
aaa

2) 配置运行参数  
在Intellij菜单栏中选择Run->Edit Configurations，在弹出来的对话框中点击+，新建一个Application配置。配置Main class为WordCount（可以点击右边的...选择），

Program arguments为input/ output/，即输入路径为刚才创建的input文件夹，输出为output

![72194-20161207141527272-63070507.png][]

**由于Hadoop的设定，下次运行时务必删除`output`文件夹！**

好了,运行程序,结果如下:

aaa 1  
aadads 1  
cckcer 1  
dfdfadgdgag 2  
fadf 1  
fuck 2  
fuckfuck 1  
fudflcl 2  
haha 1

至此,一个简单的hadoop程序完成!

[copycode.gif]: /images/20220519/3aecb01816054cbb80bf424d8503088f.png
[72194-20161207141527272-63070507.png]: /images/20220519/72e8c33eaa83402087a8ae3d969f8f18.png