Hadoop yarn报错：org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce

Dear 丶 2022-05-09 10:26 349阅读 0赞

启动hdfs和yarn后，运行wordcount程序，错误如下：

[root@master hadoop]# hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /wcinput/input.txt /wcoutput
    18/10/06 14:21:43 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.56.100:8032
    18/10/06 14:21:45 INFO input.FileInputFormat: Total input paths to process : 1
    18/10/06 14:21:46 INFO mapreduce.JobSubmitter: number of splits:1
    18/10/06 14:21:46 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1538806803872_0001
    18/10/06 14:21:47 INFO impl.YarnClientImpl: Submitted application application_1538806803872_0001
    18/10/06 14:21:47 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1538806803872_0001/
    18/10/06 14:21:47 INFO mapreduce.Job: Running job: job_1538806803872_0001
    18/10/06 14:22:00 INFO mapreduce.Job: Job job_1538806803872_0001 running in uber mode : false
    18/10/06 14:22:00 INFO mapreduce.Job:  map 0% reduce 0%
    18/10/06 14:22:02 INFO mapreduce.Job: Task Id : attempt_1538806803872_0001_m_000000_0, Status : FAILED
    Container launch failed for container_1538806803872_0001_01_000002 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
    	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    	at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
    	at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
    	at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
    	at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    	at java.lang.Thread.run(Thread.java:745)
    
    18/10/06 14:22:05 INFO mapreduce.Job: Task Id : attempt_1538806803872_0001_m_000000_1, Status : FAILED
    Container launch failed for container_1538806803872_0001_01_000003 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
    	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    	at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
    	at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
    	at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
    	at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    	at java.lang.Thread.run(Thread.java:745)
    
    18/10/06 14:22:06 INFO mapreduce.Job: Task Id : attempt_1538806803872_0001_m_000000_2, Status : FAILED
    Container launch failed for container_1538806803872_0001_01_000004 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
    	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    	at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
    	at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
    	at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
    	at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    	at java.lang.Thread.run(Thread.java:745)
    
    18/10/06 14:22:09 INFO mapreduce.Job:  map 100% reduce 100%
    18/10/06 14:22:09 INFO mapreduce.Job: Job job_1538806803872_0001 failed with state FAILED due to: Task failed task_1538806803872_0001_m_000000
    Job failed as tasks failed. failedMaps:1 failedReduces:0
    
    18/10/06 14:22:09 INFO mapreduce.Job: Counters: 13
    	Job Counters 
    		Failed map tasks=4
    		Killed reduce tasks=1
    		Launched map tasks=4
    		Other local map tasks=3
    		Data-local map tasks=1
    		Total time spent by all maps in occupied slots (ms)=4
    		Total time spent by all reduces in occupied slots (ms)=0
    		Total time spent by all map tasks (ms)=4
    		Total time spent by all reduce tasks (ms)=0
    		Total vcore-milliseconds taken by all map tasks=4
    		Total vcore-milliseconds taken by all reduce tasks=0
    		Total megabyte-milliseconds taken by all map tasks=4096
    		Total megabyte-milliseconds taken by all reduce tasks=0

其中最主要的就是：org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce，看了很多博客但是却解决不了，大家都提示修改yarn-site.xml文件但是没有提示修改集群里哪个机器的（完全分布式）

以下是在slave结点里添加：

<property>  
        <name>yarn.nodemanager.aux-services</name>  
        <value>mapreduce_shuffle</value>  
    </property>  
     
    <property>
        <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>

最终master 的yarn-site.xml:

<configuration>
    
    
    	<property>
    	    <name>yarn.resourcemanager.hostname</name>
    	    <value>master</value>
    	</property>
     
    	<property>  
    	    <name>yarn.nodemanager.aux-services</name>  
    	    <value>mapreduce_shuffle</value>  
    	</property>  
     
    	<property>
    	    <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
    	    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    	</property>
    
    	<property>  
    	    <name>yarn.resourcemanager.address</name>  
    	    <value>master:8032</value>  
    	</property> 
    
    	<property>
    	    <name>yarn.resourcemanager.scheduler.address</name>  
    	    <value> master:8030</value>  
    	</property>
    
    	<property>
    	    <name>yarn.resourcemanager.resource-tracker.address</name>  
    	    <value> master:8031</value>  
    	</property>
    
    </configuration>

三个slave的yarn-site.xml:

<configuration>
    
    
    
    	<property>  
    	    <name>yarn.resourcemanager.address</name>  
    	    <value>master:8032</value>  
    	</property> 
    
    	<property>
    	    <name>yarn.resourcemanager.scheduler.address</name>  
    	    <value> master:8030</value>  
    	</property>
    
    	<property>
    	    <name>yarn.resourcemanager.resource-tracker.address</name>  
    	    <value> master:8031</value>  
    	</property>
    
    	<property>  
    	    <name>yarn.nodemanager.aux-services</name>  
    	    <value>mapreduce_shuffle</value>  
    	</property>  
     
    	<property>
    	    <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
    	    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    	</property>
    </configuration>

以上就可以解决org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce

如有错误，请指教