Shell$ExitCodeException - Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/service/CompositeService

Tools and Technologies used in this article :

1. Apache Hadoop 2.2.0

2. Windows 7 OS

3. JDK 1.6

Exception

C:\hadoop>bin\yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 5 50
Number of Maps  = 5
Samples per Map = 50
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Starting Job
14/02/27 14:53:04 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/02/27 14:53:04 INFO input.FileInputFormat: Total input paths to process : 5
14/02/27 14:53:04 INFO mapreduce.JobSubmitter: number of splits:5
:
:
:
14/02/27 14:53:05 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1393491185191_0001
14/02/27 14:53:06 INFO impl.YarnClientImpl: Submitted application application_1393491185191_0001 to ResourceManager at /0.0.0.0:8032
14/02/27 14:53:06 INFO mapreduce.Job: The url to track the job: http://ABHIJITG:8088/proxy/application_1393491185191_0001/
14/02/27 14:53:06 INFO mapreduce.Job: Running job: job_1393491185191_0001
14/02/27 14:53:15 INFO mapreduce.Job: Job job_1393491185191_0001 running in uber mode : false
14/02/27 14:53:20 INFO mapreduce.Job:  map 0% reduce 0%
14/02/27 14:53:20 INFO mapreduce.Job: Job job_1393491185191_0001 failed with state FAILED due to: Application application_1393491185191_0001 failed 2 times due to AM Container for appattempt_1393491185191_0001_000002 exited with  exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
        at org.apache.hadoop.util.Shell.run(Shell.java:379)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

        1 file(s) moved.

java.lang.NoClassDefFoundError: org/apache/hadoop/service/CompositeService
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.service.CompositeService
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	... 13 more
Exception in thread "main"

Fix

Add all the required Hadoop jars to the property 'yarn.application.classpath' in yarn-site.xml configuration file.

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>
	<property>
	   <name>yarn.nodemanager.aux-services</name>
	   <value>mapreduce_shuffle</value>
	</property>
	<property>
	   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
	   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
	</property>
	<property>
	   <name>yarn.application.classpath</name>
	   <value>
			%HADOOP_HOME%\etc\hadoop,
			%HADOOP_HOME%\share\hadoop\common\*,
			%HADOOP_HOME%\share\hadoop\common\lib\*,
			%HADOOP_HOME%\share\hadoop\hdfs\*,
			%HADOOP_HOME%\share\hadoop\hdfs\lib\*,
			%HADOOP_HOME%\share\hadoop\mapreduce\*,
			%HADOOP_HOME%\share\hadoop\mapreduce\lib\*,
			%HADOOP_HOME%\share\hadoop\yarn\*,
			%HADOOP_HOME%\share\hadoop\yarn\lib\*
	   </value>
	</property>
</configuration>

Verify

Re-run the example to verify the fix.

C:\hadoop>bin\yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 5 50
Number of Maps  = 5
Samples per Map = 50
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Starting Job
14/02/27 15:30:42 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/02/27 15:30:43 INFO input.FileInputFormat: Total input paths to process : 5
14/02/27 15:30:44 INFO mapreduce.JobSubmitter: number of splits:5
:
:
:
14/02/27 15:30:44 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1393495167950_0001
14/02/27 15:30:44 INFO impl.YarnClientImpl: Submitted application application_1393495167950_0001 to ResourceManager at /0.0.0.0:8032
14/02/27 15:30:44 INFO mapreduce.Job: The url to track the job: http://ABHIJITG:8088/proxy/application_1393495167950_0001/
14/02/27 15:30:44 INFO mapreduce.Job: Running job: job_1393495167950_0001
14/02/27 15:31:01 INFO mapreduce.Job: Job job_1393495167950_0001 running in uber mode : false
14/02/27 15:31:01 INFO mapreduce.Job:  map 0% reduce 0%
14/02/27 15:31:24 INFO mapreduce.Job:  map 40% reduce 0%
14/02/27 15:31:25 INFO mapreduce.Job:  map 80% reduce 0%
14/02/27 15:31:27 INFO mapreduce.Job:  map 100% reduce 0%
14/02/27 15:31:49 INFO mapreduce.Job:  map 100% reduce 100%
14/02/27 15:31:50 INFO mapreduce.Job: Job job_1393495167950_0001 completed successfully
14/02/27 15:31:51 INFO mapreduce.Job: Counters: 43
        File System Counters
                FILE: Number of bytes read=116
                FILE: Number of bytes written=482348
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=1335
                HDFS: Number of bytes written=215
                HDFS: Number of read operations=23
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=3
        Job Counters
                Launched map tasks=5
                Launched reduce tasks=1
                Data-local map tasks=5
                Total time spent by all maps in occupied slots (ms)=109868
                Total time spent by all reduces in occupied slots (ms)=22635
        Map-Reduce Framework
                Map input records=5
                Map output records=10
                Map output bytes=90
                Map output materialized bytes=140
                Input split bytes=745
                Combine input records=0
                Combine output records=0
                Reduce input groups=2
                Reduce shuffle bytes=140
                Reduce input records=10
                Reduce output records=0
                Spilled Records=20
                Shuffled Maps =5
                Failed Shuffles=0
                Merged Map outputs=5
                GC time elapsed (ms)=1221
                CPU time spent (ms)=4410
                Physical memory (bytes) snapshot=1237839872
                Virtual memory (bytes) snapshot=1675640832
                Total committed heap usage (bytes)=1123549184
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=590
        File Output Format Counters
                Bytes Written=97
Job Finished in 68.624 seconds
Estimated value of Pi is 3.16800000000000000000

References

• Run Hadoop wordcount MapReduce Example on Windows
• Build, Install, Configure and Run Apache Hadoop 2.2.0 in Microsoft Windows OS
• ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path
• Maven Build Failure - Hadoop 2.2.0 - [ERROR] class file for org.mortbay.component.AbstractLifeCycle not found