小编给大家分享一下hadoop-006完全分布式问题有哪些，希望大家阅读完这篇文章之后都有所收获，下面让我们一起去探讨吧！
成都创新互联公司是一家专业提供柞水企业网站建设,专注与成都网站制作、做网站、外贸营销网站建设、H5开发、小程序制作等业务。10年已为柞水众多企业、政府机构等服务。创新互联专业网站制作公司优惠进行中。
一、运行mapreduce 任务，但是在yarn里看不到任务。
bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount /test /out1
我配置的resourceManager 为 http://192.168.31.136:8088
原因是在 mapred-site.xml 没有配置
mapreduce.framework.name
yarn
|
二、配置好问题一后，运行mr任务提示The auxService:mapreduce_shuffle does not exist
原因是在yarn-site.xml 没有配置yarn.nodemanager.aux-services 节点
yarn.nodemanager.aux-services
mapreduce_shuffle
|
报错如下
16/11/29 23:10:45 INFO mapreduce.Job: Task Id : attempt_1480432102879_0001_m_000000_2, Status : FAILED
Container launch failed for container_e02_1480432102879_0001_01_000004 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) |
-----------------------------------------------------------------------------------------------------------------
配置文件记录
已提前搭建好三个节点的zookeeper 小集群，配置可以实现 HDFS ＨＡ和 ＹＡＲＮ ＨＡ
一、hadoop-env.sh 修改了
| export JAVA_HOME=/usr/lib/jvm/jdk8/jdk1.8.0_111 |
二、yarn-env.sh 修改了
| export JAVA_HOME=/usr/lib/jvm/jdk8/jdk1.8.0_111 |
三、core-site.xml
fs.defaultFS
hdfs://mycluster
dfs.journalnode.edits.dir
/home/jxlgzwh/hadoop-2.7.2/data/jn
hadoop.tmp.dir
/home/jxlgzwh/hadoop-2.7.2/data/tmp
ha.zookeeper.quorum
master:2181,slave01:2181,slave02:2181
|
四、hdfs-site.xml
dfs.nameservices
mycluster
dfs.ha.namenodes.mycluster
nn1,nn2
dfs.namenode.rpc-address.mycluster.nn1
master:8020
dfs.namenode.rpc-address.mycluster.nn2
slave01:8020
dfs.namenode.http-address.mycluster.nn1
master:50070
dfs.namenode.http-address.mycluster.nn2
slave01:50070
dfs.namenode.shared.edits.dir
qjournal://master:8485;slave01:8485;slave02:8485/mycluster
dfs.client.failover.proxy.provider.mycluster
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
dfs.ha.fencing.methods
sshfence
dfs.ha.fencing.ssh.private-key-files
/home/jxlgzwh/.ssh/id_dsa
dfs.replication
3
dfs.permissions.enabled
false
dfs.ha.automatic-failover.enabled
true
|
五、mapred-site.xml
mapreduce.framework.name
yarn
|
六、slaves
192.168.31.136
192.168.31.130
192.168.31.229 |
七、yarn-site.xml
yarn.resourcemanager.ha.enabled
true
yarn.resourcemanager.cluster-id
cluster1
yarn.resourcemanager.ha.rm-ids
rm1,rm2
yarn.resourcemanager.hostname.rm1
master
yarn.resourcemanager.hostname.rm2
slave01
yarn.resourcemanager.webapp.address.rm1
master:8088
yarn.resourcemanager.webapp.address.rm2
slave01:8088
yarn.resourcemanager.zk-address
master:2181,slave01:2181,slave02:2181
yarn.resourcemanager.recovery.enabled
true
yarn.resourcemanager.zk-state-store.parent-path
/rmstore
yarn.resourcemanager.store.class
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore
yarn.nodemanager.recovery.enabled
true
yarn.nodemanager.address
0.0.0.0:45454
yarn.nodemanager.recovery.dir
/home/jxlgzwh/hadoop-2.7.2/data/tmp/yarn-nm-recovery
yarn.nodemanager.aux-services
mapreduce_shuffle
|
八、配置 /etc/hosts 文件 并配置 ssh免密码登录
192.168.31.136 master.com master
192.168.31.130 slave01
192.168.31.229 slave02 |
看完了这篇文章，相信你对“hadoop-006完全分布式问题有哪些”有了一定的了解，如果想了解更多相关知识，欢迎关注创新互联行业资讯频道，感谢各位的阅读！
当前题目：hadoop-006完全分布式问题有哪些
标题链接：
http://cdkjz.cn/article/iehcej.html
