介绍Hadoop常用发行版及选型Apache HadoopCDH : Cloudera Distributed HadoopHDP : Hortonworks Data Platform一般使用CDH下载地址http://archive.cloudera.com/cdh5/cdh/5/LZ下载hadoop-2.6.0-cdh5.7.0只要cdh版本号一样即可这样各个组件之间不会冲突看到链接为http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0/ // 将链接改为如下形式即可下载 http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz准备hostname设置vi/etc/sysconfig/networkNETWORKINGyesHOSTNAMEhadoop000hostname和ip地址的设置vi/etc/hosts192.168.199.100 hadoop000192.168.199.101 hadoop001192.168.199.102 hadoop002角色分配hadoop000 NameNode/DataNode ResourceManager/NodeManager hadoop001 DataNode NodeManager hadoop002 DataNode NodeManager前置安装SSH免密登陆在每台机器上运行// 生成公钥和私钥对 ssh-keygen-trsa生成的文件在~/.ssh下id_rsa和id_rsa.pub执行如下命令输入密码即可就是把hadoop000的公钥放置在hadoop000hadoop001和hadoop002上这样就直接通过ssh登陆或者自己手动吧hadoop000 ~/.ssh/id_rsa.pub的内容复制到hadoop000hadoop001和hadoop002上的~/.ssh/authorized_keys即可ssh-copy-id-i~/.ssh/id_rsa.pub hadoop000 ssh-copy-id-i~/.ssh/id_rsa.pub hadoop001 ssh-copy-id-i~/.ssh/id_rsa.pub hadoop002公钥访问时需要把用户的公钥放置在~/.ssh/authorized_keys文件中上面的命令做的就是这个功能将hadoop的tar包解压到app目录下tar-xvfhadoop-2.6.0-cdh5.7.0.tar.gz-C~/app配置用户级别的环境变量// 打开文件vi~/.bash_profile //加入如下命令exportHADOOP_HOME/data/qa/app/hadoop-2.6.0-cdh5.7.0exportPATH$HADOOP_HOME/bin:$PATH// 刷新一下source~/.bash_profile取出JAVA_HOME的路径echo$JAVA_HOME配置~/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop下的hadoop-env.sh设置JAVA_HOMEexportJAVA_HOME/data/lib/jdk8配置~/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop下的core-site.xmlconfigurationpropertynamefs.default.name/namevaluehdfs://mad01:8020/value/property/configuration配置~/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop下的hdfs-site.xmlconfigurationpropertynamedfs.namenode.name.dir/namevalue/data/qa/app/tmp/dfs/name/value/propertypropertynamedfs.datanode.data.dir/namevalue/data/qa/app/tmp/dfs/data/value/property/configuration配置~/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop下的yarn-site.xmlconfigurationpropertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/value/propertypropertynameyarn.resourcemanager.hostname/namevaluemad01/value/property/configuration配置~/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop下的mapred-site.xmlcpmapred-site.xml.template mapred-site.xmlvimapred-site.xmlconfigurationpropertynamemapreduce.framework.name/namevalueyarn/value/property/configuration配置~/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop下的slaves// 将localhost改为如下 hadoop000 hadoop001 hadoop002分发安装包到hadoop001和hadoop002节点scp-r~/app hadoophadoop001:~/scp-r~/app hadoophadoop002:~/## 分发环境变量scp~/.bash_profile hadoophadoop001:~/scp~/.bash_profile hadoophadoop002:~/## 生效source~/.bash_profile启动对HDFS NameNode做格式化只要在hadoop000上执行即可cd$HADOOP_HOME/bin ./hdfs namenode-format启动集群只要在hadoop000上执行即可cd$HADOOP_HOME/sbin ./start-all.sh## 关闭集群./stop-all.sh验证jpshadoop000NameNode SecondaryNameNode ResourceManager DataNode NodeManagerhadoop001hadoop002NodeManager DataNodewebuihttp://hadoop000:50070http://hadoop000:8088备查命令hadoop-daemons.sh start datanode 启动所有的数据节点 hadoop-daemon.sh start datanode 只启动当前的数据节点参考博客
