二、安装JDK和Hadoop1.安装JDK这步的前提是已经配置好了ssh远程登录1.1打开Xftp 8上传JDK文件1.打开左上角的“”号新建会话2.按照步骤新建会话属性注意“主机”“IP”不要用默认的哦3.编辑完会话属性后直接点“连接”4.成功远程控制的界面如下5.在本地找到所需对应JDK文件6.右键JDK将它拖至hadoop01中1.2创建目录分别在三台虚拟机上新建目录mkdir -p /export/data mkdir -p /export/servers mkdir -p /export/software1.3解压并修改环境配置1.解压jdk文件tar -zxvf /root/jdk-8u202-linux-x64.tar.gz -C /export/servers/2.vi进入/etc/profile修改环境配置vi /etc/profile进入/etc/profile后添加以下内容export JAVA_HOME/export/servers/jdk1.8.0_202 export PATH$JAVA_HOME/bin:$PATH使用source /etc/profile命令初始化系统环境变量可以使用java -version验证JDK是否安装成功1.4分发在hadoop01上分发JDK安装目录scp -r /export/servers/jdk1.8.0_202 roothadoop02:/export/servers/ scp -r /export/servers/jdk1.8.0_202 roothadoop03:/export/servers/ scp /etc/profile roothadoop02:/etc/ scp /etc/profile roothadoop03:/etc/注意文件路径中间不能空格接下来需到hadoop2和hadoop3上使用source /etc/profile初始化系统环境变量。2.安装hadoop2.1上传hadoop压缩包按照上传jdk的方式2.2解压并修改环境变量1.解压hadoop压缩包tar -zxvf /root/hadoop-3.3.1.tar.gz -C /export/servers2.修改环境变量vi /etc/profilevi进入/etc/profile文件修改内容如下export HADOOP_HOME/export/servers/hadoop-3.3.1 export PATH$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin使用source /etc/profile命令初始化系统环境变量可以使用hadoop version验证Hadoop是否安装成功2.3修改Hadoop配置文件切换目录cd/export/servers/hadoop-3.3.1/etc/hadoop1.配置Hadooop运行时的环境。vi hadoop-env.sh在文件末添加以下内容export JAVA_HOME/export/servers/jdk1.8.0_202 export HDFS_NAMENODE_USERroot export HDFS_DATANODE_USERroot export HDFS_SECONDARYNAMENODE_USERroot export YARN_RESOURCEMANAGER_USERroot export YARN_NODEMANAGER_USERroot2.配置Hadoopvi core-site.xml进入文件以后(用shiftG到达文件末)添加以下内容注意要添加在configuration和/configuration之间property namefs.defaultFS/name valuehdfs://hadoop1:9000/value /property property namehadoop.tmp.dir/name value/export/data/hadoop-3.3.1/value /property property namehadoop.http.staticuser.user/name valueroot/value /property property namehadoop.proxyuser.root.hosts/name value*/value /property property namehadoop.proxyuser.root.groups/name value*/value /property property namefs.trash.interval/name value1440/value /property3.配置HDFSvi hdfs-site.xml进入文件以后添加以下内容注意要添加在configuration和/configuration之间property namedfs.replication/name value2/value /property property namedfs.namenode.secondary.http-address/name valuehadoop2:9868/value /property4.配置MapReducevi mapred-site.xml进入文件以后添加以下内容注意要添加在configuration和/configuration之间property namemapreduce.framework.name/name valueyarn/value /property property namemapreduce.jobhistory.address/name valuehadoop1:10020/value /property property namemapreduce.jobhistory.Webapp.address/name valuehadoop1:19888/value /property property nameyarn.app.mapreduce.am.env/name valueHADOOP_MAPRED_HOME${HADOOP_HOME}/value /property property namemapreduce.map.env/name valueHADOOP_MAPRED_HOME${HADOOP_HOME}/value /property property namemapreduce.reduce.env/name valueHADOOP_MAPRED_HOME${HADOOP_HOME}/value /property5.配置YARNvi yarn-site.xml进入文件以后添加以下内容注意要添加在configuration和/configuration之间property nameyarn.resourcemanager.hostname/name valuehadoop1/value /property property nameyarn.nodemanager.aux-services/name valuemapreduce_shuffle/value /property property nameyarn.nodemanager.pmem-check-enabled/name valuefalse/value /property property nameyarn.nodemanager.vmem-check-enabled/name valuefalse/value /property property nameyarn.log-aggregation-enable/name valuetrue/value /property property nameyarn.log.server.url/name valuehttp://hadoop1:19888/jobhistory/logs/value /property property nameyarn.log-aggregation.retain-seconds/name value604800/value /property6.配置Hadoop从节点运行的服务器vi workers在文件末添加以下内容hadoop2 hadoop32.4分发1.在hadoop1上执行以下操作#Hadoop安装目录分发至虚拟机hadoop2中存放安装程序的目录 scp -r /export/servers/hadoop-3.3.1 roothadoop2:/export/servers/ scp -r /export/servers/hadoop-3.3.1 roothadoop3:/export/servers/ #将系统环境变量分发至虚拟机hadoop2上 scp /etc/profile roothadoop2:/etc scp /etc/profile roothadoop3:/etc2.在hadoop2和hadoop3上执行以下命令初始化系统环境变量source /etc/profile3.在hadoop2和hadoop3上验证Hadoop版本hadoop version2.5格式化HDFS注在格式化之前进行快照注不要格式化太多次会对文件系统造成损伤hdfs namenode -format格式化成功后会显示“successfully formatted”2.6启动Hadoop在hadoop1上启动#第一种方式 start-all.sh #第二种方式 start-dfs.sh start-yarn.sh2.7查看Hadoop运行状态2.8通过Web UI查看Hadoop的运行状态1.查看防火墙状态systemctl status firewalld2.关闭防火墙systemctl stop firewalld systemctl disable firewalld浏览器打开 http://192.168.1.132即可查看。