Hadoop 分布式存储

目录一、hadoop单机模式1.1 部署1.2 HDFS创建目录上传测试文件二、完全分布式2.1 server1部署NFS共享服务2.2 修改Hadoop核心配置文件2.3 server2、server3挂载NFS共享目录2.4 格式化2.5 HDFS集群测试创建目录上传文件三、集群节点扩容一、hadoop单机模式1.1 部署官方安装网址Index of /hadoop/common官方指导文件Apache Hadoop 3.3.5 – Hadoop: Setting up a Single Node Cluster.下载安装软件包解压缩并创建软连接创建一个普通用户hadoop并登入tar zxf hadoop-3.3.6.tar.gztar zxf jdk1.8.0_181-linux-x64.tar.gzln -s hadoop-3.3.6 hadoopln -s jdk1.8.0_181/ jdk配置Java环境变量编辑用户家目录.bashrc只对hadoop这个用户生效hadoop‑env.sh指定JAVA_HOME[hadoopserver1 hadoop]$ cd etc[hadoopserver1 etc]$ lshadoop[hadoopserver1 etc]$ cd hadoop[hadoopserver1 hadoop]$ vim hadoop-env.sh把本地xml配置文件复制到本地input文件夹[hadoopserver1 hadoop]$ cp etc/hadoop/*.xml input运行官方grep示例MR程序[hadoopserver1 hadoop]$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar grep input output dfs[a-z.][hadoopserver1 hadoop]$ cat output/*[hadoopserver1 hadoop]$ vim etc/hadoop/core-site.xmlfs.defaultFSHDFS文件系统默认地址客户端访问HDFS的入口单机写localhost:9000[hadoopserver1 hadoop]$ vim etc/hadoop/hdfs-site.xmlhdfs‑site.xml副本数配置免密对loaclhost这里就不详细展示格式化NameNode格式化会清空HDFS全部数据已经存数据的集群绝对不能重复执行[hadoopserver1 hadoop]$ bin/hdfs namenode -format生成NameNode元数据目录初始化fsimage镜像文件建立HDFS文件系统。启动HDFS[hadoopserver1 hadoop]$ sbin/start-dfs.sh9870端口浏览器正常访问1.2 HDFS创建目录上传测试文件把本地文件上传分布式文件系统 HDFS给 MapReduce 任务做输入数据源。[hadoopserver1 hadoop]$ bin/hdfs dfs -mkdir /user[hadoopserver1 hadoop]$ bin/hdfs dfs -mkdir /user/hadoop[hadoopserver1 hadoop]$ bin/hdfs dfs -mkdir input[hadoopserver1 hadoop]$ bin/hdfs dfs -put etc/hadoop/*.xml input先删除本地旧outpu交mr任务读取hdfs input目录正则匹配dfs[a‑z.]输出到hdfs output[hadoopserver1 hadoop]$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar grep input output dfs[a-z.][hadoopserver1 hadoop]$ rm -rf output[hadoopserver1 hadoop]$ bin/hdfs dfs -get output output[hadoopserver1 hadoop]$ bin/hdfs dfs -cat output/*二、完全分布式2.1 server1部署NFS共享服务server1和server2、server3上安装nfs-utils安装nfs服务组件实现目录网络共享server2、server3可以挂载server1的/home/hadoop。yum install -y nfs-utils[rootserver1 etc]# vim exports[rootserver1 etc]# systemctl enable --now nfs[rootserver1 etc]# showmount -e查看本机已经共享出来的目录验证exports配置是否生效cd hadoop/etc/hadoop2.2 修改Hadoop核心配置文件[hadoopserver1 hadoop]$ vim core-site.xmlHDFS 主节点地址完全分布式必须写NameNode主机名 server1不能写[localhost]。集群所有 DataNode 通过这个地址去找 NameNode。[hadoopserver1 hadoop]$ vim hdfs-site.xml当前集群有2台DataNodeserver2、server3副本数设置2[hadoopserver1 hadoop]$ vim workers填入dataNode主机名2.3 server2、server3挂载NFS共享目录[rootserver2 ~]# mount 192.168.159.133:/home/hadoop/ /home/hadoop/2.4 格式化停止旧服务[hadoopserver1 hadoop]$ sbin/stop-dfs.sh[hadoopserver1 hadoop]$ bin/hdfs namenode -format启动完全分布式 HDFS 集群[hadoopserver1 hadoop]$ sbin/start-dfs.sh2.5 HDFS集群测试创建目录上传文件[hadoopserver1 hadoop]$ bin/hdfs dfs -mkdir /user[hadoopserver1 hadoop]$ bin/hdfs dfs -mkdir /user/hadoop[hadoopserver1 hadoop]$ bin/hdfs dfs -mkdir input[hadoopserver1 hadoop]$ bin/hdfs dfs -put etc/hadoop/*.xml input挂载成功后也会有server1的文件等三、集群节点扩容原有集群server1 (NameNodeResourceManager)、server2 (DataNodeNodeManager)、server3 (DataNodeNodeManager)扩容目标新增server4同时作为DataNodeHDFS 数据节点 NodeManagerYARN计算节点环境沿用NFS共享/home/hadoopserver4挂载之后直接复用jdk、hadoop整套程序不用重新解压部署hadoop。新开一台虚拟机server4初始化新节点 server4[rootserver4 ~]# useradd hadoop[rootserver4 ~]# passwd hadoop[rootserver4 ~]# mount 192.168.159.133:/home/hadoop/ /home/hadoop/server1修改MapReduce配置mapred‑site.xml[hadoopserver1 hadoop]$ vim etc/hadoop/mapred-site.xmlserver1修改MapReduce配置yarn‑site.xml[hadoopserver1 hadoop]$ vim etc/hadoop/yarn-site.xmlserver4手动后台启动DataNode、NodeManager[rootserver4 ~]# su - hadoop[hadoopserver4 ~]$ hdfs --daemon start datanode[hadoopserver4 ~]$ yarn --daemon start nodemanagerserver1更新workers文件启动YARN集群查看集群节点状态[hadoopserver1 hadoop]$ vim workers[hadoopserver1 hadoop]$ sbin/start-yarn.sh[hadoopserver1 hadoop]$ hdfs dfsadmin -report[hadoopserver1 hadoop]$ vim etc/hadoop/yarn-site.xml新加入两行内容server2也需要启动[hadoopserver3 ~]$ yarn --daemon start nodemanager添加成功浏览器访问192.168.159.133:8088端口也就是server1