Hadoop 3.3.5 伪分布式部署Ubuntu 22.04 单机模拟集群 5 步配置在Ubuntu 22.04上搭建Hadoop伪分布式环境是学习大数据处理的必经之路。与单机模式不同伪分布式模式能完整模拟HDFS文件系统和YARN资源调度的工作机制让你在单台机器上就能体验真实的集群操作。下面这套经过验证的5步配置法将带你快速构建可立即投入使用的学习环境。1. 基础环境准备在开始Hadoop部署前需要确保系统环境满足以下条件Ubuntu 22.04 LTS建议使用全新安装的系统至少4GB内存Hadoop进程需要足够内存空间20GB可用磁盘空间用于存储HDFS文件系统Java 8或11Hadoop 3.3.5的官方推荐版本首先创建专用用户避免使用root账户操作sudo adduser hadoop sudo usermod -aG sudo hadoop切换到hadoop用户并配置SSH免密登录这是Hadoop节点间通信的基础su - hadoop ssh-keygen -t rsa -P -f ~/.ssh/id_rsa cat ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys验证SSH配置是否成功ssh localhost提示如果遇到Connection refused错误需先安装openssh-serversudo apt install openssh-server2. Java环境配置Hadoop依赖Java环境以下是JDK11的安装步骤sudo apt update sudo apt install openjdk-11-jdk -y配置JAVA_HOME环境变量编辑~/.bashrc文件echo export JAVA_HOME/usr/lib/jvm/java-11-openjdk-amd64 ~/.bashrc echo export PATH$PATH:$JAVA_HOME/bin ~/.bashrc source ~/.bashrc验证Java安装java -version预期输出应包含类似内容openjdk version 11.0.22 2024-01-16 OpenJDK Runtime Environment (build 11.0.227-post-Ubuntu-0ubuntu222.04.1) OpenJDK 64-Bit Server VM (build 11.0.227-post-Ubuntu-0ubuntu222.04.1, mixed mode)3. Hadoop安装与配置从Apache官网下载Hadoop 3.3.5并解压wget https://archive.apache.org/dist/hadoop/common/hadoop-3.3.5/hadoop-3.3.5.tar.gz tar -xzvf hadoop-3.3.5.tar.gz sudo mv hadoop-3.3.5 /usr/local/hadoop设置Hadoop环境变量继续编辑~/.bashrcecho export HADOOP_HOME/usr/local/hadoop ~/.bashrc echo export PATH$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin ~/.bashrc echo export HADOOP_CONF_DIR$HADOOP_HOME/etc/hadoop ~/.bashrc source ~/.bashrc现在进入核心配置环节需要修改以下四个关键文件core-site.xml- 配置HDFS地址和临时目录configuration property namefs.defaultFS/name valuehdfs://localhost:9000/value /property property namehadoop.tmp.dir/name value/usr/local/hadoop/tmp/value /property /configurationhdfs-site.xml- 配置HDFS副本数和存储路径configuration property namedfs.replication/name value1/value /property property namedfs.namenode.name.dir/name valuefile://${hadoop.tmp.dir}/dfs/name/value /property property namedfs.datanode.data.dir/name valuefile://${hadoop.tmp.dir}/dfs/data/value /property /configurationmapred-site.xml- 配置MapReduce使用YARN框架configuration property namemapreduce.framework.name/name valueyarn/value /property /configurationyarn-site.xml- 配置YARN资源管理configuration property nameyarn.nodemanager.aux-services/name valuemapreduce_shuffle/value /property property nameyarn.nodemanager.env-whitelist/name valueJAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME/value /property /configuration4. 初始化与启动集群首次启动前需要格式化NameNodehdfs namenode -format成功格式化后你会看到类似下面的输出2024-07-15 14:56:12,423 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1399603265-127.0.1.1-1721044572423 2024-07-15 14:56:12,516 INFO common.Storage: Storage directory /usr/local/hadoop/tmp/dfs/name has been successfully formatted.启动HDFS和YARN服务start-dfs.sh start-yarn.sh使用jps命令验证进程是否正常启动jps预期应该看到以下进程NameNode DataNode SecondaryNameNode ResourceManager NodeManager Jps5. 验证与测试通过Web界面验证集群状态HDFS NameNode: http://localhost:9870YARN ResourceManager: http://localhost:8088运行经典的WordCount示例测试整个集群hdfs dfs -mkdir -p /user/hadoop/input hdfs dfs -put $HADOOP_HOME/etc/hadoop/*.xml /user/hadoop/input hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.5.jar wordcount /user/hadoop/input /user/hadoop/output查看计算结果hdfs dfs -cat /user/hadoop/output/*预期会看到各个单词的统计结果例如configuration 3 deprecated 1 encoding 1常见问题排查遇到启动失败时可按以下步骤排查检查日志文件NameNode日志$HADOOP_HOME/logs/hadoop-hadoop-namenode-*.logDataNode日志$HADOOP_HOME/logs/hadoop-hadoop-datanode-*.log端口冲突问题sudo netstat -tulnp | grep java如果发现9000或9870端口被占用需修改core-site.xml中的端口配置权限问题sudo chown -R hadoop:hadoop /usr/local/hadoopJAVA_HOME未设置 编辑$HADOOP_HOME/etc/hadoop/hadoop-env.sh明确指定export JAVA_HOME/usr/lib/jvm/java-11-openjdk-amd64对于长期学习者建议将以下命令添加到~/.bashrc方便日常使用alias hstartstart-dfs.sh start-yarn.sh alias hstopstop-yarn.sh stop-dfs.sh alias hstatusjps