1. Hadoop单机版安装
Hadoop运行依赖jdk环境，请先确保安装好jdk
1.1 下载安装包
cd /home/admin
wget https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.2.2/hadoop-3.2.2.tar.gz
1.2 解压
sudo tar -zxvf hadoop-3.2.2.tar.gz -C /opt
sudo ln -s /opt/hadoop-3.2.2 /opt/hadoop-current
sudo chown -R admin:admin /opt/hadoop-*
1.3 设置环境变量
# /etc/profile
export HADOOP_HOME=/opt/hadoop-current
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:$NODE_JS_HOME/bin:$ZK_HOME/bin:$HADOOP_HOME/bin
1.4 配置hadoop
<!-- /opt/hadoop-current/etc/hadoop/core-site.xml-->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/opt/hadoop-current/tmp</value>
<description>Abase for other temporary directories.</description>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:19000</value>
</property>
</configuration>
<!-- /opt/hadoop-current/etc/hadoop/hdfs-site.xml-->
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/opt/hadoop-current/tmp/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/opt/hadoop-current/tmp/dfs/data</value>
</property>
</configuration>
1.5 初始化hdfs
/opt/hadoop-current/sbin/hdfs namenode -format
启动hadoop
/opt/hadoop-current/sbin/start-all.sh
1.6 web端访问
# HDFS
http://192.168.0.100:9870
# YARN
http://192.168.0.100:8088
2.Hadoop集群部署
安装前准备事项：
部署jdk环境并配置 JAVA_HOME 环境变量
部署Ansible集群运维工具
配置Ansible节点到Hadoop集群节点的ssh免密登录
集群物理架构
| server | ResourceManager | NameNode | SecondaryNameNode | NodeManager | DataNode | 备注 |
|---|---|---|---|---|---|---|
| 192.168.0.100 | yes | yes | Ansible节点 | |||
| 192.168.0.101 | yes | yes | yes | |||
| 192.168.0.102 | yes | yes | ||||
| 192.168.0.103 | yes | yes |
2.1 部署单节点的Hadoop
先在192.168.0.100节点部署Hadoop，具体步骤请参照第一节说明
集群配置如下：
<!-- /opt/hadoop-current/etc/hadoop/hdfs-site.xml -->
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/opt/hadoop-current/tmp/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/opt/hadoop-current/tmp/dfs/data</value>
</property>
<property>
<!-- 主节点地址 -->
<name>dfs.namenode.http-address</name>
<value>192.168.0.100:50070</value>
</property>
<property>
<!-- 第二节点地址 -->
<name>dfs.namenode.secondary.http-address</name>
<value>192.168.0.101:9001</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
<description>配置为false后，可以允许不要检查权限就生成dfs上的文件，需防止误删操作</description>
</property>
</configuration>
<!-- /opt/hadoop-current/etc/hadoop/mapred-site.xml -->
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>192.168.0.100:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>192.168.0.100:19888</value>
</property>
</configuration>
<!-- /opt/hadoop-current/etc/hadoop/yarn-site.xml -->
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>192.168.0.100:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>192.168.0.100:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>192.168.0.100:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>192.168.0.100:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>192.168.0.100:8088</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
<property>
<!-- 调度策略，设置为公平调度器 -->
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>
</configuration>
# cat /opt/hadoop-current/etc/hadoop/workers
192.168.0.101
192.168.0.102
192.168.0.103
2.2 使用Ansible拷贝hadoop安装目录
192.168.0.100节点部署完成后，打包部署好的安装目录，拷贝到剩余节点，完成安装
# 打包安装好的hadoop安装目录
sudo tar -czvf hadoop.tar.gz /opt/hadoop-current/
# 压缩目录传输到其它节点
ansible 192.168.0.101,192.168.0.102,192.168.0.103 -m copy -a 'src=~/hadoop.tar.gz dest=~/hadoop.tar.gz'
# 解压到/opt目录
ansible 192.168.0.101,192.168.0.102,192.168.0.103 -m shell -a 'sudo tar -zxvf ~/hadoop.tar.gz -C /'
ansible 192.168.0.101,192.168.0.102,192.168.0.103 -m shell -a 'sudo ln -s /opt/hadoop-3.2.2 /opt/hadoop-current'
# 目录访问权限设置
ansible 192.168.0.101,192.168.0.102,192.168.0.103 -m shell -a 'sudo chown admin:admin -R /opt/hadoop*'
2.3 启动Hadoop集群
启动集群需要登录到Master节点（192.168.0.100），然后执行启动命令
# 登录192.168.0.100节点
ssh admin@192.168.0.100
# 格式化hdfs文件系统
hdfs namenode -format
# 启动hdfs
start-dfs.sh
# 启动yarn
start-yarn.sh
查看Hadoop集群各节点进程
ansible 192.168.0.100,192.168.0.101,192.168.0.102,192.168.0.103 -m shell -a 'jps'
192.168.0.102 | CHANGED | rc=0 >>
29252 DataNode
29637 NodeManager
9151 Jps
192.168.0.100 | CHANGED | rc=0 >>
32467 Jps
29754 NameNode
30347 ResourceManager
192.168.0.101 | CHANGED | rc=0 >>
30097 SecondaryNameNode
29975 DataNode
31287 Jps
30317 NodeManager
192.168.0.103 | CHANGED | rc=0 >>
10848 DataNode
11690 NodeManager
15099 Jps
2.4 web端访问
# yarn
http://192.168.0.100:8088/
# hdfs
http://192.168.0.100:50070/
文章转载自大数据小黑屋，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。
