Hadoop配置TDE加密数据

Hadoop平台中的透明数据加密

什么是透明数据加密
TransparentData Encryption用来加密数据文件里的数据，保护从底层对数据的安全访问。所谓透明是指对使用者来说是未知的，当使用者在写入数据时，系统会自动加密，当使用数据时系统会自动解密。Oracle、SQLServer很早就支持了透明数据加密的特性，下面说一下针对HDFS文件的透明数据加密过程。
HDFS KMS透明加密配置及测试
HDFS Encryption zone加密空间是一种end-to-end(端到端)的加密模式.其中的加/解密过程对于客户端来说是完全透明的.数据在客户端读操作的时候被解密,当数据被客户端写的时候被加密,所以HDFS本身并不是一个主要的参与者,形象的说,在HDFS中,你看到的只是一堆加密的数据流.
Encryption zone原理介绍
1.每个encryption zone 会与每个encryption zone key相关联,而这个key就是会在创建encryption zone的时候同时被指定。
2.每个encryption zone中的文件会有其唯一的data encryption key数据加密key,简称就是DEK。
3.DEK不会被HDFS直接处理,取而代之的是,HDFS只处理经过加密的DEK, 就是encrypted data encryption key,缩写就是EDEK。
4.客户端询问KMS服务去解密EDEK,然后利用解密后得到的DEK去读/写数据。  Key Provider可以理解为是一个key store的保存库,其中KMS是其中的一个实现。
环境准备
hadoop2.8.5
配置KMS步骤如下：
1.core-site.xml 新增配置项

//配置客户端的地址

hadoop.security.key.provider.path kms://http@zxdbdb1:16000/kms

2.hdfs-site.xml 新增配置项

//主要配置hdfs

dfs.encryption.key.provider.uri kms://http@zxdbdb1:16000/kms

3.使用keytool生成的秘钥密码是123456 将密码直接写入到kms.keystore.password文件(文件名一定要是这个)

echo 123456 > ${HADOOP_HOME}/share/hadoop/kms/tomcat/webapps/kms/WEB-INF/classes/kms.keystore.password

${HADOOP_HOME}：表示Hadoop安装的路径

4.kms-env.sh 配置文件

export KMS_HOME=/opt/modules/hadoop-2.8.5
export KMS_LOG=${KMS_HOME}/logs/kms
export KMS_HTTP_PORT=16000
export KMS_ADMIN_PORT=16001

5、创建秘钥

keytool -genkey -alias ‘key1’ #删除的话 keytool -delete -alias ‘key1’;

Enter keystore password:123456 #-----此处创建的密码，需要和上面kms.keystore.password配置文件的密码相同

Re-enter new password:

What is your first and last name?

What is the name of your organizational unit?

Unknown:

What is the name of your organization?

Unknown:

What is the name of your City or Locality?

Unknown:

What is the name of your State or Province?

Unknown:

What is the two-letter country code for this unit?

Unknown:

Is CN=lch, OU=gdbd, O=gdbd, L=sz, ST=sz, C=sz correct?

[no]: yes #输入yes

Enter key password for <user_a_key>

(RETURN if same as keystore password):

Re-enter new password:

6、cd /opt/modules/hadoop-2.8.5/sbin
#启动KMS
/opt/modules/hadoop-2.8.5/sbin/kms.sh start
#停止KMS
/opt/modules/hadoop-2.8.5/sbin/kms.sh stop

此时通过jps能看到Bootstrap进程，表示成功启动。

7、重新启动namenode与datanode

hadoop checknative -a
启动hadoop
/opt/modules/hadoop-2.8.5/sbin/start-all.sh
停止hadoop
/opt/modules/hadoop-2.8.5/sbin/stop-all.sh

8、[root@hdp130 kms]# hadoop key create key1 — wafkey为上面通过keytool创建的

[root@hdp130 kms]# hadoop key list ---- 查看一下

[root@hdp130 kms]# hadoop fs -mkdir /sub

[root@hdp130 logs]# hdfs crypto -createZone -keyName key1 -path /sub —设置/sub为加密区
Added encryption zone /sub
hdfs crypto -listZones #—查看已加密区域
查看密钥
hadoop key list -metadata

上传一个文件到hdfs的/sub上,拿到数据块的BP的id号到对应datanode节点上，下载文件快查看，然后具体磁盘上看到数据已经加密。

ok!

若删除加密区报错：

20/06/05 17:20:27 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 1440 minutes, Emptier interval = 0 minutes.

rm: Failed to move to trash: hdfs://hdfscluster/subkey/testdata.sh: /subkey/testdata.sh can’t be moved from an encryption zone.

则删除加密区域文件或文件夹命令： hadoop fs -rm -r -skipTrash /subkey/test.logs

支持hive远程操作数据
----------添加core-site.xml ------------------------------------------------------------------------------------------------------------------------------

hadoop.proxyuser.hadoop.hosts * hadoop.proxyuser.hadoop.groups * hadoop.proxyuser.root.groups * The 'nfsserver' user is allowed to proxy all members of the 'users-group1' and 'users-group2' groups. Note that in most cases you will need to include the group "root" because the user "root" (which usually belonges to "root" group) will generally be the user that initially executes the mount on the NFS client system. Set this to '*' to allow nfsserver user to proxy any group. hadoop.proxyuser.root.hosts * This is the host where the nfs gateway is running. Set this to '*' to allow requests from any hosts to be proxied. hadoop.security.key.provider.path kms://http@zxdbdb1:16000/kms ———————————————————————————————————————— -------------------------hdfs-site.xml -----------------------------------------------------------------

<property>
  <name>dfs.webhdfs.enabled</name>
  <value>true</value>
</property>

nfs.dump.dir /tmp/.hdfs-nfs nfs.rtmax 1048576 This is the maximum size in bytes of a READ request supported by the NFS gateway. If you change this, make sure you also update the nfs mount's rsize(add rsize= # of bytes to the mount directive). nfs.wtmax 65536 This is the maximum size in bytes of a WRITE request supported by the NFS gateway. If you change this, make sure you also update the nfs mount's wsize(add wsize= # of bytes to the mount directive). nfs.exports.allowed.hosts * rw 允许所有主机对文件有rw权限 dfs.encryption.key.provider.uri kms://http@zxdbdb1:16000/kms