Ceph对象存储

一、什么是Ceph对象存储？？

作为文件系统的磁盘，操作系统不能直接访问对象存储。相反，它只能通过应用程序级别的API访问。Ceph是一种分布式对象存储系统，通过Ceph对象网关提供对象存储接口，也称为RADOS网关（RGW）接口，它构建在Ceph RADOS层之上。RGW使用librgw（RADOS Gateway library）和librados，允许应用程序与Ceph对象存储建立连接。RGW为应用程序提供了一个RESTful S3/swift兼容的接口，用于Ceph集群中对象的形式存储数据。Ceph还支持多租户对象存储，可以通过RESTful API访问。此外，RGW还支持ceph管理API，可以使用本机API调用来管理Ceph集群存储。
librados软件库非常灵活，允许用户应用程序通过C、C++、java、python和php绑定直接访问ceph存储集群。ceph对象存储还具有多站点功能，即灾难恢复提供解决方案。

二、部署对象存储

1、安装Ceph-radosgw

[root@storage01 ceph-clusters]# yum install ceph-radosgw

2、部署rgw

root@storage01 ceph-clusters]# ceph-deploy rgw create storage01 storage02 storage03

错误提示

ceph_deploy.rgw RuntimeError: config file etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite
主要原因是：配置失败的两个节点的配置文件的内容于当前节点不一致，提示使用--overwrite-conf参数去覆盖不一致的配置文件

如果要修改为80端口，可修改配置文件重启radosgw服务

vim etc/ceph/ceph.conf
[client.rgw.storage01]
rgw_frontends = "civetweb port=80"

重启radosgw服务

[root@storage01 ceph-clusters]# systemctl restart ceph-radosgw@rgw.storage01.service

该测试没有修改为80端口

3、创建存储池、设置副本数2

[root@storage01 ceph-clusters]# ceph osd pool create rados 64
pool 'rados' created
[root@storage01 ceph-clusters]# ceph osd pool set rados size 2
set pool 6 size to 2
[root@storage01 ceph-clusters]# ceph df
GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED
    540GiB     509GiB      30.8GiB          5.70 
POOLS:
    NAME                    ID     USED        %USED     MAX AVAIL     OBJECTS
    rbd                     1      7.02GiB      4.22        159GiB        1817 
    .rgw.root               2      2.08KiB         0        159GiB           6 
    default.rgw.control     3           0B         0        159GiB           8 
    default.rgw.meta        4           0B         0        159GiB           0 
    default.rgw.log         5           0B         0        159GiB         175 
    rados                   6           0B         0        239GiB           0 
[root@storage01 ceph-clusters]#

4、测试是否能访问Ceph集群

[root@storage01 ~]# ls -l var/lib/ceph/
total 0
drwxr-x---. 2 ceph ceph  6 Apr 11 21:15 bootstrap-mds
drwxr-x---. 2 ceph ceph 25 May  5 21:41 bootstrap-mgr
drwxr-x---. 2 ceph ceph 25 May  5 21:37 bootstrap-osd
drwxr-x---. 2 ceph ceph  6 Apr 11 21:15 bootstrap-rbd
drwxr-x---. 2 ceph ceph 25 May  8 16:13 bootstrap-rgw
drwxr-x---. 2 ceph ceph  6 Apr 11 21:15 mds
drwxr-x---. 3 ceph ceph 27 May  5 21:41 mgr
drwxr-x---. 3 ceph ceph 27 May  5 21:17 mon
drwxr-x---. 5 ceph ceph 45 May  5 21:38 osd
drwxr-xr-x. 3 root root 31 May  8 16:13 radosgw
drwxr-x---. 2 ceph ceph  6 May  5 21:17 tmp
[root@storage01 ~]# cp var/lib/ceph/radosgw/ceph-rgw.storage01/keyring ./
[root@storage01 ~]# ceph -s -k keyring --name client.rgw.storage01
  cluster:
    id:     a20708e0-6069-42a0-aee3-3b1617e4653c
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum storage01,storage02,storage03
    mgr: storage01(active), standbys: storage03, storage02
    osd: 9 osds: 9 up, 9 in
    rgw: 3 daemons active
 
  data:
    pools:   6 pools, 224 pgs
    objects: 2.01k objects, 7.02GiB
    usage:   30.8GiB used, 509GiB  540GiB avail
    pgs:     224 active+clean
 
You have new mail in var/spool/mail/root
[root@storage01 ~]#

三、使用S3 API访问ceph对象存储

1、创建radosgw用户

[root@storage01 ceph-clusters]# radosgw-admin user create --uid=radosgw --display-name="radosgw"
2019-05-08 16:57:54.093252 7f6e49bdadc0  0 WARNING: detected a version of libcurl which contains a bug in curl_multi_wait(). enabling a workaround that may degrade performance slightly.
{
    "user_id": "radosgw",
    "display_name": "radosgw",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
        {
            "user": "radosgw",
            "access_key": "SM02YOBTLDJDHAG4XH6V",
            "secret_key": "TSvSDrvJDoIwPneXzIfl7PtRqMcLmDwjcAb3BpIR"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw"
}

[root@storage01 ceph-clusters]#

2、安装S3cmd客户端并配置

[root@ceph-client ~]# yum install -y s3cmd
[root@ceph-client ~]# s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: SM02YOBTLDJDHAG4XH6V                               #ceph创建对象用户生成的密钥
Secret Key: TSvSDrvJDoIwPneXzIfl7PtRqMcLmDwjcAb3BpIR           #ceph创建对象用户生成的密钥
Default Region [US]: US

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]:

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]:

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:

New settings:
  Access Key: SM02YOBTLDJDHAG4XH6V
  Secret Key: TSvSDrvJDoIwPneXzIfl7PtRqMcLmDwjcAb3BpIR
  Default Region: US
  S3 Endpoint: s3.amazonaws.com
  DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.s3.amazonaws.com
  Encryption password:
  Path to GPG program: usr/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] n

Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'
[root@ceph-client ~]#

3、编辑S3配置文件

[root@ceph-client ~]# cat root/.s3cfg
[default]
access_key = SM02YOBTLDJDHAG4XH6V
access_token =
add_encoding_exts =
add_headers =
bucket_location = US
ca_certs_file =
cache_file =
check_ssl_certificate = True
check_ssl_hostname = True
cloudfront_host = 20.20.20.126:7480    #填写输入实际radosgw网关地址
content_disposition =
content_type =
default_mime_type = binary/octet-stream
delay_updates = False
delete_after = False
delete_after_fetch = False
delete_removed = False
dry_run = False
enable_multipart = True
encrypt = False
expiry_date =
expiry_days =
expiry_prefix =
follow_symlinks = False
force = False
get_continue = False
gpg_command = usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase =
guess_mime_type = True
host_base = 20.20.20.126:7480                          #填写输入实际radosgw网关地址
host_bucket = %(bucket)s.20.20.20.126:7480             #填写输入实际radosgw网关地址
human_readable_sizes = False
invalidate_default_index_on_cf = False
invalidate_default_index_root_on_cf = True
invalidate_on_cf = False
kms_key =
limit = -1
limitrate = 0
list_md5 = False
log_target_prefix =
long_listing = False
max_delete = -1
mime_type =
multipart_chunk_size_mb = 15
multipart_max_chunks = 10000
preserve_attrs = True
progress_meter = True
proxy_host =
proxy_port = 0
put_continue = False
recursive = False
recv_chunk = 65536
reduced_redundancy = False
requester_pays = False
restore_days = 1
restore_priority = Standard
secret_key = TSvSDrvJDoIwPneXzIfl7PtRqMcLmDwjcAb3BpIR
send_chunk = 65536
server_side_encryption = False
signature_v2 = False
signurl_use_https = False
simpledb_host = sdb.amazonaws.com
skip_existing = False
socket_timeout = 300
stats = False
stop_on_error = False
storage_class =
throttle_max = 100
upload_id =
urlencoding_mode = normal
use_http_expect = False
use_https = False       ##关闭https认证默认是开启
use_mime_magic = True
verbosity = WARNING
website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/
website_error =
website_index = index.html

4、创建桶并放入文件

[root@ceph-client ~]# s3cmd mb s3://First-bucket   #注意全小写可能会报错，建议用大小写并用
Bucket 's3://First-bucket/' created
[root@ceph-client ~]# s3cmd ls
2019-05-09 15:12  s3://First-bucket
2019-05-09 10:20  s3://My-test-bucket1
2019-05-09 10:20  s3://My-test-bucket2
[root@ceph-client ~]# s3cmd put etc/hosts s3://First-bucket
upload: '/etc/hosts' -> 's3://First-bucket/hosts'  [1 of 1]
 253 of 253   100% in    3s    75.19 B/s  done
[root@ceph-client ~]# s3cmd ls s3://First-bucket
2019-05-09 15:15       253   s3://First-bucket/hosts
[root@ceph-client ~]#

四、使用Swift API访问ceph对象存储

1、创建swift api子用户

[root@storage01 ~]# radosgw-admin subuser create --uid=radosgw --subuser=radosgw:swift --access=full
2019-05-09 23:20:40.650013 7f03beea5dc0  0 WARNING: detected a version of libcurl which contains a bug in curl_multi_wait(). enabling a workaround that may degrade performance slightly.
{
    "user_id": "radosgw",
    "display_name": "radosgw",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [
        {
            "id": "radosgw:swift",
            "permissions": "full-control"
        }
    ],
    "keys": [
        {
            "user": "radosgw",
            "access_key": "SM02YOBTLDJDHAG4XH6V",
            "secret_key": "TSvSDrvJDoIwPneXzIfl7PtRqMcLmDwjcAb3BpIR"
        }
    ],
    "swift_keys": [
        {
            "user": "radosgw:swift",
            "secret_key": "2DSdH4BGtFyLJwj63O1XyBIrhTDPoCem7cQLVxKa"
        }
    ],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw"
}

2、安装swift环境

[root@ceph-client ~]# yum install -y python-setuptools
[root@ceph-client ~]# easy_install pip
[root@ceph-client ~]# pip install --upgrade setuptools
[root@ceph-client ~]# pip install --upgrade python-swiftclient

3、测试

[root@ceph-client ~]# swift -A http://20.20.20.126:7480/auth/1.0 -U radosgw:swift -K 2DSdH4BGtFyLJwj63O1XyBIrhTDPoCem7cQLVxKa list
First-bucket
My-test-bucket1
My-test-bucket2
[root@ceph-client ~]#

2DSdH4BGtFyLJwj63O1XyBIrhTDPoCem7cQLVxKa 是swift api子用户密钥

[root@ceph-client ~]# swift -A http://20.20.20.126:7480/auth/1.0 -U radosgw:swift -K 2DSdH4BGtFyLJwj63O1XyBIrhTDPoCem7cQLVxKa post sencond-bucket
[root@ceph-client ~]# swift -A http://20.20.20.126:7480/auth/1.0 -U radosgw:swift -K 2DSdH4BGtFyLJwj63O1XyBIrhTDPoCem7cQLVxKa list
First-bucket
My-test-bucket1
My-test-bucket2
Sencond-bucket
sencond-bucket
[root@ceph-client ~]#
[root@ceph-client ~]# s3cmd ls
2019-05-09 15:12  s3://First-bucket
2019-05-09 10:20  s3://My-test-bucket1
2019-05-09 10:20  s3://My-test-bucket2
2019-05-09 15:40  s3://Sencond-bucket
2019-05-09 15:40  s3://sencond-bucket
[root@ceph-client ~]#