OpenStack實踐系列⑨雲硬盤服務Cinder
OpenStack實踐系列⑨雲硬盤服務Cinder
八、cinder
8.1存儲的三大分類
塊存儲：硬盤，磁盤陣列DAS，SAN存儲
文件存儲：nfs，GluserFS，Ceph（PB級分布式文件系統），MooserFS（缺點Metadata數據丟失，虛擬機就毀了）
11.2網絡類型選擇
對象存儲：swift，S3
8.2 cinder控制節點的部署
安裝cinder
[[email protected] ~]# yum install openstack-cinder python-cinderclient -y
修改cinder配置文件
[default]
rpc_backend = rabbit
glance_host = 192.168.3.199
auth_strategy = keystone
[oslo_concurrency]
lock_path = /var/lib/cinder/tmp
[oslo_messaging_rabbit]
rabbit_host = 192.168.3.199 # rabbitmq的主機
rabbit_port = 5672 # rabbitmq的端口
rabbit_userid = openstack # rabbitmq的用戶
rabbit_password = openstack # rabbitmq的密碼
[database]
connection = mysql://cinder:[email protected]
[keystone_authtoken]
auth_uri = http://192.168.3.199:5000
auth_url = http://192.168.3.199:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = service
username = cinder
password = cinder
修改後結果如下
[[email protected] cinder]# grep-n ‘^[a-Z]‘ /etc/cinder/cinder.conf 421:glance_host = 192.168.3.199 536:auth_strategy = keystone 2294:rpc_backend = rabbit 2516:connection = mysql://cinder:[email protected]/cinder 2641:auth_uri = http://192.168.3.199:5000 2642:auth_url = http://192.168.3.199:35357 2643:auth_plugin = password 2644:project_domain_id = default 2645:user_domain_id = default 2646:project_name = service 2647:username = cinder 2648:password = cinder 2874:lock_path = /var/lib/cinder/tmp 3173:rabbit_host = 192.168.3.199 3177:rabbit_port = 5672 3189:rabbit_userid = openstack 3193:rabbit_password = openstack
修改nova的配置文件
[[email protected] ~]# vim /etc/nova/nova.conf
os_region_name = RegionOne # 通知nova使用cinder [cinder]部分
執行同步數據庫操作
[[email protected] ~]# su -s /bin/sh -c "cinder-manage db sync" cinder 檢查導入數據庫結果 MariaDB [(none)]> use cinder Database changed MariaDB [cinder]> show tables; +----------------------------+ | Tables_in_cinder | +----------------------------+ | backups | | cgsnapshots | | consistencygroups | | driver_initiator_data | | encryption | | image_volume_cache_entries | | iscsi_targets | | migrate_version | | quality_of_service_specs | | quota_classes | | quota_usages | | quotas | | reservations | | services | | snapshot_metadata | | snapshots | | transfers | | volume_admin_metadata | | volume_attachment | | volume_glance_metadata | | volume_metadata | | volume_type_extra_specs | | volume_type_projects | | volume_types | | volumes | +----------------------------+ 25 rows in set (0.00 sec)
創建一個cinder用戶,加入service項目，給予admin角色
[[email protected] ~]# source admin-openrc.sh [[email protected] ~]# openstack user create --domain default --password-prompt cinder User Password: Repeat User Password: # （密碼需要配置成cinder就是/etc/cinder/cinder.conf配置文件中配置的2648行） +-----------+----------------------------------+ | Field | Value | +-----------+----------------------------------+ | domain_id | default | | enabled | True | | id | 420d7573e9fc43b3b263f31bb6dd76e2 | | name | cinder | +-----------+----------------------------------+ [[email protected] ~]# openstack role add --project service --user cinder admin
重啟nova-api服務和啟動cinder服務
[[email protected] ~]# systemctl restart openstack-nova-api.service [[email protected] ~]# systemctl enable openstack-cinder-api.service openstack-cinder-scheduler.service [[email protected] ~]# systemctl start openstack-cinder-api.service openstack-cinder-scheduler.service
創建服務（包含V1和V2）
[[email protected] ~]# openstack service create --name cinder --description "OpenStack Block Storage" volume +-------------+----------------------------------+ | Field | Value | +-------------+----------------------------------+ | description | OpenStack Block Storage | | enabled | True | | id | 6e3b2c3940d14300ab28aed272ade1d3 | | name | cinder | | type | volume | +-------------+----------------------------------+ [[email protected] ~]# openstack service create --name cinderv2 --description "OpenStack Block Storage" volumev2 +-------------+----------------------------------+ | Field | Value | +-------------+----------------------------------+ | description | OpenStack Block Storage | | enabled | True | | id | 2108489d055e4fcb8f9c88fa9d5e4e3d | | name | cinderv2 | | type | volumev2 | +-------------+----------------------------------+
分別對V1和V2創建三個環境（admin，internal，public）的endpoint
[[email protected] ~]# openstack endpoint create --region RegionOne volume public http://192.168.3.199:8776/v1/%\(tenant_id\)s +--------------+--------------------------------------------+ | Field | Value | +--------------+--------------------------------------------+ | enabled | True | | id | 007497468db7456d81157962f8740540 | | interface | public | | region | RegionOne | | region_id | RegionOne | | service_id | 6e3b2c3940d14300ab28aed272ade1d3 | | service_name | cinder | | service_type | volume | | url | http://192.168.3.199:8776/v1/%(tenant_id)s | +--------------+--------------------------------------------+ [[email protected] ~]# openstack endpoint create --region RegionOne volume internal http://192.168.3.199:8776/v1/%\(tenant_id\)s +--------------+--------------------------------------------+ | Field | Value | +--------------+--------------------------------------------+ | enabled | True | | id | e7543b96b69342bcabead7ad8a583860 | | interface | internal | | region | RegionOne | | region_id | RegionOne | | service_id | 6e3b2c3940d14300ab28aed272ade1d3 | | service_name | cinder | | service_type | volume | | url | http://192.168.3.199:8776/v1/%(tenant_id)s | +--------------+--------------------------------------------+ [[email protected] ~]# openstack endpoint create --region RegionOne volume admin http://192.168.3.199:8776/v1/%\(tenant_id\)s +--------------+--------------------------------------------+ | Field | Value | +--------------+--------------------------------------------+ | enabled | True | | id | 12e4bea586384d43b16e4de5a00afb1b | | interface | admin | | region | RegionOne | | region_id | RegionOne | | service_id | 6e3b2c3940d14300ab28aed272ade1d3 | | service_name | cinder | | service_type | volume | | url | http://192.168.3.199:8776/v1/%(tenant_id)s | +--------------+--------------------------------------------+ [[email protected] ~]# openstack endpoint create --region RegionOne volumev2 public http://192.168.3.199:8776/v2/%\(tenant_id\)s +--------------+--------------------------------------------+ | Field | Value | +--------------+--------------------------------------------+ | enabled | True | | id | 07c56b0033454fbda201f3cc58ce0a1b | | interface | public | | region | RegionOne | | region_id | RegionOne | | service_id | 2108489d055e4fcb8f9c88fa9d5e4e3d | | service_name | cinderv2 | | service_type | volumev2 | | url | http://192.168.3.199:8776/v2/%(tenant_id)s | +--------------+--------------------------------------------+ [[email protected] ~]# openstack endpoint create --region RegionOne volumev2 internal http://192.168.3.199:8776/v2/%\(tenant_id\)s +--------------+--------------------------------------------+ | Field | Value | +--------------+--------------------------------------------+ | enabled | True | | id | 66b34a18d4de456ab32ebba24831b959 | | interface | internal | | region | RegionOne | | region_id | RegionOne | | service_id | 2108489d055e4fcb8f9c88fa9d5e4e3d | | service_name | cinderv2 | | service_type | volumev2 | | url | http://192.168.3.199:8776/v2/%(tenant_id)s | +--------------+--------------------------------------------+ [[email protected] ~]# openstack endpoint create --region RegionOne volumev2 admin http://192.168.3.199:8776/v2/%\(tenant_id\)s +--------------+--------------------------------------------+ | Field | Value | +--------------+--------------------------------------------+ | enabled | True | | id | fc0bed271a7048a5aff7e63aebd0199a | | interface | admin | | region | RegionOne | | region_id | RegionOne | | service_id | 2108489d055e4fcb8f9c88fa9d5e4e3d | | service_name | cinderv2 | | service_type | volumev2 | | url | http://192.168.3.199:8776/v2/%(tenant_id)s | +--------------+--------------------------------------------+
8.3 cinder存儲節點的部署（此處使用nova的計算節點即node2.chinasoft.com）
本文中cinder後端存儲使用ISCSI（類似於nova-computer使用的kvm），ISCSI使用LVM，在定義好的VG中，每創建一個雲硬盤，就會增加一個LV，使用ISCSI發布。
在存儲節點上加一個硬盤
查看磁盤添加情況
[[email protected] ~]# fdisk -l Disk /dev/sdb: 85.9 GB, 85899345920 bytes, 167772160 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/sda: 128.8 GB, 128849018880 bytes, 251658240 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x0004c2a9 Device Boot Start End Blocks Id System /dev/sda1 * 2048 616447 307200 83 Linux /dev/sda2 616448 155811839 77597696 8e Linux LVM Disk /dev/mapper/centos-root: 32.2 GB, 32212254720 bytes, 62914560 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/mapper/centos-swap: 4294 MB, 4294967296 bytes, 8388608 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/mapper/centos-data: 42.9 GB, 42945478656 bytes, 83877888 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes
創建一個pv和vg（名為cinder-volumes）
[[email protected] ~]# pvcreate /dev/sdb Physical volume "/dev/sdb" successfully created [[email protected] ~]# vgcreate cinder-volumes /dev/sdb Volume group "cinder-volumes" successfully created
修改lvm的配置文件中添加filter，只有instance可以訪問
[[email protected] ~]# vim /etc/lvm/lvm.conf 131 filter = [ "a/sdb/", "r/.*/"]
存儲節點安裝
[[email protected] ~]# yum install openstack-cinder targetcli python-oslo-policy -y
修改存儲節點的配置文件，在這裏直接拷貝控制節點的文件
[[email protected] ~]# scp /etc/cinder/cinder.conf 192.168.3.200:/etc/cinder/
修改存儲(即計算節點node2.chinasoft.com)上的/etc/cinder/cinder.conf文件
添加如下配置：
默認沒有[lvm]需要自己創建
[lvm]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver # 使用lvm後端存儲
volume_group = cinder-volumes # vg的名稱：剛才創建的
iscsi_protocol = iscsi # 使用iscsi協議
iscsi_helper = lioadm
[[email protected] cinder]# grep -n ‘^[a-Z]‘ /etc/cinder/cinder.conf
421:glance_host = 192.168.3.199
536:auth_strategy = keystone
540:enabled_backends = lvm # lvm 使用的後端是lvm，要對應添加的[lvm]，自定義也可以
2294:rpc_backend = rabbit
2516:connection = mysql://cinder:[email protected]/cinder
2641:auth_uri = http://192.168.3.199:5000
2642:auth_url = http://192.168.3.199:35357
2643:auth_plugin = password
2644:project_domain_id = default
2645:user_domain_id = default
2646:project_name = service
2647:username = cinder
2648:password = cinder
2874:lock_path = /var/lib/cinder/tmp
3173:rabbit_host = 192.168.3.199
3177:rabbit_port = 5672
3189:rabbit_userid = openstack
3193:rabbit_password = openstack
[lvm] # 此行不是grep過濾出來的,因為是在配置文件最後添加上的，其對應的是540行的lvm
3416:volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver # 使用lvm後端存儲
3417:volume_group = cinder-volumes # vg的名稱：剛才創建的
3418:iscsi_protocol = iscsi # 使用iscsi協議
3419:iscsi_helper = lioadm
啟動存儲節點的cinder(這裏是node2)
[[email protected] cinder]# systemctl enable openstack-cinder-volume.service target.service Created symlink from /etc/systemd/system/multi-user.target.wants/openstack-cinder-volume.service to /usr/lib/systemd/system/openstack-cinder-volume.service. Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service. [[email protected] cinder]# systemctl start openstack-cinder-volume.service target.service
查看雲硬盤服務狀態（如果是虛擬機作為宿主機，時間不同步，會產生問題）
[[email protected] ~]# source admin-openrc.sh [[email protected] ~]# cinder service-list +------------------+-------------------------+------+---------+-------+----------------------------+-----------------+ | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +------------------+-------------------------+------+---------+-------+----------------------------+-----------------+ | cinder-scheduler | node1.chinasoft.com | nova | enabled | up | 2017-04-28T09:59:54.000000 | - | | cinder-volume | [email protected] | nova | enabled | up | 2017-04-28T09:59:58.000000 | - | +------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
創建一個雲硬盤,將雲硬盤掛載到虛擬機上,啟動在虛擬機實例詳情可以查看到，雲硬盤掛載的情況
具體步驟：
1.創建雲硬盤
2.點擊 動作選項下的 管理已連接雲硬盤，選擇需要的虛擬機 將雲硬盤掛載到指定的虛擬機上
在虛擬機中對掛載的硬盤進行分區格式化,如果有時不想掛載這個雲硬盤了，一定不要刪掉，生產環境一定要註意，否則虛擬機會出現error，應該使用umont確定卸載了，再使用dashboard進行刪除雲硬盤
# ssh cirros@192.168.3.103 $ sudo fdisk -l Disk /dev/vda: 1073 MB, 1073741824 bytes 255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/vda1 * 16065 2088449 1036192+ 83 Linux Disk /dev/vdb: 1073 MB, 1073741824 bytes 16 heads, 63 sectors/track, 2080 cylinders, total 2097152 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/vdb doesn‘t contain a valid partition table # 分區 $ sudo fdisk /dev/vdb Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel with disk identifier 0x3fecc8a5. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won‘t be recoverable. Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) Command (m for help): n Partition type: p primary (0 primary, 0 extended, 4 free) e extended Select (default p): p Partition number (1-4, default 1): 1 First sector (2048-2097151, default 2048): Using default value 2048 Last sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151): Using default value 2097151 Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. 查看分區，已經生成了新的分區/dev/vdb $ sudo fdisk -l Disk /dev/vda: 1073 MB, 1073741824 bytes 255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/vda1 * 16065 2088449 1036192+ 83 Linux Disk /dev/vdb: 1073 MB, 1073741824 bytes 9 heads, 8 sectors/track, 29127 cylinders, total 2097152 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0xfaacdc93 Device Boot Start End Blocks Id System /dev/vdb1 2048 2097151 1047552 83 Linux # 格式化 $ sudo mkfs.ext4 /dev/vdb1 mke2fs 1.42.2 (27-Mar-2012) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 65536 inodes, 261888 blocks 13094 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=268435456 8 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376 Allocating group tables: done Writing inode tables: done Creating journal (4096 blocks): done Writing superblocks and filesystem accounting information: done # 掛載 $ sudo mkdir /data $ sudo mount /dev/vdb1 /data $ df -h Filesystem Size Used Available Use% Mounted on /dev 242.3M 0 242.3M 0% /dev /dev/vda1 23.2M 18.0M 4.0M 82% / tmpfs 245.8M 0 245.8M 0% /dev/shm tmpfs 200.0K 72.0K 128.0K 36% /run /dev/vdb1 1006.9M 17.3M 938.5M 2% /data
方法二：從雲硬盤啟動虛擬機
①先創建一個demo2的雲硬盤
②動作-作為實例啟動
九、虛擬機創建流程：
第一階段：用戶操作
1)用戶使用Dashboard或者CLI連接keystone，發送用戶名和密碼，待keystone驗證通過，keystone會返回給dashboard一個authtoken
2）Dashboard會帶著上述的authtoken訪問nova-api進行創建虛擬機請求
3）nova-api會通過keytoken確認dashboard的authtoken認證消息。
第二階段：nova內組件交互階段
4）nova-api把用戶要創建的虛擬機的信息記錄到數據庫中.
5）nova-api使用rpc-call的方式發送請求給消息隊列
6）nova-scheduler獲取消息隊列中的消息
7）nova-scheduler和查看數據庫中要創建的虛擬機信息和計算節點的信息，進行調度
8)nova-scheduler把調度後的信息發送給消息隊列
9）nova-computer獲取nova-schedur發送給queue的消息
10）nova-computer通過消息隊列發送消息給nova-conudctor，想要獲取數據庫中的要創建虛擬機信息
11）nova-conductor獲取消息隊列的消息
12）nova-conductor讀取數據庫中要創建虛擬機的信息
13）nova-conductor把從數據庫獲取的消息返回給消息隊列
14）nova-computer獲取nova-conducter返回給消息隊列的信息
第三階段：nova和其他組件進行交互
15）nova-computer通過authtoken和數據庫返回的鏡像id請求glance服務
16）glance會通過keystone進行認證
17）glance驗證通過後把鏡像返回給nova-computer
18）nova-computer通過authtoken和數據庫返回的網絡id請求neutron服務
19）neutron會通過keystone進行認證
20）neutron驗證通過後把網絡分配情況返回給nova-computer
21）nova-computer通過authtoken和數據庫返回的雲硬盤請求cinder服務
22）cinder會通過keystone進行認證
23）cinder驗證通過後把雲硬盤分配情況返回給nova-computer
第四階段：nova創建虛擬機
24)nova-compute通過libvirt調用kvm根據已有的信息創建虛擬機，動態生成xml
25)nova-api會不斷的在數據庫中查詢信息並在dashboard顯示虛擬機的狀態
生產場景註意事項：
1、新加的一個計算節點，創建虛擬機時間會很長，因為第一次使用計算節點，沒有鏡像，計算節點要把glance的鏡像放在後端文件（/var/lib/nova/instance/_base）下，
鏡像如果很大，自然會需要很長時間，然後才會在後端文件的基礎上創建虛擬機（寫時復制copy on write）。
2、創建虛擬機失敗的原因之一：創建網橋失敗。要保證eth0網卡配置文件的BOOTPROTE是static而不是dhcp狀態。
OpenStack實踐系列⑨雲硬盤服務Cinder