**作者：張華 發表於：2016-08-05
版權宣告：可以任意轉載，轉載時請務必以超連結形式標明文章原始出處和作者資訊及本版權宣告
http://blog.csdn.net/quqi99 )**

理論基礎

1. iscsi還不能執行在容器裡（因為netlink還不支援namesapce)，本文采用rbd使用ceph代替iscsi
2. ovs, kvm通過定義profile支援執行在容器裡。ovs目前只支援security.privileged: “true”

配置LXD

參考Play with LXD一文 在ubuntu 16.04上部署LXD環境。

LXD上部署OpenStack

1, 從這個

curl https://api.jujucharms.com/charmstore/v5/openstack-base/archive -o openstack-base.zip
unzip openstack-base.zip

2, 執行’juju bootstrap’，注意：執行這一步時先不要修改profile

sudo snap install lxd --classic
#export PATH=/snap/bin:$PATH
juju bootstrap --debug --config bootstrap-series=xenial --config agent-stream=devel localhost lxd-controller
lxc exec `lxc list |grep juju- |awk -F '
 
|' '{print $2}'` bash

3, 建立model，且它會自動生成juju-openstack-model profile （’juju add-model’會自動執行這一句‘lxc profile create juju-openstack-model 2>/dev/null || echo “juju-openstack-model profile already exists”’）, 如果不定義model，就會有一個名為default的model，那麼這時下面第4步要編輯juju-default profile。

juju add-model openstack-model
 

juju models
lxc profile show juju-openstack-model

4, 編輯juju-openstack-model profile。

#sudo apt-get install --reinstall linux-image-extra-$(uname -r)
sudo modprobe nbd
sudo modprobe ip_tables
sudo modprobe openvswitch

cat << EOF > juju-openstack-model.yaml
name: juju-openstack-model
config:
  boot.autostart: "true"
  security.nesting: "true"
  security.privileged: "true"
  raw.lxc: lxc.apparmor.profile=unconfined
  linux.kernel_modules: openvswitch,nbd,ip_tables,ip6_tables,ebtables,netlink_diag,nf_nat,overlay
devices:
  eth0:
    mtu: "9000"
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  eth1:
    mtu: "9000"
    name: eth1
    nictype: bridged
    parent: lxdbr0
    type: nic
  kvm:
    path: /dev/kvm
    type: unix-char
  mem:
    path: /dev/mem
    type: unix-char
  root:
    path: /
    pool: default
    type: disk
  tun:
    path: /dev/net/tun
    type: unix-char
EOF
cat juju-openstack-model.yaml | lxc profile edit juju-openstack-model
#其他命令演示
#lxc profile set juju-openstack-model raw.lxc lxc.aa_profile=unconfined
#lxc profile device add juju-openstack-model fuse unix-char path=/dev/fuse
#sudo lxc network create lxdbr1 ipv4.address=auto ipv4.nat=true ipv6.address=none

5, 使用juju一鍵部署openstack

git clone https://github.com/openstack-charmers/openstack-on-lxd
cd openstack-on-lxd
juju deploy bundle-xenial-mitaka.yaml
juju status
juju debug-log

juju config neutron-gateway data-port=br-ex:eth1
juju resolved neutron-gateway/0

安裝過程中遇到的問題

• 如果報這個錯 - failed to bootstrap model: cannot start bootstrap instance: The container’s root device is missing the pool property， 那是要在profile中的root元素下新增：pool: default
• bootstrap時報這個錯 - FATAL: Module ip6_tables，ebtables，netlink_diag not found in directory /lib/modules/4.4.0-98-generic - 執行‘sudo apt-get install –reinstall linux-image-extra-(uname−r)′安裝模塊（/lib/modules/$(uname-r{)}^{\prime }安裝模塊（/lib/modules/$(uname -r)/kernel/net/netlink/netlink_diag.ko）。此外是因為profile中寫的這些模組名是從網頁拷過來的存在亂碼。

配置使用OpenStack

source novarc
$ cat novarc 
#!/bin/bash
_keystone_unit=$(juju status keystone --format yaml | \
    awk '/units:$/ {getline; gsub(/:$/, ""); print $1}')
_keystone_ip=$(juju run --unit ${_keystone_unit} 'unit-get private-address')
_password=$(juju run --unit ${_keystone_unit} 'leader-get admin_passwd')
export OS_USERNAME=admin
export OS_PASSWORD=${_password}
export OS_TENANT_NAME=admin
export OS_REGION_NAME=RegionOne
export OS_AUTH_URL=${OS_AUTH_PROTOCOL:-http}://${_keystone_ip}:5000/v2.0
export OS_AUTH_TYPE=password

wget http://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img
glance image-create --name xenial --file xenial-server-cloudimg-amd64-disk1.img --visibility public --progress --container-format bare --disk-format qcow2
glance image-list

cd openstack-on-lxd
./neutron-ext-net --network-type flat -g 10.0.8.1 -c 10.0.8.0/24 -f 10.0.8.201:10.0.8.254 ext_net
./neutron-tenant-net -t admin -r provider-router -N 10.0.8.1 internal 192.168.20.0/24
neutron net-list

nova keypair-add --pub-key ~/.ssh/id_rsa.pub mykey
nova boot --image xenial --flavor m1.small --key-name mykey --nic net-id=$(neutron net-list | grep internal | awk '{ print $2 }') i1
nova list

neutron floatingip-create ext_net
neutron floatingip-associate $(neutron floatingip-list |grep 10.0.8.202 |awk '{print $2}') $(neutron port-list |grep '192.168.20.3' |awk '{print $2}')

for i in $(openstack security group list | awk '/default/{ print $2 }'); do \
    openstack security group rule create $i --protocol icmp --remote-ip 0.0.0.0/0; \
    openstack security group rule create $i --protocol tcp --remote-ip 0.0.0.0/0 --dst-port 22; \
done

ssh ubuntu@<new-floating-ip>
#juju ssh neutron-gateway/0 -- sudo ip netns exec qrouter-d0e7bf5c-c0ac-4980-b042-68b4550230e5 ping 10.0.8.202

cinder --os-volume-api-version 2 create --name testvolume 1
nova volume-attach xenial $(cinder list | grep testvolume | awk '{ print $2 }') /dev/vdc
cinder --os-volume-api-version 2 create --image-id $(glance image-list |grep trusty |awk '{print $2}') --display-name bootvol 8
nova boot --key-name mykey --image trusty --flavor m1.small --nic net-id=$(neutron net-list |grep ' private ' |awk '{print $2}')  --block-device-mapping vda=$(cinder --os-volume-api-version 2 list |grep bootvol |awk '{print $2}'):::0 i1

又一例 - 部署opencontrail在lxd單機上

下面的yaml是juju2.0的，如果是juju1.x可見：http://pastebin.ubuntu.com/24170320/
實際上,opencontrail vrouter部署在容器裡會報下列錯，此例子只是說明yaml怎麼寫。

2017-03-13 11:46:06 INFO juju-log Loading kernel module vrouter
2017-03-13 11:46:06 INFO install modprobe: ERROR: ../libkmod/libkmod.c:556 kmod_search_moddep() could not open moddep file '/lib/modules/4.8.0-34-generic/modules.dep.bin'
2017-03-13 11:46:06 INFO juju-log vrouter kernel module failed to load, clearing pagecache and retrying

series: trusty
services:
  # openstack
  ubuntu:
    charm: cs:trusty/ubuntu
    num_units: 1
  ntp:
    charm: cs:trusty/ntp
  mysql:
    charm: cs:trusty/mysql
    options:
      dataset-size: 15%
      max-connections: 1000
    num_units: 1
  rabbitmq-server:
    charm: cs:trusty/rabbitmq-server
    num_units: 1
  keystone:
    charm: cs:~sdn-charmers/trusty/keystone
    options:
      admin-password: password
      admin-role: admin
      openstack-origin: cloud:trusty-mitaka
    num_units: 1
  nova-cloud-controller:
    charm: cs:trusty/nova-cloud-controller
    options:
      network-manager: Neutron
      openstack-origin: cloud:trusty-mitaka
    num_units: 1
  neutron-api:
    charm: cs:trusty/neutron-api
    options:
      manage-neutron-plugin-legacy-mode: false
      openstack-origin: cloud:trusty-mitaka
    num_units: 1
  glance:
    charm: cs:trusty/glance
    options:
      openstack-origin: cloud:trusty-mitaka
    num_units: 1
  openstack-dashboard:
    charm: cs:trusty/openstack-dashboard
    options:
      openstack-origin: cloud:trusty-mitaka
    num_units: 1
  nova-compute:
    charm: cs:trusty/nova-compute
    options:
      openstack-origin: cloud:trusty-mitaka
    num_units: 1
  # contrail
  cassandra:
    charm: cs:trusty/cassandra
    options:
      authenticator: AllowAllAuthenticator
      install_sources: |
        - deb http://www.apache.org/dist/cassandra/debian 22x main
        - ppa:openjdk-r/ppa
        - ppa:stub/cassandra
    num_units: 1
  zookeeper:
    charm: cs:~charmers/trusty/zookeeper
    num_units: 1
  kafka:
    charm: cs:~sdn-charmers/trusty/apache-kafka
    num_units: 1
  contrail-configuration:
    charm: cs:~sdn-charmers/trusty/contrail-configuration
    options:
      openstack-origin: cloud:trusty-mitaka
    num_units: 1
  contrail-control:
    charm: cs:~sdn-charmers/trusty/contrail-control
    num_units: 1
  contrail-analytics:
    charm: cs:~sdn-charmers/trusty/contrail-analytics
    num_units: 1
  contrail-webui:
    charm: cs:~sdn-charmers/trusty/contrail-webui
    num_units: 1
  neutron-api-contrail:
    charm: cs:~sdn-charmers/trusty/neutron-api-contrail
    num_units: 0
  neutron-contrail:
    charm: cs:~sdn-charmers/trusty/neutron-contrail
    num_units: 0

relations:
  # openstack
 - [ ubuntu, ntp ]
 - [ keystone, mysql ]
 - [ glance, mysql ]
 - [ glance, keystone ]
 - [ nova-cloud-controller, mysql ]
 - [ nova-cloud-controller, rabbitmq-server ]
 - [ nova-cloud-controller, keystone ]
 - [ nova-cloud-controller, glance ]
 - [ neutron-api, mysql ]
 - [ neutron-api, rabbitmq-server ]
 - [ neutron-api, nova-cloud-controller ]
 - [ neutron-api, keystone ]
 - [ neutron-api, neutron-api-contrail ]
 - [ "nova-compute:shared-db", "mysql:shared-db" ]
 - [ "nova-compute:amqp", "rabbitmq-server:amqp" ]
 - [ nova-compute, glance ]
 - [ nova-compute, nova-cloud-controller ]
 - [ nova-compute, ntp ]
 - [ openstack-dashboard, keystone ]
  # contrail
 - [ kafka, zookeeper ]
 - [ "contrail-configuration:cassandra", "cassandra:database" ]
 - [ contrail-configuration, zookeeper ]
 - [ contrail-configuration, rabbitmq-server ]
 - [ "contrail-configuration:identity-admin", "keystone:identity-admin" ]
 - [ "contrail-configuration:identity-service", "keystone:identity-service" ]
 - [ neutron-api-contrail, contrail-configuration ]
 - [ neutron-api-contrail, keystone ]
 - [ "contrail-control:contrail-api", "contrail-configuration:contrail-api" ]
 - [ "contrail-control:contrail-discovery", "contrail-configuration:contrail-discovery" ]
 - [ "contrail-control:contrail-ifmap", "contrail-configuration:contrail-ifmap" ]
 - [ contrail-control, keystone ]
 - [ "contrail-analytics:cassandra", "cassandra:database" ]
 - [ contrail-analytics, kafka ]
 - [ contrail-analytics, zookeeper ]
 - [ "contrail-analytics:contrail-api", "contrail-configuration:contrail-api" ]
 - [ "contrail-analytics:contrail-discovery", "contrail-configuration:contrail-discovery" ]
 - [ "contrail-analytics:identity-admin", "keystone:identity-admin" ]
 - [ "contrail-analytics:identity-service", "keystone:identity-service" ]
 - [ "contrail-configuration:contrail-analytics-api", "contrail-analytics:contrail-analytics-api" ]
 - [ nova-compute, neutron-contrail ]
 - [ "neutron-contrail:contrail-discovery", "contrail-configuration:contrail-discovery" ]
 - [ "neutron-contrail:contrail-api", "contrail-configuration:contrail-api" ]
 - [ neutron-contrail, keystone ]
 - [ contrail-webui, keystone ]
 - [ "contrail-webui:cassandra", "cassandra:database" ]

通過conjure-up安裝OpenStack

我們也可以通過conjure-up安裝OpenStack,

#Install a lxd container
sudo lxc init ubuntu:16.04 openstack -c security.privileged=true -c security.nesting=true -c "linux.kernel_modules=iptable_nat, ip6table_nat, ebtables, openvswitch, nbd"
printf "lxc.cap.drop=\nlxc.aa_profile=unconfined\n" | sudo lxc config set openstack raw.lxc -
sudo lxc config get openstack raw.lxc
lxc config device add openstack mem unix-char path=/dev/mem
lxc start openstack
lxc list

#Install conjure-up inside the lxd container
#lxc exec openstack bash
lxc exec openstack -- apt update
#lxc exec openstack -- apt dist-upgrade -y
lxc exec openstack -- apt install squashfuse -y
lxc exec openstack -- ln -s /bin/true /usr/local/bin/udevadm
lxc exec openstack -- snap install conjure-up --classic

#Init lxd container
#Use the “dir” storage backend (“zfs” doesn’t work in a nested container)
#Do NOT configure IPv6 networking (conjure-up/juju don’t play well with it)
#lxc exec openstack -- lxd init
lxc exec openstack -- snap install lxd
sleep 10  #avoid the error 'Unable to talk to LXD: Get http://unix.socket/1.0'
lxc exec openstack -- /snap/bin/lxd init --auto
lxc exec openstack -- /snap/bin/lxc network create lxdbr0 ipv4.address=auto ipv4.nat=true ipv6.address=none
lxc exec openstack -- /snap/bin/lxc profile show default

#Deploying OpenStack with conjure-up in nested LXD
#conjure-up is a nice, user friendly, tool that interfaces with Juju to deploy complex services.
#Step 1, select “OpenStack with NovaLXD”
#Step 2, select “localhost” as the deployment target (uses LXD)
#Step 3, select default in all middle steps, and click “Deploy all remaining applications”
lxc exec openstack -- sudo -u ubuntu -i conjure-up
[email protected]:~$ sudo lxc list
+-----------+---------+--------------------------------+------+------------+-----------+
|   NAME    |  STATE  |              IPV4              | IPV6 |    TYPE    | SNAPSHOTS |
+-----------+---------+--------------------------------+------+------------+-----------+
| openstack | RUNNING | 10.73.227.154 (eth0)           |      | PERSISTENT | 0         |
|           |         | 10.164.92.1 (lxdbr0)           |      |            |           |
|           |         | 10.101.0.1 (conjureup0)        |      |            |           |
+-----------+---------+--------------------------------+------+------------+-----------+

#Or deploy OpenStack with conjure-up in physical node
sudo snap install lxd
export PATH=/snap/bin:$PATH
sudo /snap/bin/lxd init --auto
sudo /snap/bin/lxc network create lxdbr0 ipv4.address=auto ipv4.nat=true ipv6.address=none
sudo -i
conjure-up openstack #but I hit the error 'This should _not_ be run as root or with sudo' even though I've already used root

下面粘一些使用conjure-up過程中的截圖：

參考