目錄

前言

本篇記錄了 Openstack 在建立 Instances 時，nova-scheduler 作為排程器的工作原理和程式碼實現。
Openstack 中會由多個的 Instance 共享同一個 Host，而不是獨佔。所以就需要使用排程器這種管理規則來協調和管理 Instance 之間的資源分配。

排程器

排程器：排程 Instance 在哪一個 Host 上執行的方式。
目前 Nova 中實現的排程器方式由下列幾種：

• ChanceScheduler(隨機排程器)：從所有正常執行 nova-compute 服務的 Host Node 中隨機選取來建立 Instance

• FilterScheduler(過濾排程器)：根據指定的過濾條件以及權重來挑選最佳建立 Instance 的 Host Node 。

• Caching(快取排程器)：是 FilterScheduler 中的一種，在其基礎上將 Host 資源資訊快取到本地的記憶體中，然後通過後臺的定時任務從資料庫中獲取最新的 Host 資源資訊。

為了便於擴充套件，Nova 將一個排程器必須要實現的介面提取出來成為 nova.scheduler.driver.Scheduler，只要繼承了該類並實現其中的介面，我們就可以自定義排程器。

注意：不同的排程器並不能共存，需要在 /etc/nova/nova.conf

vim /etc/nova/nova.conf

scheduler_driver = nova.scheduler.filter_scheduler.FilterScheduler

FilterScheduler排程器的工作流程

FilterScheduler 首先使用指定的 Filters(過濾器) 過濾符合條件的 Host，EG. 記憶體使用率小於 2% 。然後對得到的 Host 列表計算 Weighting 權重並排序，獲得最佳的 Host 。

Filters 過濾器

Filtering 就是首先根據各個 Host 當前可用的資源情況來過濾掉那些不能滿足 Instance 要求的 Host，然後再使用配置檔案指定的各種 Filters 去過濾掉不符合過濾條件的 Host。經過 Filters 過濾後，會得到一個 Host 列表。

這樣的話 nova-scheduler 就需要從資料庫中取得當前各個 Host 最新的資源使用情況，這些資源資料的收集和儲存都由 nova-compute 中定義的資料庫同步機制來完成。但是 nova-compute 對資料庫的更新是週期性的， nova-scheduler 在選擇最佳 Host 時需要最新的資源資料。所以在 nova-scheduler 中使用了 nova.scheduler.host_manager:HostState 來維護一份資料。這份資料僅儲存在當前程序的記憶體中，裡面包含了從上次資料庫更新到現在 Host 資源的變化情況，也就是最新的 Host 資源資料。nova-scheduler 為了保持自己所維護的資源資料是最新的，每建立一個 Instance ，nova-scheduler 都要將這份資源資料更新，並從 Host 可用資源中去掉虛擬機器使用的部分。
注意：nova-scheduler 所維護的資料不會同步到資料庫，它只會從資料庫同步資料到自身，所以 nova-scheduler 並沒有寫資料庫的功能。

Filters 型別

• ALLHostsFilter：不進行任何過濾
• RamFilter：根據記憶體的可用情況來進行過濾
• ComputeFilter：選取所有處於 Active 的 Host
• TrustedFilter：選取所有可信的 Host
• PciPassthroughFilter：選取提供 PCI SR-IOV 支援的 Host

所有的 Filters 實現都位於nova/scheduler/filters 目錄，每個 Filter 都要繼承自 nova.scheduler.filters.BaseHostFilter 。如果需要自定義一個 Filter，只需通過繼承此類並實現一個函式 host_passes()，返回的結果只有 True or False 。

在配置檔案中指定 Filters：

scheduler_available_filters=
scheduler_default_filters=

Weighting 權重

Weighting 表示對所有符合過濾條件(通過 Filters)的 Host 計算權重並以此排序從而得到最佳的一個 Host。計算 Host 權重的過程需要呼叫指定的各種 Weigher Module，得到每個 Host 的權重值。

所有的 Weigher 的實現都位於 nova/scheduler/weights 目錄下。

原始碼實現

關鍵檔案及其意義

• /nova/scheduler/driver.py: 檔案中最重要的就是 Scheduler 類，是所有排程器實現都要繼承的基類，包含了排程器必須要實現的所有介面。

• /nova/scheduler/manager.py: 主要實現了 SchedulerManager 類，定義了 Host 的管理操作函式，如：刪除 Host 中的 Instance — delete_instance_info

• /nova/scheduler/host_manager.py: 有兩個類的實現，都是描述了跟排程器相關的 Host 的操作實現，類 HostState 維護了一份最新的 Host 資源資料。類 HostManager 描述了排程器相關的操作函式， EG._choose_host_filters/get_filtered_hosts/get_weighed_hosts

• /nova/scheduler/chance.py: 只有 ChanceScheduler 類(隨機排程器)，繼承自 Scheduler 類，實現隨機選取 Host Node 的排程器

• /nova/scheduler/client: 客戶端呼叫程式的入口

• /nova/scheduler/filter_scheduler.py: 只有 FilterScheduler 類(過濾排程器)，繼承自 Scheduler 類，實現了根據指定的過濾條件來選取 Host Node 的排程器

• /nova/scheduler/filters 和 /nova/scheduler/weights: 這兩個目錄下的內容分別對應 過濾器 和 權重 的實現 。

階段一：nova-scheduler 接收 build_instances RPC 遠端呼叫

nova-conductor ==> RPC scheduler_client.select_destinations() ==> nova-sechduler

#nova.conductor.manager.ComputeTaskManager:build_instances()

    def build_instances(self, context, instances, image, filter_properties,
            admin_password, injected_files, requested_networks,
            security_groups, block_device_mapping=None, legacy_bdm=True):
        # TODO(ndipanov): Remove block_device_mapping and legacy_bdm in version
        #                 2.0 of the RPC API.

        # 獲取需要建立的 Instance 的引數資訊
        request_spec = scheduler_utils.build_request_spec(context, image,
                                                          instances)

        # TODO(danms): Remove this in version 2.0 of the RPC API
        if (requested_networks and
                not isinstance(requested_networks,
                               objects.NetworkRequestList)):
            # 請求 network 資訊
            requested_networks = objects.NetworkRequestList(
                objects=[objects.NetworkRequest.from_tuple(t)
                         for t in requested_networks])
        # TODO(melwitt): Remove this in version 2.0 of the RPC API

        # 獲取 flavor 資訊
        flavor = filter_properties.get('instance_type')
        if flavor and not isinstance(flavor, objects.Flavor):
            # Code downstream may expect extra_specs to be populated since it
            # is receiving an object, so lookup the flavor to ensure this.
            flavor = objects.Flavor.get_by_id(context, flavor['id'])
            filter_properties = dict(filter_properties, instance_type=flavor)

        try:
            scheduler_utils.setup_instance_group(context, request_spec,
                                                 filter_properties)
            # check retry policy. Rather ugly use of instances[0]...
            # but if we've exceeded max retries... then we really only
            # have a single instance.
            scheduler_utils.populate_retry(filter_properties,
                instances[0].uuid)

            # 獲取 Hosts 列表
            hosts = self.scheduler_client.select_destinations(context,
                    request_spec, filter_properties)

        except Exception as exc:
            updates = {'vm_state': vm_states.ERROR, 'task_state': None}
            for instance in instances:
                self._set_vm_state_and_notify(
                    context, instance.uuid, 'build_instances', updates,
                    exc, request_spec)
            return

        for (instance, host) in itertools.izip(instances, hosts):
            try:
                instance.refresh()
            except (exception.InstanceNotFound,
                    exception.InstanceInfoCacheNotFound):
                LOG.debug('Instance deleted during build', instance=instance)
                continue
            local_filter_props = copy.deepcopy(filter_properties)
            scheduler_utils.populate_filter_properties(local_filter_props,
                host)
            # The block_device_mapping passed from the api doesn't contain
            # instance specific information
            bdms = objects.BlockDeviceMappingList.get_by_instance_uuid(
                    context, instance.uuid)


            self.compute_rpcapi.build_and_run_instance(context,
                    instance=instance, host=host['host'], image=image,
                    request_spec=request_spec,
                    filter_properties=local_filter_props,
                    admin_password=admin_password,
                    injected_files=injected_files,
                    requested_networks=requested_networks,
                    security_groups=security_groups,
                    block_device_mapping=bdms, node=host['nodename'],
                    limits=host['limits'])

nova-conductor 在呼叫 nova-scheduler 來獲取能夠建立 Instance 的 Host 的同時也獲取了：requested_networks/flavor 等資訊。

其中獲取 Hosts 列表的程式碼塊：

            # 獲取 Hosts 列表
            hosts = self.scheduler_client.select_destinations(context,
                    request_spec, filter_properties)

下面列出了一系列為了獲取 Hosts 列表的函式呼叫跳轉：

# nova.scheduler.client.query.SchedulerQueryClient:select_destinations()

from nova.scheduler import rpcapi as scheduler_rpcapi

class SchedulerQueryClient(object):
    """Client class for querying to the scheduler."""

    def __init__(self):
        self.scheduler_rpcapi = scheduler_rpcapi.SchedulerAPI()

    def select_destinations(self, context, request_spec, filter_properties):
        """Returns destinations(s) best suited for this request_spec and
        filter_properties.

        The result should be a list of dicts with 'host', 'nodename' and
        'limits' as keys.
        """
        # 
        return self.scheduler_rpcapi.select_destinations(
            context, request_spec, filter_properties)


# nova.scheduler.rpcapi.SchedulerAPI:select_destinations()

    def select_destinations(self, ctxt, request_spec, filter_properties):
        cctxt = self.client.prepare(version='4.0')
        return cctxt.call(ctxt, 'select_destinations',
            request_spec=request_spec, filter_properties=filter_properties)

階段二：從 scheduler.rpcapi.SchedulerAPI 到 scheduler.manager.SchedulerManager

rpcapi.py 中的介面函式會在 manager.py 中實現實際操作函式。
所以跳轉到 nova.scheduler.manager.SchedulerManager:select_destinations()

# nova.scheduler.manager.SchedulerManager:select_destinations()
class SchedulerManager(manager.Manager):
    """Chooses a host to run instances on."""

    target = messaging.Target(version='4.2')

    def __init__(self, scheduler_driver=None, *args, **kwargs):
        if not scheduler_driver:
            scheduler_driver = CONF.scheduler_driver
        # 可以看出這裡的 driver 是通過配置檔案中的選項值指定的類來返回的物件 EG.nova.scheduler.filter_scheduler.FilterScheduler
        self.driver = importutils.import_object(scheduler_driver)
        super(SchedulerManager, self).__init__(service_name='scheduler',
                                               *args, **kwargs)


    def select_destinations(self, context, request_spec, filter_properties):
        """Returns destinations(s) best suited for this request_spec and
        filter_properties.

        The result should be a list of dicts with 'host', 'nodename' and
        'limits' as keys.
        """
        dests = self.driver.select_destinations(context, request_spec,
            filter_properties)
        return jsonutils.to_primitive(dests)

階段三：從 scheduler.manager.SchedulerManager 到排程器 FilterScheduler

vim /etc/nova/nova.conf

scheduler_driver = nova.scheduler.filter_scheduler.FilterScheduler

從配置檔案選項 scheduler_driver 的值可以知道，nova.scheduler.manager.SchedulerManager:driver
是 nova.scheduler.filter_scheduler.FilterScheduler 的例項化物件。
所以跳轉到 nova.scheduler.filter_scheduler.FilterScheduler:select_destinations() 。

# nova.scheduler.filter_scheduler.FilterScheduler:select_destinations()

class FilterScheduler(driver.Scheduler):
    """Scheduler that can be used for filtering and weighing."""
    def __init__(self, *args, **kwargs):
        super(FilterScheduler, self).__init__(*args, **kwargs)
        self.options = scheduler_options.SchedulerOptions()
        self.notifier = rpc.get_notifier('scheduler')

    def select_destinations(self, context, request_spec, filter_properties):
        """Selects a filtered set of hosts and nodes."""
        self.notifier.info(context, 'scheduler.select_destinations.start',
                           dict(request_spec=request_spec))

        # 需要建立的 Instances 的數量
        num_instances = request_spec['num_instances']

        # 獲取滿足笫一次過濾條件的主機列表 List (詳見上述的排程器過濾原理)
        # nova.scheduler.filter_scheduler.FilterScheduler:_schedule() ==> return selected_hosts
        selected_hosts = self._schedule(context, request_spec,
                                        filter_properties)

        # Couldn't fulfill the request_spec
        # 當請求的 Instance 數量大於合適的主機數量時，不會建立 Instance 且輸出 'There are not enough hosts available.'
        if len(selected_hosts) < num_instances:
            # NOTE(Rui Chen): If multiple creates failed, set the updated time
            # of selected HostState to None so that these HostStates are
            # refreshed according to database in next schedule, and release
            # the resource consumed by instance in the process of selecting
            # host.
            for host in selected_hosts:
                host.obj.updated = None

            # Log the details but don't put those into the reason since
            # we don't want to give away too much information about our
            # actual environment.
            LOG.debug('There are %(hosts)d hosts available but '
                      '%(num_instances)d instances requested to build.',
                      {'hosts': len(selected_hosts),
                       'num_instances': num_instances})

            reason = _('There are not enough hosts available.')
            raise exception.NoValidHost(reason=reason)

        dests = [dict(host=host.obj.host, nodename=host.obj.nodename,
                      limits=host.obj.limits) for host in selected_hosts]

        self.notifier.info(context, 'scheduler.select_destinations.end',
                           dict(request_spec=request_spec))
        return dests


 def _schedule(self, context, request_spec, filter_properties):
        # 獲取所有 Hosts 的狀態
        hosts = self._get_all_host_states(elevated)

        selected_hosts = []

        # 獲取需要建立的 Instances 數目
        num_instances = request_spec.get('num_instances', 1)

        # 遍歷 num_instances，為每個 Instance 選取合適的主機
        for num in range(num_instances):
            # Filter local hosts based on requirements ...

            # 在 for 迴圈裡，_schedule 的兩個關鍵操作，get_filtered_hosts() 和 get_weighed_hosts()
            hosts = self.host_manager.get_filtered_hosts(hosts,
                    filter_properties, index=num)
            if not hosts:
                # Can't get any more locally.
                break

            LOG.debug("Filtered %(hosts)s", {'hosts': hosts})

            weighed_hosts = self.host_manager.get_weighed_hosts(hosts,
                    filter_properties)

            LOG.debug("Weighed %(hosts)s", {'hosts': weighed_hosts})

            scheduler_host_subset_size = CONF.scheduler_host_subset_size

            # 下面兩個 if，主要為了防止 random.choice 呼叫越界
            if scheduler_host_subset_size > len(weighed_hosts):
                scheduler_host_subset_size = len(weighed_hosts)
            if scheduler_host_subset_size < 1:
                scheduler_host_subset_size = 1

            # 在符合要求的weigh過的host裡進行隨機選取
            chosen_host = random.choice(
                weighed_hosts[0:scheduler_host_subset_size])
            LOG.debug("Selected host: %(host)s", {'host': chosen_host})
            selected_hosts.append(chosen_host)

            # Now consume the resources so the filter/weights
            # will change for the next instance.
            chosen_host.obj.consume_from_instance(instance_properties)
            if update_group_hosts is True:
                if isinstance(filter_properties['group_hosts'], list):
                    filter_properties['group_hosts'] = set(
                        filter_properties['group_hosts'])
                filter_properties['group_hosts'].add(chosen_host.obj.host)
        # 迴圈為每一個例項獲取合適的主機後，返回選擇的主機列表
        return selected_hosts

上述的函式有三個非常關鍵的操作函式：

• _get_all_host_states: 獲取所有的 Host 狀態，並且將初步滿足條件的 Hosts 過濾出來。
• get_filtered_hosts：使用 Filters 過濾器將第一個函式返回的 hosts 進行再一次過濾。
• get_weighed_hosts：通過 Weighed 選取最優 Host。

這三個關鍵函式在後面會繼續介紹。

首先看看host_manager.get_filtered_hosts() 中，host_manager 是 nova.scheduler.driver.Scheduler 的成員變數 。如下：

# nova.scheduler.driver.Scheduler:__init__()

# nova.scheduler.filter_scheduler.FilterScheduler 繼承了 nova.scheduler.driver.Scheduler
 class Scheduler(object):
     """The base class that all Scheduler classes should inherit from."""

     def __init__(self):
         # 從這裡知道 host_manager 會根據配置檔案動態匯入
         self.host_manager = importutils.import_object(
                 CONF.scheduler_host_manager)
         self.servicegroup_api = servicegroup.API()

還需要注意：scheduler.filter_scheduler.FilterScheduler:_schedule() 中獲取 Hosts 狀態的函式 _get_all_host_states() 實現如下：

# nova.scheduler.host_manager.HostManager:get_all_host_states()

 def get_all_host_states(self, context):

        service_refs = {service.host: service
                        for service in objects.ServiceList.get_by_binary(
                            context, 'nova-compute')}

        # 獲取 Compute Node 資源
        compute_nodes = objects.ComputeNodeList.get_all(context)
        # nova.object.__init__()
        #     ==> nova.object.compute_node.ComputeNodeList:get_all
        seen_nodes = set()
        for compute in compute_nodes:
            service = service_refs.get(compute.host)

            if not service:
                LOG.warning(_LW(
                    "No compute service record found for host %(host)s"),
                    {'host': compute.host})
                continue
            host = compute.host
            node = compute.hypervisor_hostname
            state_key = (host, node)
            host_state = self.host_state_map.get(state_key)

            # 更新主機資訊
            if host_state:
                host_state.update_from_compute_node(compute)
            else:
                host_state = self.host_state_cls(host, node, compute=compute)
                self.host_state_map[state_key] = host_state
            # We force to update the aggregates info each time a new request
            # comes in, because some changes on the aggregates could have been
            # happening after setting this field for the first time
            host_state.aggregates = [self.aggs_by_id[agg_id] for agg_id in
                                     self.host_aggregates_map[
                                         host_state.host]]
            host_state.update_service(dict(service))
            self._add_instance_info(context, compute, host_state)
            seen_nodes.add(state_key)

        # remove compute nodes from host_state_map if they are not active
        # * 移除 not active 的節點
        dead_nodes = set(self.host_state_map.keys()) - seen_nodes


for state_key in dead_nodes:
            host, node = state_key
            LOG.info(_LI("Removing dead compute node %(host)s:%(node)s "
                         "from scheduler"), {'host': host, 'node': node})
            del self.host_state_map[state_key]

        return six.itervalues(self.host_state_map)
# get_all_host_states主要用來去除不活躍的節點

繼續往下看獲取 Compute Node 資源資訊函式 objects.ComputeNodeList.get_all(context) 的實現。

# nova.object.compute_node:get_all()

    @base.remotable_classmethod
    def get_all(cls, context):
        # 調到了 nova.db.api.compute_node_get_all()
        db_computes = db.compute_node_get_all(context)


        return base.obj_make_list(context, cls(context), objects.ComputeNode,
                                  db_computes)



#nova.db.api:compute_node_get_all()

def compute_node_get_all(context):
    """Get all computeNodes.

    :param context: The security context

    :returns: List of dictionaries each containing compute node properties
    """
    return IMPL.compute_node_get_all(context)

至此，說明 liberty 版本的 nova-scheduler 還是能夠訪問資料庫的。

問題是： nova-scheduler 是怎麼更新主機資訊的，能夠直接資料庫進行寫操作嗎？
答案是：不能，nova-scheduler 不能夠對資料庫進行寫操作，但是卻可以從資料庫中讀取 Host 資源資料並快取在程序的記憶體中。如下：

# nova.scheduler.host_manager.HostState:__init__()
class HostState(object):
    """Mutable and immutable information tracked for a host.
    This is an attempt to remove the ad-hoc data structures
    previously used and lock down access.
    """

    def __init__(self, host, node, compute=None):
        self.host = host
        self.nodename = node

        # Mutable available resources.
        # These will change as resources are virtually "consumed".
        self.total_usable_ram_mb = 0
        self.total_usable_disk_gb = 0
        self.disk_mb_used = 0
        self.free_ram_mb = 0
        self.free_disk_mb = 0
        self.vcpus_total = 0
        self.vcpus_used = 0
        self.pci_stats = None
        self.numa_topology = None

        # Additional host information from the compute node stats:
        self.num_instances = 0
        self.num_io_ops = 0

        # Other information
        self.host_ip = None
        self.hypervisor_type = None
        self.hypervisor_version = None
        self.hypervisor_hostname = None
        self.cpu_info = None
        self.supported_instances = None

nova-scheduler 並沒有寫資料庫的操作函式，但是 nova-scheduler 會將資料庫的資料快取到程序記憶體中。這樣就可以在保證了 nova-scheduler 能使用最新的 Host 資源資訊，同時下降低了對資料庫的 I/O 請求。

階段四：從排程器 FilterScheduler 到過濾器 Filters

上面的程式碼中呼叫了 Filters 函式：get_filtered_hosts()，實現如下：

# nova.scheduler.host_manager.HostManager:get_filtered_hosts()
    def get_filtered_hosts(self, hosts, filter_properties,
            filter_class_names=None, index=0):
        """Filter hosts and return only ones passing all filters."""
        # 下面定義了若干區域性函式，先省略掉
        def _strip_ignore_hosts(host_map, hosts_to_ignore):
            ignored_hosts = []
            for host in hosts_to_ignore:

        。。。。
        # 返回經過驗證的可用的過濾器；
        filter_classes = self._choose_host_filters(filter_class_names)
        。。。。
            # 呼叫了get_filtered_objects
            return self.filter_handler.get_filtered_objects(filters,
                        hosts, filter_properties, index)



# 繼續跳轉到 get_filtered_objects()
 def get_filtered_objects(self, filters, objs, filter_properties, index=0):
        list_objs = list(objs)
        LOG.debug("Starting with %d host(s)", len(list_objs))
        part_filter_results = []
        full_filter_results = []
        log_msg = "%(cls_name)s: (start: %(start)s, end: %(end)s)"
        for filter_ in filters:
            if filter_.run_filter_for_index(index):
                cls_name = filter_.__class__.__name__
                start_count = len(list_objs)
                # 關鍵的一句話
                objs = filter_.filter_all(list_objs, filter_properties)
                if objs is None:
                    LOG.debug("Filter %s says to stop filtering", cls_name)
                    return
                list_objs = list(objs)
                end_count = len(list_objs)
                part_filter_results.append(log_msg % {"cls_name": cls_name,
                        "start": start_count, "end": end_count})
                if list_objs:
                    remaining = [(getattr(obj, "host", obj),
                                  getattr(obj, "nodename", ""))
                                 for obj in list_objs]
                    full_filter_results.append((cls_name, remaining))

        return list_objs



# objs 的 return 又呼叫了 filter_.filter_all(list_objs, filter_properties)
def filter_all(self, filter_obj_list, filter_properties):
        for obj in filter_obj_list：
            if self._filter_one(obj, filter_properties):
                # 符合規則 生產一個obj
                yield obj



# 繼續呼叫 _filter_one()
def _filter_one(self, obj, filter_properties):

        # 如果符合 Filter 過濾器，就返回 TRUE，否則返回 FALSE

        return self.host_passes(obj, filter_properties)

經過一連串的呼叫跳轉，Filter 的過濾工作就完成了。

階段五：Filters 到權重計算與排序

# nova.scheduler.host_manager.HostManager:get_weighed_hosts（）
    def get_weighed_hosts(self, hosts, weight_properties):
        """Weigh the hosts."""
        return self.weight_handler.get_weighed_objects(self.weighers,
                hosts, weight_properties)


# nova.weights.BaseWeightHandler:get_weighed_objects（）
class BaseWeightHandler(loadables.BaseLoader):
    object_class = WeighedObject

    def get_weighed_objects(self, weighers, obj_list, weighing_properties):
        """Return a sorted (descending), normalized list of WeighedObjects."""
        weighed_objs = [self.object_class(obj, 0.0) for obj in obj_list]

        if len(weighed_objs) <= 1:
            return weighed_objs

        for weigher in weighers:
            weights = weigher.weigh_objects(weighed_objs, weighing_properties)

            # Normalize the weights
            weights = normalize(weights,
                                minval=weigher.minval,
                                maxval=weigher.maxval)

            for i, weight in enumerate(weights):
                obj = weighed_objs[i]
                obj.weight += weigher.weight_multiplier() * weight

        # 進行排序
        return sorted(weighed_objs, key=lambda x: x.weight, reverse=True)