1. 程式人生 > 實用技巧 >模板超程式設計 - 實現查詢最大最小值

模板超程式設計 - 實現查詢最大最小值

在某次關於釋出系統的專案中,需要呼叫ansible來發布任務,其中一段程式碼是初始化ansible的連線,並傳入一個source(目標機器)的值,程式碼段如下:

from .ansible_api import AnsibleClient
sources = '192.168.1.218'
client = AnsibleClient(sources)
...

完成後發現一直報錯,如下所示錯誤資訊:

[WARNING]: Unable to parse python/devops/192.168.1.218 as an inventory source
  看字面意思描述,是說無法解析python/devops/192.168.1.218為一個inventory源。inventory是ansible專用的一個用來儲存目標機器的檔案。通過報錯資訊大概能夠猜出,當我想要以IP地址的方式傳入目標機器時候,ansible卻把IP當做inventory檔案來解析了。

  先丟擲結論,正確的傳值方式,是在IP地址後增加一個 ',' ,讓sources = '192.168.1.218,' ,再傳入。或者傳入兩個以上的IP,如'192.168.1.218,192.168.1.219' 。

  但是傳入一個IP是我的一個正常應用場景,那麼通過原始碼分析,我們來看看為何要出現以上結果。

  例項化的AnsibleClient是在ansible_api檔案中,有一個AnsibleClient類,其中有初始化方法__init__(self, source),程式碼如下:

class AnsibleClient:
    def __init__(self, source):
        self.source = source
        self.loader = DataLoader()
        self.inventory = InventoryManager(loader=self.loader, sources=self.source)
        self.variable_manager = VariableManager(loader=self.loader, inventory=self.inventory)
        self.passwords = dict(vault_pass='secret')
        self.callback = None

  在以上程式碼中,可以看到在例項化AnsibleClient連線時,之前的192.168.1.218會由source引數傳進來,並且最終再傳入InventoryManager

  接著再檢視InventoryManager的原始碼,原始碼所在檔案位置為路徑ansible/inventory/manager.py中,程式碼片段如下所示:

class InventoryManager(object):
def init(self, loader, sources=None):
if sources is None:
self._sources = []
elif isinstance(sources, string_types):
self._sources = [sources]
else:
self._sources = sources
self.parse_sources(cache=True)
  通過以上程式碼,會發現在InventoryManager例項化時候,如果sources有值傳入,那麼sources會賦值給self._sources,並繼續走self.parse_sources(cache=True)這段,通過利用print('報錯資訊在此!!!')斷點大法,確實如此。

  接著檢視parse_sources方法,程式碼片段如下:

def parse_sources(self, cache=False):
    ''' iterate over inventory sources and parse each one to populate it'''

    parsed = False
    # allow for multiple inventory parsing
    for source in self._sources:

        if source:
            if ',' not in source:
                source = unfrackpath(source, follow=False)
            parse = self.parse_source(source, cache=cache)
            if parse and not parsed:
                parsed = True

  注意裡面有一句if ',' not in source:,此時我們會走到這一分支,並進入unfrackpath這一步,接著檢視這一方法的原始碼,如下所示:

def unfrackpath(path, follow=True, basedir=None):
b_basedir = to_bytes(basedir, errors='surrogate_or_strict', nonstring='passthru')

if b_basedir is None:
    b_basedir = to_bytes(os.getcwd(), errors='surrogate_or_strict')
elif os.path.isfile(b_basedir):
    b_basedir = os.path.dirname(b_basedir)

b_final_path = os.path.expanduser(os.path.expandvars(to_bytes(path, errors='surrogate_or_strict')))

if not os.path.isabs(b_final_path):
    b_final_path = os.path.join(b_basedir, b_final_path)

if follow:
    b_final_path = os.path.realpath(b_final_path)

return to_text(os.path.normpath(b_final_path), errors='surrogate_or_strict')

  這個方法是用來解析檔案路徑的,通過這個方法,讓我們的'192.168.1.218',最終變成了'python/devops/192.168.1.218',此時,我們得到了報錯資訊提示源是誰了。但是我們傳入的是192.168.1.218,不想被解析為檔案,所以回到parse_sources方法,將if ',' not in source變為if ',' and '.' not in source:,跳過if分支,再次執行程式,報錯資訊變成了下面這樣:

[WARNING]: Unable to parse 192.168.1.218 as an inventory source
  成功了一半了,但是為何還會出現報錯資訊,得接著查

  回到parse_sources方法接著往下,source通過跳過unfrackpathparse的if分支後,被傳入self.parse_source(source, cache=cache)方法,該方法程式碼如下:

def parse_source(self, source, cache=False):
''' Generate or update inventory for the source provided '''

    parsed = False
    display.debug(u'Examining possible inventory source: %s' % source)

    # use binary for path functions
    b_source = to_bytes(source)

    # process directories as a collection of inventories
    if os.path.isdir(b_source):
        display.debug(u'Searching for inventory files in directory: %s' % source)
        for i in sorted(os.listdir(b_source)):

            display.debug(u'Considering %s' % i)
            # Skip hidden files and stuff we explicitly ignore
            if IGNORED.search(i):
                continue

            # recursively deal with directory entries
            fullpath = to_text(os.path.join(b_source, i), errors='surrogate_or_strict')
            parsed_this_one = self.parse_source(fullpath, cache=cache)
            display.debug(u'parsed %s as %s' % (fullpath, parsed_this_one))
            if not parsed:
                parsed = parsed_this_one
    else:
        # left with strings or files, let plugins figure it out

        # set so new hosts can use for inventory_file/dir vars
        self._inventory.current_source = source

        # try source with each plugin
        failures = []
        for plugin in self._fetch_inventory_plugins():

            plugin_name = to_text(getattr(plugin, '_load_name', getattr(plugin, '_original_path', '')))
            display.debug(u'Attempting to use plugin %s (%s)' % (plugin_name, plugin._original_path))

            # initialize and figure out if plugin wants to attempt parsing this file
            try:
                plugin_wants = bool(plugin.verify_file(source))
                print(plugin_wants,source)
            except Exception:
                plugin_wants = False

            if plugin_wants:
                try:
                    # FIXME in case plugin fails 1/2 way we have partial inventory
                    plugin.parse(self._inventory, self._loader, source, cache=cache)
                    try:
                        plugin.update_cache_if_changed()
                    except AttributeError:
                        # some plugins might not implement caching
                        pass
                    parsed = True
                    display.vvv('Parsed %s inventory source with %s plugin' % (source, plugin_name))
                    break
                except AnsibleParserError as e:
                    display.debug('%s was not parsable by %s' % (source, plugin_name))
                    tb = ''.join(traceback.format_tb(sys.exc_info()[2]))
                    failures.append({'src': source, 'plugin': plugin_name, 'exc': e, 'tb': tb})
                except Exception as e:
                    display.debug('%s failed while attempting to parse %s' % (plugin_name, source))
                    tb = ''.join(traceback.format_tb(sys.exc_info()[2]))
                    failures.append({'src': source, 'plugin': plugin_name, 'exc': AnsibleError(e), 'tb': tb})
            else:
                display.vvv("%s declined parsing %s as it did not pass its verify_file() method" % (plugin_name, source))
        else:
            if not parsed and failures:
                # only if no plugin processed files should we show errors.
                for fail in failures:
                    display.warning(u'\n* Failed to parse %s with %s plugin: %s' % (to_text(fail['src']), fail['plugin'], to_text(fail['exc'])))
                    if 'tb' in fail:
                        display.vvv(to_text(fail['tb']))
                if C.INVENTORY_ANY_UNPARSED_IS_FAILED:
                    raise AnsibleError(u'Completely failed to parse inventory source %s' % (source))
    if not parsed:
        if source != '/etc/ansible/hosts' or os.path.exists(source):
            # only warn if NOT using the default and if using it, only if the file is present
            display.warning("Unable to parse %s as an inventory source" % source)

    # clear up, jic
    self._inventory.current_source = None

    return parsed

  該方法比較長,但是可以看到裡面的報錯資訊出處,在這裡"Unable to parse %s as an inventory source",是因為if not parsed條件進來的,那麼parsed為何False,還得看這段程式碼動作。

  首先最開始parsed被置為了False,然後通過print大法得知if os.path.isdir(b_source)分支沒有進入,走的是後面else分支。通過程式碼各種變數命名規則,大概能夠猜到這裡面是在做inventory plugin的判斷。

  如果parsed被置為了False,那麼一定會報錯,為了避免就一定會在這段程式碼裡parsed會有置為True的地方,那麼plugin_wants = bool(plugin.verify_file(source))這一句就至關重要。

  通過plugin.verify_file(source)接收到引數source(192.168.1.218)後,parsed被置為了False,通過方法字面意思知道verify_file是校驗source是否為檔案的。我們檢視verify_file的原始碼,檔案位置為ansible/plugins/invenroy/host_list.py,程式碼如下:

def verify_file(self, host_list):

    valid = False
    b_path = to_bytes(host_list, errors='surrogate_or_strict')
    if not os.path.exists(b_path) and ',' in host_list:
        valid = True
    return valid

  果然,如果想讓返回結果為True的話,需要滿足if條件,即檔案不存在,並且 ',' 存在。

  這裡將if not os.path.exists(b_path) and ',' in host_list:這一句修改為if not os.path.exists(b_path):,執行程式,報錯沒了。

  再回頭捋一遍整個原始碼思路,當我們將IP地址傳入ansible連線初始化時,會校驗該IP地址是否為檔案,如果是檔案,那麼就解析檔案路徑,最終看該檔案內是否有目標機器。

  不為檔案,就校驗inventory plugin的各種外掛(預設開啟5個,其中包括host_list)。如果該IP符合外掛所需特徵,就由外掛來解析IP地址,如果不符合,最終丟擲報錯資訊Unable to parse %s as an inventory source。

  要想解析這個問題,有兩個思路:

  第一個,最簡單的就是傳入IP地址為一個的時候,末尾加上 ','

  第二個,修改原始碼兩個地方,一是跳過unfrackpath分支,二是讓host_list.py的verify_file方法為真。上文均有過修改程式碼的描述。

  

  好了,整個分析結束。我猜,ansible的意思是,既然初始化ansible連線需要傳入host的list,那麼單個IP地址不能稱為list,如果想成為list,就得加個 ',' ,哈哈