1. 程式人生 > >pip2pi和pypiserver及Apache在pip本地源配置中的應用實踐

pip2pi和pypiserver及Apache在pip本地源配置中的應用實踐

pip介紹:
pip是python比較流行和強大的包管理工具,pip即將替代setuptools(setuptools也可以看成easy_install的替代詞,安裝setuptools之後
將使用easy_install命令列來安裝包),但是,pip依然是建立在setuptools基礎之上的,安裝pip之前必須安裝setuptools。

pip有強大:
Usage:   
  pip [options]

Commands:
 install                    Install packages.
 uninstall                  Uninstall packages.
 freeze                     Output installed packages in requirements format.
 list                       List installed packages.
 show                       Show information about installed packages.
 search                     Search PyPI for packages.
 wheel                      Build wheels from your requirements.
 help                       Show help for commands.

General Options:
  -h,--help                 Show help.
 --isolated                 Run pip in an isolated mode, ignoring environment variables anduser configuration.
  -v,--verbose              Give more output. Option is additive, and can be used up to 3times.
  -V,--version              Show version and exit.
  -q,--quiet                Give less output.
  --log               Path to a verbose appending log.
  --proxy            Specify a proxy in the form [user:
[email protected]
]proxy.server:port.
  --retries        Maximum number of retries each connection should attempt (default 5times).
  --timeout            Set the socket timeout (default 15 seconds).
  --exists-action    Defaultaction when a path already exists: (s)witch, (i)gnore, (w)ipe,(b)ackup.
  --trusted-host   Mark this host as trusted,even though it does not have valid or any HTTPS.
  --cert              Path to alternate CA bundle.
  --client-cert       Path to SSL client certificate, a single file containing theprivate key and the certificate in PEM format.
  --cache-dir          Store the cache data in.
 --no-cache-dir             Disable the cache.
  --disable-pip-version-check
                             Don't periodically check PyPI to determine whether a new version ofpip is available for download. Implied with --no-index.

pip預設的源(index-pypi=http://pypi.python.org/simple),pip install 和pipserach命令預設都會到pip.python.org上搜索下載pypi包
pypi的一個強大功能就是:把應用全部的依賴包及其版本全部寫入一個requirements.txt的檔案中,然後通過pipinstall -r requirements.txt
依次按照requirements.txt中的順序安裝pypi包。pip在安裝過程中,預設的安裝檔案快取在HOME/.cache/pip目錄裡面,改目錄下回生產1,2,3....,a,b,c
的子目錄用以存放快取檔案。因此,如果想要再次重新安裝可以不必通過外網,通過如下命令:pip install pypi_nmae--src $HOME/.cache或者pip install -r requirements.txt --src$HOME/.cache
既可以通過本地cache目錄進行安裝。

[
[email protected]
http]# pwd
/root/.cache/pip/http
[[email protected] http]# ll
total 0
drwx------.  9 root root 62 Jul 27 16:22 0
drwx------.  6 root root 38 Jul 27 16:23 1
drwx------. 13 root root 94 Jul 27 16:23 2
drwx------. 11 root root 78 Jul 27 16:22 3
drwx------. 13 root root 94 Jul 27 16:22 4
drwx------. 10 root root 70 Jul 27 16:23 5
drwx------. 12 root root 86 Jul 27 16:22 6
drwx------. 11 root root 78 Jul 27 16:22 7
drwx------.  9 root root 62 Jul 27 16:22 8
drwx------. 13 root root 94 Jul 27 16:23 9
drwx------. 12 root root 86 Jul 27 16:21 a
drwx------. 13 root root 94 Jul 27 16:23 b
drwx------. 11 root root 78 Jul 27 16:23 c
drwx------. 11 root root 78 Jul 27 16:23 d
drwx------.  9 root root 62 Jul 27 16:23 e
drwx------. 11 root root 78 Jul 27 16:17 f

pip本地源的搭建:


pip本地源搭建的思想就是:先下載pypi包,然後指定pip本地源
此處介紹兩種方法:
1:新建一個目錄用以存放pypi包,如/root/pypi
2.將所有依賴合併到一個檔案中:如 find . -name requirements.txt -exec cat {} \;>pip_requirement.all
3.開始下載pypi包到指定目錄:
#!/bin/bash

PIP_REQUIRE="pip_requirement.all"

while read LINE
do
  if [[ $LINE =~ ^[a-zA-Z] ]]
  then
    echo$LINE
    pip install$LINE -d /root/pypi  #僅下載不安裝
  fi
done < $PIP_REQUIRE
此步驟將會把pip_requirement.all中指定的全部依賴包下載到/root/pypi目錄中。在下載過程中,預設回到官網pypi.python.org
去下載,GFW導致下載較慢,可以採用國內豆瓣源來下載,非常快相對,具體方法:
編輯 .pip/pip.conf檔案,加入如下語句:
[global]
trusted-host = pypi.douban.com
index-url = http://pypi.douban.com/simple

或者採用阿里雲源:
[global]
trusted-host=mirrors.aliyun.com
index-url=http://mirrors.aliyun.com/pypi/simple/

語句trusted-host=xxx是因為pip新版本為認為http未不信任主機,不讓你連線。

4.更簡單的批量pypi包下載方式是安裝pip2pi工具:pip install pip2pi
下載單個包:pip2tgz /save/to/path pypi_nmae
批量下載:pip2tgz /save/to/path -r requirements.txt
採用此方式將會把requirements.txt裡面全部pypi包下載到指定的目錄(如果採用Apache部署pip源,推薦獎此路徑設為Apache的預設主目錄:
/var/www/html/pypi,如pip2tgz /var/www/html/pypi -r/root/openstack/keystone/requirements.txt)

5.生成pypi的index:dir2pi --normalize-package-names/var/www/html/pypi.命令將會在pypi裡面生成simple檔案,simple中的檔名是標準化後的
packeage名:
[[email protected] pypi]# cd /var/www/html/pypi/simple
[[email protected] simple]# ll
total 8
drwxr-xr-x. 2 root root   62 Jul28 17:53 aioeventlet
drwxr-xr-x. 2 root root   50 Jul28 17:53 alembic
drwxr-xr-x. 2 root root   57 Jul28 17:53 amqp
drwxr-xr-x. 2 root root   50 Jul28 17:53 anyjson
drwxr-xr-x. 2 root root   64 Jul28 17:53 appdirs
drwxr-xr-x. 2 root root   65 Jul28 17:53 argparse
drwxr-xr-x. 2 root root   46 Jul28 17:53 babel
drwxr-xr-x. 2 root root   60 Jul28 17:53 bandit
drwxr-xr-x. 2 root root   60 Jul28 17:53 bashate
drwxr-xr-x. 2 root root   67 Jul28 17:53 beautifulsoup4
drwxr-xr-x. 2 root root   67 Jul28 17:53 cachetools
drwxr-xr-x. 2 root root   47 Jul28 17:53 cffi
drwxr-xr-x. 2 root root   54 Jul28 17:53 contextlib2
simple目錄中的每一個子目錄都將生成對應的index.html檔案,檔案內容如:WebOb-1.4.1.tar.gz


6.配置Apache伺服器。
Apache伺服器主要是配置/etc/httpd/conf/httpd.conf檔案,就本文情況,不要修改預設的DocumentRoot"/var/www/html"
directory欄位新增 Options Indexes MultiViews FollowSymlinks
Indexes:指明在沒有index.html的情況下允許列出檔案和目錄;
MultiViews:允許根據內容情況多種檢視列出
FollowSymlinks:允許符號連線

把deny的地方改成 :Require all granted

重啟Apache:systemctl restart httpd.service
檢視Apache監聽情況:netstat -ntl |grep 80
瀏覽器輸入:http://host_ip/pypi/simple ,頁面將以超連結的形式顯示全部pypi包:





任意點選即可下載!

7.設定pip本地源。
可以使用如下形式安裝包:pip install --index-url http://host_ip/pypi/simple -rrequirements.txt
不過比較麻煩,可以修改~/.pip/pip.conf來指定預設本地pip源

[global]
trusted-host = host_ip
index-url = http://host_ip/pypi/simple

如此,即可以採用本地pip源安裝:pip install -r requirements.txt
將會發現速度非常快,比豆瓣源要快!此外,如果需要批量下載pypi包,如下:pip uninstall -rrequirements.txt -y
-y 引數表示無需確認即解除安裝。

另外一種搭建pip本地源的方式是採用pypiserver包來搭建:pip install pypiserver
pypi-server -h
pypi-server [OPTIONS] [PACKAGES_DIRECTORY...]
  start PyPI compatible package server servingpackages from
  PACKAGES_DIRECTORY. If PACKAGES_DIRECTORY is notgiven on the
  command line, it uses the default~/packages.  pypiserver scans this
  directory recursively for packages. It skipspackages and
  directories starting with a dot. Multiplepackage directories can be
  specified.

pypi-server understands the following options:

  -p, --port PORT
    listen onport PORT (default: 8080)

  -i, --interface INTERFACE
    listen oninterface INTERFACE (default: 0.0.0.0, any interface)

  -a, --authenticate (UPDATE|download|list),...
   comma-separated list of (case-insensitive) actions toauthenticate
    (requiresgiving also the -P option). For example to password-protect
    packageuploads & downloads while leaving listings public, give:
     -a update,download.
    By default,only 'update' is password-protected.

  -P, --passwords PASSWORD_FILE
    use apachehtpasswd file PASSWORD_FILE to set usernames & passwords
    used forauthentication of certain actions (see -a option).

  --disable-fallback
    disableredirect to real PyPI index for packages not found in the
    localindex

  --fallback-url FALLBACK_URL
    for packagesnot found in the local index, this URL will be used to
    redirect to(default: http://pypi.python.org/simple)

  --server METHOD
    use METHODto run the server. Valid values include paste,
    cherrypy,twisted, gunicorn, gevent, wsgiref, auto. The
    default isto use "auto" which chooses one of paste, cherrypy,
    twisted orwsgiref.

  -r, --root PACKAGES_DIRECTORY
    [deprecated]serve packages from PACKAGES_DIRECTORY

  -o, --overwrite
    allowoverwriting existing package files

  --welcome HTML_FILE
    uses theASCII contents of HTML_FILE as welcome message response.

  -v
    enableverbose logging;  repeate for moreverbosity.

  --log-file
    writelogging info into this FILE.

  --log-frmt
    the loggingformat-string.  (see `logging.LogRecord` classfrom standard python library)
    [Default:%(asctime)s|%(levelname)s|%(thread)d|%(message)s]

  --log-req-frmt FORMAT
    aformat-string selecting Http-Request properties to log; setto  '%s' to see them all.
    [Default:%(bottle.request)s]
    
  --log-res-frmt FORMAT
    aformat-string selecting Http-Response properties to log; setto  '%s' to see them all.
    [Default:%(status)s]

  --log-err-frmt FORMAT
    aformat-string selecting Http-Error properties to log; setto  '%s' to see them all.
    [Default:%(body)s: %(exception)s
%(traceback)s]

  --cache-control AGE
    Add"Cache-Control: max-age=AGE, public" header to packagedownloads.
    Pip 6+ needsthis for caching.


pypi-server -h
pypi-server --help
  show this help message

pypi-server --version
  show pypi-server's version

pypi-server -U [OPTIONS] [PACKAGES_DIRECTORY...]
  update packages in PACKAGES_DIRECTORY. Thiscommand searches
  pypi.python.org for updates and shows a pipcommand line which
  updates the package.

The following additional options can be specified with -U:

  -x
    execute thepip commands instead of only showing them

  -d DOWNLOAD_DIRECTORY
    downloadpackage updates to this directory. The default is to use
    thedirectory which contains the latest version of the package to
    beupdated.

  -u
    allowupdating to unstable version (alpha, beta, rc, dev versions)

Visit https://pypi.python.org/pypi/pypiserver for moreinformation.

pypi-server的配置相對簡單,不需要再去配置apache服務。
總結起來,pip本地源的配置大概兩種方式:
1.pip2pi + apache
2.pip2pi + pypiserver

第一張方式即是利用pip2pi的pip2tgz -r requirements.txt命令批量下載python包,然後利用dir2pipackges_dir 生成index,然後配置
Apache伺服器,通過http訪問本地repos來搭建本地源;

第二章方式在下載python包部分也是利用pip2pi命令來下載,然後將存放python包的目錄作為pypiserver的包倉庫,通過啟動pypiserver服務來
監聽埠:pypi-server -i host_ip -p port packges_dir1 packges_dir12 ...packges_dir1n &
通過netstat -ntl可以檢視埠監聽狀態,通過lsof -i:port 可以看到監聽埠的應用程式名稱和程序ID

pypi-server不需要使用dir2pi生成index,只需要將需要的python包下載到本地,然後啟動pypiserver監聽即可!

pypi-server可以使用--fall-back 引數指定在找不到需要的包時,pip到什麼地方去搜索包,如:
pypi-server -i 192.168.142.10 -p 8080 --fallback-urlhttp://192.168.142.10/pypi/simple/ pypi &

pypiserver監聽192.168.142.10:8080埠,同時使用url=http://192.168.142.10/pypi/simple/作為候選index-url(此列為前面配置的
Apacheindex),如果在~/pypi目錄中找不到需要的python包,則到http://192.168.142.10/pypi/simple中去找。

啟動pypiserver監聽之後,為了便於安裝,對~.pip/pip.conf進行配置:
[global]

##############douban mirrors###############
#trusted-host=pypi.douban.com
#index-url = http://pypi.douban.com/simple

##############pip2pi&apache local repos##############
#trusted-host=192.168.142.10
#index-url = http://192.168.142.10/pypi/simple

################pypiserver local repos################
trusted-host=192.168.142.10
index-url = http://192.168.142.10:8080/simple/


完成之後,即可pip install -r requirements.txt 進行安裝!

注:如果要停止pypiserver,需要lsof -i:8080 檢視到程序ID ,然後kill PID!