ubuntu14.04下利用pypi-server搭建本地的pip库

应实验室需要，自己动手实现了一个pip本地仓库。其实搭建起来很简单，重点在于对本地不存在包的处理。这里我设置的机制是当本地不存在所请求的包时继续维持连接，同时从阿里的pip镜像源中将该包下载到本地以供用户下载。于是，问题主要在于获取所缺失包的信息。

我的解决方法是：当pypi-server在本地找不到所请求包时跳转到主机的另一端口，在此端口上运行一个监听程序，从HTTP请求中获取所要下载的包，然后从阿里的pip镜像库中将该包下载到本地。

1.下载pypi-server

跳转下载

2.pypi-server的使用

pypi-server [option] packages_path

中间可以跟相关的参数，最后跟上本地存储路径

这里只说明一些我所用到的参数：

-p 指明pypi-server所在的端口

--disable-fallback 禁止在找不到包的时候跳转到pypi官网链接

--fallback-url [url]指明找不到包需要跳转的链接

如：pypi-server -p 8080 --fallback-url http://localhost:9000 ~/pypi-packages

pypi-server运行在主机的8080端口，若找不到所请求的包，则跳转到主机的9000端口，本地存储路径为~/pypi-packages

3.监听程序

前一个版本的在使用过程中遇到问题，即pip用户在找不到包时会不断的发送请求，于是本地仓库中便会有大量重复的包。

解决方法：下载包之前先判断当前目录下是否已经存在，存在则直接跳过.....

<span style="font-size:18px;">
</span>

<span style="font-size:18px;">#coding:utf-8
import threading
import socket
import re
import os
import urllib
import urllib.request as urllib2
encoding = 'utf-8'
BUFSIZE = 1024


# a read thread, read data from remote
class Reader(threading.Thread):
<span style="white-space:pre">	</span>def __init__(self, client):
<span style="white-space:pre">		</span>threading.Thread.__init__(self)
<span style="white-space:pre">		</span>self.client = client
<span style="white-space:pre">		</span>self.index_url="http://mirrors.aliyun.com/pypi/simple/"
<span style="white-space:pre">		</span>self.dir_url="http://mirrors.aliyun.com/pypi/packages/source/"
<span style="white-space:pre">		</span>self.pkgname=""
<span style="white-space:pre">		</span>self.pkgdir=[]
<span style="white-space:pre">		</span>self.pkglist=[]
<span style="white-space:pre">		</span>self.storepath="~/pypi/"
        
<span style="white-space:pre">	</span>def run(self):
<span style="white-space:pre">		</span>while True:
<span style="white-space:pre">			</span>f=open("listen.log","a+")
<span style="white-space:pre">			</span>data = self.client.recv(BUFSIZE)
<span style="white-space:pre">			</span>if(data):
<span style="white-space:pre">				</span>string = bytes.decode(data, encoding)
<span style="white-space:pre">				</span>f.writelines(string)
<span style="white-space:pre">				</span>self.pkgname=re.findall("GET /(.*?)/",string)
<span style="white-space:pre">				</span>self.pkgname="".join(self.pkgname)
<span style="white-space:pre">				</span>f.writelines("*******the pkgname: "+self.pkgname+"*******")
<span style="white-space:pre">				</span>self.getpkgdir()
<span style="white-space:pre">				</span>pkgdir=self.pkgdir
<span style="white-space:pre">				</span>for pkg in pkgdir:
<span style="white-space:pre">					</span>if os.path.exists(self.storepath+pkg[1]):
<span style="white-space:pre">						</span>print("pkg exists!")
<span style="white-space:pre">						</span>break
<span style="white-space:pre">					</span>else:
<span style="white-space:pre">						</span>self.download(self.dir_url+pkg[0])
<span style="white-space:pre">			</span>else:
<span style="white-space:pre">				</span>break
<span style="white-space:pre">		</span>peername=self.client.getpeername()
<span style="white-space:pre">		</span>f.writelines("close:"+str(peername))
<span style="white-space:pre">		</span>f.writelines("\n=============================================\n")
<span style="white-space:pre">		</span>f.close()


<span style="white-space:pre">	</span>def getpkgdir(self):


<span style="white-space:pre">		</span>pkg_url=self.index_url+self.pkgname
<span style="white-space:pre">		</span>request=urllib2.Request(pkg_url)
<span style="white-space:pre">		</span>response=urllib2.urlopen(request)
<span style="white-space:pre">		</span>data=response.read().decode("utf-8")
<span style="white-space:pre">		</span>self.pkgdir=re.findall('<a href=".*?source/(.*?)" rel="internal">(.*?)</a>',data)
<span style="white-space:pre">	</span>def download(self,url):
<span style="white-space:pre">		</span>os.system("wget -P %s %s"%(self.storepath,url))
    
<span style="white-space:pre">	</span>def readline(self):
<span style="white-space:pre">		</span>rec = self.inputs.readline()
<span style="white-space:pre">		</span>if rec:
<span style="white-space:pre">			</span>string = bytes.decode(rec, encoding)
<span style="white-space:pre">			</span>if len(string)>2:
<span style="white-space:pre">				</span>string = string[0:-2]
<span style="white-space:pre">			</span>else:
<span style="white-space:pre">				</span>string = ' '
<span style="white-space:pre">		</span>else:
<span style="white-space:pre">			</span>string = False
<span style="white-space:pre">		</span>return string


# a listen thread, listen remote connect
# when a remote machine request to connect, it will create a read thread to handle
class Listener(threading.Thread):
<span style="white-space:pre">	</span>def __init__(self, port):
<span style="white-space:pre">		</span>threading.Thread.__init__(self)
<span style="white-space:pre">		</span>self.port = port
<span style="white-space:pre">		</span>self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
<span style="white-space:pre">		</span>self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
<span style="white-space:pre">		</span>self.sock.bind(("0.0.0.0", port))
<span style="white-space:pre">		</span>self.sock.listen(0)
<span style="white-space:pre">	</span>def run(self):
<span style="white-space:pre">		</span>print("listener started")
<span style="white-space:pre">		</span>while True:
<span style="white-space:pre">			</span>client, cltadd = self.sock.accept()
<span style="white-space:pre">			</span>Reader(client).start()
<span style="white-space:pre">			</span>cltadd = cltadd
<span style="white-space:pre">			</span>print("accept a connect")
lst  = Listener(8080)   # create a listen thread
lst.start() # then start
</span>

<span style="font-size:18px;">
</span>

4.pip下载

pip install [packagename] -i http://localhost:8080/simple

利用上述命令则可以从本地pip库中下载所需的包

或者利用下述办法，可直接利用pip install [packagename]命令从本地源下载：

<span style="font-size:18px;"></span><pre name="code" class="plain"><span style="font-size:18px;">//在主文件夹下新建.pip文件夹</span>

<span style="font-size:18px;">mkdir ~/.pip</span>

<span style="font-size:18px;">
</span>

<span style="font-size:18px;">cd ~/.pip</span>

<span style="font-size:18px;"></span><pre name="code" class="plain"><span style="font-size:18px;">//在文件夹.pip中新建pip.conf文件</span>

vim pip.conf

<span style="font-size:18px;">
</span>

<span style="font-size:18px;">
</span>

<span style="font-size:18px;">//在pip.conf文件中添加以下内容
[global]
index-url = http://localhost:8080/simple
</span>

<span style="font-size:18px;">
</span>