kaggle api下载时出现网络链接问题的解决方案

kaggle api下载时出现网络链接问题的解决方案

问题描述

通过kaggle api下载数据时，可能会产生下面的错误。我是在使用服务器的时候，遇到这个问题的。服务器是通过代理连接网络的，代理地址为http://172.16.17.164:3128，在kaggle api安装的时候，代理地址有可能是没有设置的，因此在调用kaggle competitions download -c hubmap-kidney-segmentation下载数据的时候，就会报网络相关的错误。

kaggle competitions download -c hubmap-kidney-segmentation


2021-04-09 13:39:14,850 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.Ver$
fiedHTTPSConnection object at 0x7f0564096910>: Failed to establish a new connection: [Errno 110] Connection timed out',)': /api/v1/competitions/data/list/hubmap-kidney-segmentation

^CTraceback (most recent call last):
  File "/usr/local/bin/kaggle", line 9, in <module>
    load_entry_point('kaggle==1.1.0', 'console_scripts', 'kaggle')()
  File "/usr/local/lib/python2.7/dist-packages/kaggle/cli.py", line 18, in main
    out = args.func(**command_args)
  File "/usr/local/lib/python2.7/dist-packages/kaggle/api/kaggle_api_extended.py", line 246, in competitionDownloadCli
    self.competitionDownloadFiles(competition, path, force, quiet)
  File "/usr/local/lib/python2.7/dist-packages/kaggle/api/kaggle_api_extended.py", line 231, in competitionDownloadFiles
    files = self.competitionListFiles(competition)
  File "/usr/local/lib/python2.7/dist-packages/kaggle/api/kaggle_api_extended.py", line 192, in competitionListFiles
    competitionListFilesResult = self.process_response(self.competitions_data_list_files_with_http_info(id = competition))
  File "/usr/local/lib/python2.7/dist-packages/kaggle/api/kaggle_api.py", line 224, in competitions_data_list_files_with_http_info
    collection_formats=collection_formats)
  File "/usr/local/lib/python2.7/dist-packages/kaggle/api_client.py", line 318, in call_api
    _preload_content, _request_timeout)
  File "/usr/local/lib/python2.7/dist-packages/kaggle/api_client.py", line 149, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/local/lib/python2.7/dist-packages/kaggle/api_client.py", line 339, in request
    headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/kaggle/rest.py", line 235, in GET
    query_params=query_params)
  File "/usr/local/lib/python2.7/dist-packages/kaggle/rest.py", line 208, in request
    headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/request.py", line 66, in request
    **urlopen_kw)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/request.py", line 87, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/poolmanager.py", line 321, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 668, in urlopen
    **response_kw)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connection.py", line 284, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python2.7/dist-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python2.7/dist-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
  File "/usr/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)

解决方案

参考链接：kaggle api下载错误解决

I have the same issue so I tried to change the value of my proxy using this command line:
kaggle config set -n proxy -v [server] [1]
but now it says
The specified proxy blablabla is not valid, please check your proxy settings
so I tried to unset it using command line
kaggle config unset -n proxy
and it keeps saying [1]

在我们的case中只需执行如下的命令即可：

kaggle config set -n proxy -v

kaggle config set -n proxy -v http://172.16.17.164:3128

效果

在执行完上述的命令之后，再通过kaggle api进行数据的下载，效果如下：

kaggle competitions download -c hubmap-kidney-segmentation
Warning: Looks like you're using an outdated API Version, please consider updating (server 1.5.12 / client 1.1.0)
4ef6695ce-anatomical-structure.json: Downloaded 2KB of 2KB
1e2425f28.json: Downloaded 763KB of 763KB
26dc41664.json: Downloaded 844KB of 844KB
54f2eec69.json: Downloaded 275KB of 275KB
26dc41664.tiff.zip: Downloaded 2GB of 2GB2GB
0486052bb-anatomical-structure.json: Downloaded 4KB of 4KB
0486052bb.tiff.zip: Downloaded 239MB of 239MB
26dc41664-anatomical-structure.json: Downloaded 4KB of 4KB
095bf7a1f.json.zip: Downloaded 205KB of 205KB
2f6ecfcdf-anatomical-structure.json: Downloaded 3KB of 3KB
095bf7a1f-anatomical-structure.json: Downloaded 944B of 944B
1e2425f28.tiff.zip: Downloaded 2GB of 2GB2GB
1e2425f28-anatomical-structure.json: Downloaded 3KB of 3KB
095bf7a1f.tiff.zip: Downloaded 2GB of 2GB2GB
54f2eec69-anatomical-structure.json: Downloaded 2KB of 2KB
0486052bb.json: Downloaded 271KB of 271KB
2f6ecfcdf.tiff.zip: Downloaded 203MB of 203MB
4ef6695ce.tiff.zip: Downloaded 4GB of 4GB4GB
4ef6695ce.json.zip: Downloaded 428KB of 428KB
2f6ecfcdf.json: Downloaded 493KB of 493KB
d488c759a-anatomical-structure.json: Downloaded 4KB of 4KB
3589adb90.tiff.zip: Downloaded 170MB of 170MB
2ec3f1bb9-anatomical-structure.json: Downloaded 3KB of 3KB
sample_submission.csv: Downloaded 68B of 68B
aa05346ff-anatomical-structure.json: Downloaded 3KB of 3KB
d488c759a.tiff.zip: Downloaded 2GB of 2GB2GB
57512b7f1.tiff.zip: Downloaded 2GB of 2GB2GB
aa05346ff.tiff.zip: Downloaded 3GB of 3GB3GB
2ec3f1bb9.tiff.zip: Downloaded 450MB of 450MB
57512b7f1-anatomical-structure.json: Downloaded 3KB of 3KB
3589adb90-anatomical-structure.json: Downloaded 2KB of 2KB
train.csv.zip: Downloaded 7MB of 7MB
sample_submission.csv: Skipping, found more recently modified local copy (use --force to force download)
2021-04-09 14:07:33,874 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, u'EOF occurred in vi
olation of protocol (_ssl.c:590)'),)': /kagglesdsdata/datasets/1136396/2008726/HuBMAP-20-dataset_information.csv?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=
1618207213&Signature=RxWMkqKQjuEA8ZVthn2uwgiGXN1%2BN4DuY6%2BEri1NE5kKB4xLncwl7fgcYTe1ceYLqCgULuu1moN7XBZ%2FC84vL1n7jnwSnTYAjpKfSc9BmNKM4W18SdGWXKB6DrO9RR8jtZki0T3AbKyQkuW%2BvdECmdN9ts
AsT2rQT63jZfgBZsFsvcx%2FIhQnyTTHJpe3CDlwA2XsXDFb4VcJrq2zaOCbqPpTR3RcwJwR4DsC335IEKhRfj6%2FUUKLTGTAiPMPEEW%2Bk3CTgEhU8PiePNkUj6G%2FnUauFValCXdvPHVvS%2FuJfSbiWDU9lBAmtqV36fiXFq%2F06%2Bf
uuOc4Ci%2FklgEJMkedcA%3D%3D&response-content-disposition=attachment%3B+filename%3DHuBMAP-20-dataset_information.csv
2021-04-09 14:14:54,706 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, u'EOF occurred in vi
olation of protocol (_ssl.c:590)'),)': /kagglesdsdata/datasets/1136396/2008726/HuBMAP-20-dataset_information.csv?GoogleAccessId=web-data@kaggle-161607.iam.gserviceaccount.com&Expires=
1618207213&Signature=RxWMkqKQjuEA8ZVthn2uwgiGXN1%2BN4DuY6%2BEri1NE5kKB4xLncwl7fgcYTe1ceYLqCgULuu1moN7XBZ%2FC84vL1n7jnwSnTYAjpKfSc9BmNKM4W18SdGWXKB6DrO9RR8jtZki0T3AbKyQkuW%2BvdECmdN9ts
AsT2rQT63jZfgBZsFsvcx%2FIhQnyTTHJpe3CDlwA2XsXDFb4VcJrq2zaOCbqPpTR3RcwJwR4DsC335IEKhRfj6%2FUUKLTGTAiPMPEEW%2Bk3CTgEhU8PiePNkUj6G%2FnUauFValCXdvPHVvS%2FuJfSbiWDU9lBAmtqV36fiXFq%2F06%2Bf
uuOc4Ci%2FklgEJMkedcA%3D%3D&response-content-disposition=attachment%3B+filename%3DHuBMAP-20-dataset_information.csv
HuBMAP-20-dataset_information.csv: Downloaded 3KB of 3KB