python pandas read_csv 迭代器使用方法_使用Python读取Dropbox共享的csv文件的内容

I am new to Dropbox API. A colleague has shared a dropbox folder containing large (various sizes between 450 MB and 17 GB) csv files.

I want to read the files on Dropbox without downloading them. I tried many things including reading the Dropbox API tutorials and documentation as well as referring to Stackoverflow post.

My question is: What is a good way to read csv files shared by somebody else and what path I need to use to read them with pandas.read_csv() if size allows me or line by line if size is too large.

Thanks in advance.

解决方案

I didn't try it, but ...

pandas.read_csv(filepath_or_buffer, ...

filepath_or_buffer : str, pathlib.Path, py._path.local.LocalPath or any object with a read() method (such as a file handle or StringIO)

get_file(from_path,...

Returns

A dropbox.rest.RESTResponse

RESTResponse

Responses to requests can come in the form of RESTResponse. These are thin wrappers around the socket file descriptor. read() and close() are implemented.

Then, should be as easy as:

with client.get_file('/magnum-opus.txt') as f:

p=pandas.read_csv(f)

Where pandas.read_csv calls read method on f ( dropbox file handler )

Edited thanks @greg

If you are downloading file with Dropbox V2 API then you may check if raw property of the request response is on stream mode and has read method.:

_, res = dbx.files_download(path)

p=pandas.read_csv(res.raw)

sorry, I didn't try it, just try and come back. If not then you must use iterators and hand chunks. More detail at Raw Response Content