MapD表格导出测试

\gpu

1.1万行数据测试：

COPY (SELECT * FROM flights_2008_10k) TO '/mapd-core/build/sample_datasets/flights_2008_new/flights_2008_new.csv';

验证：

create table test (
..> flight_year smallint,
..> flight_month smallint,
..> flight_dayofmonth smallint,
..> flight_dayofweek smallint,
..> deptime smallint,
..> crsdeptime smallint,
..> arrtime smallint,
..> crsarrtime smallint,
..> uniquecarrier text encoding dict,
..> flightnum smallint,
..> tailnum text encoding dict,
..> actualelapsedtime smallint,
..> crselapsedtime smallint,
..> airtime smallint,
..> arrdelay smallint,
..> depdelay smallint,
..> origin text encoding dict,
..> dest text encoding dict,
..> distance smallint,
..> taxiin smallint,
..> taxiout smallint,
..> cancelled smallint,
..> cancellationcode text encoding dict,
..> diverted smallint,
..> carrierdelay smallint,
..> weatherdelay smallint,
..> nasdelay smallint,
..> securitydelay smallint,
..> lateaircraftdelay smallint,
..> dep_timestamp timestamp(0),
..> arr_timestamp timestamp(0),
..> carrier_name text encoding dict,
..> plane_type text encoding dict,
..> plane_manufacturer text encoding dict,
..> plane_issue_date date,
..> plane_model text encoding dict,
..> plane_status text encoding dict,
..> plane_aircraft_type text encoding dict,
..> plane_engine_type text encoding dict,
..> plane_year smallint,
..> origin_name text encoding dict,
..> origin_city text encoding dict,
..> origin_state text encoding dict,
..> origin_country text encoding dict,
..> origin_lat real,
..> origin_lon real,
..> dest_name text encoding dict,
..> dest_city text encoding dict,
..> dest_state text encoding dict,
..> dest_country text encoding dict,
..> dest_lat real,
..> dest_lon real,
..> origin_merc_x real,
..> origin_merc_y real,
..> dest_merc_x real,
..> dest_merc_y real
..> );

copy test from '/mapd-core/build/sample_datasets/flights_2008_new/flights_2008_new.csv' with (quoted='true');
Result
Loaded: 10000 recs, Rejected: 0 recs in 28.290000 secs

2.20万行数据测试：

COPY (SELECT * FROM flights_2008_200k) TO '/mapd-core/build/sample_datasets/flights_2008_new/flights_2008_new1.csv';

验证：

copy test from '/mapd-core/build/sample_datasets/flights_2008_new/flights_2008_new1.csv' with (quoted='true');
Result
Loaded: 200000 recs, Rejected: 0 recs in 1.924000 secs

3.700万行数据测试：

COPY (SELECT * FROM flights_2008_7M) TO '/mapd-core/build/sample_datasets/flights_2008_new/flights_2008_new2.csv';

报错：GPU内存不足：E0628 10:31:13.320468 30484 MapDHandler.cpp:2384] Exception: Query couldn't keep the entire working set of columns in GPU memory
Exception: Query couldn't keep the entire working set of columns in GPU memor

\cpu

1.直接测700万行数据：

COPY (SELECT * FROM flights_2008_7M) TO '/mapd-core/build/sample_datasets/flights_2008_new/flights_2008_new2.csv';（花费时间：40-60分钟）

验证：

copy test from '/mapd-core/build/sample_datasets/flights_2008_new/flights_2008_new2.csv' with (quoted='true');
Result
Loaded: 7009728 recs, Rejected: 0 recs in 60.079000 secs

2.测2800万行数据：

COPY (SELECT * FROM flights_2008_28M) TO '/mapd-core/build/sample_datasets/flights_2008_new/flights_2008_new3.csv';（花费时间：259分钟）

（%cpu:100   %mem:16）

验证：

copy test from '/mapd-core/build/sample_datasets/flights_2008_new/flights_2008_new3.csv' with (quoted='true');

Result
Loaded: 28248912 recs, Rejected: 0 recs in 344.980000 secs

（%cpu:3000-4000   %mem:6）

3..测试1亿行：

COPY (SELECT * FROM flights_2008_100M) TO '/mapd-core/build/sample_datasets/flights_2008_new/flights_2008_new3.csv';

报错：Thrift: Wed Jun 28 12:38:38 2017 TSocket::write_partial() send() <Host: localhost Port: 9091>Broken pipe
Thrift: Wed Jun 28 12:38:38 2017 TSocket::open() connect() <Host: localhost Port: 9091>Connection refused
Thrift: Wed Jun 28 12:38:38 2017 TSocket::open() connect() <Host: localhost Port: 9091>Connection refused
Thrift error: connect() failed: Connection refused

第二次尝试：

Thrift: Wed Jun 28 13:38:09 2017 TSocket::write_partial() send() <Host: localhost Port: 9091>Broken pipe
Thrift: Wed Jun 28 13:38:09 2017 TSocket::open() connect() <Host: localhost Port: 9091>Connection refused
Thrift: Wed Jun 28 13:38:09 2017 TSocket::open() connect() <Host: localhost Port: 9091>Connection refused
Thrift error: connect() failed: Connection refused

 

转载于:https://www.cnblogs.com/laozhuang/p/7089340.html