odps dship客户端使用

最新推荐文章于 2022-03-01 20:50:18 发布

cuiba1921

最新推荐文章于 2022-03-01 20:50:18 发布

阅读量1.2k

点赞数

文章标签： java 运维操作系统

1、下载DShip插件
下载地址：http://www.aliyun.com/product/odps/最下面的“开发者资源”，“数据上传下载工具”，下载文件为odps-dship.zip。
odps-dship\dship.bat即为windows下的执行文件。
2、配置DShip
odps-dship\odps.conf为DShip的配置文件，（需要jdk1.6以上），配置文件原来是这样的：
#odps dship config
endpoint=http://service.odps.aliyun.com/api
id=
key=
project=
配置好项目名称，access ID，access Key。
正确配置请参照文档下面部分的odps.conf配置
3、打开客户端
注意：要到命令行cd到相应的目录下面，执行命令dfish可以看到dfish的帮助信息。
C:\Users\yangswa>cd C:\Users\yangswa\Desktop\odps-dship

C:\Users\yangswa\Desktop\odps-dship>dship
Usage: dship [options] [args]
Type 'dship help ' for help on a specific subcommand.
Type 'dship --version' to see the program version.

Available subcommands:
    upload (u)
    download (d)
    resume (r)
    show (s)
    config (c)
    purge (p)
    help (h)

Valid global options:
    -c   [--charset] ARG                 : file charset
    -dfp [--date-format-pattern] ARG     : date format pattern
    -e   [--endpoint] ARG                : specify endpoint
    -fd [--field-delimiter] ARG         : field delimiter
    -i   [--id] ARG                      : access id
    -k   [--key] ARG                     : access key
    -ni [--null-indicator] ARG          : null indicator string
    -p   [--project] ARG                 : default project
    -rd [--record-delimiter] ARG        : record delimiter
    -te [--tunnel-endpoint] ARG         : specify tunnel endpoint

dship is tool for odps data transfer.
4、在odps的yangsw_test项目中有一个dual表
表结构：
id，类型为bigint。
Name, 类型为string
Dt,类型为string，该字段为分区字段

odps:sql:yangsw_test> desc dual;
+------------------------------------------------------------------------------------+
| Table: dual                                                                        |
| Owner: ALIYUN$**************     | Project: yangsw_test                       |
| TableComment:                                                                      |
+------------------------------------------------------------------------------------+
| CreatedTime:              2014-11-11 10:14:25                                      |
| LastMetaModifiedTime:     2014-11-11 10:14:25                                      |
| LastDataModifiedTime:     1970-01-01 08:00:00                                      |
+------------------------------------------------------------------------------------+
| Type : Table                 | Size: 0 Bytes                                       |
+------------------------------------------------------------------------------------+
| Native Columns:                                                                    |
+------------------------------------------------------------------------------------+
| Field           | Type       | Comment                                             |
+------------------------------------------------------------------------------------+
| id              | BIGINT     |                                                     |
| name            | STRING     |                                                     |
+------------------------------------------------------------------------------------+
| Partition Columns:                                                                 |
+------------------------------------------------------------------------------------+
| dt              | STRING     |                                                     |
+------------------------------------------------------------------------------------+
我们给dual表加一个分区
odps:sql:yangsw_test> alter table dual add partition(dt='2014');
InstanceId: 20141111021906223gw514il
OK
5、准备数据文件
Dual_2014.txt
1,tom
2,jacky
3,jet
4,rose
5,marry
6,obama

6、用dship上传数据
C:\Users\yangswa\Desktop\odps-dship> dship upload -fd "," -rd "\r\n" C:\Users\yangswa\Desktop\dual_2014.txt dual/dt="2014";
ERROR: Unknown error -
405 Not Allowed

405 Not Allowed

nginx

查看日志文件odps-dship\sessions\.sidnull\log.txt
2014-11-11 11:10:56 - java.lang.Exception: ERROR: Unknown error -
405 Not Allowed

405 Not Allowed

nginx

at com.aliyun.odps.ship.DShip.parseCommandLine(DShip.java:101)
at com.aliyun.odps.ship.DShip.main(DShip.java:35)
Caused by: java.lang.RuntimeException:
405 Not Allowed

405 Not Allowed

nginx

at com.aliyun.odps.tunnel.TunnelException.(TunnelException.java:62)
at com.aliyun.odps.tunnel.TableTunnel$UploadSession.initiate(TableTunnel.java:462)
at com.aliyun.odps.tunnel.TableTunnel$UploadSession.(TableTunnel.java:402)
at com.aliyun.odps.tunnel.TableTunnel.createUploadSession(TableTunnel.java:95)
at com.aliyun.odps.ship.upload.TunnelUploadSession.(TunnelUploadSession.java:61)
at com.aliyun.odps.ship.DShip.parseCommandLine(DShip.java:62)
... 1 more
Caused by: Parse response failed
at com.aliyun.odps.tunnel.TunnelException.loadFromJson(TunnelException.java:138)
at com.aliyun.odps.tunnel.TunnelException.(TunnelException.java:60)
... 6 more
Caused by: org.codehaus.jackson.JsonParseException: Unexpected character ('
at [Source: java.io.ByteArrayInputStream@a5464c; line: 1, column: 2]
at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433)
at org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521)
at org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:442)
at org.codehaus.jackson.impl.Utf8StreamParser._handleUnexpectedValue(Utf8StreamParser.java:2090)
at org.codehaus.jackson.impl.Utf8StreamParser._nextTokenNotInObject(Utf8StreamParser.java:606)
at org.codehaus.jackson.impl.Utf8StreamParser.nextToken(Utf8StreamParser.java:492)
at org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2770)
at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2718)
at org.codehaus.jackson.map.ObjectMapper.readTree(ObjectMapper.java:1507)
at com.aliyun.odps.tunnel.TunnelException.loadFromJson(TunnelException.java:127)
... 7 more
难道是多个分号？？？？？
C:\Users\yangswa\Desktop\odps-dship>dship upload -fd "," -rd "\r\n" C:\Users\yangswa\Desktop\dual_2014.txt dual/dt="2014"
ERROR: TunnelException – null
查看日志文件odps-dship\sessions\.sidnull\log.txt

2014-11-11 14:41:38 - null
at com.aliyun.odps.tunnel.TableTunnel$UploadSession.initiate(TableTunnel.java:467)
at com.aliyun.odps.tunnel.TableTunnel$UploadSession.(TableTunnel.java:402)
at com.aliyun.odps.tunnel.TableTunnel.createUploadSession(TableTunnel.java:95)
at com.aliyun.odps.ship.upload.TunnelUploadSession.(TunnelUploadSession.java:61)
at com.aliyun.odps.ship.DShip.parseCommandLine(DShip.java:62)
at com.aliyun.odps.ship.DShip.main(DShip.java:35)
Caused by: java.net.UnknownHostException: null
at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at sun.net.NetworkClient.doConnect(Unknown Source)
at sun.net.www.http.HttpClient.openServer(Unknown Source)
at sun.net.www.http.HttpClient.openServer(Unknown Source)
at sun.net.www.http.HttpClient.(Unknown Source)
at sun.net.www.http.HttpClient.New(Unknown Source)
at sun.net.www.http.HttpClient.New(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.net.HttpURLConnection.getResponseCode(Unknown Source)
at com.aliyun.odps.commons.transport.DefaultConnection.getResponse(DefaultConnection.java:101)
at com.aliyun.odps.tunnel.TableTunnel$UploadSession.initiate(TableTunnel.java:457)
... 5 more
还是报错。。。。。。
http://bbs.aliyun.com/read/181537.html?spm=5176.7189909.0.0.YeZ2UV这位大神的配置起作用：
最终在windows环境下我得odps.conf改为下面就可以上传数据了（注意上面两行）
\u00EF\u00BB\u00BF\#odps=dship config
tunnel-endpoint=http\://dt.odps.aliyun.com
id=*******
key=********
project=yangsw_test
最终的执行结果如下：
C:\Users\yangswa\Desktop\odps-dship>dship upload -fd "," -rd "\r\n" C:\Users\yangswa\Desktop\dual_2014.txt yangsw_test.dual/dt="2014"
Upload session: 201411111456542581870a0024a407
2014-11-11 14:56:25     scanning file: 'dual_2014.txt'
2014-11-11 14:56:26     uploading file: 'dual_2014.txt'
2014-11-11 14:56:28     'dual_2014.txt' uploaded
OK

7、用odps sql查一下dual表的数据
odps:sql:yangsw_test> select * from dual;
InstanceId: 20141111070328172gre5pdx5

+------------+------+----+
| id         | name | dt |
+------------+------+----+
| 1          | tom | 2014 |
| 2          | jacky | 2014 |
| 3          | jet | 2014 |
| 4          | rose | 2014 |
| 5          | marry | 2014 |
| 6          | obama | 2014 |

至此，使用dship上传数据成功了。