在953版本使用gbfs加载时,我遇到了无法成功加载orc、avro文件的情况,经过试验及了解,发现是版本不对应导致的,目前gbfs还不支持这两个文件类型的加载,特此记录。
1、问题现象:
(1)avro文件加载报错: avro dataparser error - EOF reached
--执行语句及结果举例如下:
gbase> load data infile 'gbfs://10.10.1.1:6666/test/test1.avro' into table t1 data avro;
ERROR 1733 (HY000): (GBA-01EX-700) Gbase general error: Task 12 failed, [10.10.1.2:5050](GBA-02AD-0005)Failed to query in gnode:
DETAIL: (GBA-01EX-700) Gbase general error: avro dataparser error -EOF reached
(2)orc文件加载卡住,ctrl+c后宕机
--执行语句及结果举例如下:
gbase> load data infile 'gbfs://10.10.1.1:6666/test/test2.orc' into table t2 data orc;
^CQuery aborted by Ctrl+C
ERROR 2013 (HY000): Lost connection to GBase server during query
--ctrl+c前查看卡住的进程信息如下:
+----+-----+---------+----------+-----+-----+----------+-------+--------+------------------------+
|Id |Tid |User |Host |vc |db |Command |Time |State |Info |
+----+-----+---------+----------+-----+-----+----------+-------+--------+------------------------+
|696 |15199|root |localhost |vc1 |test |Query |445 |LOADING | load data infile 'gbfs://10.10.1.1:6666/test/test2.orc' into table t2 data orc |
+----+-----+---------+----------+-----+-----+----------+-------+--------+------------------------+
--此时查看该卡住进程的堆栈如下:
Thread 1(process 15199):
#0 0x00007feb8d6ee0e3 in epoll_wait() from /lib64/libc.so.6
#1 0x00000000019972ce in ??()
#2 0x000000000198af63 in BaseEventLoop()
#3 0x000000000198a9eb in event_base_dispatch()
#4 0x000000000108f688 in CEventAsyncSqlDispatcher::WaitAllQueryFinished()()
#5 0x0000000001189897 in CGCLoadStatement::ExecuteLoad(std::string&,SqlJiaoHuan*,std::string const&,std::string const&,bool)()
#6 0x0000000001191ada in CGCLoadStatement::ExecuteLoad(std::string&)()
#7 0x0000000000b44681 in GExecCommand(SessionCtx*)()
--ctrl+c后出现的宕机信息如下:
/opt/gcluster/serber/bin/gclusterd(GDayinStacktrace+0x27)[0x1a02fc7]
/opt/gcluster/serber/bin/gclusterd(handle_segfault+0x39f)[0x120f6df]
/lib64/libpthread.so.0(+0xf630)[0x7f09c7a6d630]
/lib64/libc.so.6(gsignal+0x37)[0x7f09c2f85387]
/lib64/libc.so.6(abort+0x148)[0x7f09c2f86a78]
/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x165)[0x7f09c367fa95]
/lib64/libstdc++.so.6(+0x5ea06)[0x7f09c367da06]
/lib64/libstdc++.so.6(+0x5ea33)[0x7f09c367da33]
/lib64/libstdc++.so.6(+0x5ec53)[0x7f09c367dc53]
/opt/gcluster/server/bin/gclusterd(_ZN22DataSessionSPtrWrapper15getEachSkipInfoEN5boost10shared_ptrIN3gns11DataSessionEEERiS5_+0x4c4)[0x12df7c4]
/opt/gcluster/server/bin/gclusterd(_ZN22DataSessionSPtrWrapper11getSkipInfoERiS0_+0x81)[0x12df9d1]
/opt/gcluster/server/bin/gclusterd(_ZN16CGCLoadStatement11ExecuteLoadERSsP11SqlJiaoHuanRKSsS4_S4_b+0x3fc0)[0x118acd0]
/opt/gcluster/server/bin/gclusterd(_ZN16CGCLoadStatement11ExecuteLoadERSs+0x9a)[0x1191ada]
/opt/gcluster/server/bin/gclusterd(_Z12GExecCommandP10SessionCtx+0x16231)[0xb44681]
/opt/gcluster/server/bin/gclusterd(_Z6GParseP10SessionCtxPKcjPS2_+0x2e8)[0xb52958]
/opt/gcluster/server/bin/gclusterd(_Z17DistributeCommand19enum_server_commandP10SessionCtxPcj+0x1331)[0xb54121]
/opt/gcluster/server/bin/gclusterd(_Z11ExecCommandP10SessionCtx+0x150)[0xb55140]
/opt/gcluster/server/bin/gclusterd(ChuliOneLianjie+0x221)[0xb0f6c1]
/lib64/libpthread.so.0(+0x7ea5)[0x7f09c7a65ea5]
/lib64/libc.so.6(clone+0x6d)[0x7f09c304db0d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
2、问题分析:
经过了解,发现了以上问题的引发原因:是gbfs版本与集群版本不同步导致,953.22的gbfs里没有新文件类型orc和avro的相应处理内容,会导致后续的报错、卡住、宕机情况。
3、解决或绕行方案:
考虑未来使用适配的能支持这些类型文件加载的gbfs版本。
由于当前无适用版本,又需要加载orc和avro文件,建议换成ftp/sftp/http加载,避免延误加载使用。