一、查询结果导出到hdfs需要特别注意的事项
查询结果导出语句示例:
create table t(a bigint,b varchar(20)) distributed by ('a');
insert into t values(10,'aaa'),(20,'bbb'),(30,'ccc'),(40,'ddd'),(50,'eee');
set gbase_hdfs_namenodes="ip1,ip2" ;
show variables like 'gbase_hdfs_namenodes';
select * from test.t into outfile 'hdp://hdfs@ip1:ip1_port/output/t.tbl' outfilemode by hdfs writemode by overwrites;
注意:如果导出时遇到报错:failed with error - Fail to Create HDFS File,先排查部署hadoop环境的节点Namenode和Datanode服务是否正常,若服务未启动则需要重新启动,如果服务显示正常,则需要排查集群各节点/etc/hosts中是否添加了Hadoop的Namenode和Datanode的IP地址和主机名映射,
例如:
[gbase@www-node-1 gcinstall]$ cat /etc/hosts
ip1 www-node-1
ip2 www-node-2
二、查询结果导出到hdfs曾遇到的问题
[hdfs@www-node-1 hadoop-3.1.0]$ jps
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000085c00000, 5767168, 0) failed; error='Cannot allocate memory' (errno=12)## There is insufficient memory for the Java Runtime Environment to continue.# Native memory allocation (mmap) failed to map 5767168 bytes for committing reserved memory.# An error report file with more information is saved as:# /home/hdfs/hadoop/hadoop-3.1.0/hs_err_pid21392.log
问题原因:jvm分配的内存不够
解决办法:修改配置文件参数
/home/hdfs/hadoop/hadoop-3.1.0/etc/hadoop/hadoop-env.sh
添加内容:
export HADOOP_HEAPSIZE_MAX=4G(根据具体情况进行设置即可)
然后,执行停启服务,使修改后的配置文件生效
cd ${HADOOP_HOME}
sbin/stop-dfs.sh
sbin/start-dfs.sh