nifi mysql to mysql_NiFi 实战

最新推荐文章于 2024-08-07 10:09:04 发布

咪爷

最新推荐文章于 2024-08-07 10:09:04 发布

阅读量724

点赞数

文章标签： nifi mysql to mysql

本文链接：https://blog.csdn.net/weixin_28235513/article/details/113468793

版权

本文演示了如何使用NiFi将大量CSV文件从本地文件系统批量上传到HDFS，然后转换并导入到MySQL数据库。内容涵盖GetFile、PutHDFS和LogAttribute处理器的配置，以及内存溢出问题的解决策略。

摘要由CSDN通过智能技术生成

Demo 1 local to HDFS

1 场景：文件批量上传HDFS

2 数据源

local文件系统中, /opt/nifiData/raw下批量文件：

root@machine001:/opt/nifiData/raw# cd /opt/nifiData/raw

root@machine001:/opt/nifiData/raw# ls

machine000_metric000.csv machine001_metric005.csv machine003_metric000.csv machine004_metric005.csv machine006_metric000.csv machine007_metric005.csv machine009_metric000.csv

machine000_metric001.csv machine001_metric006.csv machine003_metric001.csv machine004_metric006.csv machine006_metric001.csv machine007_metric006.csv machine009_metric001.csv

machine000_metric002.csv machine001_metric007.csv machine003_metric002.csv machine004_metric007.csv machine006_metric002.csv machine007_metric007.csv machine009_metric002.csv

machine000_metric003.csv machine001_metric008.csv machine003_metric003.csv machine004_metric008.csv machine006_metric003.csv machine007_metric008.csv machine009_metric003.csv

machine000_metric004.csv machine001_metric009.csv machine003_metric004.csv machine004_metric009.csv machine006_metric004.csv machine007_metric009.csv machine009_metric004.csv

machine000_metric005.csv machine002_metric000.csv machine003_metric005.csv machine005_metric000.csv machine006_metric005.csv machine008_metric000.csv machine009_metric005.csv

machine000_metric006.csv machine002_metric001.csv machine003_metric006.csv machine005_metric001.csv machine006_metric006.csv machine008_metric001.csv machine009_metric006.csv

machine000_metric007.csv machine002_metric002.csv machine003_metric007.csv machine005_metric002.csv machine006_metric007.csv machine008_metric002.csv machine009_metric007.csv

machine000_metric008.csv machine002_metric003.csv machine003_metric008.csv machine005_metric003.csv machine006_metric008.csv machine008_metric003.csv machine009_metric008.csv

machine000_metric009.csv machine002_metric004.csv machine003_metric009.csv machine005_metric004.csv machine006_metric009.csv machine008_metric004.csv machine009_metric009.csv

machine001_metric000.csv machine002_metric005.csv machine004_metric000.csv machine005_metric005.csv machine007_metric000.csv machine008_metric005.csv

machine001_metric001.csv machine002_metric006.csv machine004_metric001.csv machine005_metric006.csv machine007_metric001.csv machine008_metric006.csv

machine001_metric002.csv machine002_metric007.csv machine004_metric002.csv machine005_metric007.csv machine007_metric002.csv machine008_metric007.csv

machine001_metric003.csv machine002_metric008.csv machine004_metric003.csv machine005_metric008.csv machine007_metric003.csv machine008_metric008.csv

machine001_metric004.csv machine002_metric009.csv machine004_metric004.csv machine005_metric009.csv machine007_metric004.csv machine008_metric009.csv

3 Processes选择

3.1 GetFile

3.1.1 Properties