nifi apache
介绍 (Introduction)
In this article, I’ll present the problem with having large amount of data processed in our NiFi data flows regularly — while being bounded by IOPS. Since NiFi is IOPS intensive, this issue can become even worse when running over containers and using NFS. Moreover, we’ll talk about keeping our data flows monitored, without increasing dramatically the IOPS. We’ll design a new architecture for NiFi clusters and monitoring — running NiFi with its repositories over RAM (and by that also boost our data flows performance), while still being able to monitor it, with having any kind of IOPS bottleneck.
在本文中，我将介绍在NiFi数据流中定期处理大量数据的问题-同时受IOPS限制。 由于NiFi是IOPS密集型设备，因此在容器上运行并使用NFS时，此问题会变得更加严重。 此外，我们将讨论在不大幅提高IOPS的情况下保持对数据流的监控。 我们将为NiFi群集和监视设计一种新的体系结构—在RAM上通过其存储库运行NiFi(这也将提高我们的数据流性能)，同时仍然能够监视它，并具有任何IOPS瓶颈。
监控NiFi (Monitoring NiFi)
Apache NiFi is an easy to use and powerful system to process and distribute data. Using the large variety of already provided processors (and the ability to create new ones as we wish), we can build powerful data flows for extracting data from sources, transform it as we wish, and push it to a different endpoint.
Apache NiFi是一个易于使用且功能强大的系统，用于处理和分发数据。 使用各种已经提供的处理器(以及根据需要创建新处理器的能力)，我们可以构建强大的数据流，以从源中提取数据，按需要进行转换，并将其推送到其他端点。
Using NiFi, we can stream huge amounts of data constantly –hundreds of GB (or even TB), either by creating a few “heavy” data flows or by creating thousands of data flows. Both can be done easily with NiFi.
使用NiFi，我们可以通过创建一些“大量”数据流或创建数千个数据流，持续不断地传输大量数据-数百GB(甚至TB)。 两者都可以使用NiFi轻松完成。
When getting into big data technologies and specifically data flows systems, one cannot ignore one of the main issues — reliability. We must trust our system to get all the data from our sources, transform it exactly as we wish, and send it to the correct output systems. Besides NiFi being a highly reliable technology, we’d like to make sure that the data flows that we created work as we wanted, and that our data won’t be gone somewhere in our data flows, either because of a bug in our data flow, or problems with specific NiFi nodes (crashed, disconnected, etc..).
当进入大数据技术，特别是数据流系统时，人们不能忽视主要问题之一-可靠性。 我们必须信任我们的系统，以从源中获取所有数据，完全按照我们的期望进行转换，并将其发送到正确的输出系统。 除了NiFi是一种高度可靠的技术外，我们还想确保创建的数据流能够按需工作，并且不会因为数据中的错误而将数据流到数据流中的某处流量或特定NiFi节点的问题(崩溃，断开等)。
To allow us monitoring our data flows, NiFi have created the Provenance Repository. Whenever a processor finish processing a flow file, it will produce a Provenance event, a log that states what happened with that specific flow file — whether it was sent to external process, cloned into another flow file, its content was modified, etc. The Provenance Repository is an internal Lucene that comes with the NiFi itself.
为了允许我们监视数据流，NiFi已创建了Provenance Repository。 每当处理器完成对流文件的处理时，它将产生一个Provenance事件，该日志说明该特定流文件的情况-是否将其发送到外部流程，克隆到另一个流文件中，对其内容进行修改等。来源存储库是NiFi本身随附的内部Lucene。
As for NiFi 1.12.0, there are 2 Reporting Tasks regarding Provenance Events — AzureLogAnalyticsProvenanceReportingTask and SiteToSiteProvenanceReportingTask. In this article I’ll speak only about the SiteToSiteProvenanceReportingTask, as it is more widely used.
对于NiFi 1.12.0，有两个关于出处事件的报告任务-AzureLogAnalyticsProvenanceReportingTask和SiteToSiteProvenanceReportingTask。 在本文中，我将只谈论SiteToSiteProvenanceReportingTask，因为它的使用更为广泛。
The SiteToSiteProvenanceReportingTask allows us to send the Provenance Events to another instance of NiFi. In the second NiFi instance, we can open an input port and do whatever we’d like with the Provenance events. Basically, creating a data flow for ingesting our provenance events from the previous instance of NiFi, that contains our main data flows.
SiteToSiteProvenanceReportingTask允许我们将Provenance Events发送到另一个NiFi实例。 在第二个NiFi实例中，我们可以打开一个输入端口，然后执行Provenance事件。 基本上，创建一个数据流以从以前的NiFi实例中提取我们的出处事件，其中包含我们的主要数据流。
If you’re asking yourself why not using the SiteToSiteProvenanceReportingTask to send the Provenance Events to the same instance of NiFi, where our main data flows are, remember that this means you’ll use the same resources (CPU, RAM and disk) for processing your data and your Provenance Events. Separating it for 2 instances, we know that the Provenance events won’t have any kind of impact over our main data flows.
如果您问自己为什么不使用SiteToSiteProvenanceReportingTask将Provenance事件发送到我们主要数据流所在的NiFi相同实例，请记住，这意味着您将使用相同的资源(CPU，RAM和磁盘)进行处理您的数据和出处事件。 将其分开两个实例，我们知道Provenance事件不会对我们的主要数据流产生任何影响。
In our Provenance events data flow, on the second instance of NiFi, we can add any further processing as we wish, and send it to any other data sink.
在我们的Provenance事件数据流中，在NiFi的第二个实例上，我们可以根据需要添加任何进一步的处理，并将其发送到任何其他数据接收器。
So, we have our first NiFi cluster, with our main data flows, sends its Provenance data to another cluster of NiFi, that process our events and send it to an endpoint of our choice, allowing us to have advanced investigation of our data flows and the data ingested through it.
因此，我们有了第一个带有主要数据流的NiFi集群，将其出处数据发送到另一个NiFi集群，该集群处理事件并将其发送到我们选择的端点，从而使我们能够对数据流和通过它摄取的数据。
IOPS问题 (The IOPS Issue)
Leaving our Provenance Events NiFi cluster aside, let’s look at our main cluster of NiFi, that process TB of data a day. Every processor that reads/modify/write the flow files content, will cause a disk IO operation. Having thousands of data flows, that process thousands of flow files concurrently, we’ll end up with thousands of IO operations per second (“IOPS”).
暂不考虑“起源事件” NiFi集群，让我们看一下主要的NiFi集群，它每天处理TB的数据。 每个读取/修改/写入流文件内容的处理器都将导致磁盘IO操作。 拥有成千上万的数据流，可同时处理成千上万的流文件，我们最终将获得每秒数千的IO操作(“ IOPS”)。
Moreover, in this era of docker and containers, if you’re like me and you run NiFi over Kubernetes, and the storage is actually NFS, having large amount of IOPS become significantly worse — we’re now dependent on our network for every disk operation.
此外，在这个Docker和容器时代，如果您像我一样，并且在Kubernetes上运行NiFi，而存储实际上是NFS，则大量IOPS会变得更糟-我们现在每个磁盘都依赖于我们的网络操作。
Having heavy and huge amount of IOPS, will force us getting better machines for our NiFi and spend more money just because we process more data.
拥有大量的IOPS，将迫使我们为NiFi购买更好的机器，并因为我们处理更多的数据而花更多的钱。
As you can understand, the main problem is within the Content Repository, where every flow file content is kept. However, NiFi comes already packed with a multiple implementations of the Content Repository — for both persistent (on disk) and ephemeral (on RAM).
如您所知，主要问题在于内容存储库中，其中保留了每个流文件的内容。 但是，NiFi已经打包了内容存储库的多种实现-永久性(在磁盘上)和临时性(在RAM上)。
The implementation we wish to use can be configured in the nifi.properties file, under nifi.content.repository.implementation. The default value is FileSystemRepository, and we’d like to use VolatileContentRepository. It also make sense to use the VolatileFlowFileContentRepository, since if our content is delete, there is no reason for the Flow File repository to be persistent — it’ll keep the metadata of already deleted content.
我们希望使用的实现可以在nifi.properties文件进行配置，在nifi.content.repository.implementation。 默认值为FileSystemRepository ，我们想使用VolatileContentRepository 。 使用VolatileFlowFileContentRepository也是有意义的，因为如果我们的内容被删除，则流文件存储库没有理由是持久的，它将保留已删除内容的元数据。
By changing the content repository implementation, we can encounter new problems, such as Content Repository out of space error (as we’d probably have less RAM than disk), JavaHeapSpace, etc. Thus, when using the VolatileContentRepository, we should configure 2 new properties — nifi.volatile.content.repository.max.size and nifi.volatile.content.repository.block.size. Configuring these properties according to the resources we can provide to our NiFi cluster, will prevent us from having the errors mentioned above.
通过更改内容存储库的实现，我们会遇到新的问题，例如内容存储库空间不足错误(因为我们的RAM可能比磁盘少)，JavaHeapSpace等。因此，在使用VolatileContentRepository时 ，我们应该配置2个新的属性-nifi.volatile.content.repository.max.size和nifi.volatile.content.repository.block.size 。 根据我们可以提供给NiFi群集的资源配置这些属性，可以防止我们遇到上述错误。
By changing the content repository from disk-based to RAM-based, we have not only dramatically reduced the IOPS that NiFi causes, but have also improved our data flows performance, as obviously RAM operations are much faster than disk operations (and even faster, when running over containers, and the operations are based on network). One might say that we need now more expensive RAM instead of SSD/check Hard Disk, however since we have the performance boost, and our data won’t remain in our content repository for so long, we don’t need as much RAM as disk we needed for disk.
通过将内容存储库从基于磁盘的更改为基于RAM的内容，我们不仅大大降低了NiFi引起的IOPS，而且还改善了我们的数据流性能，因为显然RAM操作比磁盘操作要快得多(甚至更快，在容器上运行时，并且操作基于网络)。 也许有人会说我们现在需要更昂贵的RAM来代替SSD / check硬盘，但是由于我们的性能得到了提高，而且我们的数据不会在内容存储库中保存很长时间，因此我们不需要那么多的RAM我们需要磁盘的磁盘。
For example, If I ran a NiFI cluster over Kubernetes, with total of 7 nodes of 500 GB HDD, 32 GB RAM and 12 CPU cores, I could now have a cluster of 3 nodes, of 50 GB HDD, 50 GB RAM and 8 CPU cores. The reduction of CPU comes from the better performance of the RAM Content Repository. So we can have a smaller cluster, with less resources (besides a bit more RAM), and I’ve managed to reduce the IOPS from around 8,000 per node to around 400. Of course, beside the save in costs of resources, we’ve also boosted our data flows performance, allowing our NiFi cluster to process about ~5 times more data a day.
例如，如果我在Kubernetes上运行NiFI群集，总共有7个500 GB HDD，32 GB RAM和12个CPU内核的节点，那么我现在可以拥有3个节点，50 GB HDD，50 GB RAM和8个节点的群集CPU核。 CPU减少的原因是RAM内容存储库的性能更好。 因此，我们可以拥有一个较小的群集，并使用更少的资源(除了更多的RAM)，而且我设法将IOPS从每个节点的8,000降低到了大约400。当然，除了节省资源成本之外，我们还ve还提高了数据流性能，使我们的NiFi集群每天可以处理大约5倍的数据。
Regarding the IOPS, in fact, the only IOPS remain in our NiFi cluster is the Provenance Repository (which could be also volatile, but as mentioned before, we used it to monitor our data flows, so we’d like to keep it persistent).
关于IOPS，实际上，我们的NiFi集群中唯一保留的IOPS是Provenance Repository(它也可能是易失的，但是如前所述，我们使用它来监视我们的数据流，因此我们希望保持其持久性) 。
However, we face now 2 new problems:
但是，我们现在面临两个新问题：
1. Our NiFi is not reliable anymore — when our content repository was on disk, we could be sure that even if a NiFi nodes crashes, our data will remain forever (at least up to disk corruption). Now, that our content on RAM, our data will be deleted.
1.我们的NiFi不再可靠-当我们的内容存储库位于磁盘上时，我们可以确定，即使NiFi节点崩溃了，我们的数据也将永远存在(至少直到磁盘损坏为止)。 现在，我们在RAM上的内容将被删除。
2. Remember our second NiFi cluster, for ingesting Provenance events? Now, when our first cluster can process much more data per second, it will produce even more Provenance events, our second cluster will cause even more IOPS.
2.还记得我们的第二个NiFi群集，用于接收“源”事件吗？ 现在，当我们的第一个集群每秒可以处理更多数据时，它将产生更多的出处事件，我们的第二个集群将导致更多的IOPS。
Let’s solve both problems:
让我们解决两个问题：
1. As mentioned before, reliability is probably one of the most important issues when getting into Big Data. Moving to Volatile Content Repository lowered the reliability of our data flows. The thing is, NiFi is a stream data flows system. This means that the data we process is ingested into our flows from different sources. If we could simply reflow the data on demand, it would solve it — whenever a NiFi node crashes, we just need to reflow the data. If that’s not the case, we need to have some kind of “Store First” before sending the data to our NiFi cluster. This could be done, for example, by having Kafka topics between the sources and our NiFi — and configure the Kafka to high retention time, enough for us to re consume the data from Kafka in case of a node crash.
1.如前所述，可靠性可能是进入大数据时最重要的问题之一。 迁移到易失性内容存储库降低了我们数据流的可靠性。 事实是，NiFi是一个流数据流系统。 这意味着我们处理的数据从不同的来源被提取到我们的流中。 如果我们可以简单地按需对数据进行重熔，它将解决该问题-每当NiFi节点崩溃时，我们只需要对数据进行重熔即可。 如果不是这种情况，在将数据发送到我们的NiFi集群之前，我们需要某种“先存储”。 例如，可以通过在源和NiFi之间设置Kafka主题来完成此操作，并将Kafka配置为高保留时间，这样足以使我们在节点崩溃的情况下重新使用来自Kafka的数据。
If we cannot control how we get the data from the sources, we could have another instance of NiFi, again a persistent one, with a very short flows — listen to the sources as we always to and publish to Kafka. Since we only need to write once and read once each message, and the rest of the processing flow will be on our Volatile Content NiFi, we might have slightly more IOPS, but not as much as we had before. For example, if we’ve had a single flow with 10 processors that each one needs to read/write the flow file content, we’ll have now 10 times less multiplied by the amount of flow files processed by that flow. Now multiple it by every processor that causes IOP, and you get can understand what we’ve done here.
如果我们无法控制如何从源中获取数据，则可以使用另一个NiFi实例，这又是一个持久实例，流量非常短-像往常一样收听源并将其发布到Kafka。 由于我们只需要编写一次并读取每个消息一次，而其余的处理流程将在我们的易失性内容NiFi上进行，因此我们的IOPS可能会略多一些，但不会像以前那样多。 例如，如果我们有一个带有10个处理器的流，每个流需要读取/写入流文件内容，那么现在乘以该流处理的流文件的数量将减少10倍。 现在，将每个引起IOP的处理器乘以它，您可以了解我们在这里所做的事情。
2. After changing the first cluster to use Volatile Content Repository, we have reduced significantly the IOPS. However, we solved only half of the IOPS problem, as we have the second NiFi cluster. When having a boundary of allowed IOPS, we cannot allow ourselves having the high amount of IOPS that the Provenance Events cluster does.
2.在将第一个群集更改为使用“易失性内容存储库”之后，我们大大降低了IOPS。 但是，由于有了第二个NiFi集群，我们仅解决了IOPS问题的一半。 当具有允许的IOPS边界时，我们不能允许自己拥有“来源事件”集群所具有的大量IOPS。
We could also change the second cluster to use Volatile repository as well, but the Provenance Events are used to verify hermeticity of our data, so we must have reliable flows for ingesting the Provenance Events. Don’t get me wrong — the data we process is obviously much more important than the Provenance Data, but when we’re using the Volatile Content Repository, we can hold less data in our NiFi cluster, meaning we might have back pressure on peaks. In that case, the Provenance data can become much more important — we can keep being updated about the delayed data. So, we don’t want to use Volatile repository, and the File System repository is too IO intensive. Thus, we’d like to find a solution for ingesting our Provenance Events without another instance of NiFi. The problem is that NiFi come only with the 2 mentioned above Reporting Tasks. We’ll solve this by writing a new Reporting Task, which consumes Provenance Events and publishes to Kafka topics.
我们也可以将第二个群集更改为也使用Volatile存储库，但是Provenance Events用于验证数据的密封性，因此我们必须具有可靠的流来提取Provenance Events。 不要误会我的意思-我们处理的数据显然比来源数据重要得多，但是当我们使用挥发性内容存储库时，我们可以在NiFi集群中保存较少的数据，这意味着我们可能会对峰值施加压力。 在这种情况下，出处数据可能变得更加重要-我们可以不断更新有关延迟数据的信息。 因此，我们不想使用Volatile存储库，而File System存储库的IO占用过多。 因此，我们希望找到一种无需其他NiFi实例即可提取“来源事件”的解决方案。 问题在于NiFi仅随附上述2个报告任务。 我们将通过编写一个新的报告任务来解决此问题，该报告任务将使用Provenance Events并发布到Kafka主题。
Why Kafka? Because Kafka is a streaming platform built to run using low hardware requirement, while having almost and no IOPS at all. In fact, thanks to the OS Cache page, if our consumer is consuming continuously, we can have now IOPS at all of our data.
为什么选择卡夫卡？ 因为Kafka是构建为使用低硬件要求运行的流平台，同时几乎没有IOPS。 实际上，由于有了OS Cache页面，如果我们的使用者不断消耗，我们现在可以在所有数据上使用IOPS。
Writing this new reporting task wasn’t difficult at all — I simply had to combine the ProvenanceSiteToSiteReportingTask with and PublishKafka processor (with the relevant Kafka client version).
编写此新的报告任务一点也不困难-我只需要将ProvenanceSiteToSiteReportingTask与PublishKafka处理器(与相关的Kafka客户端版本)结合起来。
When our Provenance Events are published to Kafka, we can consume it to any other data pipeline or CI/CD systems and send it to any endpoint of our choice. I found that the most suitable tech for my case is Logstash. Logstash allow us to listen to Kafka, process (“filter”) the Provenance Events as we wish (with dedicated filter plugins or anything else we’d like with the Ruby plugin) and send it to where ever we want. In my case, wanted to use the Ruby plugin to flatten any nested JSON, and send it to ElasticSearch. The most important benefit of the Logstash in our scenario, is the fact that the Logstash cause almost and no IOPS at all — when processing the data, it all go through its RAM. Moreover, Logstash is very simple and easy to install and maintain, especially running over Kubernetes. So, we can achieve the processing of our second NiFi cluster, just without the IOPS, and probably with much less resources needed — as the Logstash require barely resources while having great performance. So there is no reason to have as much resources as we would have used for the NiFi cluster.
当我们的来源事件发布到Kafka时，我们可以将其使用到任何其他数据管道或CI / CD系统，并将其发送到我们选择的任何端点。 我发现最适合我的情况的技术是Logstash。 Logstash允许我们收听Kafka，根据需要(使用专用过滤器插件或我们希望使用Ruby插件的其他任何东西)处理(“过滤”)出处事件，并将其发送到我们想要的任何地方。 在我的情况下，想使用Ruby插件来拼合任何嵌套的JSON，然后将其发送到ElasticSearch。 Logstash在我们的方案中最重要的好处是，Logstash几乎几乎不会引起IOPS-在处理数据时，它们全部通过其RAM。 而且，Logstash非常简单，易于安装和维护，尤其是在Kubernetes上运行。 因此，我们无需使用IOPS就可以完成第二个NiFi集群的处理，并且可能需要的资源要少得多，因为Logstash几乎不需要资源，而性能却很高。 因此，没有理由拥有与NiFi集群一样多的资源。
摘要 (Summary)
In conclusion, Using this new architecture, we’ve managed to take a large NiFi cluster, which causes much IOPS, and using NiFi built-in properties combined with IOPS friendly systems to build a reliable, cheaper, monitored and boosted NiFi cluster without much effort at all, that causes barely IOPS, for the Provenance Data used to monitor our flows only.
总之，使用这种新架构，我们成功地采用了一个大型NiFi集群，该集群会导致大量IOPS，并利用NiFi内置属性与IOPS友好系统相结合，构建了可靠，便宜，可监控和增强的NiFi集群努力，几乎不会造成IOPS，因为仅用于监控我们流程的出处数据。
翻译自: https://medium.com/@tomergarber/apache-nifi-the-iops-problem-2974d21eb67c
nifi apache
316
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
