In this article, we will explore the capacity of the ELK stack to get the most from a microservices approach to software development. Microservices bring plenty of opportunity for a more efficient approach to creating software products, along with some drawbacks and potential risks. When properly used, ELK has the potential to reduce these risks and help you deliver quality in an efficient way.
在本文中，我们将探讨ELK堆栈的功能，以从微服务方法进行软件开发中获得最大收益。 微服务为创建软件产品的更有效方法带来了很多机会，同时也带来了一些缺点和潜在风险。 正确使用ELK可以降低这些风险，并帮助您有效地交付质量。
序幕 (Prologue)
We have no idea which was first: the chicken or the egg, but we know that the Monolith was the igniter of changes that we observe every day in IT. The Monolith, the big, the scary, the unchangeable, the breakable, the Return of Investment (ROI) eater. And then out of nowhere, Microservices arrived. Those small, pretty small pieces of software that conquered the Monolith, split it into pieces, and made changes to software easy again. But you, reader, should know best that requirements always change. Code becomes obsolete and hard to maintain. You may have a feeling that behind the horizon stands new danger for your ROI. What may it be? How to deal with that?
我们不知道是第一个是鸡还是鸡蛋，但是我们知道Monolith是我们每天在IT中观察到的变化的点火器。 巨石，是大的，令人恐惧的，不变的，易碎的，投资回报率(ROI)的食用者。 然后从哪儿冒出来，微服务到达。 那些征服了Monolith的小巧软件片断，将其拆分为几块，并再次轻松更改软件。 但是，您，读者，应该最了解需求总是在变化。 代码变得过时且难以维护。 您可能会觉得，在地平线后面为您的投资回报带来新的危险。 可能是什么？ 怎么处理呢？
为什么要使用微服务方法？ (Why use a Microservices Approach?)
Getting into microservices seems easy at first glance. When you need to create new shiny features for your clients, developers create from scratch small software pieces. Developers love new software — it is so clean, manageable, no dependencies, no legacy stuff. They can use new technologies, learn while doing their job, thus their morale is high. You are happy that features are created quickly, developers are happy, but where is the catch?
乍一看，进入微服务似乎很容易。 当您需要为客户创建新的闪亮功能时，开发人员可以从头开始创建小型软件。 开发人员喜欢新软件-它是如此干净，可管理，没有依赖关系，没有遗留物。 他们可以使用新技术，边工作边学习，因此他们的士气很高。 您很高兴能快速创建功能，开发人员很高兴，但是收获在哪里？
As we wrote size matters. Creating tens, hundreds, or thousands of Microservices creates a huge network of connections and dependencies between them (Microservices hell). Before Microservices, the Monolith crash was obvious — none of your clients could use it anymore. Nowadays failures may be intermittent and hard to find. Are you sure that your software product meets the Service Layer Agreement (SLA)? Are you sure that all payment transactions end with success? Are you sure that your clients love your product and will not give up before your goals are met?
当我们写大小时。 创建数十个，数百个或数千个微服务会在它们之间建立一个庞大的连接和依赖关系网络(微服务hell )。 在微服务之前， Monolith崩溃很明显-您的客户都无法再使用它了。 如今，故障可能是断断续续的，很难发现。 您确定您的软件产品符合服务层协议(SLA)吗？ 您确定所有付款交易都以成功结束吗？ 您确定您的客户喜欢您的产品并且在实现目标之前不会放弃吗？
你能为这个做什么？ (What can you do about it?)
The ELK Stack may help you with that. This is a group of tools called ElasticSearch, Logstash and Kibana, which were created to help monitor your systems’ infrastructure and business goals. It’s a single point of information where you can find all the data that you need.
ELK Stack可能会帮助您。 这是一组名为ElasticSearch，Logstash和Kibana的工具，它们的创建是为了帮助监视系统的基础结构和业务目标。 这是一个信息点，您可以在其中找到所需的所有数据。
ELK堆栈理论 (Theory of the ELK Stack)
ElasticSearch is a noSQL, document database, meant to store data that is time-related. It stores events containing data relevant to your needs. It may be logs from production, metrics of operation systems, or even metrics of your CI/CD process. ElasticSearch implements many aggregation functions to enable fast information retrieval for long periods of time (e.g. What browser was the most popular in your solution last year. Should you still support Internet Explorer?).
ElasticSearch是一个noSQL文档数据库，用于存储与时间相关的数据。 它存储包含与您的需求相关的数据的事件。 它可能是生产日志，操作系统指标，甚至是CI / CD流程指标。 ElasticSearch实现了许多聚合功能，可实现长时间的快速信息检索(例如，去年您的解决方案中最流行的浏览器是哪种浏览器。您是否仍支持Internet Explorer？)。
is a piece of software that can have multiple roles. It was designed to be log-files parser, because text parsing may be a resource-intensive process. It also can be a point of aggregation for all of your services. It may tag events from your system exactly as you need it and store those in ElasticSearch or other types of services or databases.
是一款可以扮演多个角色的软件。 它被设计为日志文件解析器，因为文本解析可能会占用大量资源。 它也可以是所有服务的聚合点。 它可以完全根据需要标记系统中的事件，并将事件存储在ElasticSearch或其他类型的服务或数据库中。
我在ELK Stack上的经验 (My experience with the ELK Stack)
Some time ago I was working at a company that delivered some products with certain SLA. The system evolved from the Monolith into a distributed services network, and then started to change into Microservices. It was monitored by a dedicated team who used logs that developers left in the code. That model worked fine, but sometimes there were problems not indicated by the logs, but by clients calling, angry that they cannot use our product. It made me wonder how it can be?
不久前，我在一家提供某些SLA产品的公司工作。 该系统从Monolith演变为分布式服务网络，然后开始转变为Microservices。 它由一个专门的团队进行监视，该团队使用了开发人员留在代码中的日志。 该模型运行良好，但有时出现的问题不是日志显示的，而是客户打电话来的，他们生气他们不能使用我们的产品。 这让我想知道怎么回事？
I spoke with a dedicated monitoring team, the Ops team, the Networking team and they saw no issues in their data. Looking for help, I found the ELK Stack and, after some negotiations with the manager and the Ops and Monitoring teams, we deployed it. We started with what I think is the most important part monitoring the input/output of our system: the access.log files.
我与专门的监视团队，Ops团队，网络团队进行了交谈，他们发现数据没有问题。 在寻求帮助的过程中，我找到了ELK堆栈，在与经理，运营和监控团队进行了一些协商之后，我们进行了部署。 我们从监视系统输入/输出最重要的部分开始： access.log文件。
This magic file is created by a web server and contains a lot of information about your system. For example, what endpoint was called, by whom, was the response successful or not, how big was the payload, how long did it take your system to process it, or even what browser your user used. A lot of useful data. Using that data we discovered some bad things. Some services had nearly 10% of error responses that we weren’t aware of. There were also periods when requests took a lot of time that we didn’t know about either. It took some drilling to fix that, but this example showed us that such a simple thing as an access.log file may improve your SLA and users' experience.
该魔术文件是由Web服务器创建的，其中包含有关系统的许多信息。 例如，调用哪个端点，响应的对象是谁，响应是否成功，有效负载有多大，系统处理该负载花了多长时间甚至用户使用了哪种浏览器。 很多有用的数据。 使用这些数据，我们发现了一些不好的东西。 一些服务有将近10％的错误响应是我们不知道的。 在某些情况下，请求花了很多时间，我们都不知道。 需要花一些时间来解决这个问题，但是此示例向我们展示了诸如access.log文件之类的简单内容可以改善您的SLA和用户体验。
ELK堆栈的下一步是什么？ (What’s next for the ELK Stack?)
ELK Stack becomes more and more powerful, the more data you feed it. There are unlimited scenarios that you can use it for. These are some examples:
ELK Stack变得越来越强大，您提供的数据越多。 您可以使用无限的方案。 这些是一些示例：
- Find out what percentage of clients that put at least one thing to the shopping cart, actually finished checkout. If that percentage is small, it may indicate too complicated processes or payment-provider problems.
-找出将至少一件商品放入购物车，实际结帐的客户百分比。 如果该百分比很小，则可能表明流程太复杂或付款提供商出现问题。
- Measuring ROI of new features. Knowing the cost of a feature allows you to monitor its ROI in real time. It may be that it’s not used enough to pay off its investment and it may be cheaper to remove it.
-测量新功能的投资回报率。 了解功能的成本可以让您实时监控其投资回报率。 可能是因为它的使用量不足以还清投资，并且删除它可能更便宜。
- The most expensive code in production is the one you’re not using. You could write an event per API-call in ElasticSearch and measure usage (or use APM — more on that below). Maybe some of your Microservices are not used anymore. Now you have proof that you can safely decommission them.
-生产中最昂贵的代码是您不使用的代码。 您可以在ElasticSearch中针对每个API调用编写一个事件，并衡量使用情况(或使用APM，有关更多信息，请参见下文)。 也许您的某些微服务已不再使用。 现在，您有证据可以安全地停用它们。
- You can gather historical data and for example: imagine you invested in optimization of software. Now you can observe how much it paid off .
-您可以收集历史数据，例如：假设您投资于软件优化。 现在您可以观察到它还清了多少。
The ELK stack may also help you during migration from the Monolith into Microservices. You could begin migration with documenting features in the system. Then, each use of a feature may be sent to ElasticSearch via REST API. Those metrics help find hot spots in the system and help to decide what needs to be migrated first. Libraries such Metrics.NET.ElasticSearch will help you with that kind of task.
从Monolith迁移到微服务期间，ELK堆栈也可能为您提供帮助。 您可以开始使用系统中的文档功能进行迁移。 然后，功能的每次使用都可以通过REST API发送到ElasticSearch。 这些度量标准有助于找到系统中的热点，并有助于确定首先需要迁移的内容。 诸如Metrics.NET.ElasticSearch之类的库将帮助您完成此类任务。
The ELK stack is great in aggregating logs, metrics and other information from many sources in one place. It shipswith small services called that will do all the work of reading and structuring requests for ElasticSearch for you. Just remember to add custom fields to each event (using Logstash mutate plugin or Beats configuration , for example) to indicate environment, machine, client and service name. A single Kibana dashboard can then be used to display the same kind of information for multiple environments.
ELK堆栈非常适合在一个地方汇总来自许多来源的日志，指标和其他信息。 它附带了称为的小型服务，它将为您完成所有对ElasticSearch的读取和构建请求的工作。 只要记住要向每个事件添加自定义字段(例如，使用Logstash mutate插件或Beats配置)来指示环境，机器，客户端和服务名称。 然后，可以使用单个Kibana仪表板为多个环境显示相同类型的信息。
The newest and most exciting part of the ELK Stack is — Application Performance Monitoring. If you have Elastic Cloud or ElasticSearch on-premise, you can use it to monitor internals of your applications with just a few lines of code. delivers ready-to-use libraries for languages such as Java, .NET, node.js, python, Ruby or even Go. It can deliver information on performance issues to developers, as well as how services communicate with each other in production, and how they are interconnected.
ELK堆栈中最新最激动人心的部分是-应用程序性能监视。 如果您具有本地弹性云或ElasticSearch，则可以使用它通过几行代码来监视应用程序的内部。 提供适用于Java，.NET，node.js，python，Ruby甚至Go的语言的现成库。 它可以将有关性能问题的信息提供给开发人员，以及服务在生产中如何相互通信以及如何相互连接。
但… (But…)
Pointing out all these features, you might get the impression that those tools will solve all of your problems! As always there’s a catch. Investing in the ELK Stack may become pricey. You might worry over how free, open-source tools can overwhelm your wallet. I’ll try to explain and give you some ideas on how to deal with that.
指出所有这些功能，您可能会觉得这些工具将解决您所有的问题！ 和往常一样，有一个陷阱。 投资ELK堆栈可能会变得昂贵。 您可能会担心免费的开放源代码工具如何使您的钱包不堪重负。 我将尽力解释并给您一些有关如何处理该问题的想法。
The first obvious reason why the ELK Stack price is high is hosting. Regardless of whether cloud-hosted or on-premises hosted, it will consume resources. ElasticSearch is pretty smart about data and how it’s stored. In general, new data isn’t immediately written to disk storage. Depending on the configuration, new data is stored in the RAM to be accessible faster. It is especially useful for dashboards showing real-time data.
ELK Stack价格高的第一个显而易见的原因是托管。 无论是云托管还是本地托管，它都会消耗资源。 ElasticSearch对于数据及其存储方式非常聪明。 通常，新数据不会立即写入磁盘存储中。 根据配置，新数据将存储在RAM中，以便可以更快地访问。 对于显示实时数据的仪表板特别有用。
If you’re planning to ingest a lot of data, then parse or pre-process it, you may need additional resources. In general, Logstash works better when it has access to multiple threads. Those then process events in batches of hundreds or thousands, which may be a resource-intensive task.
如果您打算摄取大量数据，然后对其进行解析或预处理，则可能需要其他资源。 通常，Logstash可以访问多个线程时效果更好。 然后，它们以成百上千的批量处理事件，这可能是一项资源密集型任务。
If you’re not interested in all the features that Kibana has in its extended version, then consider using. Like Kibana, it’s a data visualizer tool, but unlike Kibana, it’s also able to display data from multiple sources at the same time (ElasticSearch, InfluxDB, Prometheus or even direct SQL queries). You can use the cloud-hosted version, or deploy it by yourself using the install package, a Docker image, or even install it on Raspberry Pi.
如果您对Kibana扩展版中的所有功能不感兴趣，请考虑使用。 与Kibana一样，它是一个数据可视化工具，但与Kibana不同，它还能够同时显示多个来源的数据(ElasticSearch，InfluxDB，Prometheus或直接SQL查询)。 您可以使用云托管版本，也可以使用安装包，Docker映像自行部署它，甚至可以将其安装在Raspberry Pi上。
课程 (The Lesson)
Microservices have changed the way software is designed nowadays. They are cool, small, easy, and cheap to make. Unfortunately, they also have drawbacks such as their role in increasing overall system complexity. To be able to manage that, tools like the ELK Stack help to harness their potential and also bring other benefits to your company. The ELK Stack seems to be an all-in-one, single-source-of-truth solution that deals with tasks like infrastructure monitoring, log aggregation, usage statistics, and business process performance data gathering. This single point of information is invaluable when you want to deliver quality. Just give it a try!
如今，微服务已经改变了软件设计的方式。 它们很酷，小巧，容易且制造便宜。 不幸的是，它们还具有缺点，例如它们在增加整体系统复杂性方面的作用。 为了能够进行管理，ELK Stack之类的工具有助于发挥其潜力，并为您的公司带来其他好处。 ELK Stack似乎是一种多合一的，真实的单一来源解决方案，可处理诸如基础架构监视，日志聚合，使用情况统计以及业务流程性能数据收集之类的任务。 如果您想提供高质量的信息，那么这单点信息将无价之宝。 试一试！
220
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
