我们如何建造Beampipe

I wanted to kick this blog off with a discussion of some of the technical decisions we made while building beampipe. However, I guess I can’t really get away without talking at all about what the product is so here goes…

我想通过讨论在构建射线管时做出的一些技术决策来开始本博客。 但是，我想我真的不能完全不谈论产品是什么，所以这里走了……

Beampipe is a simple, privacy-focussed web analytics tool. It’s somewhat similar in concept to Fathom and Plausible but with some slightly different goals, particularly with regards to our future roadmap. Put simply, and most fundamentally, we think web analytics tools should be easy and fun to use. Secondly, we think that where possible, these tools should respect user’s privacy. We take the same cookieless approach as the two aforementioned tools and explicitly not that of Google Analytics. Thirdly and hopefully distinctively, we think web analytics tools should be powerful. They should give you useful, actionable insights. We feel that there is no tool on the market ticking all three of these boxes and that we can build it!

Beampipe是一个简单的，注重隐私的Web分析工具。 它的概念与Fathom和Plausible相似，但目标略有不同，尤其是关于我们未来的路线图。 简而言之，最根本的是，我们认为网络分析工具应该易于使用且有趣 。 其次，我们认为这些工具应尽可能尊重用户的隐私 。 我们采用与上述两个工具相同的无cookie方法，但显然不是Google Analytics(分析)。 第三，希望与众不同， 我们认为网络分析工具应该功能强大 。 他们应该给您有用的，可行的见解。 我们认为市场上没有工具可以在这三个方框中打勾，我们可以构建它！

Ok so now that I’ve kept our marketing department happy, how are we building it? I’m going to start out by describing how a web analytics service works.

好吧，现在我让营销部门感到高兴，我们如何构建它？ 我将首先描述Web分析服务的工作方式。

All of the products I name-dropped in the preamble function in basically the same way. You drop a little (or not-so-little in the case of Google Analytics) chunk of Javascript onto your page. That Javascript will then make a connection to a backend service and phone home with a little blob of data describing (to various levels of detail) who the user is and what they’re doing on your site. There are some variations here: for instance that you can forego Javascript in favour of an invisible image. But that’s pretty muchthe size of it.

我用相同的方式在前导函数中命名的所有产品。 您将一小段Javascript(或在Google Analytics(分析)中不是那么小)放到页面上。 然后，该Javascript将通过少量数据(以各种详细程度)描述用户是谁以及他们在您的网站上所做的工作，从而连接到后端服务和电话回家。 这里有一些变化：例如，您可以放弃Javascript而使用不可见的图像。 但这几乎就是它的大小。

So the first piece to any web analytics service is to build this tracking script and a suitable backend to process and store the data that it spits out. So what did we choose here?

因此，任何网络分析服务的第一步都是构建此跟踪脚本和一个合适的后端，以处理和存储其吐出的数据。 那么我们在这里选择了什么？

An actual beampipe. Courtesy of CERN Courier.

实际的射线管。 由CERN Courier提供。

Kotlin + Micronaut (Kotlin + Micronaut)

I’m a big fan of Kotlin, a relatively new programming language from the JetBrains. Kotlin is built upon the Java Virtual Machine (JVM), inheriting a huge base of libraries and frameworks as well as an extremely performant runtime. On top of this, and much more importantly from my perspective, it is extremely fun and productive to write code in. It mixes in just enough functional programming magic to make you feel clever and powerful without quite the learning curve, cognitive burden and multiple competing styles that comes with languages like Haskell (or to a lesser degree, Scala).

我是Kotlin的忠实拥护者 ，它是JetBrains的一种相对较新的编程语言。 Kotlin建立在Java虚拟机(JVM)的基础上，继承了庞大的库和框架基础以及性能卓越的运行时。 最重要的是，从我的角度来看，更重要的是，编写代码非常有趣且富有成效。它混合了足够的功能编程魔术，使您在没有学习曲线，认知负担和多重竞争的情况下感到聪明和强大。 Haskell等语言(或在较小程度上为Scala)随附的样式。

On top of these strengths, Kotlin also has some very nice support for asynchronous programming using coroutines, which can be quite an advantage from two perspectives:

除了这些优点之外，Kotlin还对使用协程的异步编程提供了很好的支持，从两个角度来看，这是一个很大的优势：

• Performance. I/O heavy code can share operating system threads. This helps us keep our costs down.
  性能。 I / O繁重的代码可以共享操作系统线程。 这有助于我们降低成本。
• Reasoning about concurrency. We have several features within the application which call out to external APIs and may perform a chain of operations on the results, handling errors appropriately. Coroutines, in my opinion at least, give a nice readable way of structuring this logic.
  关于并发的推理。 我们在应用程序中具有多项功能，这些功能可以调用外部API，并且可以对结果执行一系列操作，从而适当地处理错误。 至少在我看来，协程提供了一种很好的可读性来构造这种逻辑。

On top of Kotlin, I had to pick a web framework. For this I chose Micronaut which is a fairly full-featured JVM-based web framework with some specific support for Kotlin. Whilst I’ve previously preferred less full-featured frameworks, I’ve actually found that, once through the learning curve, Micronaut gives you a lot of useful stuff out of the box without forcing you to use it (e.g. security/authentication, database connection pools, GraphQL).

在Kotlin之上，我必须选择一个Web框架。 为此，我选择了Micronaut ，它是一个功能齐全的基于JVM的Web框架，对Kotlin具有特定的支持。 尽管我以前偏爱功能较弱的框架，但实际上我发现，通过学习曲线，Micronaut可以为您提供很多有用的东西，而不会强迫您使用它(例如，安全性/身份验证，数据库连接池(GraphQL)。

数据库 (Database)

So here we come to perhaps the most important choice. Web analytics tools have some fairly specific needs from a database point of view and ones I shan’t pretend to be an expert in. Write rates are high (at least once people are using the tool). So we need something that can support a fairly high sustained write rate (equivalent to the peak simultaneous traffic for all of the sites using the service). Secondly, we are dealing with timeseries data — something for which certain databases have specific support. Query patterns against the data are often going to be so-called OLAP queries — i.e. aggregations over various axes in the recorded data.

因此，在这里我们可能是最重要的选择。 从数据库的角度来看，Web分析工具有一些相当具体的需求，而我不假装自己是专家。写入率很高(至少在人们使用该工具时)。 因此，我们需要一些可以支持相当高的持续写入率(相当于使用该服务的所有站点的峰值同时通信量)。 其次，我们正在处理时间序列数据-某些数据库具有特定的支持。 针对数据的查询模式通常被称为OLAP查询-即记录数据中各个轴上的聚合。

There are a lot of potentially good database choices here: influxdb, clickhouse, bigquery, redshift etc. However, two concerns led our decision: low overall ops overhead and low cost (since we want to be able to offer a free tier for the service). For this reason, we chose TimescaleDB which is a plugin on top of the excellent PostgreSQL database. This means we only have to live with a single database (including our non-timeseries data) and let’s us get started quickly and at low cost. In future, we may well switch out for something else, but this has served us well so far.

这里有很多潜在的良好数据库选择： influxdb ， clickhouse ，bigquery， redshift等。 但是 ，两个问题导致了我们的决定：较低的总体运营开销和较低的成本(因为我们希望能够为服务提供免费的服务)。 因此，我们选择了TimescaleDB ，它是优秀PostgreSQL数据库之上的插件。 这意味着我们只需要使用一个数据库(包括我们的非时间序列数据)即可，让我们以低成本快速入门。 将来，我们可能会改用其他产品，但这到目前为止对我们很有帮助。

蜜蜂 (APIs)

Having strong API support is part of the longer term vision for the product. We want our customers to be able to integrate deeply with their analytics data. We also want to build a fantastic frontend experience and do it quickly (the first version of the product was launched within a week).

拥有强大的API支持是该产品长期愿景的一部分。 我们希望我们的客户能够与他们的分析数据进行深度集成。 我们还希望构建出色的前端体验并Swift做到(该产品的第一个版本在一周之内发布了)。

Thus we chose GraphQL. Why? Web analytics data is not well represented by a graph and thus doesn’t really benefit from GraphQL’s namesake feature. On the other hand, we’ve found it very useful for building rich APIs with more sophisticated querying capabilities. It is also strongly typed, meaning that users of the API can have some confidence in the shape of the data structures returned. Lastly, alongside frontend libraries like apollo-client or urql we have a very opinionated, simple and powerful way to hook our web interface up to backend queries. This makes development easy and code readable.

因此，我们选择了GraphQL。 为什么？ Web分析数据不能很好地用图形表示，因此并不能真正受益于GraphQL的同名功能。 另一方面，我们发现它对于构建具有更复杂查询功能的丰富API很有用。 它也是强类型的，这意味着API的用户可以对返回的数据结构的形状有一定的信心。 最后，除了前端库(如apollo-client或urql)，我们还有一种非常自以为是的，简单而强大的方法，可将我们的Web界面连接到后端查询。 这使开发变得容易并且代码可读。

Very rough architecture diagram for the backend.

后端的架构图非常粗糙。

构建/开发运营 (Build/DevOps)

So first an admission, despite the pronouns employed in this blog post, Beampipe is a single engineer at the moment — namely me! As a full stack engineer with a lot of ground to cover, automation is key. On the other hand, I have to pick my battles. Continuous Integration (CI) tooling is essential. I simply don’t have time to be manually testing, building and uploading things every time I make a change, at least not if we want to maintain velocity on the product. But I also don’t have time for the kind of build and deployment pipeline you might find on bigger engineering teams.

因此，首先承认一下，尽管本博客文章中使用了代词，但Beampipe目前还是一名工程师-即我！ 作为具有很多基础的全栈工程师，自动化是关键。 另一方面，我必须选择我的战斗。 持续集成(CI)工具至关重要。 我根本没有时间去做每次更改时手动测试，构建和上传东西，至少如果我们想保持产品的速度，至少没有时间。 但是我也没有足够的时间来进行大型工程团队的构建和部署。

For this reason, I build the backend code as a Docker image using GitHub Actions. The build itself is coordinated with Gradle and makes use of Google’s handy jib plugin which produces small, cacheable docker images.

因此，我使用GitHub Actions将后端代码构建为Docker映像。 该版本本身与Gradle进行了协调，并利用了Google便捷的三角臂插件 ，该插件可生成可缓存的小型Docker映像。

Deployment is another area where there were trade-offs to be made. I’d love to use something like Heroku and really not have to worry about deploying the application. On the other hand, these tend to be quite cost prohibitive and not necessarily a good fit for how I wanted to build the app. For that reason, I picked Kubernetes on Scaleway’s hosted Kapsule service. I’ve used this for a few projects now and it’s been excellent.

部署是需要权衡的另一个领域。 我很乐意使用Heroku之类的东西，并且真的不必担心部署该应用程序。 另一方面，这些功能往往成本很高，而且不一定适合我想要构建应用程序的方式。 因此，我选择了Scaleway托管的Kapsule服务上的Kubernetes 。 我现在已经在一些项目中使用了它，这非常好。

Currently the backend is deployed as a monolith but it should be fairly easy to split things up should we need to scale component individually (e.g. separating the data ingest service from the query layer).

当前，后端是作为一个整体部署的，但是如果我们需要分别扩展组件(例如，将数据提取服务与查询层分离)，则应该很容易地将其拆分。

So that pretty much wraps it up on the backend pieces of the app. In the next article we’ll discuss the frontend in a similar manner. Hit me up with questions or feedback on hello@beampipe.io Cheers!

这样就可以将其包装到应用程序的后端。 在下一篇文章中，我们将以类似方式讨论前端。 在hello@beampipe.io上提问或反馈让我兴奋不已！

翻译自: https://medium.com/beampipe/how-we-build-beampipe-50618730a0fd