谷歌firebase_如何使用Firebase和Google Cloud Firestore构建事件记录系统

谷歌firebase

Firebase and Google Cloud Platform offer an excellent way to build apps and systems that can rapidly scale while providing a fantastic developer experience. At EstateSync, we use GCP services as the backend for our API to allow the fast distribution of real estate data to multiple vendors. Given the requirement to be able to scale to large amounts of data, we built our architecture on the premise of asynchronicity.

Firebase和Google Cloud Platform提供了一种极好的方式来构建可快速扩展的应用程序和系统，同时提供出色的开发人员体验。 在EstateSync ，我们将GCP服务用作API的后端，以允许将房地产数据快速分发给多个供应商。 考虑到能够扩展到大量数据的要求，我们在异步性的前提下构建了体系结构。

Because of this, there is an obvious need to keep users informed about what is going on within the system.

因此，很明显需要使用户了解系统中正在发生的事情。

问题 (The Problem)

Naturally in an asynchronous system, we can only provide the user with direct feedback for direct requests (e.g. when a request resource is invalid). To check if some event has occurred in the background, the user would need to query the API every time.

自然地，在异步系统中，我们只能为用户提供直接请求的直接反馈(例如，当请求资源无效时)。 为了检查后台是否发生了某些事件，用户每次都需要查询API。

This can be solved by webhooks — instead of having the user query our service for changes, we can notify the user. In Firebase, asynchronous work usually happens in Cloud Functions. Easily enough, at the end of whatever we do in a Cloud Function, we can just fire the user’s webhook by making a request to an URL he provided with a relevant payload (e.g. the entity that has been created in the background).

这可以通过webhooks来解决-我们可以通知用户，而不是让用户查询我们的服务以进行更改。 在Firebase中，异步工作通常在Cloud Functions中进行。 足够容易地，在我们执行Cloud Function的任何操作结束时，我们只要向用户提供的URL提供相关有效负载(例如，已在后台创建的实体)的请求，就可以触发用户的Webhook。

This does not scale, though. If the webhook response takes long (even with a reasonable timeout) this setup will prolong the processing time of the Cloud Function. It also means that we would be tying two systems (the asynchronous work and the webhook) together that should be independent. Also, this setup provides no option to deal with retries for failed webhook calls — no state is persisted, which means there is no easy way to retry failed requests or log errors.

但是，这不会扩展。 如果Webhook响应花费的时间很长(即使超时时间合理)，此设置也会延长Cloud Function的处理时间。 这也意味着我们将把两个相互独立的系统(异步工作和Webhook)捆绑在一起。 另外，此设置没有选项来处理失败的Webhook调用的重试-没有状态保持不变，这意味着没有简单的方法可以重试失败的请求或记录错误。

Also, there is a different kind of information that can be produced asynchronously: Errors. These are the kind of events that developers receive email notifications about and that need to be addressed by the developer (e.g. by changing a configuration). For these there is usually no webhook implemented because they are supposed to not occur again after the initial cause is fixed. They too come with some form of payload to help with debugging. The user often wants access to these kinds of errors through a UI to check past issues and have a single source of truth and not just email notifications.

另外，还可以异步产生另一种信息：错误。 这些是开发人员接收有关电子邮件通知的事件，开发人员需要处理这些事件(例如，通过更改配置)。 对于这些，通常不实施任何Webhook，因为在确定了最初的原因之后，它们应该不会再出现。 它们也带有某种形式的有效负载以帮助调试。 用户通常希望通过UI访问这些类型的错误以检查过去的问题，并拥有单一的真实来源，而不仅仅是电子邮件通知。

So here we are, asynchronously generating both recurring system information and occasional errors. How can we build a robust architecture that takes both of them into account?

因此，在这里，我们异步生成重复出现的系统信息和偶然的错误。 我们如何建立一个将两者都考虑在内的健壮架构？

一个办法 (A Solution)

We can come up with a solution by looking at the commonalities of the two types of information we generate. By abstracting the underlying system it becomes obvious that both are „events“ — something that has occurred in our system, be it an error or some finished task. Let’s model the architecture around the concept of events.

通过查看生成的两种信息的共性，我们可以提出一个解决方案。 通过抽象化底层系统，很明显，两者都是“事件”，即在我们的系统中已经发生的事情，无论是错误还是完成的任务。 让我们围绕事件的概念对架构进行建模。

步骤1：计划如何构造数据 (Step 1: Plan how to structure the data)

Create a collection events within Firestore. Each of the future documents should have the following fields:

在Firestore中创建一个收集events 。 每个将来的文档都应具有以下字段：

• name: a string used for identification purposes, e.g. „user.processed“

  name ：用于标识目的的字符串，例如“ user.processed”

• payload: a map used to store arbitrary data of the event

  payload ：用于存储事件的任意数据的映射

• isCritical: a boolean to indicate that this is an event that is not supposed to happen (also called „Error“)

  isCritical ：一个布尔值，指示这是不应该发生的事件(也称为“错误”)

• createdAt: a timestamp used for sorting and display

  createdAt ：用于排序和显示的时间戳

The isCritical field could alternatively be named „level“ and store an integer according to the widely used syslog severity levels („debug“, „informational“, „warning“, „error“, …). This way we could allow the user to request events with certain levels, e.g. utilising the „in“ query operator. In our case we did not expect a future need for levels so we settled with a simple isCritical flag.

也可以将isCritical字段命名为“ level”，并根据广泛使用的syslog严重性级别 (“ debug”，“ informational”，“ warning”，“ error”等)存储一个整数。 这样，我们可以允许用户以特定级别请求事件，例如，利用“ in”查询运算符。 在我们的案例中，我们并不预期将来会需要级别，因此我们使用了一个简单的isCritical标志来解决。

步骤2：存储事件 (Step 2: Store the events)

Whenever an asynchronous action in a Cloud Function finishes, let the Function create an event in the events collection.

每当Cloud Function中的异步操作完成时，请让Function在events集合中创建一个事件。

I would recommend creating a separate module that takes care of persisting events. Each event can be represented by a class that knows about the details of the event type (e.g. how to format its payload). This is to follow the Single Responsibility Principle.

我建议创建一个单独的模块来处理持久事件。 每个事件都可以由一个类来表示，该类知道事件类型的详细信息(例如，如何格式化其有效负载)。 这是遵循单一责任原则 。

Let’s say our system can host multiple accounts. Upon creation of a user within the account we do some processing. When done, we want to store the event of finishing this processing. An example Cloud Function could look like this:

假设我们的系统可以托管多个帐户。 在帐户内创建用户后，我们会进行一些处理。 完成后，我们要存储完成此处理的事件。 示例云功能可能如下所示：

const processUser = functions.firestore
  .document("accounts/{accountId}/users/{userId}")
  .onCreate(async (snapshot, context) => {
    await processTheUser(snapshot);
    const event = new UserProcessed(snapshot);
    await new Account(context.params.accountId).addEvent(event);
  })

The UserProcessed class (and its interface) might look like this:

UserProcessed类(及其接口)可能如下所示：

interface Event {
  public isCritical(): boolean;
  public getName(): string;
  public getPayload(): object;
}


class UserProcessed implements Event {
  constructor(protected user: firestore.DocumentSnapshot) {}


  public isCritical() {
    return true;
  }


  public getName() {
    return "user.processed";
  }


  public getPayload() {
    return {
      id: this.user.id,
      name: this.user.data().name,
      email: this.user.data().email
    };
  }
}

The Account class that is used to store the event might look like this:

用于存储事件的Account类可能如下所示：

class Account {
  constructor(protected accountId: string) {}


  public async addEvent(event: Event) {
    return firestore()
      .collection("accounts")
      .doc(this.accountId)
      .collection("events")
      .add({
        name: event.getName(),
        payload: event.getPayload(),
        isCritical: event.isCritical(),
        createdAt: firestore.FieldValue.serverTimestamp(),
      });
  }
}

The naming convention we used in this case follows the syntax {entity}.{action}. It is worth spending time on finding a clean syntax in order to avoid cluttering your database and to help future developers and users to understand how the system works. Segment has an excellent guide on naming events.

在这种情况下，我们使用的命名约定遵循语法{entity}.{action} 。 值得花时间在寻找干净的语法上，以避免使数据库混乱，并帮助将来的开发人员和用户了解系统的工作方式。 该部分对命名事件有很好的指导 。

The example above deals with the sort of information a user would create a webhook for — a recurring system event. What would the Cloud Function look like if we want to store an error event? Like this:

上面的示例处理了用户将为之创建Webhook的一种信息-重复发生的系统事件。 如果我们要存储错误事件，Cloud Function会是什么样？ 像这样：

const processUser = functions.firestore
  .document("accounts/{accountId}/users/{userId}")
  .onCreate(async (snapshot, context) => {
    try {
      await processTheUser(snapshot);
      const event = new UserProcessed(snapshot);
      await new Account(context.params.accountId).addEvent(event);
    } catch (error) {
      const event = new UserProcessingFailed(snapshot, error);
      await new Account(context.params.accountId).addEvent(event);
    }
  })

Notice how we pass the error into the event. This way the event class can check if it recognizes the error and put a helpful message into the payload.

注意我们如何将错误传递给事件。 这样，事件类可以检查它是否能够识别错误并将有用的消息放入有效负载中。

Storing events like this will often lead to duplicate data that is stored not only in its original place but also in the event document. In the example, the data of the created user is now sitting both in the user and the event doc. This is alright since the event represents a snapshot of the data at this time and Firestore as a non-relational database is fine with duplicate data (Fireship has a great video about that).

像这样存储事件通常会导致重复的数据不仅存储在原始位置，而且存储在事件文档中。 在示例中，创建的用户的数据现在位于用户和事件文档中。 没关系，因为该事件表示此时的数据快照，并且Firestore作为非关系数据库可以使用重复数据(Fireship提供了与此相关的精彩视频 )。

However, be aware to consider this when thinking about privacy: in case you handle personal data, you might need a system that keeps track of where you store a user's data and be able to retrieve, anonymize or delete it. By storing duplicate data in the event, you need to keep track of an additional data location. The firebase team provides a „Delete User Data“ extension just for dealing with that.

但是，在考虑隐私时要注意这一点：如果处理个人数据，则可能需要一个跟踪用户数据存储位置并能够检索，匿名或删除它的系统。 通过在事件中存储重复数据，您需要跟踪其他数据位置。 firebase团队提供了一个“删除用户数据”扩展名来解决该问题。

步骤3：显示事件 (Step 3: Display the events)

In your UI you can display and access the events collection like usual. You might want to include a switch to filter for only critical events (based on the isCritical flag). If you are using the Firebase libraries this will also give you realtime updates when new events are created. This can greatly enhance the developer experience because it allows to „watch“ how your app is working in the background.

在用户界面中，您可以像往常一样显示和访问events集合。 您可能希望包括一个仅针对关键事件进行过滤的开关(基于isCritical标志)。 如果您使用的是Firebase库，那么当创建新事件时，这还将为您提供实时更新。 这可以极大地改善开发人员的体验，因为它可以“观察”您的应用程序在后台的工作方式。

When you first filter based on isCritical and sort on createdAt, Firestore will ask you to create a composite index for that. Just follow the instructions on the link they provide with that and you are good to go.

当你第一次筛选基于isCritical和排序createdAt ，公司的FireStore会要求你创建一个综合指数。 只需按照他们提供的链接上的说明进行操作，就可以了。

第4步：为新事件触发网络鸣叫 (Step 4: Fire webhooks for new events)

Thanks to the fact that you have a concrete list of event types now, you can just pipe them through a webhook system. This way the user can decide for which events he wants to set up a webhook and if he wants to listen for error events.

由于您现在有了事件类型的具体列表，因此您可以将它们通过Webhook系统进行管道传输。 这样，用户可以决定要为其设置网络挂钩的事件以及是否要侦听错误事件。

I assume that you already have a way to store webhooks with their URL and the event name they are supposed to be fired on. In order to fire the correct webhooks, create a Cloud Function that is invoked upon the creation of new event documents. In the Function, query for all webhooks that are supposed to be fired for the newly created event (based on its name). The request to the webhook URL can be made with the event data as request body, giving the developer a consistent format to work with for all events.

我假设您已经可以用其URL和应该触发它们的事件名称来存储webhooks。 为了触发正确的webhook，请创建在创建新事件文档时调用的Cloud Function。 在“函数”中，查询应该为新创建的事件触发的所有Webhooks(基于其名称)。 可以使用事件数据作为请求正文来发出对Webhook URL的请求，从而为开发人员提供用于所有事件的一致格式。

You should also think about retrying failed webhook request, ideally giving the user time to fix the problem until the webhook is fired again (Exponential Backoff). Building such a system is outside the scope of this article, but in general, one could go about this by storing a request attempt in a subcollection on the webhook, making it available for retry later if it fails.

您还应该考虑重试失败的Webhook请求，最好是给用户时间来解决问题，直到再次触发Webhook为止( 指数回退 )。 构建这样的系统不在本文讨论的范围之内，但是通常，可以通过将请求尝试存储在webhook的子集合中来解决此问题，并使其在以后失败时可以重试。

步骤5：将严重事件通知用户 (Step 5: Notify the user of critical events)

In the same Cloud Function that fires the webhook, you can check if the event has the isCritical flag set. If it does, you might want to notify the consumer of the issue by email or even push notification. Through the API of a service like Sendgrid or Mailgun, one can send an email template filled with the relevant event details.

在触发isCritical的同一Cloud Function中，您可以检查事件是否设置了isCritical标志。 如果是这样，您可能希望通过电子邮件甚至推送通知将问题通知消费者。 通过诸如Sendgrid或Mailgun之类的服务的API，可以发送包含相关事件详细信息的电子邮件模板。

And that’s it. By abstracting the initial problem one is able to build a robust system that plays on the strengths of the Firebase ecosystem. We are now able to persist different types of events and notify the user about them through webhooks or the UI. Let me know if you have any questions or feedback about the proposed solution.

就是这样。 通过抽象出最初的问题，人们可以构建一个强大的系统，发挥Firebase生态系统的优势。 现在，我们可以保留不同类型的事件，并通过webhooks或UI通知用户有关事件的信息。 如果您对建议的解决方案有任何疑问或反馈，请告诉我。

翻译自: https://medium.com/@richartkeil/how-to-build-an-event-logging-system-with-firebase-and-google-cloud-firestore-8a1a457c1522

谷歌firebase