jira批量导入任务_我们如何使用批量操作扩展用户导入任务

jira批量导入任务

We have functionality in our loyalty product to allow our clients to import their customers into the program. Our team wrote this as a Celery task which would read each row from a CSV file and call our existing user enrolment method. This method would insert a user into the database and send out emails to the user.

我们的会员产品具有功能，可让我们的客户将其客户导入该计划。 我们的团队将此作为Celery任务编写，该任务将从CSV文件读取每一行并调用我们现有的用户注册方法。 此方法会将用户插入数据库，然后向用户发送电子邮件。

When we released this feature, it worked perfectly fine for small customers who would import around 20k-30k customers into the program. Then one fine day, horror-struck when one of our customers wanted to import 700k users. All of the nodes (celery workers in our case) were only working for this task to get complete. Our RabbitMQ queue started getting filled up and there were continuous memory alerts. Our database connections increased and became astonishingly high making the system super slow. The import took almost a day to complete (during which we faced a lot of customer support queries) and post this nightmare we decided to improve the functionality.

当我们发布此功能时，它对于将大约2万至3万名客户导入该程序的小型客户来说效果很好。 然后是美好的一天，当我们的一位客户想要导入70万用户时，震惊不已。 所有节点(在我们的案例中为芹菜工人)都只是为了完成此任务而工作。 我们的RabbitMQ队列开始变满，并且持续出现内存警报。 我们的数据库连接增加并变得异常庞大，从而使系统变得非常慢。 导入花了将近一天的时间(在此期间，我们面临许多客户支持查询)，并且发布了这一噩梦，我们决定改进功能。

We went through the import users task once again and figured out a few performance issues.

我们再次完成了导入用户任务，并找出了一些性能问题。

• We were making a database call for each user to fetch configurations from the database like email notification details, merchant configuration settings, etc.
  我们正在为每个用户进行数据库调用，以从数据库中获取配置，例如电子邮件通知详细信息，商家配置设置等。
• Each time there was a database query to insert a user into the database.
  每次进行数据库查询以将用户插入数据库。

We realized that we were doing a lot of reparative processes. Let’s compare this with our public transport system. Suppose there are 50 employees working in an organization and living in the same society as well. Everyone uses their own vehicle to commute to the office. For everyone, the security perso