数据工程 数据科学_什么是数据工程？

数据工程 数据科学

This is the first in a series of posts on Data Engineering. If you like this and want to know when the next post in the series is released, you can subscribe at the bottom of the page.

这是有关数据工程的系列文章中的第一篇。 如果您喜欢这种方式，并且想知道该系列的下一篇文章何时发布，可以在页面底部进行订阅 。

From helping cars drive themselves to helping Facebook tag you in photos, data science has attracted a lot of buzz recently. Data scientists have become extremely sought after, and for good reason – a skilled data scientist can add incredible value to a business.

从帮助汽车驾驶到帮助Facebook在照片中标记您的身份，数据科学最近吸引了很多关注。 数据科学家受到了极大的追捧 ，并且有充分的理由-熟练的数据科学家可以为企业增加不可思议的价值。

Data scientists and engineers help power self-driving cars.

数据科学家和工程师为自动驾驶汽车提供动力。

But a data scientist is only as good as the data they have access to. Most companies store their data in variety of formats across databases and text files. This is where data engineers come in – they build pipelines that transform that data into formats that data scientists can use. Data engineers are just as important as data scientists, but tend to be less visible because they tend to be further from the end product of the analysis.

但是，数据科学家的素质仅与他们可以访问的数据一样好。 大多数公司在数据库和文本文件中以各种格式存储数据。 这就是数据工程师进来的地方–他们建立了将数据转换成数据科学家可以使用的格式的管道。 数据工程师与数据科学家同等重要，但由于它们离分析的最终产品更远，因此它们的知名度通常较低。

A good analogy is a race car builder vs a race car driver. The driver gets the excitement of speeding along a track, and thrill of victory in front of a crowd. But the builder gets the joy of tuning engines, experimenting with different exhaust setups, and creating a powerful, robust, machine. If you’re the type of person that likes building and tweaking systems, data engineering might be right for you. In this post, we’ll explore the day to day of a data engineer, and discuss the skills required for the role.

一个很好的类比是赛车制造商与赛车手。 驾驶员兴奋地沿着轨道行驶，并在人群面前获得胜利的快感。 但是，制造商可以通过调整引擎，尝试不同的排气设置以及创建功能强大，坚固的机器来获得乐趣。 如果您是喜欢构建和调整系统的人，那么数据工程可能适合您。 在本文中，我们将探讨数据工程师的日常工作，并讨论该角色所需的技能。

数据工程师的角色 (The data engineer role)

The data science field is incredibly broad, encompassing everything from cleaning data to deploying predictive models. However, it’s rare for any single data scientist to be working across the spectrum day to day. Data scientists usually focus on a few areas, and are complemented by a team of other scientists and analysts.

数据科学领域极为广阔，涵盖了从清理数据到部署预测模型的所有内容。 但是，很少有任何数据科学家每天都在整个频谱上工作。 数据科学家通常专注于几个领域，并由其他科学家和分析师团队进行补充。

Data engineering is also a broad field, but any individual data engineer doesn’t need to know the whole spectrum of skills. In this section, we’ll sketch the broad outlines of data engineering, then walk through more specific descriptions that illustrate specific data engineering roles.

数据工程也是一个广阔的领域，但是任何个人数据工程师都不需要了解全部技能。 在本节中，我们将概述数据工程的概述，然后遍历更具体的描述，以说明特定的数据工程角色。

A data eng