8 Real Applications
One group of 14 busy clusters with 8069 total tablet servers saw an aggregate volume of more than 1.2 million requests per second, with incoming RPC traffic of about 741 MB/s and outgoing RPC traffic of about 16 GB/s.
the tables range widely in total size, average cell size, percentage of data served from memory, and complexity of the table schema.
In the rest of this section, we briefly describe how three product teams use Bigtable.
8实际应用
截至2006年8月，共有388个未测试的BigTable集群在各种Google机器集群中运行，共计约24,500个 tablet 服务器。
表1显示了每个群集的 tablet 服务器的粗略分布。
这些集群中的许多都被用于开发目的，因此在很长时间内闲置。
一组14个繁忙的集群，总共有8069个 tablet 服务器，每秒总共需要超过120万个请求，传入的RPC流量约为741 MB/s，出货RPC流量约为16 GB/s。
表2提供了有关当前使用的几个表的一些数据。
一些表存储为用户提供的数据，而其他表存储用于批处理的数据;
这些表的范围广泛在总大小，平均单元大小，从内存服务的数据的百分比以及表模式的复杂性。
在本节的其余部分，我们简要介绍三个产品团队如何使用BigTable。
8.1 Google Analytics
Google Analytics (analytics.google.com) is a service that helps webmasters analyze traffic patterns at their web sites.
It provides aggregate statistics, such as the number of unique visitors per day and the page views per URL per day, as well as site-tracking reports, such as the percentage of users that made a purchase, given that they earlier viewed a specific page.
To enable the service, webmasters embed a small JavaScript program in their web pages.
This program is invoked whenever a page is visited.
It records various information about the request in Google Analytics, such as a user identifier and information about the page being fetched.
Google Analytics summarizes this data and makes it available to webmasters.
We briefly describe two of the tables used by Google Analytics.
The raw click table ( ̃200 TB) maintains a row for each end-user session.
The row name is a tuple containing the website’s name and the time at which the session was created.
This schema ensures that sessions that visit the same web site are contiguous, and that they are sorted chronologically.
This table compresses to 14% of its original size.
The summary table ( ̃20 TB) contains various predefined summaries for each website.
This table is generated from the raw click table by periodically scheduled MapReduce jobs.
Each MapReduce job extracts recent session data from the raw click table.
The overall system’s throughput is limited by the throughput of GFS.
This table compresses to 29% of its original size.
8.1 Google Analytics（分析）
Google Analytics（分析）是一种可帮助网站管理员分析其网站流量模式的服务。
它提供汇总统计信息，例如每天的唯一身份访问者数量和每个网址每天的网页浏览量，以及网站跟踪报告（例如，进行购买的用户的百分比），因为他们早先查看了特定网页。
要启用该服务，网站管理员会在其网页中嵌入一个小型JavaScript程序。
每当访问页面时调用此程序。
它记录有关Google Analytics（分析）中的请求的各种信息，例如用户标识符和有关正在获取的页面的信息。
Google Analytics（分析）总结了这些数据，并将其提供给网站管理员。
我们简要介绍了Google Analytics（分析）使用的两个表格。
原始点击表（200 TB）为每个最终用户会话维护一行。
行名称是一个包含网站名称和创建会话时间的元组。
此模式确保访问同一网站的会话是连续的，并且它们按时间顺序排序。
该表压缩到其原始尺寸的14％。
汇总表（20 TB）包含各个网站的各种预定义摘要。
此表是通过定期安排的MapReduce作业从原始点击表生成的。
每个MapReduce作业从原始点击表中提取最近的会话数据。
整个系统的吞吐量受到GFS吞吐量的限制。
该表压缩到其原始尺寸的29％。
8.2 Google Earth
Google operates a collection of services that provide users with access to high-resolution satellite imagery of the world’s surface, both through the web-based Google Maps interface (maps.google.com) and through the Google Earth (earth.google.com) custom client software.
These products allow users to navigate across the world’s surface: they can pan, view, and annotate satellite imagery at many different levels of resolution.
This system uses one table to preprocess data, and a different set of tables for serving client data.
The preprocessing pipeline uses one table to store raw imagery.
During preprocessing, the imagery is cleaned and consolidated into final serving data.
This table contains approximately 70 terabytes of data and therefore is served from disk.
The images are efficiently compressed already, so Bigtable compression is disabled.
Table 2: Characteristics of a few tables in production use.
Table size (measured before compression) and # Cells indicate approximate sizes.
Compression ratio is not given for tables that have compression disabled.
Each row in the imagery table corresponds to a single geographic segment.
Rows are named to ensure that adjacent geographic segments are stored near each other.
The table contains a column family to keep track of the sources of data for each segment.
This column family has a large number of columns: essentially one for each raw data image.
Since each segment is only built from a few images, this column family is very sparse.
The preprocessing pipeline relies heavily on MapReduce over Bigtable to transform data.
The overall system processes over 1 MB/sec of data per tablet server during some of these MapReduce jobs.
The serving system uses one table to index data stored in GFS.
This table is relatively small ( ̃500 GB), but it must serve tens of thousands of queries per second per datacenter with low latency.
As a result, this table is hosted across hundreds of tablet servers and contains in-memory column families.
8.2 Google地球
Google通过基于网络的Google地图界面（maps.google.com）和Google地球（earth.google.com），为用户提供了世界各地高分辨率卫星图像的服务，定制客户端软件。
这些产品允许用户浏览世界各地的表面：他们可以以许多不同的分辨率平移，查看和注释卫星图像。
该系统使用一个表来预处理数据，以及一组用于提供客户端数据的不同表。
预处理管道使用一个表来存储原始图像。
在预处理期间，图像被清理并整合成最终的投放数据。
该表包含大约70 TB的数据，因此从磁盘提供。
图像已被有效地压缩，因此禁用Bigtable压缩。
表2：生产使用中几张表的特征。
表尺寸（压缩前测量）和＃单元格表示近似尺寸。
没有给压缩比压缩禁用的表。
图像表中的每一行对应于单个地理区域。
命名行以确保相邻的地理区段彼此靠近存储。
该表包含一个列系列，用于跟踪每个段的数据源。
此列系列具有大量列：对于每个原始数据图像，基本上为一列。
由于每个段仅由几个图像构建，所以该列族是非常稀疏的。
预处理流水线严重依赖于BigTable中的MapReduce来转换数据。
在某些MapReduce作业期间，整个系统处理每个 tablet 服务器超过1 MB/秒的数据。
服务系统使用一个表来索引存储在GFS中的数据。
此表格相对较小（500 GB），但每个数据中心每秒可能需要数万次查询，具有低延迟。
因此，该表托管在数百个 tablet 服务器上，并包含内存中的列系列。
8.3 Personalized Search
Personalized Search (www.google.com/psearch) is an opt-in service that records user queries and clicks across a variety of Google properties such as web search, images, and news.
Users can browse their search histories to revisit their old queries and clicks, and they can ask for personalized search results based on their historical Google usage patterns.
Personalized Search stores each user’s data in Bigtable.
Each user has a unique userid and is assigned a row named by that userid.
All user actions are stored in a table.
A separate column family is reserved for each type of action (for example, there is a column family that stores all web queries).
Each data element uses as its Bigtable timestamp the time at which the corresponding user action occurred.
Personalized Search generates user profiles using a MapReduce over Bigtable.
These user profiles are used to personalize live search results.
The Personalized Search data is replicated across several Bigtable clusters to increase availability and to reduce latency due to distance from clients.
The Personalized Search team originally built a client-side replication mechanism on top of Bigtable that ensured eventual consistency of all replicas.
The current system now uses a replication subsystem that is built into the servers.
The design of the Personalized Search storage system allows other groups to add new per-user information in their own columns, and the system is now used by many other Google properties that need to store per-user configuration options and settings.
Sharing a table amongst many groups resulted in an unusually large number of column families.
To help support sharing, we added a simple quota mechanism to Bigtable to limit the storage consumption by any particular client in shared tables; this mechanism provides some isolation between the various product groups using this system for per-user information storage.
8.3个性化搜索
个性化搜索（www.google.com/psearch）是一种选择性服务，用于记录用户查询和点击各种Google资源（如网络搜索，图像和新闻）。
用户可以浏览他们的搜索历史记录来重新访问他们的旧查询和点击次数，他们可以根据Google的历史使用模式要求个性化搜索结果。
个性化搜索将每个用户的数据存储在Bigtable中。
每个用户都有一个唯一的用户标识，并分配一个由该用户标识的行。
所有用户操作都存储在表中。
为每种类型的操作保留单独的列系列（例如，存在所有Web查询的列系列）。
每个数据元素使用相应用户操作发生的时间作为其Bigtable时间戳。
个性化搜索使用MapReduce通过BigTable生成用户配置文件。
这些用户配置文件用于个性化实时搜索结果。
个性化搜索数据在多个Bigtable集群中复制，以增加可用性并减少由于与客户端的距离而造成的延迟。
个性化搜索团队最初在Bigtable之上构建了一个客户端复制机制，确保所有副本的最终一致性。
当前系统现在使用内置在服务器中的复制子系统。
个性化搜索存储系统的设计允许其他组在其自己的列中添加新的每用户信息，并且系统现在被许多其他Google属性使用，需要存储每个用户的配置选项和设置。
在许多团体之间分享一张桌子导致了非常多的列家庭。
为了帮助支持共享，我们在Bigtable中添加了一个简单的配额机制来限制共享表中任何特定客户端的存储消耗;
这种机制为使用该系统的各个用户信息存储的各种产品组之间提供了一些隔离。
As of August 2006, there are 388 non-test Bigtable clusters running in various Google machine clusters, with a combined total of about 24,500 tablet servers.
Table 1 shows a rough distribution of tablet servers per cluster.
One group of 14 busy clusters with 8069 total tablet servers saw an aggregate volume of more than 1.2 million requests per second, with incoming RPC traffic of about 741 MB/s and outgoing RPC traffic of about 16 GB/s.
Table 2 provides some data about a few of the tables currently in use.
the tables range widely in total size, average cell size, percentage of data served from memory, and complexity of the table schema.
In the rest of this section, we briefly describe how three product teams use Bigtable.
8实际应用
截至2006年8月，共有388个未测试的BigTable集群在各种Google机器集群中运行，共计约24,500个 tablet 服务器。
表1显示了每个群集的 tablet 服务器的粗略分布。
这些集群中的许多都被用于开发目的，因此在很长时间内闲置。
一组14个繁忙的集群，总共有8069个 tablet 服务器，每秒总共需要超过120万个请求，传入的RPC流量约为741 MB/s，出货RPC流量约为16 GB/s。
表2提供了有关当前使用的几个表的一些数据。
一些表存储为用户提供的数据，而其他表存储用于批处理的数据;
这些表的范围广泛在总大小，平均单元大小，从内存服务的数据的百分比以及表模式的复杂性。
在本节的其余部分，我们简要介绍三个产品团队如何使用BigTable。
8.1 Google Analytics
Google Analytics (analytics.google.com) is a service that helps webmasters analyze traffic patterns at their web sites.
It provides aggregate statistics, such as the number of unique visitors per day and the page views per URL per day, as well as site-tracking reports, such as the percentage of users that made a purchase, given that they earlier viewed a specific page.
To enable the service, webmasters embed a small JavaScript program in their web pages.
This program is invoked whenever a page is visited.
It records various information about the request in Google Analytics, such as a user identifier and information about the page being fetched.
Google Analytics summarizes this data and makes it available to webmasters.
We briefly describe two of the tables used by Google Analytics.
The raw click table ( ̃200 TB) maintains a row for each end-user session.
The row name is a tuple containing the website’s name and the time at which the session was created.
This schema ensures that sessions that visit the same web site are contiguous, and that they are sorted chronologically.
This table compresses to 14% of its original size.
The summary table ( ̃20 TB) contains various predefined summaries for each website.
This table is generated from the raw click table by periodically scheduled MapReduce jobs.
Each MapReduce job extracts recent session data from the raw click table.
The overall system’s throughput is limited by the throughput of GFS.
This table compresses to 29% of its original size.
8.1 Google Analytics（分析）
Google Analytics（分析）是一种可帮助网站管理员分析其网站流量模式的服务。
它提供汇总统计信息，例如每天的唯一身份访问者数量和每个网址每天的网页浏览量，以及网站跟踪报告（例如，进行购买的用户的百分比），因为他们早先查看了特定网页。
要启用该服务，网站管理员会在其网页中嵌入一个小型JavaScript程序。
每当访问页面时调用此程序。
它记录有关Google Analytics（分析）中的请求的各种信息，例如用户标识符和有关正在获取的页面的信息。
Google Analytics（分析）总结了这些数据，并将其提供给网站管理员。
我们简要介绍了Google Analytics（分析）使用的两个表格。
原始点击表（200 TB）为每个最终用户会话维护一行。
行名称是一个包含网站名称和创建会话时间的元组。
此模式确保访问同一网站的会话是连续的，并且它们按时间顺序排序。
该表压缩到其原始尺寸的14％。
汇总表（20 TB）包含各个网站的各种预定义摘要。
此表是通过定期安排的MapReduce作业从原始点击表生成的。
每个MapReduce作业从原始点击表中提取最近的会话数据。
整个系统的吞吐量受到GFS吞吐量的限制。
该表压缩到其原始尺寸的29％。
8.2 Google Earth
Google operates a collection of services that provide users with access to high-resolution satellite imagery of the world’s surface, both through the web-based Google Maps interface (maps.google.com) and through the Google Earth (earth.google.com) custom client software.
These products allow users to navigate across the world’s surface: they can pan, view, and annotate satellite imagery at many different levels of resolution.
This system uses one table to preprocess data, and a different set of tables for serving client data.
The preprocessing pipeline uses one table to store raw imagery.
During preprocessing, the imagery is cleaned and consolidated into final serving data.
This table contains approximately 70 terabytes of data and therefore is served from disk.
The images are efficiently compressed already, so Bigtable compression is disabled.
Table 2: Characteristics of a few tables in production use.
Table size (measured before compression) and # Cells indicate approximate sizes.
Compression ratio is not given for tables that have compression disabled.
Each row in the imagery table corresponds to a single geographic segment.
Rows are named to ensure that adjacent geographic segments are stored near each other.
The table contains a column family to keep track of the sources of data for each segment.
This column family has a large number of columns: essentially one for each raw data image.
Since each segment is only built from a few images, this column family is very sparse.
The preprocessing pipeline relies heavily on MapReduce over Bigtable to transform data.
The overall system processes over 1 MB/sec of data per tablet server during some of these MapReduce jobs.
The serving system uses one table to index data stored in GFS.
This table is relatively small ( ̃500 GB), but it must serve tens of thousands of queries per second per datacenter with low latency.
As a result, this table is hosted across hundreds of tablet servers and contains in-memory column families.
8.2 Google地球
Google通过基于网络的Google地图界面（maps.google.com）和Google地球（earth.google.com），为用户提供了世界各地高分辨率卫星图像的服务，定制客户端软件。
这些产品允许用户浏览世界各地的表面：他们可以以许多不同的分辨率平移，查看和注释卫星图像。
该系统使用一个表来预处理数据，以及一组用于提供客户端数据的不同表。
预处理管道使用一个表来存储原始图像。
在预处理期间，图像被清理并整合成最终的投放数据。
该表包含大约70 TB的数据，因此从磁盘提供。
图像已被有效地压缩，因此禁用Bigtable压缩。
表2：生产使用中几张表的特征。
表尺寸（压缩前测量）和＃单元格表示近似尺寸。
没有给压缩比压缩禁用的表。
图像表中的每一行对应于单个地理区域。
命名行以确保相邻的地理区段彼此靠近存储。
该表包含一个列系列，用于跟踪每个段的数据源。
此列系列具有大量列：对于每个原始数据图像，基本上为一列。
由于每个段仅由几个图像构建，所以该列族是非常稀疏的。
预处理流水线严重依赖于BigTable中的MapReduce来转换数据。
在某些MapReduce作业期间，整个系统处理每个 tablet 服务器超过1 MB/秒的数据。
服务系统使用一个表来索引存储在GFS中的数据。
此表格相对较小（500 GB），但每个数据中心每秒可能需要数万次查询，具有低延迟。
因此，该表托管在数百个 tablet 服务器上，并包含内存中的列系列。
8.3 Personalized Search
Personalized Search (www.google.com/psearch) is an opt-in service that records user queries and clicks across a variety of Google properties such as web search, images, and news.
Users can browse their search histories to revisit their old queries and clicks, and they can ask for personalized search results based on their historical Google usage patterns.
Personalized Search stores each user’s data in Bigtable.
Each user has a unique userid and is assigned a row named by that userid.
All user actions are stored in a table.
A separate column family is reserved for each type of action (for example, there is a column family that stores all web queries).
Each data element uses as its Bigtable timestamp the time at which the corresponding user action occurred.
Personalized Search generates user profiles using a MapReduce over Bigtable.
These user profiles are used to personalize live search results.
The Personalized Search data is replicated across several Bigtable clusters to increase availability and to reduce latency due to distance from clients.
The Personalized Search team originally built a client-side replication mechanism on top of Bigtable that ensured eventual consistency of all replicas.
The current system now uses a replication subsystem that is built into the servers.
The design of the Personalized Search storage system allows other groups to add new per-user information in their own columns, and the system is now used by many other Google properties that need to store per-user configuration options and settings.
Sharing a table amongst many groups resulted in an unusually large number of column families.
To help support sharing, we added a simple quota mechanism to Bigtable to limit the storage consumption by any particular client in shared tables; this mechanism provides some isolation between the various product groups using this system for per-user information storage.
8.3个性化搜索
个性化搜索（www.google.com/psearch）是一种选择性服务，用于记录用户查询和点击各种Google资源（如网络搜索，图像和新闻）。
用户可以浏览他们的搜索历史记录来重新访问他们的旧查询和点击次数，他们可以根据Google的历史使用模式要求个性化搜索结果。
个性化搜索将每个用户的数据存储在Bigtable中。
每个用户都有一个唯一的用户标识，并分配一个由该用户标识的行。
所有用户操作都存储在表中。
为每种类型的操作保留单独的列系列（例如，存在所有Web查询的列系列）。
每个数据元素使用相应用户操作发生的时间作为其Bigtable时间戳。
个性化搜索使用MapReduce通过BigTable生成用户配置文件。
这些用户配置文件用于个性化实时搜索结果。
个性化搜索数据在多个Bigtable集群中复制，以增加可用性并减少由于与客户端的距离而造成的延迟。
个性化搜索团队最初在Bigtable之上构建了一个客户端复制机制，确保所有副本的最终一致性。
当前系统现在使用内置在服务器中的复制子系统。
个性化搜索存储系统的设计允许其他组在其自己的列中添加新的每用户信息，并且系统现在被许多其他Google属性使用，需要存储每个用户的配置选项和设置。
在许多团体之间分享一张桌子导致了非常多的列家庭。
为了帮助支持共享，我们在Bigtable中添加了一个简单的配额机制来限制共享表中任何特定客户端的存储消耗;
这种机制为使用该系统的各个用户信息存储的各种产品组之间提供了一些隔离。
7041
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
