What is the difference between Rack-local map task

最新推荐文章于 2021-02-25 10:25:33 发布

weixin_34082789

最新推荐文章于 2021-02-25 10:25:33 发布

阅读量97

点赞数

文章标签： python

原文链接：https://my.oschina.net/forrest420/blog/185938

版权

2019独角兽企业重金招聘Python工程师标准>>>

In a data-local task, nothing needs to be copied. That's because the block is physically on the same server like the computation.

The next tier is the rack-local task, here the data must be copied, because there is no local copy of the desired block available. Note that rack-local does only copy within the rack-local switching of the network.

There is also the worst case, where the data isn't available local, nor on the same rack. So this must be copied over two switches to the hosts where the computation runs. I don't know if there is a counter for that, but basically this must be #all tasks - #data-local tasks - #rack-local tasks.

转载于:https://my.oschina.net/forrest420/blog/185938