[tsubame]

node architecture

IOH: Input-Output Hub. data communication between a CPU and a mother board.
QPI: QuickPath Interconnect. point-to-point processor interconnect
PCIe: Peripheral Component Interconnect Express. high-speed serial computer expansion bus standard.
PCEe 2.0x16. 2.0是总线接口的规范版本号. 16是总线接口的位宽-the amount of lanes the slot has.
GPU1&GPU2在同一个clipset(mother board)上.

caffe parallel part:
–without P2P access, for example, crossing PCIe root comples, data is copied through host and effective exchage bandwidth is greatly reduced.
–You can do through PCIe bridges, but not across socket level links at this time. e.g. across CPU sockets on a multi-socket mother board.

storage

filesystem

Lustre system is set available, besides the available home space in TSUBAME 2.5.

“/data0” has been being newly prepared as GPFS area since the spring of 2011. “/work0” can achieve a high throughput performance even by a large-scale and parallel application of the HPC simulation. “data0” composes a hierarchical filesystem that synchronizes with the tape library system. Therefore, it is suitable for use by a data that needs the data of the TB scale intensive application.

data

每个node有各自的SSD, 彼此间不能访问.
每次提交一个job给某个node,这个node会从你的directory(work0/,work1/,local directory)下读数据,但是这样很慢,所以可以用scratch.

4.6.4 Global Scratch
Global scratch consists of parallel file system (Lustre) is available on all the nodes. The environmental variables $TMPDIR is set to the global scratch, and you might set it as workspace in a shell script as follows.

4.6.5 Local scratch
High-speed scratch SSD is available as local scratch, and it is used as stagein/stageout.

//to be continued