ID1217 Concurrent Programming 课程笔记 第一讲

课程教材 • Course Material – Book: Foundations of Multithreaded, Parallel and Distributed Programming, by G. R. Andrews, Addison Wesley – Lecture notes and Lab PM – URL: http://www.imit.kth.se/courses/ID1217/ 推荐阅读 • Recommended additional text book – The Art of Multiprocessor Programming, by Maurice Herlihy, and Nir Shavit, Elsevier Science & Technology Books (Morgan Kaufmann Publishers), 2008; ISBN 978-0-12-370591-4 – Principles of Concurrent and Distributed Programming, by M. Ben- Ari, 2-nd edition, Prentice-Hall, 2006, ISBN 0-13-711821-X --------------------------------------------------------------------------- Week 4. Processes and synchronization. Semantics of concurrent execution. Lecture 1 Introduction. Parallel Programming Concepts, Models and Paradigms Sequential Program Versus Parallel Program Sequential program – Represents a sequence of actions that produces a result – Contains a single thread of control – Usually, it is called a task (to be executed by/in a process) • Parallel (concurrent, distributed) program – Contains two or more processes that cooperate in solving a problem • Each process executes a sequential program and has its own state • Processes communicate via shared memory or/and by message passing – Can be executed • Concurrently on a single-processor machine or • In parallel on a multiprocessor or on several computers. Programming with Processes • Concurrent Programming – Several concurrent processes (threads) share a single CPU – Shared memory programming model – Motivations: • Improving performance, utilization and scalability • Modeling of concurrency in the (real) world • Distributed Programming – Processes distributed among computers communicate over network – Message passing programming model – For distributed computing: Users, data, machines are geographically distributed – For high performance (scalable) computing • Parallel Programming – Parallel (concurrent) processes execute on their own processors – Both, shared memory and message passing programming models – For high performance computing : solve a problem faster or/and solve alarger problem 并行执行的硬件实现 Hardware • For multiprogramming and concurrent programming with shared memory – A single-CPU processor • For concurrent and parallel programming with shared memory – Shared-memory multiprocessor • Centralized shared memory (UMA) • Distributed shared memory (NUMA) • For distributed and parallel programming with message passing – Distributed memory multiprocessor – Computer clusters – Computer networks • For distributed and parallel programming with message passing and sh.memory – Hierarchical multiprocessor: distributed memory MP with SMP nodes 第一类 Multiprogramming on a Single Processor The CPU time is shared between several programs by time-slicing • OS controls and schedules processes – If a time slice period has expired or the process blocks for some reason (e.g. IO operation, blocking synchronization operation), OS makes a process (context) switch • Process switching: Suspend the current process and restore a new process: 1. Save the current process state to the memory; Add the process to the tail of the ready queue or to a wait queue; 2. Take a process from the head of the ready queue; Restore the proc state and make it running. • Resume a blocked process: – Move a process from the wait queue to the ready queue. 第二类 Shared-Memory Multiprocessors A shared-memory MP offers a shared address space – Processes executing on different processors have access to a common (shared) memory • Symmetric multiprocessors (SMP, UMA) – Centralized shared memory Memory Bus – Bus – Snoopy cache coherence protocols • Distributed shared memory MP (NUMA) – Shared memory is distributed among nodes – Scalable interconnection network – Directory-based cache coherency protocols 第三类 Multi-Core Processors • Combines several (two or more) independent cores into a single package on the same die. – Tightly-coupled multiprocessor – May share L2 cache or have separate caches – Shared the same interconnect to the rest of the system (memory) • TLP – thread-level-parallelism on the chip • Important R&D areas (by Erik Hagersten, Uppsala Univerisy) – Algorithms – Parallelization – Verification – Modeling/Simulation/Tools – Bandwidth optimizations – Managing data locality – … • “There are no indications auto parallelization will radically improve any time soon.” – Tim Mattson, Intel 第四类 Distributed Memory Multiprocessors and Multi-Computers • There is no HW-supported shared memory – Each node has its own local memory – Processes on different nodes communicates by message passing – Major problem is the communication latency • Massively parallel processors (MPP) and computer clusters – Tightly coupled computers connected with a high-speed interconnection network • Computer networks – Loosely coupled computers (LAN or WAN) 并行编程原理 Parallel Programming Concepts • Task – an arbitrary piece of un-decomposed work in parallel or sequential computation – Executed sequentially; concurrency is only across tasks – Fine-grained versus coarse-grained tasks • Process (thread) – an abstract entity that performs tasks assigned to it – Each process has its state and a unique ID – Processes communicate and synchronize to perform their tasks via shared memory or/and by message passing – Three types of processes: • Heavy-weight processes • Light-weight processes • Threads 并行编程中的元素 Parallel Programming Concepts (cont’d) • A process state (context) includes all information needed to execute the process: – In CPU: contents of PC, NPC, PSW, SP, registers – In memory: contents of text segment, data segment, heap, stack • Can be cached in data and instruction caches • A portion (but not stack) can be shared with other processes • A context switch – terminate or suspend the current process and start or resume another process • Performed by an OS kernel • The CPU state of the current process must be saved to the memory • “Dirty” cache lines can be written back to memory on replacement • Processor – a physical engine on which processes execute – Processes virtualize machine to programmer • First write program in term of processes, then map to processors • Concurrent program (execution) is formed of several processes (threads) that share a (multi)processor • Usually more processes than processors • Parallel program (execution) is formed of several processes each executes on its own processor • Usually more processors than processes • Distributed program (execution) is formed of processes that are distributed among processors and communicate over network • No shared memory 重量级进程 Heavy-Weight Processes (HWP) • A heavy-weight process has its own virtual address space not shared with other HWPs – Entire process state is private – A HW process can be multithreaded – A HWP can create another HWP (a copy of itself or a different process) ------------------------------------------------------------------------------------- 进入正题 Threads • A thread is essentially a program counter, an execution stack, and a set of registers – thread context. – All the other data structures and the code belong to a HWP where threads are created, and are shared by the threads. – Each thread is assigned a unique thread ID. • Example: Pthreads – POSIX (IEEE Portable OS Interface) threads – pthread_create creates a new thread • The thread executes a given function, has its own stack and registers, but share global variables with other threads. – Threads can be “joined” by pthread_join in parent and return or pthread_exit() in child 并行编程模式 Parallel Programming Models • A programming model defines how processes communicate and synchronize • Shared memory programming model, a.k.a. Shared address space (SAS) model. • Portions of virtual address spaces of processes are shared (common) • Processes communicate via shared memory (common variables) by conventional reads and writes • Explicit synchronization mechanisms (libraries) such as locks, barriers, semaphores, monitors,in an imperative (sequential) language, e.g. C or Java. • Distributed memory programming model, a.k.a. Message passing programming model • No shared memory, only communication channels are shared • Processes communicate by sending/receiving messages over the channels. • Synchronization is implicit: a message to be received must be sent 同步与通信机制 Existing Synchronization and Communication Mechanisms • Synchronization mechanisms – for shared memory programming – Locks – Barriers – Condition variables – Semaphores – Monitors • Communication mechanisms – for distributed memory programming – Asynchronous message passing – Synchronous message passing (e.g. CSP: Communicating Sequential Processes) – Remote procedure call (RPC) and Rendezvous – Remote method invocation (RMI)