Here are some raw notes and books I recommend about parallel programming that I used recently for a project. You will find really interesting that the definition of "parallel computing", "cluster", "cloud computing" are extremely loose. Each book will define the terms in a very different way.
Definitions
Parallel computing = program parts running simultaneously on multiple processors in the same computer.
Distributed computing = a form of parallel computing but in multiple computers. Distributed computing differs from cluster computing in that computers in a distributed computing environment are typically not exclusively running "group" tasks, whereas clustered computers are usually much more tightly coupled. Distributed computing also often consists of machines which are widely separated geographically.
Grid computing = uses the resources of many separate computers, loosely connected (needs little or no inter-node communication), by a network usually the Internet. Grid computing is optimized for workloads which consist of many independent jobs or packets of work, which do not have to share data between the jobs during the computation process.
CPU-scavenging, creates a “grid” from the unused resources in a network of participants
Computer cluster = a group of linked computers, working together closely so that in many respects they form a single computer. The components of a cluster are commonly, but not always, connected to each other through fast local area networks
Parallel Virtual Machine = The Parallel Virtual Machine (PVM) is a software tool for parallel networking of computers. It is designed to allow a network of heterogeneous Unix and/or Windows machines to be used as a single distributed parallel processor.
Message Passing Interface (MPI) is a specification for an API that allows many computers to communicate with one another. It is used in computer clusters and supercomputers. MPI's goals are high performance, scalability, and portability. MPI remains the dominant model used in high-performance computing today.
Documentatie: Chapter 1. Introduction to Parallel Programming - http://www.redbooks.ibm.com/redbooks/pdfs/sg245380.pdf
Amdahl's law
The speedup of a program using multiple processors in parallel computing is limited by the time needed for the sequential fraction of the program. For example, if a program needs 20 hours using a single processor core, and a particular portion of 1 hour cannot be parallelized, while the remaining promising portion of 19 hours (95%) can be parallelized, then regardless of how many processors we devote to a parallelized execution of this program, the minimal execution time cannot be less than that critical 1 hour. Hence the speed up is limited up to 20x, as the diagram illustrates.
Beowulf (computer cluster)
Beowulf is a multi-computer architecture which can be used for parallel computations. It is a system which usually consists of one server node, and one or more client nodes connected together via Ethernet or some other network. It is a system built using commodity hardware components, like any PC capable of running a Unix-like operating system, with standard Ethernet adapters, and switches. It does not contain any custom hardware components and is trivially reproducible. Beowulf also uses commodity software like the Linux or Solaris operating system, Parallel Virtual Machine (PVM) and Message Passing Interface (MPI). The server node controls the whole cluster and serves files to the client nodes. It is also the cluster's console and gateway to the outside world. Large Beowulf machines might have more than one server node, and possibly other nodes dedicated to particular tasks, for example consoles or monitoring stations. In most cases client nodes in a Beowulf system are dumb, the dumber the better. Nodes are configured and controlled by the server node, and do only what they are told to do. In a disk-less client configuration, client nodes don't even know their IP address or name until the server tells them what it is.
The typical setup of a beowulf cluster
Parallel computing = program parts running simultaneously on multiple processors in the same computer.
Distributed computing = a form of parallel computing but in multiple computers. Distributed computing differs from cluster computing in that computers in a distributed computing environment are typically not exclusively running "group" tasks, whereas clustered computers are usually much more tightly coupled. Distributed computing also often consists of machines which are widely separated geographically.
Grid computing = uses the resources of many separate computers, loosely connected (needs little or no inter-node communication), by a network usually the Internet. Grid computing is optimized for workloads which consist of many independent jobs or packets of work, which do not have to share data between the jobs during the computation process.
CPU-scavenging, creates a “grid” from the unused resources in a network of participants
Computer cluster = a group of linked computers, working together closely so that in many respects they form a single computer. The components of a cluster are commonly, but not always, connected to each other through fast local area networks
Parallel Virtual Machine = The Parallel Virtual Machine (PVM) is a software tool for parallel networking of computers. It is designed to allow a network of heterogeneous Unix and/or Windows machines to be used as a single distributed parallel processor.
Message Passing Interface (MPI) is a specification for an API that allows many computers to communicate with one another. It is used in computer clusters and supercomputers. MPI's goals are high performance, scalability, and portability. MPI remains the dominant model used in high-performance computing today.
Documentatie: Chapter 1. Introduction to Parallel Programming - http://www.redbooks.ibm.com/redbooks/pdfs/sg245380.pdf
Amdahl's law
The speedup of a program using multiple processors in parallel computing is limited by the time needed for the sequential fraction of the program. For example, if a program needs 20 hours using a single processor core, and a particular portion of 1 hour cannot be parallelized, while the remaining promising portion of 19 hours (95%) can be parallelized, then regardless of how many processors we devote to a parallelized execution of this program, the minimal execution time cannot be less than that critical 1 hour. Hence the speed up is limited up to 20x, as the diagram illustrates.
Beowulf (computer cluster)
Beowulf is a multi-computer architecture which can be used for parallel computations. It is a system which usually consists of one server node, and one or more client nodes connected together via Ethernet or some other network. It is a system built using commodity hardware components, like any PC capable of running a Unix-like operating system, with standard Ethernet adapters, and switches. It does not contain any custom hardware components and is trivially reproducible. Beowulf also uses commodity software like the Linux or Solaris operating system, Parallel Virtual Machine (PVM) and Message Passing Interface (MPI). The server node controls the whole cluster and serves files to the client nodes. It is also the cluster's console and gateway to the outside world. Large Beowulf machines might have more than one server node, and possibly other nodes dedicated to particular tasks, for example consoles or monitoring stations. In most cases client nodes in a Beowulf system are dumb, the dumber the better. Nodes are configured and controlled by the server node, and do only what they are told to do. In a disk-less client configuration, client nodes don't even know their IP address or name until the server tells them what it is.
The typical setup of a beowulf cluster
One of the main differences between Beowulf and a Cluster of Workstations (COW) is the fact that Beowulf behaves more like a single machine rather than many workstations. In most cases client nodes do not have keyboards or monitors, and are accessed only via remote login or possibly serial terminal. Beowulf nodes can be thought of as a CPU + memory package which can be plugged in to the cluster, just like a CPU or memory module can be plugged into a motherboard. Beowulf is not a special software package, new network topology or the latest kernel hack. Beowulf is a technology of clustering computers to form a parallel, virtual supercomputer. Although there are many software packages such as kernel modifications, PVM and MPI libraries, and configuration tools which make the Beowulf architecture faster, easier to configure, and much more usable, one can build a Beowulf class machine using standard Linux distribution without any additional software. If you have two networked computers which share at least the /home file system via Network File System (protocol), and trust each other to execute remote shells (rsh), then it could be argued that you have a simple, two node Beowulf machine.
Parallel programming books
Introduction to Parallel Computing, 2003
Very good
Principles of Concurrent and Distributed Programming, 2006
Principles of Concurrent and Distributed Programming, 2006
http://proquest.safaribooksonline.com/9780321312839
Server Architectures: Multiprocessors, Clusters, Parallel Systems, Web Servers, and Storage Solutions
Price: 27 pounds
2004
Server Architectures: Multiprocessors, Clusters, Parallel Systems, Web Servers, and Storage Solutions
Price: 27 pounds
2004
See part III: Optimizing Parallel Applications
See chapter 14. Optimization of Message-Passing Programs
No comments:
Post a Comment