Concurrent, Parallel, Distributed

CONCURRENT, PARALLEL, DISTRIBUTED

If you look at some other books on parallel computing or you search the Web for information on parallel computing, you’re likely to also run across the terms concurrent computing and distributed computing. Although there isn’t complete agreement on the distinction between the terms parallel, distributed, and concurrent, many authors make the following distinctions:

. In concurrent computing, a program is one in which multiple tasks can be in progress

at any instant [4].

In parallel computing, a program is one in which multiple tasks cooperate closely

to solve a problem.

In distributed computing, a program may need to cooperate with other programs to solve a problem.

So parallel and distributed programs are concurrent, but a program such as a mul-titasking operating system is also concurrent, even when it is run on a machine with only one core, since multiple tasks can be in progress at any instant. There isn’t a clear-cut distinction between parallel and distributed programs, but a parallel program usually runs multiple tasks simultaneously on cores that are physically close to each other and that either share the same memory or are connected by a very high-speed network. On the other hand, distributed programs tend to be more “loosely coupled.” The tasks may be executed by multiple computers that are separated by large dis-tances, and the tasks themselves are often executed by programs that were created independently. As examples, our two concurrent addition programs would be con-sidered parallel by most authors, while a Web search program would be considered distributed.

But beware, there isn’t general agreement on these terms. For example, many authors consider shared-memory programs to be “parallel” and distributed-memory programs to be “distributed.” As our title suggests, we’ll be interested in par-allel programs—programs in which closely coupled tasks cooperate to solve a problem.