Multiprocess Programming

Multiprocess Programming

An alternative to multithreaded applications is multiprocess applications, as suggested in Chapter 1, “Hardware, Processes, and Threads.” The main advantage of multiprocess pro-gramming is that a failure of one process does not cause all the processes to die, and as a result, it might be possible to recover from such failures.

It is very easy to start multiple processes and have these processes load common initialization parameters from a file or from the command line to start communicating. However, it is often useful to do this programmatically.

The UNIX model of process creation is the Fork-Exec model. The fork() call makes a child process that receives an exact duplicate of the parent’s memory. The exec() call replaces the current process with a new executable. The calls often go together so that one application calls fork() to create a new process; then the child process immediately calls exec() to replace itself with a new executable.

The fork() call is interesting because both the child and parent processes will return from this call. The only difference is the return value of the call. The child process will return with a value of zero, and the parent process will return with the process ID of the child process.

The code in Listing 5.55 uses fork() to create a new child process. The child process will execute a sleep command with a parameter of 10 so that it will sleep for ten sec-onds. The parent process will wait for the child to terminate and then report the exit sta-tus of the child process. The execl() call is used to execute the sleep command. The execl() call takes the path to the executable plus the arguments to be passed to that executable, and the first argument should be the name of the executable itself.

Listing 5.55   Using Fork to Create a Child Process

#include <unistd.h> #include <stdio.h> #include <sys/wait.h>

int main()

{

int status;

pid_t f = fork();

if ( f == 0 )

{ /* Child process */

execl( "/usr/bin/sleep", "/usr/bin/sleep", "10" );

}

else

{

waitpid( f, &status, 0 );

printf( "Status = %i\n", status );

}

}

If the fork() call is not followed by an exec() call, we have two copies of the same process. The process state, up to the point at which the fork call was made, is duplicated in both processes. We will see how this can be useful in the following sections on setting up communications between multiple processes.

Sharing Memory Between Processes

Usually different processes share nothing; however, it is possible to set up multiple processes to share the same memory. The shm_open() call creates a shared memory seg-ment. This call takes a name for the segment, a size, and a set of flags for the created shared memory, together with the permission bits. The return value from the call is a file descriptor of the shared memory segment.

The name is a string with the first character as /, and no subsequent slashes. The flags used by the call are familiar from creating files. O_RDONLY will create a read-only seg-ment. O_RDWR will create a segment that permits reading and writing. O_CREAT will create the segment if it does not exist or return a handle to it if it does exist. O_EXCL will return an error if the segment already exists. The final parameters are the file access permissions.

The memory segment will be created, if it does not already exist, with a size of 0 bytes. The size of the reserved segment can be set with a call to ftruncate(), passing the file descriptor of the segment together with the requested size.

Once the segment exists, the process can attach to it with a call to mmap(). Table 5.1 shows the parameters to mmap.

Table 5.1      Parameters Passed to mmap()

The return value of the call to mmap() is a pointer to the shared memory segment. Once the process has finished with the shared memory segment, it can unmap it from its address space using the munmap() call, which takes a pointer to the memory region together with its size as parameters.

The shared memory can be removed from the system with a call to shm_unlink(), which takes the name for the shared region originally given to shm_open(). This causes the shared memory to be deleted when the last process detaches from it.

Listing 5.56 shows an example of creating using and deleting shared memory. When compiling this code on Solaris, it is necessary to define at least _POSIX_C_SOURCE= 199309L in order to get the header files to define shm_open() and shm_unlink().

Listing 5.56   Creating, Using, and Deleting Shared Memory

#include <sys/mman.h> #include <fcntl.h> #include <unistd.h>

int main()

{

int handle = shm_open( "/shm", O_CREAT|O_RDWR, 0777 ); ftruncate( handle, 1024*1024*sizeof(int) );

char * mem = (char*) mmap( 0, 1024*1024*sizeof(int), PROT_READ|PROT_WRITE, MAP_SHARED, handle, 0 );

for( int i=0; i<1024*1024; i++ ) { mem[i] = 0; }

munmap( mem, 1024*1024*sizeof(int) );

shm_unlink( "/shm" );

}

One use for shared memory is as a location for placing mutexes shared between processes. Listing 5.57 illustrates how a process can form a child process and share a mutex with the child process.

Listing 5.57   Sharing a Mutex Between Processes

#include <sys/mman.h> #include <sys/wait.h> #include <fcntl.h> #include <unistd.h> #include <stdio.h> #include <pthread.h>

int main()

{

pthread_mutex_t * mutex; pthread_mutexattr_t attributes; pthread_mutexattr_init( &attributes );

pthread_mutexattr_setpshared( &attributes, PTHREAD_PROCESS_SHARED );

int handle = shm_open( "/shm", O_CREAT|O_RD_WR, 0777 );

ftruncate( handle, 1024*sizeof(int) );

char * mem = mmap( 0, 1024*sizeof(int), PROT_READ|PROT_WRITE, MAP_SHARED, handle,0 );

mutex = (pthread_mutex_t*)mem;

pthread_mutex_init( mutex, &attributes );

pthread_mutexattr_destroy( &attributes );

int ret = 0;

int * pcount = (int*)( mem + sizeof(pthread_mutex_t) ); *pcount = 0;

pid_t pid = fork(); if (pid == 0)

{  /* Child process */

pthread_mutex_lock( mutex );

(*pcount)++;

pthread_mutex_unlock( mutex );

ret = 57;

}

else

{

int status;

waitpid( pid, &status, 0 );

printf( "Child returned %i\n", WEXITSTATUS(status) );

pthread_mutex_lock( mutex );

(*pcount)++;

pthread_mutex_unlock( mutex );

printf( "Count = %i\n", *pcount ); pthread_mutex_destroy( mutex );

}

munmap( mem, 1024*sizeof(int) ); shm_unlink( "/shm" );

return ret;

}

The first thing the parent process does is to set up a mutex that is shared between the parent and child processes. Once this is complete, the parent process forks a child process. The parent process then waits for the child to complete.

When the child process is forked, it receives a copy of the memory of the parent process. This gives it access to the shared memory segment as well as the mutex and vari-ables contained in the shared memory segment. The child process acquires the mutex, increments the shared variable, and releases the mutex before unmapping and unlinking the shared memory segment and exiting.

Once the child process has exited, the parent process continues to execute, obtaining the return value of the child process from the call to waitpid(). The macro WEXITSTATUS converts the exit status from waitpid() into the return value from the child process.

The parent process also acquires the mutex and increments the shared variable before releasing the mutex. It then prints the value of the shared variable, which has the value two, indicating that both the parent and the child process incremented it. The final actions of the parent process are to destroy the mutex and then unmap and unlink the shared memory segment.

One thing to pay attention to is the alignment of objects created in shared memory segments. Depending on the operating system, there may be constraints on the align-ment. For example, Solaris requires that mutexes are aligned on 8-byte boundaries.

Sharing Semaphores Between Processes

As discussed in the section on semaphores, it is very easy to create a named semaphore that is shared between multiple processes. Listing 5.58 shows a parent process creating a child process. Both the parent and child process open the same semaphore, and this sem-aphore is used to ensure that the child process completes before the parent process.

Listing 5.58   Sharing a Named Semaphore

#include <unistd.h> #include <stdio.h> #include <semaphore.h>

int main()

{

int status; pid_t f = fork(); sem_t * semaphore;

semaphore = sem_open( "/my_semaphore", O_CREAT, 0777, 1 );

if ( f == 0 )

{ /* Child process */

printf( "Child process completed\n" ); sem_post( semaphore );

sem_close( semaphore );

}

else

{

sem_t * semaphore;

sem_wait( semaphore );

printf( "Parent process completed\n" );

sem_close( semaphore );

sem_unlink( "/my_semaphore" );

}

}

Message Queues

Message queues are a method of passing messages between threads or processes. Messages can be placed into a queue, and they will be read out of the queue in a prioritized first-in, first-out order.

To attach to a message queue, the mq_open() function needs to be called, which will return a handle to the message queue. This takes a minimum of the name of the message queue and some flags. The name of the message queue should start with a / and then be up to 13 further characters with no additional / characters. The flags need to be one of O_RDONLY, O_WRONLY, or O_RDWR to produce a read-only message queue, a write-only message queue, or a read-write message queue.

The open specified in this way will open an existing message queue or fail if the message queue does not exist. If the additional flag O_CREAT is passed, then the call will create the message queue if it does not already exist. The additional flag O_EXCL can be passed if the call to open the queue should succeed only if the message queue does not exist. If the flag O_CREAT is passed, the call to mq_open() requires two more parameters, a mode setting that is used to set the access permissions for the message queue and a pointer to the attributes for the message queue; if this pointer is null, default values are used for the message queue attributes.

The attributes for a message queue are held in an mq_attr structure, which contains the fields mq_maxmsg, which is the maximum number of messages that can be held in the message queue, and mq_msgsize, which is the maximum size of the messages that can be stored in the message queue.

The other flag that can be passed to mq_open() is O_NONBLOCK. If this flag is set, any attempts to write to a full message queue or read from an empty one will fail and return immediately. The default is for the thread to block until the message queue has space for sending an additional message, or has a message in it.

The function mq_close(), which takes the handle to a message queue as a parame-ter, will close the connection to the message queue. The message queue will continue to exist. To remove the message queue from the system, it is necessary to call mq_unlink(), which takes the name of the message queue as a parameter. After a call to mq_unlink(), the message queue will be removed once all current references to the message queue have been closed.

The code in Listing 5.59 demonstrates creating and deleting a message queue.

Listing 5.59   Opening and Closing a Message Queue

#include <mqueue.h>

int main()

{

mq_attr attr;

mqd_t mqueue;

attr.mq_maxmsg = 1000;

attr.mq_msgsize = 500;

mqueue = mq_open( "/messages", O_RDWR+O_CREAT, 0777, &attr );

...

mq_close( mqueue ); mq_unlink( "/messages" );

}

Message queues are prioritized first-in, first-out (FIFO) queues. Messages are added to the queue using the call mq_send() and received from the queue using the call mq_receive(). If the message queue has been created with the O_NONBLOCK attribute, then these functions will return immediately whether or not they were successful. There are versions of these functions available, which will return after a timeout if they are unsuccessful; these are mq_timedsend() and mq_timedreceive(), which take an absolute time, and mq_reltimedsend_np() and mq_reltimedreceive_np(), which take a relative time.

The parameters to mq_send() are the message queue, a pointer to a buffer contain-ing the message, the size of the message, and a priority. Messages with a higher priority will be placed before messages with a lower priority and after messages with the same or higher priorities. The message is copied from the buffer into the queue. The call to mq_send() will fail if the message size is greater than the mq_msgsize attribute for the queue.

The call to mq_receive() takes the message queue, a pointer to a buffer where the message can be copied, the size of the buffer, and either a null pointer or a pointer to an unsigned int where the priority of the message will be written. If the size of the buffer is smaller than the size of the message, then the call will fail.

Note that sending messages requires at least two copy operations to be performed on the message—once to copy it into the queue and once to copy it out of the queue. Hence, it is advantageous to send short messages and perhaps use shared memory to pass longer items of information.

Listing 5.60 shows an example of sending messages between a parent and child process. On Solaris, the code will need to be linked with the real-time extensions library using the compiler flag -lrt.

Listing 5.60   Passing Messages Between a Parent Process and a Child Process

#include <unistd.h> #include <stdio.h> #include <mqueue.h> #include <string.h>

int main()

{

int status; pid_t f = fork();

if ( f == 0 )

{ /* Child process */ mqd_t * queue; char message[20];

queue = mq_open( "/messages", O_WRONLY+O_CREAT, 0777, 0 ); strncpy( message, "Hello", 6 );

printf( "Send message %s\n", message ); mq_send( queue, message, strlen(message)+1, 0 ); mq_close( queue );

printf( "Child process completed\n" );

}

else

{

mqd_t * queue; char message[2000];

queue = mq_open( "/messages", O_RDONLY+O_CREAT, 0777, 0 );

mq_receive( queue, message, 2000, 0 );

printf( "Receive message %s\n", message );

mq_close( queue );

mq_unlink( "/messages" );

printf( "Parent process completed\n" );

}

}

Both the child and the parent process open the message queue with the O_CREAT flag, meaning that the queue will be created if it does not already exist. The parent process then waits for a message from the child. The child sends a message and then closes its connection to the message queue. The parent receives the message and then closes the connection to the message queue and deletes the queue.

Pipes and Named Pipes

A pipe is a connection between two processes. It can be either an anonymous pipe between two processes or a named pipe, which uses an entity in the file system for communication between processes or threads. The pipe is a streamed first-in, first-out structure.

A named pipe is created by a call to pipe() typically before the child process forks. The pipe call creates two file descriptors, one for reading from the pipe and a second for writing into the pipe. After the fork, both the parent and child inherit both file descrip-tors. Typically one pipe would be used for unidirectional communication between the parent and the child.

Reading and writing to pipes can use the functions that take file descriptors as parameters, such as read and write.

Listing 5.61 shows an example of using an anonymous pipe to communicate between a child and parent process.

Listing 5.61 Using an Anonymous Pipe to Communicate Between a Parent and Child Process

#include <unistd.h> #include <stdio.h>

int main()

{

int status; int pipes[2];

pipe( pipes );

pid_t f = fork(); if ( f == 0 )

{ /* Child process */ close( pipes[0] );

write( pipes[1], "a", 1 ); printf( "Child sent 'a'\n" ); close( pipes[1] );

}

else

{

char buffer[11];

close( pipes[1] );

int len = read( pipes[0], buffer, 10 );

buffer[len] = 0;

printf( "Parent received %s\n", buffer );

close (pipes[0] );

}

return 0;

}

The code creates file descriptors for the two pipes before forking. The parent closes the descriptor indicted by pipes[1] and then waits to receive data from pipes[0]. The child process closes the descriptor pipes[0] and then sends a character to pipes[1] to be read by the parent process. The child process then closes their copy of the write file descriptor. The parent process prints the character sent by the child before closing the pipe and exiting.

Named pipes are created using a call to mknod(), which takes the path to the file that is to be used as the identifier for the pipe; the mode, which is S_IFIFO for a named pipe; and the access permissions for the file. The two processes can then call open() to open the file and treat the returned handles in the same way as before. Once the processes have finished with the named pipe, it can be removed by calling unlink(). The code in Listing 5.62 implements the same parent and child communication as Listing 5.61, this time using named pipes instead of anonymous pipes.

Listing 5.62  Parent and Child Process Communicating Using Named Pipes

#include <unistd.h> #include <stdio.h> #include <sys/stat.h> #include <fcntl.h>

int main()

{

int status;

mknod( "/tmp/pipefile", S_IFIFO|S_IRUSR|S_IWUSR, 0 );

pid_t f = fork(); if ( f == 0 )

{ /* Child process */

int mypipe = open( "/tmp/pipefile", O_WRONLY ); write( mypipe, "a", 1 );

printf( "Child sent 'a'\n" ); close( mypipe );

}

else

{

int mypipe = open( "/tmp/pipefile", O_RDONLY );

char buffer[11];

int len = read( mypipe, buffer, 10 );

buffer[len] = 0;

printf( "Parent received %s\n", buffer );

close( mypipe );

}

unlink( "/tmp/pipefile" );

return 0;

}

The parent process calls mknod() to create the pipe and then forks. Both the child and parent processes open the pipe—the child for writing and the parent for reading. The child writes into the pipe, closes the file descriptor, unlinks the pipe, and then exits. The parent process reads from the pipe before it too closes it, unlinks it, and exits.

Using Signals to Communicate with a Process

Signals are used extensively in UNIX and UNIX-like operating systems. Pressing ^C on a terminal keyboard to stop an application actually results in the SIGKILL signal being sent to that process. It is relatively straightforward to set up a signal handler for the vari-ous signals that might be sent to a process. It is not possible to install a handler for SIGKILL, but many of the other signals can be intercepted and handled.

A signal handler is installed by calling signal() with the signal number and the rou-tine that should handle the signal. A signal can be sent by calling kill() with the PID of the process that the signal should go to and the signal number. Listing 5.63 shows an example of this.

Listing 5.63   Sending and Receiving a Signal

#include <signal.h> #include <stdio.h> #include <unistd.h>

void hsignal( int signal )

{

}

int main()

{

signal( SIGRTMIN+4, hsignal );

kill( getpid(), SIGRTMIN+4 );

return 0;

}

The example installs a handler for the signal SIGRTMIN+4. The values SIGRTMIN and SIGRTMAX are system-specific values that represent the range of values that are reserved for user-defined signals.

It is tempting to imagine that the code in Listing 5.64 would be appropriate for a sig-nal handler. The problem with this code is that the function printf() is not guaranteed to be signal-safe. That is, if the application happened to be performing a printf() call when the signal arrived, it would not be safe to call the printf() in the signal handler.

Listing 5.64   Unsafe Signal Handler Code

void hsigseg( int signal )

{

printf( "Got signal\n" );

}

The POSIX guarantees that a set of function calls are async-signal-safe, in particular that the write() call can be used in a signal handler, so the code in Listing 5.65 can be used.

Listing 5.65   Printing Output in a Signal Handler

#include <stdio.h> #include <unistd.h>

void hsignal( int signal )

{

write( 1, "Got signal\n", 11 );

}

Running this on an Ubuntu system produces the expected output shown in Listing 5.66.

Listing 5.66   Output Printed by Signal Handler

$ gcc signal.c

$ ./a.out

Got signal

Sometimes, the code already performs an operation on receiving a particular signal, but it is desired to add an additional handler. What should be done is to add a signal handler to the chain and then call the default one.

The function that allows us to create a chain of signal handlers is sigaction(). This takes a signal number and two sigaction structures. The first sigaction structure contains information about the new signal handler, while the second returns information about the existing signal handler.

The code in Listing 5.67 installs a new handler for the SIGPROF signal but stores the details of the old handler. Then, when a signal is received, the new handler does its pro-cessing and then installs the old handler to perform the default action. The handler structure has an sa_sigaction field, which indicates the routine to be called in the event of a signal arriving. It also has an sa_mask field, which sets the list of signals that are to be blocked while this signal is handled. The other field of interest is sa_flags, which allows tuning of the behavior of the signal handler.

Listing 5.67   Chaining Signal Handlers

#include <signal.h> #include <unistd.h>

struct sigaction oldhandler;

void hsignal( int signal, siginfo_t* info, void* other )

{

write( 1, "Got signal\n", 11 ); if (oldhandler.sa_sigaction)

{

oldhandler.sa_sigaction( signal, info, other );

}

}

int main()

{

struct sigaction newhandler; newhandler.sa_sigaction = hsignal; newhandler.sa_flags = 0; sigemptyset( &newhandler.sa_mask );

sigaction( SIGPROF, &newhandler, &oldhandler ); kill( getpid(), SIGPROF );

}

Signals can also be used for communicating between processes. Listing 5.68 demon-strates a parent process sending a signal to a child process.

Listing 5.68   Parent Process Signaling to a Child Process

#include <unistd.h> #include <stdio.h> #include <signal.h> #include <sys/wait.h>

volatile int go = 0;

void handler( int sig )

{

go = 1;

write( 1, "Signal arrived\n", 16 );

}

int main()

{

signal( SIGRTMIN+4, handler );

pid_t f = fork(); if ( f == 0 )

{ /* Child process */ while ( !go ){}

printf( "Child completed\n" );

}

else

{

kill( f, SIGRTMIN+4 );

waitpid( f, 0, 0 );

printf( "Parent completed\n" );

}

}

Compiling and running the code produces the output shown in Listing 5.69.

Listing 5.69  Output from Parent Process Communicating with Child Process

$ gcc sigchild.c

$ ./a.out

Signal arrived

Child completed

Parent completed

It might be sufficient to signal another process, but it is more useful to be able to pass data between the processes. It is possible to do this using sigaction(), as the code in Listing 5.70 demonstrates. The code also uses sigqueue() to send the signal containing the data; on Solaris, this is found in the real-time extensions library, and the application requires linking with -lrt.

Listing 5.70   Using Signals to Transfer Information

#include <unistd.h> #include <stdio.h> #include <signal.h> #include <sys/wait.h>

volatile int go = 0;

struct sigaction oldhandler;

void handler( int sig, siginfo_t *info, void *context )

{

go = (int)info->si_value.sival_int;

write( 1, "Signal arrived\n", 16 );

}

int main()

{

struct sigaction newhandler;

newhandler.sa_sigaction = handler;

newhandler.sa_flags = SA_SIGINFO;

sigemptyset( &newhandler.sa_mask );

sigaction( SIGRTMIN+4, &newhandler, &oldhandler );

pid_t f = fork(); if ( f == 0 )

{ /* Child process */ while ( !go ){}

printf( "Child completed go=%i\n", go );

}

else

{

union sigval value;

value.sival_int = 7;

sigqueue( f, SIGRTMIN+4, value );

waitpid( f, 0, 0 );

printf( "Parent completed\n" );

}

}

The signal handler is set up with sa_flags including SA_SIGINFO. This flag causes the signal handler to receive the siginfo_t data. If the flag is not specified, the signal handler will not receive the data. The signal is sent by the parent process by calling sigqueue(). This takes a sigval union as a parameter, and the code sets the integer field of this union to the value seven. When the child process receives the signal, it can extract the value of this field from the sigval union passed into it.

Compiling and running the code produces the output shown in Listing 5.71.

Listing 5.71   Parent Process Sending Information to the Child Process

$ gcc sigchild2.c

$ ./a.out

Signal arrived

Child completed go=7

Parent completed

The parent process creates a child process and then sends a SIGRTMIN+4 signal to the child. The child process loops until the variable go becomes nonzero. When the child process receives the signal, it sets the variable go to be nonzero, and this enables the process to print a message and exit. In the meantime, the parent process has been waiting for the child process to exit. When the child process does eventually exit, the parent process prints a message and also exits.

One significant problem with using signals is that they can disrupt a system call that the thread is making at the time the signal is received. If this happens, the system call will set errno to the value EINTR indicating that the call should be retried. However, the exact behavior is system dependent and should be carefully explored before relying on signals.