MPI ERRORS

 

Some students have reported getting errors when running MPICH programs. The error message usually states that there are no free ports in the MPICH_PORT_RANGE. 

 

No Free Ports in MPICH_PORT_RANGE 

Cause:

The errors are usually caused because of program crashes which do not free the sockets available to the system. Hence over successive runs of mpiexec or mpirun the ports all become used up (leaving none free to start new programs). 

Fix:

Type:

killall mpd
killall python2.3

in a normal command line window. This should kill all of the running MPICH daemons - not this will kill running MPI programs as well (not just the 'zombie' ones). 

 

Signal 11 or Signal 9

Cause:

Usually caused by incorrect use of pointers or incrementing outside an array. Technical cause is usually a segmentation fault for Signal 11 - i.e. a pointer pointed to a location in memory outside of the programs space. Signal 9 usually means a pointer pointed to some area of code within the program which it should not.

Fix:

Debug code, check pointers and array subscripts in particular. For MPI programs check that you are not sending more data than there is in an array.

 

Signal 10

Cause:

Signal 10 is very rare on UNIX/LINUX systems and indicates that there has been a 'bus error.' Apparently a large proportion of these errors come from incorrect assembly instructions being written to the CPU. This can occur in poorly written software by accident - some compilers will emit badly grouped instructions. You may also need to check that you have used the correct 'bit' compiler -i.e. have you used the 64-bit compiler on a 32-bit platform?

Fix:

Check your pointer and memory references - this error can occur if a reference/pointer is poorly assembled (using addition/multiplication). It is very rare and very difficult to find in code. Also check the compiler and Operating System you are using (uname -a).

 

Signal 13

Cause:

Pipe failure - one process is trying to write to a process but there is no process to receive the data. This is quite unlikely but can happen with some MPICH programs where the runtime is listening for application output to root to the parent node.

Fix:

Exit all MPD's and restart the application run. If you keep getting signal 13 you should check errors to do with sends - these are almost always written to console with an MPICH error to help you diagnose what is wrong.