How can we decide whether MPI, Pthreads or OpenMP is best for our application?

In general, distributed-memory systems can provide considerably more main memory than shared-memory systems, so if the memory requirements are very large, you may need to write the application using MPI.

It’s often possible to reuse considerably more serial code in a shared-memory program than a distributed-memory program. It’s more likely that serial data structures can be easily adapted to a shared-memory system. If this is the case, the development effort for the shared-memory program will probably much less. This especially for the OpenMP programs.

If the processes/threads do little communication, an MPI program should be fairly easy to develop, and very scalable.