localBlock在java,Spark-Java IOException：无法在/ tmp / blockmgr *中创建本地目录

I was trying to run a long running Spark Job. After few hours of execution, I get exception below :

Caused by: java.io.IOException: Failed to create local dir in /tmp/blockmgr-bb765fd4-361f-4ee4-a6ef-adc547d8d838/28

Tried to get around it by checking:

Permission issue in /tmp dir. The spark server is not running as root. but /tmp dir should be writable to all users.

/tmp Dir has enough space.

解决方案

Assuming that you are working with several nodes, you'll need to check every node participate in the spark operation (master/driver + slaves/nodes/workers).

Please confirm that each worker/node have enough disk space (especially check /tmp folder), and right permissions.