最近开始学习用Linux系统工作。不得不说,Linux用来做批处理真的很方便。
用数据处理文件process.py处理单个data_index.npz文件的方式,处理后生成.output 文件
python process.py data_index
当文件多起来的时候,用手在命令行里挨个敲显然会让人非常痛苦。所以要批处理data_index.npz。这一步的操作不难,只需要列出文件夹中以.npz结尾的文件,再用循环处理即可:
#!/bash/bin
. /your_anaconda_installation_path/anaconda3/etc/profile.d/conda.sh
conda activate environment # where you run process.py
for file in *.npz; do
data_index=${file&.npz}
echo "Execution: python process.py $data_index"
python process.py $data_index
done
这样还不够,每次运行的时候,不管你的数据有没有被处理过,它都要重新处理一遍,很浪费时间。所以我想在处理之前检查一遍文件夹,避免已经处理过而且存在相应的.output的某个data_index.npz被重复处理。怎么办呢?只需要加上一个简单的判断即可:
我们知道,每次处理之后,都会产生一个对应的data_index.output文件。那么,只要检查这个output文件存不存在,不就能挑出那些没被处理过的文件进行处理了吗?
所以代码变成了这样
#!/bash/bin
. /your_anaconda_installation_path/anaconda3/etc/profile.d/conda.sh
conda activate environment # where you run process.py
mkdir processed
for file in *.npz; do
data_index=${file&.npz}
if [ -f "${number}.output" ]; then
# if the .npz file has been processed
echo "execution: mv ${number}.npz processed/"
mv "${number}.npz" "processed/"
else
# Perform operation B if .output file does not exist
echo "execution: python process.py $number"
python process.py "$number"
done
另外,还可以在代码里加上并行命令,让处理过程更快结束
这时就没有必要在循环里处理命令了,循环部分改成:
for file in *.npz; do
data_index=${file&.npz}
if [ -f "${number}.output" ]; then
# if the .npz file has been processed
echo "execution: mv ${number}.npz processed/"
mv "${number}.npz" "processed/"
done
然后加上并行指令:
NUM_JOBS=4
find . -maxdepth 1 -type f -name ".npz" | sed "find a way to move suffix"| parallel --ungroup -j $NUM_JOBS 'echo "Execution: python process.py data_index"; python process.py data_index'
这样,省心又省事还快的批处理代码就写好啦!
给它取个名字吧,就叫“反复自省的多线自我PUA脚本”!