项目日常记录 20211227

1.shell中的其他用法

cp 复制软链接的问题

scp和cp 默认复制，如果为软链接，会将源文件一同复制过去。

解决方法：cp -d  

加上-d参数，复制相同目录，不复制软链接下的内容。

删除集群命令

qstat |sed '1,2d'|awk 'BEGIN{FS=" "} {print $1}'|xargs qdel	

压缩文件夹

tar -cvzf kegg.tar.gz kegg
zip -r  kegg.zip kegg

查看contig文件是否存在

grep yangkang config |cut -d  '=' -f 2 |xargs ls

for循环变量定义

for i in  /share4/users/yangkang/project/2020_Conjointanlaysis/MWXS-20-1085D_xiangjiao_36/Conanalysis/data/diff/diff.1/*diff.xls;do num=$(basename $i);  awk -F "\t"  'BEGIN{OFS="\t" } {print $1,$2,$5,$3,$4} ' $i  > /share4/users/yangkang/project/2020_Conjointanlaysis/MWXS-20-1085D_xiangjiao_36/Conanalysis/data/diff/diff.2/$num;done

单个命令跑

for i in `ls s* `;do echo "sh $i >$i.log 2>&1";done  >work.sh

单个样品当组，进行差异分析

 华为云路径：/data/yangkang/project/2020_Denovo/MWXS-20-974-01_jianguo_24/analysis2/out

/data/yangkang/project/2020_RefGenome/MWXS-20-974-01_jianguo_24/analysis2/out/6.Difference_analysis/1.DESeq/intermediate

本地路径：/share4/users/wangping/project/2020_RefGenome/MWXS-20-669_xiaomai_jun_34/analysis_jun2/out/4.Difference_analysis/SH/work.sh

本地无参转录本数量过多，减少转录本数量

华为云：/data/yangkang/project/2020_Denovo/MWXS-20-975D/analysis/out/2.Assembly/1.assembly/work.sh
本地：/share4/users/yangkang/project/2020_RefGenome/MWXS-20-1460Y-01_ningjinxie_9/analysis2/out/2.Assembly/1.assembly/work.sh

gtf增加一列

awk -F "\t" 'BEGIN{OFS="\t"} {old=$0;sub("transcript_id","gene_id",$9);print old" "$9}' gene.gtf.old  >gene.gtf

gff文件添加外显子exon，gff文件可以作为参考

/data/yangkang/project/2021_RefGenome/MWXS-20-2163Y_dabaicai_18/analysis/ref/exon.pl

添加gtf的geneid

awk -F "\t" 'BEGIN{OFS='\t'} {if ($9~/gene_id/) {print $0}else{old=$0;sub("transcript_id","gene_id",$9);print $9;sub("transcript","gene",$9);s9=$9;print old"\t"s9;} }' 
/share4/users/yangkang/project/2021_RefGenome/MWXS-20-2350D_xiaoshu_6/analysis/out/3.Quantitative/1.genome/gene.gtf

MD5分文件夹检测

for i in */*txt ;do file=$(echo $i|cut -d '/' -f 1); cd $file; md5sum -c *txt ;cd ../;done >md5.log &

按kmeans分组进行kegg、go

华为云路径： /data/yangkang/personalized_analysis/MWXS-17-090/BIOINF-214226/test/Class4/work.sh
本地：/lustrefs/share4/users/Personalized_Analysis/MWXS-20-984-01/BIOINF-214743/test/S/kegg.test.sh   go.test.sh

kegg、go通路分析，各个分组

/data/yangkang/personalized_analysis/MWXS-19-1043_xigua_36/BIOINF-6127/work.sh

kegg、go、gsea、kog差异基因分析例子

/data/personalized_Analysis/2021_Q4/MWXS-21-2203DJ-851_huanggua_Ref/BIOINF-217156/test
/data/personalized_Analysis/2021_Q4/MWXS-21-2308DJ_yumi_12_Ref/BIOINF-217360/test/kegg.pl
/lustrefs/share4/personalizedAnalysis/2021_Q4/MWXS-19-786_24/BIOINF-217319/test2/kegg.pl    

联合分析时，KO号没有注释时加–

/share4/users/yangkang/project/2021_Conjointanlaysis/MWXS-20-1165D_chahua_21/Conanalysis/data/KO.ready.pl

去除括号的例子，自己写的perl脚本，计算pathcount

/lustrefs/share4/users/yangkang/project/2021_Conjointanlaysis/MWXS-20-1867D_shihu_27/Conanalysis/data/rawdata/path/work.sh

老师提供的转录数据KO号用自己数据库替换

/lustrefs//share4/users/yangkang/project/2021_Conjointanlaysis/MWXS-19-1577Y/Conanalysis/data/rawdata/work.sh

对pathcount文件添加K号

/lustrefs/share4/users/Personalized_Analysis/MWXS-20-1265/ConjointAnalysis/data/rawdata/work.sh

提取特定列

/lustrefs//share4/users/Personalized_Analysis/MWXS-21-678D/BIOINF-213512/huangtong/test.r

人建立GO库

/lustrefs/share4/users/yangkang/project/2021_Medical/MWY-21-340_ren_6/analysis/out/go_built/work.sh

perl中值替换

($ID)=$a[8]=~ /Transcript (.*)/   

利用perl去除括号

/lustrefs/share4/users/yangkang/test/conanalysis/data_ready/get.cor.test.pl

查找文件大小大

find ./ -type d -size +500M

调整kegg富集图的图例上下调整

/lustrefs//share4/users/Personalized_Analysis/MWY-20-592/BIOINF-211130/plotKeggEnrichment.R

进化树图的绘制

/lustrefs//share4/users/yangkang/personalized_analysis/MWXS-20-1133D_ganshu_12/BIOINF-211025/work.sh

pwd高级用法

for i in *;do if [[ -d $i ]];then work=`pwd` ;cd  `pwd`/$i;  ls ; cd $work ;fi;done

相对路径转换绝对路径

readlink -f ./
realpath ./

share5等于

sed 's@/share5/Bioinfo/users/yangkang/@/lustrefs/share4/users/yangkang/project/share5/@g'    

xargs 数据传输命令并实现该名称

for i in `ls *txt `;do echo  ${i#Nodes_}|sed 's#.txt##g'|xargs -n 1  -I {}  mkdir -p  {} ;cp $i   `echo $i|sed -e 's@Nodes_@@g' -e 's#.txt##g' `;done 

删除kegg GO数据

rm -r */KEGG/*{txt,sh,xls,vel} */GO/*{sh,txt,xls,tall}

相关性画热图

/lustrefs/share4/personalizedAnalysis/2021_Q4/MWY-21-1003_dashu_18_16S_Meta/BIOINF-216793/test/combine_heatmap

---------------------------------------------------------------------------------

2.awk的用法

linux中利用awk转置数据

awk '{for(i=1;i<=NF;i=i+1){a[NR,i]=$i}}END{for(j=1;j<=NF;j++){str=a[1,j];for(i=2;i<=NR;i++){str=str " " a[i,j]}print str}}' concentration.txt
## awk+if 的用法

选择列和不为0的数

less meta.xls|sed 's/N\/A/0/g'|awk 'BEGIN{OFS="\t"}  {if ($1 =="Index"||$2+$3+$4+$5+$6+$7+$8+$9+$10 != 0)  {print $0} }'##选择列和不为0的数

利用logfc信息更改up、down

for i in *DEG.xls; do cut -f 1,4-6 $i |awk 'BEGIN{OFS="\t"} {if($2<0 || $1 == "Gene_id") print$1,$2,"down",$3,$4;else if ($2>0 || $1 == "Gene_id") print$1,$2,"up",$3,$4}'>../transcript/${i%DEG.xls}diff.xls; done
##  awk+if中是if else

awk的变量使用，需要用到双引号的是字符，而不是变量

awk 'BEGIN{rawdata="/backup3/RNAseq/2020/November/MWXS-20-1133D_ganshu_12/rawdata/"} {print"ln -s" rawdata$1"*"$3"r_1.fq.gz" $2"_1.fq.gz" "\n" 
 "ln -s" rawdata$1"*"$3"r_2.fq.gz" $2"_2.fq.gz\n" }' lian.jie.txt >lian.sh

awk高级用法均值替代

awk -F "\t" 'BEGIN{OFS="\t"}  ARGIND==1 {INDEX[$1]=$7} ARGIND==2 {if ($1 in INDEX) LAST[$1]=($7+INDEX[$1])*0.5}  ARGIND==3  {if ($1 in INDEX) {print $1,$2,$3,$4,$5,$6,LAST[$1]}}'   RCK1d-2.read_counts.sim RCK1d-3.read_counts.sim  RCK1d-1.read_counts.sim.old >RCK1d-1.read_counts.sim1
awk -F "\t" 'BEGIN{OFS="\t"} {sub(/\.5/,"",$7);print $1,$2,$3,$4,$5,$6,$7} ' RCK1d-1.read_counts.sim1 >RCK1d-1.read_counts.sim

---------------------------------------------------------------------------------

3.sed的项目用法

添加行用法

for i in *pathcount.xls;do sed -i '1i #Term\tID\tInput\tnumberBackground\tgroupdiff\tgroup\tgeneid' $i;done 
## 替换行用法
for i in *pathcount.xls;do sed -i '1i #Term\tID\tInput\tnumberBackground\tgroupdiff\tgroup\tgeneid' $i;done 

删除指定行

sed -i   "/NAME/d"  CK_vs_AS.diff.xls  #双引号内斜杠匹配，后面接字母