文章目录
本文的内容介绍、代码下载主要参考了网上多个文件汇总而成，本文仅作汇总，学习请翻阅原文。
1. 常用数据库
近几年在生物医药圈一直有脱钩的传闻，随着24年美国大选临近，对于trump上台后的预期政策担忧也愈演愈烈。在自己吓自己的前提下，对各类数据库进行了整理，并逐步进行下载。
2. TCGA数据下载
TCGA（The Cancer Genome Atlas）数据库是一个由美国国家癌症研究所（NCI）和国家人类基因组研究所（NHGRI）合作建立的大规模癌症基因组学数据库。该数据库中收集了来自多个癌症类型的临床数据、基因组数据和表达数据。
2.1 TCGA介绍
2005.12.13日，美国国家癌症研究所（NCI）和国家人类基因组研究所（NHGRI）联合启动肿瘤基因组图谱（TCGA）项目。
TCGA是一项以基因组为基础的大科学研究计划，以人类基因组计划的成果为基础，研究癌症中基因组的变化。与HGP专注于疾病的遗传因素不同，TCGA更关心人类出生后细胞中的基因变化。
33种癌症类型的关键基因组变化的多维图谱。超过2PB的基因组数据公开。超过1万名病人提供配对的癌症和对照组织。7种数据类型。
组织结构完善，由8种组成，形成数据闭环。
由TSS收集样本，交由BCR进行数据整理，由GSC、GCC进行测序、数据鉴定，CGHub、GDAC进行把关，在DCC进行数据最后的整合，最后由NCI、NHGRI项目组发布。
对样本的纳入有严格的标准，患者是原发性种类，有配对样本。肿瘤组织需冷冻保存、且大小适宜。肿瘤细胞含量在80%以上。
2.2 数据类型详解
临床数据：TCGA数据库中包含了多种癌症类型的患者的临床信息，如年龄、性别、病理分期、生存期等。这些数据有助于研究人员了解癌症的发展和预后。
indexed clinical: 使用 XML 文件创建的精炼临床数据
XML: 原始临床数据
处理 XML 格式的临床数据分为两步：
使用 GDCquery 和 GDCDownload 来查询和下载 Biospecimen 或 Clinical XML 文件
使用 GDCprepare_clinic 来解析文件
BCR Biotab: 解析 XML 文件之后的 tsv 文件
基因组数据：TCGA数据库中的基因组数据包括了DNA测序和突变数据。这些数据可以帮助研究人员分析癌症基因组的变化，寻找与癌症发展相关的基因变异。
表达数据：TCGA数据库中的表达数据主要是通过RNA测序获得的。这些数据可以反映癌症细胞内基因的表达水平，帮助研究人员发现与癌症发展和治疗相关的基因。
甲基化数据：TCGA数据库中的甲基化数据可以帮助研究人员了解癌症细胞中DNA甲基化的模式和变化。甲基化是一种常见的基因组调控方式，与癌症的发生和发展密切相关。
蛋白质组数据：TCGA数据库中的蛋白质组数据包括了癌症细胞中蛋白质的表达水平和翻译后修饰的信息。这些数据对于研究癌症细胞的功能和信号通路具有重要意义。
2.3 TCGA数据等级
受控数据主要是采集样本的患者隐私，测序原始数据（BAM/FASTQ），SNP6、EXON芯片的1、2级数据，生信分析的VCF、MAF等中间结果。
从上表可知，1、2级数据为病人原始结果，涉及医学伦理学问题，一般处于受控状态。3、4级数据是加工处理后的数据，由TCGA整合发表。，3级部分开放，4级数据完全开放。 仅有少量的数据是open状态。
详细可阅读说明文档GDC_Introduction
2.4 TCGA网页筛选页面介绍
2.5 各文件每个字段的含义
可看官网的说明文档。
各表格的字段含义
2.6 biospecimen IDs介绍(样本ID)
目前UUID成为主要标识符，下载数据时以UUID代指一个样本。
2.7 下载代码
# 0. 起始 -------------------------------------------------------------------
library(TCGAbiolinks)
projects <- TCGAbiolinks::getGDCprojects()$project_id
projects <- projects[grepl('^TCGA', projects, perl=TRUE)]
# 1. Clinical -------------------------------------------------------------
## 1.1 clinical ------------------------------------------------------------
### XML -------------------------------------------------------------------------
# 使用 GDCquery 和 GDCDownload 来查询和下载 Biospecimen 或 Clinical XML 文件
# 使用 GDCprepare_clinic 来解析文件
### 下载所有临床数据, 并将结果汇总在一个文件中 ########
### TCGA-READ、TCGA-LGG、仅下载,未合并,因为有重复,需针对性处理
library(TCGAbiolinks)
library(data.table)
library(dplyr)
library(regexPipes)
# 获取所有索引信息
clinical <- TCGAbiolinks:::getGDCprojects()$project_id %>%
regexPipes::grep("TCGA", value = TRUE) %>%
sort %>%
plyr::alply(1, GDCquery_clinic, .progress = "text") %>%
rbindlist(fill = TRUE)
readr::write_csv(clinical, file = "all_clin_indexed.csv")
# 解析 XML 文件并信息获取对应的信息
getclinical <- function(proj) {
message(proj)
result <- NULL
attempt <- 1
max_attempts <- 5 # 设置最大尝试次数
while(attempt <= max_attempts) {
result <- tryCatch({
query <- GDCquery(project = proj, data.category = "Clinical", data.format = "bcr xml")
GDCdownload(query)
clinical <- GDCprepare_clinic(query, clinical.info = "patient")
clinical_data <- list(clinical)
for(i in c("admin", "radiation", "follow_up", "drug", "new_tumor_event")){
message(i)
aux <- GDCprepare_clinic(query, clinical.info = i)
if(is.null(aux) || nrow(aux) == 0) next
# 处理重复的列名
replicated <- which(grep("bcr_patient_barcode", colnames(aux), value = TRUE, invert = TRUE) %in% colnames(clinical))
colnames(aux)[replicated] <- paste0(colnames(aux)[replicated], ".", i)
if(!is.null(aux)) clinical <- merge(clinical, aux, by = "bcr_patient_barcode", all = TRUE)
}
# 保存临床数据到csv文件
readr::write_csv(clinical, path = paste0("TCGA_alldata/", proj, "_clinical_from_XML.csv"))
return(clinical)
}, error = function(e) {
message(paste0("Error clinical: ", proj, " Attempt: ", attempt))
attempt <<- attempt + 1 # 增加尝试次数
NULL
})
# 如果成功获取数据，则跳出循环
if (!is.null(result)) break
}
# 如果多次尝试后仍然失败，返回NULL并发出警告
if (is.null(result)) {
warning(paste0("Failed to get clinical data for project: ", proj, " after ", max_attempts, " attempts."))
}
return(result)
}
# 患者信息
# 如果内存溢出,可分批次下载, 或单独下载某个数据集
clinical <- TCGAbiolinks:::getGDCprojects()$project_id %>%
regexPipes::grep("TCGA", value = T) %>%
sort %>%
plyr::alply(1, getclinical, .progress = "text") %>%
rbindlist(fill = TRUE) %>%
setDF %>%
subset(!duplicated(clinical))
readr::write_csv(clinical, path = "TCGA_alldata/all_clin_XML.csv")
## 1.2 supplement ---------------------------------------------------------
### 1.2.1 clinical-supplement-bcr biotab, 解析 XML 文件之后的 tsv 文件-------------------------------------------------------------------------
sapply(projects, function(project){
cat("Processing project:", project, "\n")
query <- GDCquery(
project = project,
data.category = "Clinical",
data.type = "Clinical Supplement",
data.format = "BCR Biotab",
access = "open"
)
GDCdownload(query, method = "api", files.per.chunk = 100)
prepared_data <- GDCprepare(query, save = TRUE, save.filename = file.path(paste0(project, "_clinical_supplement_bcr_biotab.Rdata")))
})
### 1.2.2 Biospecimen 获取采样信息 -----------------------------------------------------------------------
sapply(projects, function(project){
cat("Processing project:", project, "\n")
query <- GDCquery(
project = project,
data.category = "Biospecimen",
data.type = "Biospecimen Supplement",
data.format = "BCR Biotab",
access = "open"
)
GDCdownload(query, method = "api", files.per.chunk = 100)
prepared_data <- GDCprepare(query, save = TRUE, save.filename = file.path(paste0(project, "_biospecimen_supplement_bcr_biotab.Rdata")))
})
### 1.2.3 Indexed 使用 XML 文件创建的精炼临床数据 -------------------------------------------------------------------------
clinical <- GDCquery_clinic(project = "TCGA-BRCA", type = "clinical")
clinical <- GDCquery_clinic(project = "TARGET-RT", type = "clinical")
### 1.2.4 诊断切片（SVS 格式）-------------------------------------------------------------------------
# 单个数据集50-60G,暂不下载
sapply(projects, function(project){
cat("Processing project:", project, "\n")
query <- GDCquery(
project = project,
data.category = "Biospecimen",
data.type = 'Slide Image',
experimental.strategy = "Diagnostic Slide", # 实验策略有两种,Tissue Slide和Diagnostic Slide(患者的)
# barcode = c("TCGA-RU-A8FL", "TCGA-AA-3972")
access = "open"
)
GDCdownload(query, method = "api", files.per.chunk = 100)
prepared_data <- GDCprepare(query, save = TRUE, save.filename = file.path(paste0(project, "_biospecimen_slide_images.Rdata")))
})
# 2. RNA -------------------------------------------------------------------------
## 2.1 mRNA-------------------------------------------------------------------------
sapply(projects, function(project){
# 查询
query <- GDCquery(
project = project,
data.category = "Transcriptome Profiling",
data.type = "Gene Expression Quantification",
workflow.type = "STAR - Counts"
)
# 下载
GDCdownload(query, method = "api", files.per.chunk = 100)
# 整理
GDCprepare(query, save = T, save.filename = file.path(paste0(project, "_mRNA.Rdata")))
}
)
## 2.2 microRNA-------------------------------------------------------------------------
sapply(projects, function(project){
query <- GDCquery(project = project,
data.category = "Transcriptome Profiling",
data.type = "miRNA Expression Quantification"
)
GDCdownload(query)
GDCprepare(query, save = T, save.filename = file.path(paste0(project, "_miRNA.Rdata")))
})
## 2.3 isoform -------------------------------------------------------------------------
sapply(projects, function(project){
query.isoform <- GDCquery(
project = project,
experimental.strategy = "miRNA-Seq",
data.category = "Transcriptome Profiling",
data.type = "Isoform Expression Quantification"
)
GDCdownload(query.isoform, method = "api", files.per.chunk = 100)
GDCprepare(query.isoform, save = T, save.filename = file.path(paste0(project, "_mirna-isoform.Rdata")))
})
# 3.SNP-------------------------------------------------------------------------
# Can't combine `..17$Tumor_Seq_Allele2` <character> and `..18$Tumor_Seq_Allele2` <logical>. 部分数据库的样本字段不一致,导致合并出错概率高, 先按癌种-样本下载好
sapply(projects, function(project){
query <- GDCquery(
project = project,
data.category = "Simple Nucleotide Variation",
data.type = "Masked Somatic Mutation",
access = "open"
)
GDCdownload(query, method = "api", files.per.chunk = 100)
# GDCprepare(query, save = T,save.filename = file.path(project_dir, paste0(project, "_SNP.Rdata")))
})
# 4.Protein-------------------------------------------------------------------------
# "TCGA-LAML-fail" "TCGA-THCA-fail", 这两个数据集没有蛋白结果
sapply(projects, function(project){
query <- GDCquery(
project = project,
data.category = "Proteome Profiling",
data.type = "Protein Expression Quantification"
)
GDCdownload(query, method = "api", files.per.chunk = 100)
GDCprepare(query, save = T, save.filename = file.path(paste0(project, "_protein.Rdata")))
})
# 5.methy-------------------------------------------------------------------------
# Beta值数据 分3个平台, Illumina Human Methylation 27、 Illumina Human Methylation 450、 Illumina Methylation Epic
# 每个癌种下,不一定含有全部3种平台的数据
# 本次先下载Illumina Human Methylation 27的
# IDAT是原始的荧光信号强度数据，而Beta值是这些原始数据的标准化表示, 本次下载优先BEta值
## 5.1 Illumina Human Methylation 27---------------------------------------------------------------------
### Methylation Beta Value --------------------------------------------------------------------
# "TCGA-BRCA" "TCGA-SARC—fail,数据集无此结果" "TCGA-ACC-fail" "TCGA-UCEC" "TCGA-KIRC" "TCGA-LAML" "TCGA-SKCM-fail" "TCGA-PAAD-fail" "TCGA-TGCT-fail" "TCGA-CESC-fail" "TCGA-ESCA-fail" "TCGA-THCA-fail" "TCGA-LIHC-fail" "TCGA-PRAD-fail" "TCGA-READ"
# "TCGA-OV" "TCGA-UVM-fail" "TCGA-BLCA-fail" "TCGA-CHOL-fail" "TCGA-GBM" "TCGA-UCS-fail" "TCGA-PCPG-fail" "TCGA-MESO-fail" "TCGA-DLBC-fail" "TCGA-COAD" "TCGA-STAD" "TCGA-KIRP" "TCGA-THYM-fail" "TCGA-KICH-fail" "TCGA-LGG-fail"
# "TCGA-LUSC" "TCGA-LUAD" "TCGA-HNSC-fail"
projects <- c("TCGA-BRCA", "TCGA-UCEC", "TCGA-KIRC", "TCGA-LAML", "TCGA-READ", "TCGA-OV", "TCGA-GBM", "TCGA-COAD", "TCGA-STAD", "TCGA-KIRP", "TCGA-LUSC", "TCGA-LUAD")
projects <- c( "TCGA-LUAD")
sapply(projects, function(project){
coad_methy <- GDCquery(
project = project,
data.category = "DNA Methylation",
data.type = "Methylation Beta Value",
platform = "Illumina Human Methylation 27"
)
GDCdownload(coad_methy, method = "api", files.per.chunk = 100)
GDCprepare(coad_methy,save = T, save.filename= file.path(paste0(project, "_METHY_beta_27.Rdata")))
})
### Masked Intensities --------------------------------------------------------------------
# "TCGA-BRCA" "TCGA-SARC—fail" "TCGA-ACC-fail" "TCGA-UCEC" "TCGA-KIRC" "TCGA-LAML" "TCGA-SKCM-fail" "TCGA-PAAD-fail" "TCGA-TGCT-fail" "TCGA-CESC-fail" "TCGA-ESCA-fail" "TCGA-THCA-fail" "TCGA-LIHC-fail" "TCGA-PRAD-fail" "TCGA-READ"
# "TCGA-OV" "TCGA-UVM-fail" "TCGA-BLCA-fail" "TCGA-CHOL-fail" "TCGA-GBM" "TCGA-UCS-fail" "TCGA-PCPG-fail" "TCGA-MESO-fail" "TCGA-DLBC-fail" "TCGA-COAD" "TCGA-STAD" "TCGA-KIRP" "TCGA-THYM-fail" "TCGA-KICH-fail" "TCGA-LGG-fail"
# "TCGA-LUSC" "TCGA-LUAD" "TCGA-HNSC-fail"
projects <- c("TCGA-BRCA", "TCGA-UCEC", "TCGA-KIRC", "TCGA-LAML", "TCGA-READ", "TCGA-OV", "TCGA-GBM", "TCGA-COAD", "TCGA-STAD", "TCGA-KIRP", "TCGA-LUSC", "TCGA-LUAD")
sapply(projects, function(project){
coad_methy <- GDCquery(
project = project,
data.category = "DNA Methylation",
data.type = "Masked Intensities",
platform = "Illumina Human Methylation 27"
)
GDCdownload(coad_methy, method = "api", files.per.chunk = 50)
})
## 5.2 Illumina Human Methylation 450 -----------------------------------------------------------------
### Methylation Beta Value --------------------------------------------------------------------
sapply(projects, function(project){
coad_methy <- GDCquery(
project = project,
data.category = "DNA Methylation",
data.type = "Methylation Beta Value",
platform = "Illumina Human Methylation 450" # Illumina Human Methylation 450
)
GDCdownload(coad_methy, method = "api", files.per.chunk = 30)
})
### Masked Intensities --------------------------------------------------------------------
sapply(projects, function(project){
coad_methy <- GDCquery(
project = project,
data.category = "DNA Methylation",
data.type = "Masked Intensities",
platform = "Illumina Human Methylation 450" # Illumina Human Methylation 450
)
GDCdownload(coad_methy, method = "api", files.per.chunk = 30)
})
# 6. CNV----------------------------------------------------------------------
## 6.1 Masked Copy Number Segment-------------------------------------------------------------------------
sapply(projects, function(project){
query <- GDCquery(
project = project,
data.category = "Copy Number Variation",
data.type = "Masked Copy Number Segment",
access = "open"
)
GDCdownload(query, method = "api", files.per.chunk = 100)
GDCprepare(query, save = T,save.filename = file.path(paste0(project, "_CNV.Rdata")))
})
## 6.2 Copy Number Segment -------------------------------------------------------------------
sapply(projects, function(project){
query <- GDCquery(
project = project,
data.category = "Copy Number Variation",
data.type = "Copy Number Segment",
access = "open"
)
GDCdownload(query, method = "api", files.per.chunk = 300)
})
## 6.3 Allele-specific Copy Number Segment ------------------------------------------------------------------
# There are more than one file for the same case. Please verify query results. You can use the command View(getResults(query)) in rstudio
sapply(projects, function(project){
query <- GDCquery(
project = project,
data.category = "Copy Number Variation",
data.type = "Allele-specific Copy Number Segment",
access = "open"
)
GDCdownload(query, method = "api", files.per.chunk = 300)
})
## 6.4 Gene Level Copy Number -------------------------------------------------------------------------
# Warning: There are more than one file for the same case. Please verify query results. You can use the command View(getResults(query)) in rstudio
projects <- c("TCGA-STAD", "TCGA-KIRP", "TCGA-THYM", "TCGA-KICH", "TCGA-LGG", "TCGA-LUSC", "TCGA-LUAD", "TCGA-HNSC")
sapply(projects, function(project){
query <- GDCquery(
project = project,
data.category = "Copy Number Variation",
data.type = "Gene Level Copy Number",
access = "open"
)
GDCdownload(query, method = "api", files.per.chunk = 30)
})
参考文件
（1）全网最全！2021最新常用肿瘤生信数据库收藏级汇总！（1）
（2）全网最全！2021最新常用肿瘤生信数据库收藏级汇总！（2）
（3）收藏：常用医学公共数据库（含临床数据库，生信数据库和机器学习数据库）
（4）6大药敏性分析数据库大汇总，助力肿瘤相关生信分析！/SCI论文/科研/研究生/生信分析热
（5）新版TCGA数据库学习：批量下载新版TCGA数据
（6）TCGA 数据下载 —— TCGAbiolinks 简单使用
（7）4年新版TCGA GDC data portal 2.0界面介绍及数据下载教程
（8）TCGA 数据下载 —— TCGAbiolinks 数据分析
（9）【TCGA数据库介绍及应用】
（10）【TCGA数据库数据分析】
869
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
