文章目录
- EInfo: Obtaining information about the Entrez databases
- ESearch: Searching the Entrez databases
- EPost: Uploading a list of identifiers
- EFetch: Downloading full records from Entrez
- ELink: Searching for related items in NCBI Entrez
- Parsing Medline records
- Parsing GEO records
- Parsing UniGene records
- ESpell: Obtaining spelling suggestions
- ESummary: Retrieving summaries from primary IDs
- example 1
- example 2
- example 3
- Searching, downloading, and parsing GenBank records
- Using the history and WebEnv -- most import
- Searching for and downloading abstracts using the history
Entrez (https://www.ncbi.nlm.nih.gov/Web/Search/entrezfs.html) is a data retrieval system that provides users access to NCBI’s databases such as PubMed, GenBank, GEO, and many others. You can access Entrez from a web browser to manually enter queries, or you can use Biopython’s Bio.Entrez module for programmatic access to Entrez. The latter allows you for example to search PubMed or download GenBank records from within a Python script.
The Bio.Entrez module makes use of the Entrez Programming Utilities (also known as EUtils), consisting of eight tools that are described in detail on NCBI’s page at https://www.ncbi.nlm.nih.gov/books/NBK25501/. Each of these tools corresponds to one Python function in the Bio.Entrez module, as described in the sections below.
Before using Biopython to access the NCBI’s online resources (via Bio.Entrez or some of the other modules), please read the NCBI’s Entrez User Requirements. If the NCBI finds you are abusing their systems, they can and will ban your access!
• For any series of more than 100 requests, do this at weekends or outside USA peak times. This is up
to you to obey.
• Use the https://eutils.ncbi.nlm.nih.gov address, not the standard NCBI Web address. Biopython
uses this web address.
• If you are using a API key, you can make at most 10 queries per second, otherwise at most 3 queries per
second.
• Use the optional email parameter so the NCBI can contact you if there is a problem. You can either explicitly set this as a parameter with each call to Entrez (e.g. include email=“A.N.Other@example.com”
in the argument list), or you can set a global email address:
>>> from Bio import Entrez
>>> Entrez.email = "A.N.Other@example.com"
• For large queries, the NCBI also recommend using their session history feature
EInfo: Obtaining information about the Entrez databases
# EInfo: Obtaining information about the Entrez databases
# EInfo provides field index term counts, last update, and available links for each of NCBI’s databases
# In addition, you can use EInfo to obtain a list of all database names accessible through the Entrez utilities
from Bio import Entrez
Entrez.email = "A.N.Other@example.com" # Always tell NCBI who you are
handle = Entrez.einfo()
result = handle.read()
handle.close()
print(result)
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE eInfoResult PUBLIC "-//NLM//DTD einfo 20190110//EN" "https://eutils.ncbi.nlm.nih.gov/eutils/dtd/20190110/einfo.dtd">
<eInfoResult>
<DbList>
<DbName>pubmed</DbName>
<DbName>protein</DbName>
<DbName>nuccore</DbName>
<DbName>ipg</DbName>
<DbName>nucleotide</DbName>
<DbName>structure</DbName>
<DbName>genome</DbName>
<DbName>annotinfo</DbName>
<DbName>assembly</DbName>
<DbName>bioproject</DbName>
<DbName>biosample</DbName>
<DbName>blastdbinfo</DbName>
<DbName>books</DbName>
<DbName>cdd</DbName>
<DbName>clinvar</DbName>
<DbName>gap</DbName>
<DbName>gapplus</DbName>
<DbName>grasp</DbName>
<DbName>dbvar</DbName>
<DbName>gene</DbName>
<DbName>gds</DbName>
<DbName>geoprofiles</DbName>
<DbName>homologene</DbName>
<DbName>medgen</DbName>
<DbName>mesh</DbName>
<DbName>ncbisearch</DbName>
<DbName>nlmcatalog</DbName>
<DbName>omim</DbName>
<DbName>orgtrack</DbName>
<DbName>pmc</DbName>
<DbName>popset</DbName>
<DbName>proteinclusters</DbName>
<DbName>pcassay</DbName>
<DbName>protfam</DbName>
<DbName>biosystems</DbName>
<DbName>pccompound</DbName>
<DbName>pcsubstance</DbName>
<DbName>seqannot</DbName>
<DbName>snp</DbName>
<DbName>sra</DbName>
<DbName>taxonomy</DbName>
<DbName>biocollections</DbName>
<DbName>gtr</DbName>
</DbList>
</eInfoResult>
# Bio.Entrez’s parser parse XML
handle = Entrez.einfo()
record = Entrez.read(handle)
record.keys()
dict_keys(['DbList'])
record["DbList"]
['pubmed', 'protein', 'nuccore', 'ipg', 'nucleotide', 'structure', 'genome', 'annotinfo', 'assembly', 'bioproject', 'biosample', 'blastdbinfo', 'books', 'cdd', 'clinvar', 'gap', 'gapplus', 'grasp', 'dbvar', 'gene', 'gds', 'geoprofiles', 'homologene', 'medgen', 'mesh', 'ncbisearch', 'nlmcatalog', 'omim', 'orgtrack', 'pmc', 'popset', 'proteinclusters', 'pcassay', 'protfam', 'biosystems', 'pccompound', 'pcsubstance', 'seqannot', 'snp', 'sra', 'taxonomy', 'biocollections', 'gtr']
For each of these databases, we can use EInfo again to obtain more information
# For each of these databases, we can use EInfo again to obtain more information:
handle = Entrez.einfo(db="pubmed")
record = Entrez.read(handle)
record["DbInfo"] #输出一个字典,可以再次引用keys然后读取响应的信息
ESearch: Searching the Entrez databases
# ESearch: Searching the Entrez databases
# To search any of these databases, we use Bio.Entrez.esearch(). For example, let’s search in PubMed for
# publications that include Biopython in their title
handle = Entrez.esearch(db="pubmed", term="biopython[title]", retmax="40" )
record = Entrez.read(handle)
# "19304878" in record["IdList"]
print(record["IdList"])
['34434786', '22909249', '19304878']
# You can also use ESearch to search GenBank. Here we’ll do a quick search for the matK gene in Cypripedioideae orchids
handle = Entrez.esearch(db="nucleotide", term="Cypripedioideae[Orgn] AND matK[Gene]", idtype="acc")
record = Entrez.read(handle)
print(record["Count"])
print(record["IdList"]) #Each of the IDs is a GenBank identifier
585
['NC_058834.1', 'NC_058833.1', 'NC_058832.1', 'OK120861.1', 'OK120860.1', 'NC_058212.1', 'NC_058211.1', 'NC_058210.1', 'NC_058209.1', 'NC_058208.1', 'NC_058207.1', 'NC_058206.1', 'MN315109.1', 'MN315108.1', 'MN315107.1', 'MN315106.1', 'MN315105.1', 'NC_056912.1', 'MZ150831.1', 'MZ150830.1']
EPost: Uploading a list of identifiers
# EPost: Uploading a list of identifiers
# EPost uploads a list of UIs for use in subsequent search strategies
# To give an example of when this is useful, suppose you have a long list of IDs you want to download
# using EFetch (maybe sequences, maybe citations { anything). When you make a request with EFetch your
# list of IDs, the database etc, are all turned into a long URL sent to the server. If your list of IDs is long,
# this URL gets long, and long URLs can break (e.g. some proxies don’t cope well)
id_list = ["19304878", "18606172", "16403221", "16377612", "14871861", "14630660"]
print(Entrez.epost("pubmed", id=",".join(id_list)).read())
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE ePostResult PUBLIC "-//NLM//DTD epost 20090526//EN" "https://eutils.ncbi.nlm.nih.gov/eutils/dtd/20090526/epost.dtd"><ePostResult>
<QueryKey>1</QueryKey>
<WebEnv>MCID_61a9ccaf13217214767bf8b1</WebEnv>
</ePostResult>
# The returned XML includes two important strings, QueryKey and WebEnv which together define your history
# session. You would extract these values for use with another Entrez call such as EFetch
id_list = ["19304878", "18606172", "16403221", "16377612", "14871861", "14630660"]
search_results = Entrez.read(Entrez.epost("pubmed", id=",".join(id_list)))
webenv = search_results["WebEnv"]
query_key = search_results["QueryKey"]
EFetch: Downloading full records from Entrez
# EFetch: Downloading full records from Entrez
# EFetch is what you use when you want to retrieve a full record from Entrez
# For most of their databases, the NCBI support several different file formats. Requesting a specific file
# format from Entrez using Bio.Entrez.efetch() requires specifying the rettype and/or retmode optional arguments.
handle = Entrez.efetch(db="nucleotide", id="EU490707", rettype="gb", retmode="text")
print(handle.read())
LOCUS EU490707 1302 bp DNA linear PLN 26-JUL-2016
DEFINITION Selenipedium aequinoctiale maturase K (matK) gene, partial cds;
chloroplast.
ACCESSION EU490707
VERSION EU490707.1
KEYWORDS .
SOURCE chloroplast Selenipedium aequinoctiale
ORGANISM Selenipedium aequinoctiale
Eukaryota; Viridiplantae; Streptophyta; Embryophyta; Tracheophyta;
Spermatophyta; Magnoliopsida; Liliopsida; Asparagales; Orchidaceae;
Cypripedioideae; Selenipedium.
REFERENCE 1 (bases 1 to 1302)
AUTHORS Neubig,K.M., Whitten,W.M., Carlsward,B.S., Blanco,M.A., Endara,L.,
Williams,N.H. and Moore,M.
TITLE Phylogenetic utility of ycf1 in orchids: a plastid gene more
variable than matK
JOURNAL Plant Syst. Evol. 277 (1-2), 75-84 (2009)
REFERENCE 2 (bases 1 to 1302)
AUTHORS Neubig,K.M., Whitten,W.M., Carlsward,B.S., Blanco,M.A.,
Endara,C.L., Williams,N.H. and Moore,M.J.
TITLE Direct Submission
JOURNAL Submitted (14-FEB-2008) Department of Botany, University of
Florida, 220 Bartram Hall, Gainesville, FL 32611-8526, USA
FEATURES Location/Qualifiers
source 1..1302
/organism="Selenipedium aequinoctiale"
/organelle="plastid:chloroplast"
/mol_type="genomic DNA"
/specimen_voucher="FLAS:Blanco 2475"
/db_xref="taxon:256374"
gene <1..>1302
/gene="matK"
CDS <1..>1302
/gene="matK"
/codon_start=1
/transl_table=11
/product="maturase K"
/protein_id="ACC99456.1"
/translation="IFYEPVEIFGYDNKSSLVLVKRLITRMYQQNFLISSVNDSNQKG
FWGHKHFFSSHFSSQMVSEGFGVILEIPFSSQLVSSLEEKKIPKYQNLRSIHSIFPFL
EDKFLHLNYVSDLLIPHPIHLEILVQILQCRIKDVPSLHLLRLLFHEYHNLNSLITSK
KFIYAFSKRKKRFLWLLYNSYVYECEYLFQFLRKQSSYLRSTSSGVFLERTHLYVKIE
HLLVVCCNSFQRILCFLKDPFMHYVRYQGKAILASKGTLILMKKWKFHLVNFWQSYFH
FWSQPYRIHIKQLSNYSFSFLGYFSSVLENHLVVRNQMLENSFIINLLTKKFDTIAPV
ISLIGSLSKAQFCTVLGHPISKPIWTDFSDSDILDRFCRICRNLCRYHSGSSKKQVLY
RIKYILRLSCARTLARKHKSTVRTFMRRLGSGLLEEFFMEEE"
ORIGIN
1 attttttacg aacctgtgga aatttttggt tatgacaata aatctagttt agtacttgtg
61 aaacgtttaa ttactcgaat gtatcaacag aattttttga tttcttcggt taatgattct
121 aaccaaaaag gattttgggg gcacaagcat tttttttctt ctcatttttc ttctcaaatg
181 gtatcagaag gttttggagt cattctggaa attccattct cgtcgcaatt agtatcttct
241 cttgaagaaa aaaaaatacc aaaatatcag aatttacgat ctattcattc aatatttccc
301 tttttagaag acaaattttt acatttgaat tatgtgtcag atctactaat accccatccc
361 atccatctgg aaatcttggt tcaaatcctt caatgccgga tcaaggatgt tccttctttg
421 catttattgc gattgctttt ccacgaatat cataatttga atagtctcat tacttcaaag
481 aaattcattt acgccttttc aaaaagaaag aaaagattcc tttggttact atataattct
541 tatgtatatg aatgcgaata tctattccag tttcttcgta aacagtcttc ttatttacga
601 tcaacatctt ctggagtctt tcttgagcga acacatttat atgtaaaaat agaacatctt
661 ctagtagtgt gttgtaattc ttttcagagg atcctatgct ttctcaagga tcctttcatg
721 cattatgttc gatatcaagg aaaagcaatt ctggcttcaa agggaactct tattctgatg
781 aagaaatgga aatttcatct tgtgaatttt tggcaatctt attttcactt ttggtctcaa
841 ccgtatagga ttcatataaa gcaattatcc aactattcct tctcttttct ggggtatttt
901 tcaagtgtac tagaaaatca tttggtagta agaaatcaaa tgctagagaa ttcatttata
961 ataaatcttc tgactaagaa attcgatacc atagccccag ttatttctct tattggatca
1021 ttgtcgaaag ctcaattttg tactgtattg ggtcatccta ttagtaaacc gatctggacc
1081 gatttctcgg attctgatat tcttgatcga ttttgccgga tatgtagaaa tctttgtcgt
1141 tatcacagcg gatcctcaaa aaaacaggtt ttgtatcgta taaaatatat acttcgactt
1201 tcgtgtgcta gaactttggc acggaaacat aaaagtacag tacgcacttt tatgcgaaga
1261 ttaggttcgg gattattaga agaattcttt atggaagaag aa
//
# save the sequence data to a local file, and then parse it with Bio.SeqIO
import os
from Bio import SeqIO
from Bio import Entrez
Entrez.email = "A.N.Other@example.com" # Always tell NCBI who you are
filename = "EU490707.gbk"
if not os.path.isfile(filename):
# Downloading...
net_handle = Entrez.efetch(
db="nucleotide", id="EU490707", rettype="gb", retmode="text")
out_handle = open(filename, "w")
out_handle.write(net_handle.read())
out_handle.close()
net_handle.close()
print("Saved")
print("Parsing...")
record = SeqIO.read(filename, "genbank")
print(record)
ELink: Searching for related items in NCBI Entrez
# ELink: Searching for related items in NCBI Entrez
# can be used to find related items in the NCBI Entrez databases
# Let’s use ELink to find articles related to the Biopython application note published in Bioinformatics in 2009. The PubMed ID of this article is 19304878
from Bio import Entrez
Entrez.email = "A.N.Other@example.com" # Always tell NCBI who you are
pmid = "19304878"
record = Entrez.read(Entrez.elink(dbfrom="pubmed", id=pmid)) #The record variable consists of a Python list
# Since we specified only one PubMed ID to search for, record contains only one item. This item is a dictionary
# containing information about our search term, as well as all the related items that were found
# record[0] #list of one element
len(record[0]["LinkSetDb"]) #The "LinkSetDb" key contains the search results, a list consisting of one item for each target database
for linksetdb in record[0]["LinkSetDb"]:
print(linksetdb["DbTo"], linksetdb["LinkName"], len(linksetdb["Link"]))
record[0]["LinkSetDb"][0]["Link"][0] #The actual search results are stored as under the "Link" key
for link in record[0]["LinkSetDb"][0]["Link"]:
print(link["Id"])
pubmed pubmed_pubmed 168
pubmed pubmed_pubmed_alsoviewed 22
pubmed pubmed_pubmed_citedin 1332
pubmed pubmed_pubmed_combined 6
pubmed pubmed_pubmed_five 6
pubmed pubmed_pubmed_refs 17
pubmed pubmed_pubmed_reviews 15
pubmed pubmed_pubmed_reviews_five 8
{'Id': '19304878'}
Parsing Medline records
# Parsing Medline records
# You can find the Medline parser in Bio.Medline . MEDLINE format used in PubMed
from Bio import Medline
with open("pubmed_result1.txt") as handle:
record = Medline.read(handle) #The record now contains the Medline record as a Python dictionary
record["PMID"]
record["AB"]
#To parse a file containing multiple Medline records, you can use the parse function instead
with open("pubmed_result2.txt") as handle:
for record in Medline.parse(handle):
print(record["TI"])
Parsing GEO records
# Parsing GEO records
# The Bio.Geo module can be used to parse GEO-formatted data
# The following code fragment shows how to parse the example GEO file GSE16.txt into a record and print the record:
from Bio import Geo
handle = open("GSE16.txt")
records = Geo.parse(handle)
for record in records:
print(record)
Parsing UniGene records
# Parsing UniGene records
# UniGene is an NCBI database of the transcriptome, with each UniGene record showing the set of transcripts
# that are associated with a particular gene in a specific organism
# This particular record shows the set of transcripts (shown in the SEQUENCE lines)
# The PROTSIM lines show proteins with significant similarity to target
# the STS lines show the corresponding sequence-tagged sites in the genome
from Bio import UniGene
input = open("myunigenefile.data")
record = UniGene.read(input)
record.ID
record.title
record.sts[0].acc
record.sts[0].unists
input = open("unigenerecords.data")
records = UniGene.parse(input) #iterators
for record in records:
print(record.ID)
ESpell: Obtaining spelling suggestions
# ESpell: Obtaining spelling suggestions
# ESpell retrieves spelling suggestions. In this example, we use Bio.Entrez.espell() to obtain the correct spelling of Biopython
from Bio import Entrez
Entrez.email = "A.N.Other@example.com" # Always tell NCBI who you are
handle = Entrez.espell(term="biopythooon")
record = Entrez.read(handle)
record["Query"]
'biopythooon'
record["CorrectedQuery"]
'biopython'
ESummary: Retrieving summaries from primary IDs
# ESummary: Retrieving summaries from primary IDs
# ESummary retrieves document summaries from a list of primary IDs
handle = Entrez.esummary(db="nlmcatalog", id="101660833")
record = Entrez.read(handle)
info = record[0]["TitleMainList"][0]
print("Journal info\nid: {}\nTitle: {}".format(record[0]["Id"], info["Title"]))
Journal info
id: 101660833
Title: IEEE transactions on computational imaging.
example 1
# example 1
from Bio import Entrez
Entrez.email = "A.N.Other@example.com" # Always tell NCBI who you are
handle = Entrez.esearch(db="pubmed", term="biopython")
record = Entrez.read(handle)
record["IdList"]
['19304878', '18606172', '16403221', '16377612', '14871861', '14630660', '12230038']
#We now use Bio.Entrez.efetch to download these Medline records:
idlist = record["IdList"]
handle = Entrez.efetch(db="pubmed", id=idlist, rettype="medline", retmode="text")
# Here, we specify rettype="medline", retmode="text" to obtain the Medline records in plain-text Medline
# format. Now we use Bio.Medline to parse these records
from Bio import Medline
records = Medline.parse(handle)
for record in records:
print(record["AU"])
example 2
# example 2
# In this example, we will query PubMed for all articles having to do with orchids
from Bio import Entrez
Entrez.email = "A.N.Other@example.com" # Always tell NCBI who you are
handle = Entrez.egquery(term="orchid") #check how many of such articles there are
record = Entrez.read(handle)
for row in record["eGQueryResult"]:
if row["DbName"]=="pubmed":
print(row["Count"])
# Now we use the Bio.Entrez.efetch function to download the PubMed IDs of these 463 articles
handle = Entrez.esearch(db="pubmed", term="orchid", retmax=463)
record = Entrez.read(handle)
handle.close()
idlist = record["IdList"] #This returns a Python list containing all of the PubMed IDs of articles related to orchids
# get the corresponding Medline records and extract the information from them
from Bio import Medline
handle = Entrez.efetch(db="pubmed", id=idlist, rettype="medline",retmode="text")
records = Medline.parse(handle)
records = list(records)
for record in records:
print("title:", record.get("TI", "?"))
print("authors:", record.get("AU", "?"))
print("source:", record.get("SO", "?"))
print("")
example 3
Searching, downloading, and parsing Entrez Nucleotide records
# example 3
# Searching, downloading, and parsing Entrez Nucleotide records
from Bio import Entrez
Entrez.email = "A.N.Other@example.com" # Always tell NCBI who you are
handle = Entrez.egquery(term="Cypripedioideae") # EGQuery will tell us how many search results were found in each of the databases
record = Entrez.read(handle)
for row in record["eGQueryResult"]:
if row["DbName"]=="nuccore": # we are only interested in nucleotides
print(row["Count"])
handle = Entrez.esearch(db="nucleotide", term="Cypripedioideae", retmax=814, idtype="acc")
# use the retmax argument to restrict the maximum number of records retrieved
record = Entrez.read(handle)
handle.close()
# We can download these records using efetch
idlist = ",".join(record["IdList"][:5])
print(idlist)
# KX265015.1, KX265014.1, KX265013.1, KX265012.1, KX265011.1]
handle = Entrez.efetch(db="nucleotide", id=idlist, retmode="xml")
records = Entrez.read(handle)
len(records)
Searching, downloading, and parsing GenBank records
# Searching, downloading, and parsing GenBank records
from Bio import Entrez
Entrez.email = "A.N.Other@example.com" # Always tell NCBI who you are
handle = Entrez.egquery(term="Opuntia AND rpl16")
record = Entrez.read(handle)
for row in record["eGQueryResult"]:
if row["DbName"]=="nuccore":
print(row["Count"])
# download the list of GenBank identifiers
handle = Entrez.esearch(db="nuccore", term="Opuntia AND rpl16")
record = Entrez.read(handle)
gi_list = record["IdList"]
gi_list
37
['1972904692', '1972904685', '1972904678', '1972904671', '1972904664', '1972904657', '1972904650', '1972904643', '1972904636', '1972904629', '1972904622', '1841709044', '377581039', '330887241', '330887240', '330887239', '330887238', '330887237', '330887236', '330887235']
# Now we use these GIs to download the GenBank records
gi_str = ",".join(gi_list)
handle = Entrez.efetch(db="nuccore", id=gi_str, rettype="gb", retmode="text")
text = handle.read()
# or download friendly
handle = Entrez.efetch(db="nuccore", id=gi_str, rettype="gb", retmode="text")
records = SeqIO.parse(handle, "gb") #iterators
Using the history and WebEnv – most import
# Using the history and WebEnv -- most import
# the NCBI prefer you to take advantage of their history support - for example combining ESearch and EFetch
# Another typical use of the history support would be to combine EPost and EFetch
# Searching for and downloading sequences using the history
# To do this, call Bio.Entrez.esearch() as normal, but with the additional argument of usehistory="y"
from Bio import Entrez
Entrez.email = "history.user@example.com" # Always tell NCBI who you are
search_handle = Entrez.esearch(db="nucleotide",term="Opuntia[orgn] and rpl16",usehistory="y", idtype="acc")
search_results = Entrez.read(search_handle)
search_handle.close()
acc_list = search_results["IdList"]
len(acc_list) #the XML output includes the first retmax search results, with retmax defaulting to 20
count = int(search_results["Count"])
webenv = search_results["WebEnv"]
query_key = search_results["QueryKey"]
# You use the retstart and retmax parameters to specify which range of search results you want
# returned (starting entry using zero-based counting, and maximum number of results to return)
batch_size = 3
out_handle = open("orchid_rpl16.fasta", "w")
for start in range(0, count, batch_size):
end = min(count, start + batch_size)
print("Going to download record %i to %i" % (start + 1, end))
fetch_handle = Entrez.efetch(
db="nucleotide",
rettype="fasta",
retmode="text",
retstart=start,
retmax=batch_size,
webenv=webenv,
query_key=query_key,
idtype="acc",
)
data = fetch_handle.read()
fetch_handle.close()
out_handle.write(data)
out_handle.close()
Searching for and downloading abstracts using the history
# Searching for and downloading abstracts using the history
from Bio import Entrez
Entrez.email = "history.user@example.com"
search_results = Entrez.read(Entrez.esearch(db="pubmed", term="Opuntia[ORGN]", reldate=365, datetype="pdat", usehistory="y"))
count = int(search_results["Count"])
print("Found %i results" % count)
batch_size = 10
out_handle = open("recent_orchid_papers.txt", "w")
for start in range(0, count, batch_size):
end = min(count, start + batch_size)
print("Going to download record %i to %i" % (start + 1, end))
fetch_handle = Entrez.efetch(
db="pubmed",
rettype="medline",
retmode="text",
retstart=start,
retmax=batch_size,
webenv=search_results["WebEnv"],
query_key=search_results["QueryKey"],
)
data = fetch_handle.read()
fetch_handle.close()
out_handle.write(data)
out_handle.close()