Searching PubMed with Biopython

search_biopython.py

# This code uses Biopython to retrieve lists of articles from pubmed
# you need to install Biopython first.

# If you use Anaconda:
# conda install biopython

# If you use pip/venv:
# pip install biopython

# Full discussion:
# https://marcobonzanini.wordpress.com/2015/01/12/searching-pubmed-with-python/

from Bio import Entrez

def search(query):
    Entrez.email = '[email protected]'
    handle = Entrez.esearch(db='pubmed', 
                            sort='relevance', 
                            retmax='20',
                            retmode='xml', 
                            term=query)
    results = Entrez.read(handle)
    return results

def fetch_details(id_list):
    ids = ','.join(id_list)
    Entrez.email = '[email protected]'
    handle = Entrez.efetch(db='pubmed',
                           retmode='xml',
                           id=ids)
    results = Entrez.read(handle)
    return results

if __name__ == '__main__':
    results = search('fever')
    id_list = results['IdList']
    papers = fetch_details(id_list)
    for i, paper in enumerate(papers['PubmedArticle']):
        print("{}) {}".format(i+1, paper['MedlineCitation']['Article']['ArticleTitle']))

Have you encountered the issue that the server will cut you off after querying ~2000 IDs?

Should the Line 32, 33 be changed into:
for i, paper in enumerate(papers['PubmedArticle']): print("%d) %s" % (i+1, paper['MedlineCitation']['Article']['ArticleTitle']))

@ThitherShore is correct - I can verify that suggested fix makes this gist functional.

@hongtao510 we download 100 articles in an infinite loop. We see no blocking info since one week.

Please update code with ThitherShore's comment.

Whenever there is a "+" sing in the content (for example ArticleTitle or AbstractText), it returns a string only after the "+" sign. Does anyone have a way to get around this?
e.g. The title "The Clinicopathological and Prognostic Implications of FoxP3+ Regulatory T Cells in Patients with Colorectal Cancer: A Meta-Analysis." will return "+ Regulatory T Cells in Patients with Colorectal Cancer: A Meta-Analysis"

It looks like the format returned by the efetch method is slightly different now

If you replace papers with papers[‘PubmedArticle’] you should get the list or papers,

@ThitherShore is correct we should use enumerate(papers['PubmedArticle']) instead of enumerate(papers)

for line 36

print(json.dumps(papers['PubmedArticle'][0], indent=2, separators=(',', ':')))
instead of:
print(json.dumps(papers[0], indent=2, separators=(',', ':')))

I get abstracts and not the full text. Any reason why?

I get abstracts and not the full text. Any reason why?

Pubmed does not contain full texts of papers. Abstracts only

How may I save the result in CSV, with Title and Abstract columns?

Quick question: is there a extra ")" (or missing "(" in line 39?
print("%d) %s" % (i+1, paper['MedlineCitation']['Article']['ArticleTitle']))

#edited for formatting

How may I save the result in CSV, with Title and Abstract columns?

@MLZTazim - I'm in the same boat: learning how to use python to drive json to the result. Good fun!
A hint: https://www.geeksforgeeks.org/json-dumps-in-python/

Quick question: is there a extra ")" (or missing "(" in line 39?
print("%d) %s" % (i+1, paper['MedlineCitation']['Article']['ArticleTitle']))

#edited for formatting

@sidewinder02139 the syntax is correct: note the first ")" on that line is part of the output string

Quick question: is there a extra ")" (or missing "(" in line 39?
print("%d) %s" % (i+1, paper['MedlineCitation']['Article']['ArticleTitle']))
#edited for formatting

@sidewinder02139 the syntax is correct: note the first ")" on that line is part of the output string

DOH! LOL
Have a brilliant weekend and stay healthy!
btw, I love the code. Well done!

ThitherShore is correct. Your code wont work until you enumerate papers['PubmedArticle']
Thank you for the example, but please change this soon so as not to confuse others.
I spent a while trying to figure out what was wrong.

While were at it the last line doesn't work either for the same reason, should be:
import json
print(json.dumps(papers['PubmedArticle'][0], indent=2, separators=(',', ':')))

ThitherShore is correct. Your code wont work until you enumerate papers['PubmedArticle']
Thank you for the example, but please change this soon so as not to confuse others.
I spent a while trying to figure out what was wrong.

While were at it the last line doesn't work either for the same reason, should be:
import json
print(json.dumps(papers['PubmedArticle'][0], indent=2, separators=(',', ':')))

@jajkelle Updated (better late than never), thank you all for pointing it out

This is awesome thanks and works pretty well. I'm stuck on trying to get the details of the authors in a succinct way, can anybody help with how to do that? paper['MedlineCitation']['Article']['AuthorList'] isnt right....
Thanks in advance!

Thanks this is very helpful. Somewhat similar to @echorule, I'm having trouble obtaining the full abstract of a paper as one string. What I get looks like this:

StringElement(' ... ', attributes={'Label': 'BACKGROUND', 'NlmCategory': 'BACKGROUND'}), StringElement(' ... ', attributes={'Label': 'METHODS', 'NlmCategory': 'METHODS'}), StringElement('...' ...)

Is there a more reasonable way to go about this?

Thanks this is very helpful. Somewhat similar to @echorule, I'm having trouble obtaining the full abstract of a paper as one string. What I get looks like this:

StringElement(' ... ', attributes={'Label': 'BACKGROUND', 'NlmCategory': 'BACKGROUND'}), StringElement(' ... ', attributes={'Label': 'METHODS', 'NlmCategory': 'METHODS'}), StringElement('...' ...)

Is there a more reasonable way to go about this?

@rsgoncalves did you manage to resolve this? You can fetch abstract by using: paper['MedlineCitation']['Article']['Abstract']['AbstractText']

So your full code retrieving for author, title and abstract could look something like:

from Bio import Entrez

def search(query):
    Entrez.email = '[email protected]'
    handle = Entrez.esearch(db='pubmed',
                            sort='relevance',
                            retmax='20',
                            retmode='xml',
                            term=query)
    results = Entrez.read(handle)
    return results

def fetch_details(id_list):
    ids = ','.join(id_list)
    Entrez.email = '[email protected]'
    handle = Entrez.efetch(db='pubmed',
                           retmode='xml',
                           id=ids)
    results = Entrez.read(handle)
    return results

def get_abstract(paper):
    abstract = ''
    if 'Abstract' in paper['MedlineCitation']['Article']:
        abstract = paper['MedlineCitation']['Article']['Abstract']['AbstractText']
        if isinstance(abstract, list):
            abstract = ' '.join(abstract)
    return abstract

if __name__ == '__main__':
    results = search('fever')
    id_list = results['IdList']
    papers = fetch_details(id_list)
    for i, paper in enumerate(papers['PubmedArticle']):
        title = paper['MedlineCitation']['Article']['ArticleTitle']
        author_list = paper['MedlineCitation']['Article']['AuthorList']
        authors = ', '.join([author.get('LastName', '') for author in author_list])
        abstract = get_abstract(paper)
        print("{}) Title: {}".format(i+1, title))
        print("   Authors: {}".format(authors))
        print("   Abstract: {}".format(abstract))
        print()