This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
accession2url <- function(x) { | |
prefix <- "ftp://ftp.sra.ebi.ac.uk/vol1/fastq" | |
dir1 <- paste0("/",substr(x,1,6)) | |
dir2 <- ifelse(nchar(x) == 9, "", | |
ifelse(nchar(x) == 10, paste0("/00",substr(x,10,10)), | |
ifelse(nchar(x) == 11, paste0("/0",substr(x,10,11)), | |
paste0("/",substr(x,10,12))))) | |
paste0(prefix,dir1,dir2,"/",x) | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import os | |
from subprocess import check_output | |
from tempfile import mkstemp | |
import sys | |
import boto3 | |
import requests | |
s3 = boto3.client('s3') |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# -*- coding: utf-8 -*- | |
"""A simple tool to document how to control AWS resources. | |
AWS AUTHENTICATION | |
------------------- | |
In order to run any of the code below, you need a profile with AWS credentials | |
set up on your computer. It's very easy to do this. Google how to configure | |
your profile with boto3, or visit the docs: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from sqlalchemy.dialects import postgresql | |
def bulk_upsert(session: Session, | |
items: Sequence[Mapping[str, Any]]): | |
session.execute( | |
postgresql.insert(MyModel.__table__) | |
.values(items) | |
.on_conflict_do_update( | |
index_elements=[MyModel.id], | |
set_={MyModel.my_field.name: 'new_value'}, |