I have a lot of files, in a lot of nested directories: 347430 directories, 379286 files
To anonymize what I’m doing we’ll say I have two types of files:
apples.csv.gz and oranges.csv.gz (if you don’t know- “gz” means the
Chase Clark chasemc
chasemc
/ gist:d3a40fcb2afd64a7fba1affaa65f1fb9
Created
August 13, 2020 14:12
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| library(ggplot2) | |
| library(data.table) | |
| library(geofacet) | |
| library(magrittr) | |
| raw_data <- data.table::fread("http://covidtracking.com/api/states/daily.csv") | |
| raw_data$date <- as.Date(as.character(raw_data$date), "%Y%m%d") | |
| raw_data <- raw_data[date > "2020-03-15", ] |
chasemc
/ update_idbac_db.R
Created
September 4, 2020 13:51
Update to new IDBAc db schema (currently need "prioritizer" git branch of IDBac)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| old_files_path <- "/home/user/Downloads/db/old" | |
| a <- list.files(old_files_path, full.names = FALSE) | |
| a <- tools::file_path_sans_ext(a) | |
| for(i in a) { | |
| pool <- IDBacApp::idbac_connect(fileName = i, | |
| filePath = old_files_path)[[1]] | |
| IDBacApp::idbac_update_db(pool = pool, | |
| copy_overwrite = "copy") |
chasemc
/ rich_print_table.py
Created
December 3, 2020 23:35
Print delimited files to terminal with https://github.com/willmcgugan/rich (Dependencies= pandas, rich)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import os | |
| import pathlib | |
| from rich.console import Console | |
| from rich.table import Table | |
| from rich.table import Column | |
| import pandas as pd | |
| def make_rich(df, title="mytitle"): | |
| table = Table(title=title) |
chasemc
/ rmarkdown.md
Last active
January 18, 2021 21:04
Count how many lines/rows contain X, from many files
chasemc
/ _run.sh
Created
January 24, 2021 19:14
— forked from jexp/_run.sh
Rendering large graphs with vivagraph.js, neo4j-javscript-driver (binary-bolt), meetup dataset and compiled runtime. Oh the joy :)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| npm install neo4j-driver | |
| node test-neo-driver.js |
chasemc
/ antismash_get_cds.sh
Created
January 28, 2021 14:21
Get CDS IDs from all antismash genbank files in all subdirectories
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| find -name "*region*gbk" | xargs grep "CDS " -A3 | grep "/ID=" | cut -d'"' -f2 |
chasemc
/ threaded_interproscan.sh
Last active
February 1, 2021 20:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| https://github.com/Micromeda/InterProScan-Docker/blob/master/LICENSE | |
| Apache License | |
| Version 2.0, January 2004 | |
| http://www.apache.org/licenses/ | |
| TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION | |
| 1. Definitions. |
chasemc
/ ga_and_len.sh
Created
February 21, 2021 18:56
Extract genomic_accessions and lengths from "ftp.ncbi.nlm.nih.gov/genomes............._assembly_report.txt"
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/bash | |
| curl -s $1 |\ | |
| sed -ne '/# Sequence-Name\tSequence-Role\tAssigned-Molecule\tAssigned-Molecule-Location\/Type\tGenBank-Accn\tRelationship\tRefSeq-Accn\tAssembly-Unit\tSequence-Length\tUCSC-style-name/,$ p' |\ | |
| awk -F"\t" 'NR==1 {for (i=1; i<=NF; i++) {f[$i] = i}}{ print $(f["RefSeq-Accn"]), $(f["Sequence-Length"])}' |\ | |
| sed 1d |
chasemc
/ mzml_to_csv.R
Created
March 17, 2021 21:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env Rscript | |
| args = commandArgs(trailingOnly=TRUE) | |
| message("Installing necessary libraries if not already installed") | |
| if (!requireNamespace("BiocManager", quietly = TRUE)) | |
| install.packages("BiocManager") | |
| if (!requireNamespace("mzR", quietly = TRUE)) | |
| install.packages("mzR") | |
| if (!requireNamespace("data.table", quietly = TRUE)) | |
| install.packages("data.table") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/bash | |
| # $1 is the file(s') name to find and hash | |
| # $2 is the extension to be given to each renamed file | |
| find $1 -print0 | xargs -0 md5sum | | |
| while read -r newname oldname; do | |
| mv -v "$oldname" "$newname".$2 | |
| done | |