Skip to content

Instantly share code, notes, and snippets.

# a method to quickly loop through all the folders/subfolders in a path and create a list of all csv files
# used regex to include only files with "output" in the name
ls.file <- list.files(path = "Z:/GIS/DataLibrary/test", pattern = "^.*output.*.csv$", full.names = TRUE, recursive = TRUE)
lapply(ls.file, function(i) {
ExtractDates(i)
})
# this takes a csv with a beginning & end address, pulls google walking directions between those points,
# and then saves the file in the ESRI Shapefile format, with each path as a feature
import googlemaps
import csv
import arcpy
from shapely.geometry import LineString, mapping, shape
from fiona import collection
from datetime import datetime
from osgeo import ogr
@black-tea
black-tea / CleanRiitsMetadata.R
Created December 21, 2015 02:41
Clean Traffic Volume Metadata
library(data.table)
### Inputs ###
meta.file <- "/Volumes/Seagate Backup Plus Drive/Volumes/arterial/arterial-config.csv"
### Load Data ###
meta.df <- fread(meta.file, header=T,sep="|",stringsAsFactors=F)
### Clean XY Data ###
x <- gsub('.+\\((-[0-9]+\\.[0-9]+),([0-9]+\\.[0-9]+).+','\\1',meta.df$START_LAT_LONG)
@black-tea
black-tea / ArcPy_AssignCollisiontoSegmentandInt.py
Last active December 30, 2015 21:04
Assigns collisions to nearest segment and then the nearest intersection associated with that segment.
# This script is intended to follow the ArcPy_AssignNodeIDstoSegment.py
# Once the segments have the Node IDs, this script will assign collisions
# to each segment and intersection.
import arcpy
from arcpy import env
##### Input #####
env.workspace = "Z:/GIS/DataLibrary/Transportation/BOE_Centerline_Intersections150930/collisiontoInt_test.gdb"
env.overwriteOutput = True #so we can rerun this script and overwrite our output feature class
@black-tea
black-tea / InrixDataProcessing.R
Created December 30, 2015 19:13
Used to comb through 6 months of INRIX speed data and pull out two weeks' worth. Each of the raw csv files is processed in 1-million row chunks. Key improvements to speed from implementing "fread" and "fastPOSIXct" and creating a list of data frames to combine at the end.
library(data.table)
library(fasttime)
start.time <- proc.time()
#parameters that will apply to the data
chunkSize <- 1000000
colnames <- c('Date.Time','XD.Segment','Speed','Score')
#import csv with list of XD.Segments to filter through
clipped.csv <- "Z:/GIS/DataLibrary/Inrix/LA County_15.2_Shapefiles/ClipforVZTechnicalAnalysis/ClippedInrixSegIDList.csv"
@black-tea
black-tea / ArcPy_AssignNodeIDstoSegment.py
Created December 30, 2015 22:42
This script will take the centerline layer, and copy the assetIDs of all intersecting points to the attribute table of the centerline layer. It will also make sure that merged intersections are appropriately accounted for as well.
import arcpy
from arcpy import env
##### Input #####
env.workspace = "Z:/GIS/DataLibrary/Transportation/BOE_Centerline_Intersections150930/collisiontoInt_test.gdb"
env.overwriteOutput = True #so we can rerun this script and overwrite our output feature class
#input fc
col_fc = "Collisions_2009to2013"
input_centerline_fc = "Street_Centerline_Raw"
library(data.table)
### Inputs ###
meta.file <- "Z:/GIS/DataLibrary/Riits/TrafficVolumes/data dump/EXPORT - 2015/config.dsv"
### Load Data ###
meta.df <- fread(meta.file, header=T,sep="|",stringsAsFactors=F)
### Clean XY Data ###
x <- gsub('.+\\((-[0-9]+\\.[0-9]+) {1}([0-9]+\\.[0-9]+).+','\\1',meta.df$START_LAT_LONG)
@black-tea
black-tea / ArcPySpatialJoin.py
Last active April 8, 2022 03:57
Basic Spatial Join in ArcPy. In this case, a one-to-many that will create new rows for every match.
#### This script will take the Riits Sensor Shp, and copy the assetIDs of all intersecting points to the attribute table of the Riits layer
import arcpy
from arcpy import env
env.workspace = "Z:/GIS/DataLibrary/Transportation/BOE_Centerline_Intersections150930/collisiontoInt_test.gdb"
env.overwriteOutput = True #so we can rerun this script and overwrite our output feature class
#input
Riits_sensor_shp = "Z:/GIS/DataLibrary/Riits/TrafficVolumes/data dump/EXPORT - 2015/RiitsSensorsproj.shp"
@black-tea
black-tea / RiitsVolumeProcessing.R
Created January 7, 2016 19:34
This script summarizes volume data we obtained from the RIITS program. It filters data for only those sensors performing above 90% reporting and then sums over each LinkID for the day/week.
library(data.table)
library(fasttime)
#input
week1.file = "Z:/GIS/DataLibrary/Riits/TrafficVolumes/data dump/EXPORT - 2015/october.dsv"
#parameters that will apply to the data
chunkSize <- 500000
#date ranges
@black-tea
black-tea / ArcPyMatchStreetNames_with_DifferentSuffix.py
Created February 1, 2016 19:24
This script splits a field called "AllStreets," which contains multiple streets separated by " & " and then checks to see if any matches a street name in the "Cross_St" field and updates a field called "Doyourname" with 0,1,2
import arcpy
from arcpy import env
##### Input #####
shp = "C:/Users/dotcid034/Desktop/Export_Output (1)/Export_Output.shp"
col_fields = ["Cross_St","AllStreets","Doyourname"]
cur = arcpy.da.UpdateCursor(shp,col_fields)
for row in cur:
val = None