sudo apt-get install python-setuptoolswget http://downloads.sourceforge.net/project/s3tools/s3cmd/1.5.0-alpha1/s3cmd-1.5.0-alpha1.tar.gztar xvfz s3cmd-1.5.0-alpha1.tar.gzcd s3cmd-1.5.0-alpha1python setup.py installs3cmd --configure
Brendan Collins brendancol
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from __future__ import print_function | |
| # std lib | |
| import os | |
| import sys | |
| # 3rd party | |
| import pandas | |
| import numpy | |
| numpy.random.seed(10) |
brendancol
/ pyspark environment setup
Created
May 30, 2015 04:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # install-on-workers.sh | |
| while read slave_url; do | |
| echo "Installing dependencies to ${slave_url}" | |
| ssh root@${slave_url} 'bash -s' < worker-install.sh | |
| done <persistent-hdfs/conf/slaves | |
| # worker-install.sh | |
| yum install libgeos-devel -y | |
| yum install python-devel -y |
brendancol
/ fix_spark_ebs.sh
Last active
October 1, 2015 02:14
When deploying Spark with spark-ec2 with --ebs-vol-size option, the mount location is wrong by default, this script: stops cluster, fixes mount location, reformats hdfs and restarts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env bash | |
| ~/persistent-hdfs/sbin/stop-all.sh | |
| sed -i 's#vol/persistent-hdfs#vol0/persistent-hdfs#g' ~/persistent-hdfs/conf/core-site.xml | |
| ~/spark-ec2/copy-dir.sh ~/persistent-hdfs/conf/core-site.xml | |
| ~/persistent-hdfs/bin/hadoop namenode -format -y | |
| ~/persistent-hdfs/sbin/start-all.sh |
brendancol
/ s3cmd_install.md
Last active
August 29, 2015 14:14
— forked from dominicsayers/s3cmd_install.md
brendancol
/ soe_utils.py
Created
October 17, 2013 18:22
SOE Utils (taken from ArcGIS Server Admin Toolbox) requires requests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ''' | |
| This script will upload and register an SOE | |
| ==Inputs== | |
| ServerName | |
| Port | |
| AdminUser | |
| AdminPassword (sent in clear text) | |
| SOE (file) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| def write_csv(outputRowDictionaries, outputFile, fieldNames=None, defaultFieldValue='', removeDuplicates=False, sortField=None): | |
| dictionaryList = [] | |
| if removeDuplicates: | |
| for row in outputRowDictionaries: | |
| if row not in dictionaryList: | |
| dictionaryList.append(row) | |
| else: | |
| dictionaryList = outputRowDictionaries | |
brendancol
/ gist:6639651
Created
September 20, 2013 15:52
Loosely ordered with the commands I use most towards the top. Sublime also offer full documentation.
| Ctrl+C | copy current line (if no selection) |
| Ctrl+X | cut current line (if no selection) |
| Ctrl+⇧+K | delete line |
| Ctrl+↩ | insert line after |
brendancol
/ BoundingCurve.cs
Created
August 30, 2013 21:07
— forked from erussell/BoundingCurve.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| using System; | |
| using System.Collections.Generic; | |
| using ESRI.ArcGIS.DataSourcesRaster; | |
| using ESRI.ArcGIS.Geodatabase; | |
| using ESRI.ArcGIS.Geometry; | |
| internal enum CurveDirection { RIGHT, UP, LEFT, DOWN }; | |
| public sealed class BoundingCurve | |
| { |
brendancol
/ overlapping_polygon_example.cs
Created
August 8, 2013 16:09
Example of handling overlapping polygons when using ArcObjects IZonalOps
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| using System; | |
| using System.Collections.Generic; | |
| using System.Linq; | |
| using System.Text; | |
| using System.Collections.Specialized; | |
| using System.Runtime.InteropServices; | |
| using ESRI.ArcGIS.esriSystem; | |
| using ESRI.ArcGIS.Server; |
brendancol
/ zonal_stats.py
Created
July 24, 2013 20:11
— forked from perrygeo/zonal_stats.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| """ | |
| Zonal Statistics | |
| Vector-Raster Analysis | |
| Copyright 2013 Matthew Perry | |
| Usage: | |
| zonal_stats.py VECTOR RASTER | |
| zonal_stats.py -h | --help | |
| zonal_stats.py --version |