Skip to content

Instantly share code, notes, and snippets.

@pydemo
pydemo / hdfs_pq_access.py
Created June 11, 2020 13:11 — forked from vepetkov/hdfs_pq_access.py
Python HDFS + Parquet (hdfs3, PyArrow + libhdfs, HdfsCLI + Knox)
##################################################################
## Native hdfs access (only on the cluster)
# conda install -c conda-forge libhdfs3=2.3.0=1 hdfs3 --yes
import hdfs3
import pandas as pd
nameNodeHost = 'hadoopnn1.localdomain'
nameNodeIPCPort = 8020
hdfs = hdfs3.HDFileSystem(nameNodeHost, port=nameNodeIPCPort)
@pydemo
pydemo / build-static-python.sh
Last active February 3, 2021 23:43 — forked from maddouri/build-static-python.sh
A simple script that builds static versions of Python and LibPython using musl-libc
We were unable to process your request.
We're required by law to inform you of the exact fees you will incur for international wires, including fees from other banks. For this request, we do not have the exact fees from other banks and therefore are not able to process it. If your request was in US dollars, you may try again in foreign currency -- this may help us determine the exact fees incurred for this transaction and enable us to send the wire request successfully. If you have received this message for both US dollars and foreign currency, it means we cannot process this request. We are sorry for any inconvenience.
#!/bin/bash
# set -eux
# This a simple script that builds static versions of Python and LibPython using musl-libc
# Find the associated article at: http://general-purpose.io/2015/12/06/compiling-python-and-libpython-statically-using-musl-libc/
@pydemo
pydemo / multiprocessing-queue-manager-client.py
Created July 20, 2021 16:40 — forked from mrjohannchang/multiprocessing-queue-manager-client.py
Python multiprocessing server client inter-process communication (IPC) over Queue. https://stackoverflow.com/a/62608473/1592410
import multiprocessing
import multiprocessing.managers
import os
import sys
from typing import AnyStr, Union
class QueueManager(multiprocessing.managers.BaseManager):
def get_queue(self, ident: Union[AnyStr, int, type(None)] = None) -> multiprocessing.Queue:
@pydemo
pydemo / busmon.py
Created August 23, 2021 16:07 — forked from kennovation1/busmon.py
Continuously monitor and pretty print Amazon EventBridge traffic as captured in CloudWatch Logs
# busmon.py
# Monitor CloudWatch Logs for EventBridge traffic.
#
# This script assumes that there is a Lambda function snooping an EventBridge bus
# and logging the events to a CloudWatch Logs group.
#
# Your format will likely be different and will require slight adjustments to the parsing
# logic below. The default format assumed for this version is:
# timestamp and other stuff|field|field|field|SNOOPED detail-type: <eventName> jsonString
# That is 5 pipe-delimited fields, where the last field is the only field used.
@pydemo
pydemo / slack-messages-by-user.py
Created November 14, 2021 17:18 — forked from demmer/slack-messages-by-user.py
count slack messages posted to a channel by user
#!/usr/bin/python
'''
Script to count messages by user posted to a channel for a given date range.
Install:
# sudo pip install slackclient
Also you will need to obtain a slack API token:
https://api.slack.com/docs/oauth-test-tokens
@pydemo
pydemo / sfdc.py
Created April 6, 2022 13:49 — forked from peterlyeung/sfdc.py
Lambda function to copy files from a SFTP server and add it to Salesforce as an attachment
import paramiko
import requests
import base64
import json
from simple_salesforce import Salesforce
#---------------------------------
# FTP portion
#---------------------------------
@pydemo
pydemo / flickr_uploader.py
Created May 4, 2023 21:40 — forked from dodo5522/flickr_uploader.py
Python script to upload a photo into flickr.
#!/usr/bin/env python2
# -*- encoding:utf-8 -*-
from __future__ import print_function
import argparse
import datetime
import exifread
import flickrapi
import os
import sys