Justin Joyce joocer

Lead Developer of Opteryx, a federated SQL query engine in Python.

joocer / read_large_file.py

Last active October 24, 2020 09:48

read an arbitrary long file in python, line by line

	def read_file(filename, chunk_size=1024*1024, delimiter='\n'):
	with open(filename, 'r', encoding="utf8") as f:
	carry_forward = ''
	chunk = 'INITIALIZED'
	while len(chunk) > 0:
	chunk = f.read(chunk_size)
	augmented_chunk = carry_forward + chunk
	lines = augmented_chunk.split(delimiter)
	carry_forward = lines.pop()
	yield from lines

joocer / gist:b64b3c82e805ec50f3d2d073b1bed4b5

Last active August 11, 2020 08:56

qualys to off sample (note to jsonl)

	import pandas as pd
	import xmltodict
	import json
	import off

	FINDINGS_FILE = r'qualys\qualys_hosts.xml'
	SEVERITIES = ['Low', 'Low', 'Medium', 'Medium', 'High']
	CONFIDENCES = { 'Potential': 'Medium', 'Confirmed': 'High' }
	OFF_FILE = r'offl.json'

joocer / setup_env.bat

Last active July 11, 2020 15:13

A simple script to create and activate a virtual env, install requirements and running a main.py script

	@echo off

	set "VIRTUAL_ENV=%CD%\temp"
	set "PATH=%VIRTUAL_ENV%\Scripts;%PATH%"
	set "PROMPT=(VIRTUAL) $P$G%"

	python -m venv temp
	pip install -r requirements.txt
	python main.py