Skip to content

Instantly share code, notes, and snippets.

View ricklentz's full-sized avatar

RWL ricklentz

View GitHub Profile
@ricklentz
ricklentz / data_warehouse_notes.txt
Created June 3, 2017 02:39
Strategic Information Notes
make decisions to be competitive
attributes:
integrated (view of the global state)
truthful (accurate and consistent with business rules)
accessable (inforamtion consumers)
credible (no contradictions)
timely (when needed)
reasons for failure in early decision support systems:
lack flexibility to adapt
Data Science -
harness the value of data
Source: IBM - Computer Programming / Statistics / Domain & Business Knowledge
Build culture to improve decision making
On the business side, data scientists are strategic thinkers. They are always focused on solving complex organizational problems using data. They have a good understanding of how the business works so that they can ask right questions and engage with the stakeholders. They're always curious. They're good communicator and storytellers.
import os
dir = r'/home/...'
for filename in os.listdir(dir):
with open(dir + '/' + filename, encoding='utf-8') as f:
for idx, line in enumerate(f):
print(filename + ' ' + str(idx))
import pandas as pd
import os
path =r'/home/wrk' # use your path
allFiles = os.listdir(path)
frame = pd.DataFrame()
list_ = []
for file_ in allFiles:
df = pd.read_csv(path + '/' + file_,index_col=None, header=0)
list_.append(df)
@ricklentz
ricklentz / fix_trace_issue_analysis.txt
Created June 8, 2017 00:10
FIX trace issue analysis
install wireshark and launch a capture session with the tcp filter enabled
install ARC Welder, link to the .apk, and launch with metadata {"stderrLog": "V"}
set up the app by inspecting chrome://inspect/#apps and then opening the plugin shell plugin.shell('adbd') via the javascript console
launch adb using shell: .platform-tools/adb shell -e 0
pm list packages -f
@ricklentz
ricklentz / intro_to_scapy.txt
Created June 10, 2017 00:04
scapy analysis replay
characterize the protocol using wireshark, trigger different operating conditions if possible, and save the aggregate pcap data
a=rdpcap("/Users/stanford/Downloads/dump.pcapng")
notice the index is one less than you see in the wireshark gui.
view packet info at index: a[278]
@ricklentz
ricklentz / dev_and_pm_data_warehouse.txt
Created June 12, 2017 21:39
Development and Management of Data Warehouse projects
Many DW projects fail and are abandoned midstream
Fail due to inadequate project management
Evaluate if really needed, evaluate opportunity and viability
Compare cost of failure vs project not started at all
Bottom up vs Top down
Enterprise-wide centralized data warehouse
individual data marts first
faster cheaper, more limited benefits
100% in house programming (not a good idea)
You can use package manager (pm) over adb shell to list packages:
adb shell pm list packages
and to display where the .apk file is:
adb shell pm path com.king.candycrushsaga
package:/data/app/com.king.candycrushsaga-1/base.apk
And adb pull to download the apk.
adb pull data/app/com.king.candycrushsaga-1/base.apk
@ricklentz
ricklentz / convert_obj_to_ply.py
Created June 14, 2017 04:52
convert obj to ply
# export PYMESH_PATH=/home/cbios/github/PyMesh
import os
import pymesh
cold_store_dir = r'/media/cbios/ResearchData/Models/train_normal'
for filename in os.listdir(cold_store_dir):
mesh = pymesh.load_mesh(cold_store_dir +'/' + filename)
pymesh.save_mesh_raw(cold_store_dir +'/' + filename.replace(".obj",".ply"), mesh.vertices, mesh.faces, mesh.voxels)
Common theme to organizational differentiators of comprehensive analytics competitor:
Top management has communicated that the organization competes on quantitative, fact-based analysis
Widespread use of predictive modeling and optimization
Analyze the heck out of their data
Comprehensive understanding of their competitors
Optimize business strategy and planning, can determine impact of unexpected constraint, simulate alternatives
Model relationship between operational costs and performance
Can measure lift of intervention strategies and apply results to improve future analysis
For each subgroup, perform regression analysis to identify factors that most closely correlate with *business event*
Information-based strategy - Programs operate under common leadership, technology, and tools