Skip to content

Instantly share code, notes, and snippets.

View ricklentz's full-sized avatar

RWL ricklentz

View GitHub Profile
@ricklentz
ricklentz / stored_procedure_starter.txt
Created July 26, 2017 01:36
beginner stored procedures
Run the following Stored Procedures on SQL Server 2014 using the SQL Server Management Studio.
Example 1
The following example runs the uspgetBillOfMaterials stored procedure. The procedure returns a hierarchical list of components used to manufacture the Road-550-W Yellow, 44 product (ProductID800).
USE AdventureWorks2014;
GO
DECLARE @CheckDate DATETIME;
SET @CheckDate = GETDATE();
--23,969,155,169 as of July 30, 2017
SELECT
Total_Rows= SUM(st.row_count)--object_name(object_id),*
FROM
sys.dm_db_partition_stats st
WHERE (index_id < 2) AND object_name(object_id) like 'Message_%'
@ricklentz
ricklentz / killing_it_with_kylin.txt
Created July 30, 2017 19:09
comments on OLAP, wait times, and not having to wait
I first heard about Online Analytical Processing (OLAP) during my M.S. program in computer information systems (2000-2003). Fast forward a few years, and I'll introduce my favorite tool for avoiding the thing I hate most about OLAP, waiting for analysis jobs.
If you have 23 TB of SQL Server data consisting of billions of data rows, stuck in a relational SQL Server database, you have to wait for analysis jobs. This is where Kylin enters the scene. "Apache Kylin™ lets you query massive data set at sub-second latency".
The key is in part distributing your data on the Hadoop cluster of your choosing. Then use Kylin to specify your cube based on star schema tables that on Hadoop. Then connect via ODBC, REST, or Tableau and just let it happen.
Behind the scenes, Kylin uses map reduce to generate aggregate data files, called HFiles. HBase uses HFiles to make and persist cube metadata and cube aggregates for each N-cuboid in HBase. Hive is also used, to store base level data, and for data modeling. Toge
"Thus, in an environment of declining prices and general tightening of lending, the necessity to deleverage can produce an ongoing cycle of forced sales, erosion of equity capital, and continued downward pressure on price."
"margin spiral situation results in a lower overall position value and lower borrowing amounts versus a loss spiral situation"
"A clearinghouse serves as a hub of information about the network of contracting parties and their positions and obligations. In this capacity, a clearinghouse can facilitate multilateral netting arrangements, which eliminates counterparty credit risk, and, thus, the need for additional funding protection."
"Shadow bank. A shadow bank is a financial institution other than a regulated depository institution. Examples of regulated depository institutions are commercial banks, thrifts, and credit unions. Examples of shadow banks are private equity funds, investment banks, hedge funds, mortgage lenders, and insurance companies."
https://fred.stlouisfed.org/series/A
SELECT State, COUNT(Supp_ID) "#Suppliers"
FROM Tb_Supplier
GROUP BY ROLLUP(State)
SELECT State, COUNT(Supp_ID) "#Suppliers"
FROM Tb_Supplier
GROUP BY State
SELECT City, COUNT(Supp_ID) "#Suppliers"
FROM Tb_Supplier
@ricklentz
ricklentz / information_gain_example.txt
Created August 5, 2017 20:37
basic data mining techniques demo
https://docs.google.com/spreadsheets/d/1Hm6Km-shwL6FHtkmdt3Mk3bYpfLsXTT-DeOV4KNIDpc/edit?usp=sharing

--apex - grand total SELECT { Measures.[Quantity - Tb Transactions], [Measures].[Tb Transactions Count], Measures.[Price - Tb Transactions] } ON COLUMNS FROM DS715

--base SELECT NON EMPTY { [Measures].[Price - Tb Transactions], [Measures].[Quantity - Tb Transactions] } ON COLUMNS,

If you are working in data mining and machine learning research, most of the problems are difficult. The thing is, sometimes the reason why they are hard can be solved in other ways—like better/different data, changing re-quirements, or adding special cases. So saying a problem would be “really hard” wasn’t a good enough reason not to try to solve it; and yet, it was con-sistently being used as the reason why things were not happening.
“And we found out that as long as a pregnant woman thinks she hasn’t been
spied on, she’ll use the coupons. She just assumes that everyone else on her block got
the same mailer for diapers and cribs. As long as we don’t spook her, it works.”
#!/usr/bin/python
import numpy
import matplotlib
matplotlib.use('agg')
import matplotlib.pyplot as plt
from studentRegression import studentReg
from class_vis import prettyPicture, output_image