mesmacosta
/ Dockerfile
Created
September 7, 2019 19:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| FROM python:3.7 | |
| WORKDIR /app | |
| # Copy the requirements.txt file and install all dependencies. | |
| COPY requirements.txt . | |
| RUN pip install -r requirements.txt | |
| # Downloading and installing gcloud package. | |
| RUN curl https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.tar.gz > /tmp/google-cloud-sdk.tar.gz | |
| RUN mkdir -p /usr/local/gcloud \ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env bash | |
| source ./env/bin/activate | |
| pytest --cov --cov-report html --cov-report term-missing | |
| open /Applications/Google\ Chrome.app htmlcov/index.html |
mesmacosta
/ docker on compute engine
Last active
September 23, 2019 18:06
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Login as super user | |
| sudo -s | |
| apt-get install curl | |
| curl -sSL https://get.docker.com/ | sh |
mesmacosta
/ hive hooks
Last active
September 24, 2019 00:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ╔═════════════════════════════╦════════════════════════════════════════════════════════════════╗ | |
| ║ Property ║ Interface Or Abstract class ║ | |
| ╠═════════════════════════════╬════════════════════════════════════════════════════════════════╗ | |
| ║ hive.exec.pre.hooks ║ org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext ║ | |
| ║ hive.exec.post.hooks ║ org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext ║ | |
| ║ hive.exec.failure.hooks ║ org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext ║ | |
| ║ hive.metastore.init.hooks ║ org.apache.hadoop.hive.metastore.MetaStoreInitListener ║ | |
| ║ hive.exec.driver.run.hooks ║ org.apache.hadoop.hive.ql.HiveDriverRunHook ║ | |
| ║ hive.semantic.analyser.hook ║ org.apache.hadoop.hive.ql.parse.AbstractSemanticAnalyzerHook ║ | |
| ╚═════════════════════════════╩════════════════════════════════════════════════════════════════╝ |
mesmacosta
/ Hive Metastore Listeners
Last active
November 18, 2019 00:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ╔═══════════════════════════════════════╦═══════════════════════════════════════════════════════════════╗ | |
| ║ Property ║ Abstract class ║ | |
| ╠═══════════════════════════════════════╬═══════════════════════════════════════════════════════════════╗ | |
| ║ hive.metastore.pre.event.listeners ║ org.apache.hadoop.hive.metastore.MetaStorePreEventListener ║ | |
| ║ hive.metastore.end.function.listeners ║ org.apache.hadoop.hive.metastore.MetaStoreEndFunctionListener ║ | |
| ║ hive.metastore.event.listeners ║ org.apache.hadoop.hive.metastore.MetaStoreEventListener ║ | |
| ╚═══════════════════════════════════════╩═══════════════════════════════════════════════════════════════╝ |
mesmacosta
/ Hive Metastore Events
Created
September 24, 2019 22:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ╔═══════════════════════════════════════════════╦════════════════════════╗ | |
| ║ Method ║ Parameter ║ | |
| ╠═══════════════════════════════════════════════╬════════════════════════╗ | |
| ║ MetaStoreEventListener.onConfigChange ║ ConfigChangeEvent ║ | |
| ║ MetaStoreEventListener.onCreateTable ║ CreateTableEvent ║ | |
| ║ MetaStoreEventListener.onDropTable ║ DropTableEvent ║ | |
| ║ MetaStoreEventListener.onAlterTable ║ AlterTableEvent ║ | |
| ║ MetaStoreEventListener.onAddPartition ║ AddPartitionEvent ║ | |
| ║ MetaStoreEventListener.onDropPartition ║ DropPartitionEvent ║ | |
| ║ MetaStoreEventListener.onAlterPartition ║ AlterPartitionEvent ║ |
mesmacosta
/ package.json
Last active
September 26, 2019 18:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "name": "sample-pubsub", | |
| "version": "0.0.1", | |
| "dependencies": { | |
| "@google-cloud/pubsub": "^0.18.0" | |
| } | |
| } |
mesmacosta
/ CustomHook.java
Last active
July 15, 2020 16:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package com.medium.hive.hook; | |
| import com.fasterxml.jackson.databind.ObjectMapper; | |
| import org.apache.hadoop.hive.ql.QueryPlan; | |
| import org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext; | |
| import org.apache.hadoop.hive.ql.hooks.HookContext; | |
| import org.apache.hadoop.hive.ql.hooks.HookContext.HookType; | |
| import org.apache.hadoop.hive.ql.hooks.ReadEntity; | |
| import org.apache.hadoop.hive.ql.hooks.WriteEntity; | |
| import org.apache.hadoop.hive.ql.plan.HiveOperation; |
mesmacosta
/ CustomListener.java
Last active
September 29, 2019 15:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package com.medium.hive.metastore.listerner; | |
| import org.apache.hadoop.conf.Configuration; | |
| import org.apache.hadoop.hive.metastore.events.AlterTableEvent; | |
| import org.apache.hadoop.hive.metastore.MetaStoreEventListener; | |
| import org.apache.hadoop.hive.metastore.events.CreateTableEvent; | |
| import org.codehaus.jackson.map.ObjectMapper; |
mesmacosta
/ memory_df.py
Last active
September 27, 2019 18:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| new_metadata = metadata.head(1).copy() | |
| for i in range(100): | |
| row = metadata.head(1) | |
| row['table_name'] = row['table_name'] + str(i) | |
| new_metadata = new_metadata.append(row) | |
| @classmethod | |
| def __get_df_memory_usage(cls, key_value, df): | |
| import io | |
| buf = io.StringIO() |