Skip to content

Instantly share code, notes, and snippets.

View freemandealer's full-sized avatar
🎯
Working on my personal projects

zhengyu freemandealer

🎯
Working on my personal projects
47 followers · 44 following
View GitHub Profile
@freemandealer
freemandealer / get_profile_from_stepid.py
Last active June 25, 2024 06:50
通过性能平台的 stepid 获取对应测试的 log 和指定 sql 的 profile
#/bin/env python3
# -*- coding: utf-8 -*-
# @Time : 2024/06/01
# @Author : zhengyu ([email protected])
# @Description: 通过性能平台的 stepid 获取对应测试的 log 和指定 sql 的 profile
# 用法：python3 perf.py --stepid=stepid --sql=sql --idx=idx --output=output
# 必要参数：
# stepid: stepid
# sql: 如 q09、q19 (tpch格式)、query01、query68 (tpcds格式)
# 可选参数：
@freemandealer
freemandealer / data_annotate_util.html
Created August 3, 2023 05:44
<!DOCTYPE html>
<html>
<body>
<h1>对话生成评估</h1>
<p>选择TSV文件:</p>
<input type="file" id="inputTSV" onchange="loadFiles()"/>
<p>选择Model1预测:</p>
@freemandealer
freemandealer / get_orc_size.py
Created March 21, 2023 09:41
get memory consumption of data from an apache ORC format file
import pyorc
import sys
# 打开要读取的orc文件
with open("./output.orc", "rb") as data:
# 创建一个读取器对象
reader = pyorc.Reader(data)
# 初始化一个变量来存储总大小
total_size = 0
# 遍历文件中的每一行记录
@freemandealer
freemandealer / sparse_data_gen.py
Created February 16, 2023 12:50
Generate LowCardinality Data as JSON & ORC format
#!/usr/bin/env python
# pip install pyspark before execution
import random
import json
import os
import sys
from pyspark.sql import SparkSession
from pyspark import SparkContext
ROW_NUM = 500
@freemandealer
freemandealer / flaten_json.py
Created August 4, 2022 13:08
flaten json file
#!/bin/env python3
import json
import sys
def data_flatten(key,val,con_s='_',basic_types=(str,int,float,bool,complex,bytes)):
"""
数据展开生成器,以键值对为最基础的数据
param key: 键，默认为基础类型数据，不做进一步分析
param val: 值，判断值的数据类型，如果为复杂类型就做进一步分析
param con_s: 拼接符，当前级的键与父级键拼接的连接符，默认为_
@freemandealer
freemandealer / sfx_bench_io_generator.cpp
Created July 6, 2020 08:03
inline void create_bytes_random(char *data_buf, int size) {
int data = 0;
int i = 0;
char *data_ptr = data_buf;
char data_char;
for ( i = 0; i < size/2; i ++) {
data = rand();
data_char = (char)data;
if (data_char ==0)
@freemandealer
freemandealer / tasker_auto_vpn.pseudo
Last active June 28, 2020 03:59
Android tasker pseudo script to automate vpn on/off when using Chrome
# turn on chrome
if VPNISON ~ 0
connect vpn
set VPNISON 1
wait 30sec // wait 30 sec and set disconnectable to avoid frequent on/off
set VPNISON 2 // disconnectable
endif
# turn on others (turn off chrome)
if VPNISON ~ 2
@freemandealer
freemandealer / lz4_bench.c
Last active January 9, 2020 03:01
test lz4 compress/decompress performance to different buffer size
#include <stdio.h> // For printf()
#include <string.h> // For memcmp()
#include <stdlib.h> // For exit()
#include <assert.h>
#include <time.h> /* for clock_gettime() */
#include <string.h>
#include <linux/types.h>
#include <stdint.h>
#include "lz4.h"
@freemandealer
freemandealer / SimpleHTTPServerWithUpload.py
Created January 7, 2020 13:15 — forked from 591342534/SimpleHTTPServerWithUpload.py
Simple Python Http Server with Upload
#!/usr/bin/env python
"""Simple HTTP Server With Upload.
This module builds on BaseHTTPServer by implementing the standard GET
and HEAD requests in a fairly straightforward manner.
"""