Skip to content

Instantly share code, notes, and snippets.

View Fokko's full-sized avatar
🚀

Fokko Driesprong Fokko

🚀
335 followers · 4 following
View GitHub Profile
@Fokko
Fokko / Avro 1.11.4 test
Created August 22, 2024 09:10
This file has been truncated, but you can view the full file.
➜ avro-src-1.11.4 ./build.sh docker-test
+ cd .
+ VERSION=1.11.4
+ DOCKER_RUN_XTRA_ARGS=
+ DOCKER_RUN_ENTRYPOINT=bash
+ DOCKER_BUILD_XTRA_ARGS=
+ DOCKER_IMAGE_NAME=
+ DOCKER_EXTRA_CONTEXT='lang/ruby/Gemfile lang/ruby/avro.gemspec lang/ruby/Manifest share/VERSION.txt'
+ (( 1 == 0 ))
+ (( 1 ))
@Fokko
Fokko / 1.5.1-verify.txt
Created April 23, 2024 21:29
➜ Desktop gpg --verify apache-iceberg-1.5.1.tar.gz.asc
gpg: assuming signed data in 'apache-iceberg-1.5.1.tar.gz'
gpg: Signature made do 18 apr 21:03:16 2024 CEST
gpg: using RSA key 18E542BBB3CF0A8E0A3197ADEE9B65F08022E944
gpg: Good signature from "Amogh Jahagirdar <[email protected]>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 18E5 42BB B3CF 0A8E 0A31 97AD EE9B 65F0 8022 E944
➜ Desktop shasum -a 512 --check apache-iceberg-1.5.1.tar.gz.sha512
@Fokko
Fokko / pyiceberg-0.6.1-docutils
Created April 18, 2024 11:13
➜ pyiceberg-0.6.1 docker run -t -i -v `pwd`:/vo python:3.10 bash
root@7984a20cf808:/# cd /vo/
root@7984a20cf808:/vo# make install
pip install poetry==1.7.1
Collecting poetry==1.7.1
Downloading poetry-1.7.1-py3-none-any.whl (235 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 236.0/236.0 kB 10.0 MB/s eta 0:00:00
Collecting keyring<25.0.0,>=24.0.0
Downloading keyring-24.3.1-py3-none-any.whl (38 kB)
Collecting installer<0.8.0,>=0.7.0
@Fokko
Fokko / pyiceberg-0.6.0rc3.txt
Created April 18, 2024 10:56
~ svn checkout https://dist.apache.org/repos/dist/dev/iceberg/pyiceberg-0.6.1rc3/ /tmp/pyiceberg/
for name in $(ls /tmp/pyiceberg/pyiceberg-*.whl /tmp/pyiceberg/pyiceberg-*.tar.gz)
do
gpg --verify ${name}.asc ${name}
done
A /tmp/pyiceberg/pyiceberg-0.6.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.asc
A /tmp/pyiceberg/pyiceberg-0.6.1-cp311-cp311-musllinux_1_1_x86_64.whl.asc
A /tmp/pyiceberg/pyiceberg-0.6.1-cp311-cp311-win_amd64.whl.asc
A /tmp/pyiceberg/pyiceberg-0.6.1-cp38-cp38-macosx_11_0_x86_64.whl.asc
@Fokko
Fokko / validate-release.txt
Created April 17, 2024 16:20
➜ ~ svn checkout https://dist.apache.org/repos/dist/dev/iceberg/pyiceberg-0.6.1rc2/ /tmp/pyiceberg/
➜ ~ for name in $(ls /tmp/pyiceberg/pyiceberg-*.whl /tmp/pyiceberg/pyiceberg-*.tar.gz)
do
gpg --verify ${name}.asc ${name}
done
gpg: Signature made di 16 apr 05:24:22 2024 CEST
gpg: using RSA key 18F404FF049EA261A89AF6550461C12D82881E8B
gpg: issuer "[email protected]"
gpg: Good signature from "Honah J. (CODE SIGNING KEY) <[email protected]>" [unknown]
gpg: WARNING: The key's User ID is not certified with a trusted signature!
@Fokko
Fokko / after_tuning.txt
Created January 26, 2023 16:24
2023-01-26T13:53:29.161 [206 Partial Content] s3.GetObject minio:9000/warehouse/wh/nyc/taxis/metadata/00006-84798267-d930-4ade-bd98-69e9a56d80fa.metadata.json 172.18.0.3 847µs ↑ 169 B ↓ 11 KiB
2023-01-26T13:53:29.203 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/snap-4964512940758329203-1-0c16c49c-2497-4706-95b3-da2a0f056e83.avro 172.18.0.1 505µs ↑ 153 B ↓ 0 B
2023-01-26T13:53:29.206 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/snap-4964512940758329203-1-0c16c49c-2497-4706-95b3-da2a0f056e83.avro 172.18.0.1 1.023ms ↑ 159 B ↓ 3.9 KiB
2023-01-26T13:53:29.225 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/89e24c2b-36ef-4fb1-bbf7-0ff7056b77f6-m0.avro 172.18.0.1 332µs ↑ 153 B ↓ 0 B
2023-01-26T13:53:29.225 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/973a3c07-6f09-41ff-b0dc-5d52747f42d4-m0.avro 172.18.0.1 279µs ↑ 153 B ↓ 0 B
2023-01-26T13:53:29.2
@Fokko
Fokko / before_tuning.txt
Created January 26, 2023 16:24
2023-01-26T13:54:37.006 [206 Partial Content] s3.GetObject minio:9000/warehouse/wh/nyc/taxis/metadata/00006-84798267-d930-4ade-bd98-69e9a56d80fa.metadata.json 172.18.0.3 1.752ms ↑ 169 B ↓ 11 KiB
2023-01-26T13:54:37.017 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/snap-4964512940758329203-1-0c16c49c-2497-4706-95b3-da2a0f056e83.avro 172.18.0.1 578µs ↑ 153 B ↓ 0 B
2023-01-26T13:54:37.020 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/snap-4964512940758329203-1-0c16c49c-2497-4706-95b3-da2a0f056e83.avro 172.18.0.1 1.176ms ↑ 159 B ↓ 3.9 KiB
2023-01-26T13:54:37.038 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/dedc80dd-0878-4cfb-869d-7f5c445b03af-m0.avro 172.18.0.1 376µs ↑ 153 B ↓ 0 B
2023-01-26T13:54:37.039 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/973a3c07-6f09-41ff-b0dc-5d52747f42d4-m0.avro 172.18.0.1 240µs ↑ 153 B ↓ 0 B
2023-01-26T13:54:37.
@Fokko
Fokko / after_tuning.txt
Created January 26, 2023 08:32
2023-01-26T08:26:42.769 [206 Partial Content] s3.GetObject minio:9000/warehouse/wh/nyc/taxis/metadata/00006-84798267-d930-4ade-bd98-69e9a56d80fa.metadata.json 172.18.0.3 2.845ms ↑ 169 B ↓ 11 KiB
2023-01-26T08:26:42.784 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/snap-4964512940758329203-1-0c16c49c-2497-4706-95b3-da2a0f056e83.avro 172.18.0.1 519µs ↑ 153 B ↓ 0 B
2023-01-26T08:26:42.788 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/snap-4964512940758329203-1-0c16c49c-2497-4706-95b3-da2a0f056e83.avro 172.18.0.1 1.968ms ↑ 159 B ↓ 3.9 KiB
2023-01-26T08:26:42.808 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/89e24c2b-36ef-4fb1-bbf7-0ff7056b77f6-m0.avro 172.18.0.1 648µs ↑ 153 B ↓ 0 B
2023-01-26T08:26:42.808 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/0c16c49c-2497-4706-95b3-da2a0f056e83-m0.avro 172.18.0.1 335µs ↑ 153 B ↓ 0 B
2023-01-26T08:26:42.
@Fokko
Fokko / before_tuning.txt
Created January 26, 2023 08:31
2023-01-26T08:28:57.406 [206 Partial Content] s3.GetObject minio:9000/warehouse/wh/nyc/taxis/metadata/00006-84798267-d930-4ade-bd98-69e9a56d80fa.metadata.json 172.18.0.3 1.018ms ↑ 169 B ↓ 11 KiB
2023-01-26T08:28:57.416 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/snap-4964512940758329203-1-0c16c49c-2497-4706-95b3-da2a0f056e83.avro 172.18.0.1 440µs ↑ 153 B ↓ 0 B
2023-01-26T08:28:57.419 [206 Partial Content] s3.GetObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/snap-4964512940758329203-1-0c16c49c-2497-4706-95b3-da2a0f056e83.avro 172.18.0.1 868µs ↑ 159 B ↓ 3.9 KiB
2023-01-26T08:28:57.437 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/89e24c2b-36ef-4fb1-bbf7-0ff7056b77f6-m0.avro 172.18.0.1 433µs ↑ 153 B ↓ 0 B
2023-01-26T08:28:57.437 [200 OK] s3.HeadObject 127.0.0.1:9000/warehouse/wh/nyc/taxis/metadata/f371bff9-d197-4314-ae33-75afd24e8781-m0.avro 172.18.0.1 793µs ↑ 153 B ↓ 0 B
2023-01-26T08:28:57.4
@Fokko
Fokko / Dockerfile
Created September 2, 2022 09:44
Hive Metastore Dockerfile
FROM openjdk:8-jre
RUN apt-get update && apt-get install -y libpostgresql-jdbc-java procps libsasl2-modules libsasl2-dev && rm -rf /var/lib/apt/lists/*
# Install Apache Hadoop
ENV HADOOP_VERSION=3.1.0
ENV HADOOP_HOME /opt/hadoop-$HADOOP_VERSION
ENV HADOOP_CONF_DIR=$HADOOP_HOME/conf
ENV PATH $PATH:$HADOOP_HOME/bin
RUN curl -L \