akki
/ mongo-hadoop-core-2.0.2.jar
Created
June 1, 2018 11:55
JAR files for MongoDB-Hadoop connector
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| " zip.vim version v27 | |
| " Browsing zipfile /home/hadoopuser/hadoop/lib/mongo-hadoop-core-2.0.2.jar | |
| " Select a file with cursor and press ENTER | |
| META-INF/ | |
| META-INF/MANIFEST.MF | |
| com/ | |
| com/mongodb/ | |
| com/mongodb/hadoop/ | |
| com/mongodb/hadoop/MongoOutputFormat.class |
akki
/ kafka.service
Last active
May 25, 2019 13:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [Unit] | |
| Requires=zookeeper.service | |
| After=zookeeper.service | |
| [Service] | |
| Type=simple | |
| User=kafka | |
| ExecStart=/bin/sh -c '/home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties > /home/kafka/kafka/kafka.log 2>&1' | |
| ExecStop=/home/kafka/kafka/bin/kafka-server-stop.sh | |
| Restart=on-abnormal |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| version: '3' | |
| services: | |
| postgres: | |
| image: postgres:9.6-alpine | |
| environment: | |
| - POSTGRES_USER=test | |
| - POSTGRES_PASSWORD=test | |
| - POSTGRES_DB=test | |
| pod: |
akki
/ benchmark-spark-magic-committer.py
Last active
March 28, 2024 14:22
Spark app with magic committer
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # These extra-configs enable the magic committer. | |
| # A _SUCCESS file with data is generated if these extra configs are passed. The file would otherwise be zero-byte in size, indicating that magic commiter did NOT get enabled. | |
| extra_configs = { | |
| "spark.sql.sources.commitProtocolClass": "org.apache.spark.internal.io.cloud.PathOutputCommitProtocol", | |
| "spark.sql.parquet.output.committer.class": "org.apache.hadoop.mapreduce.lib.output.BindingPathOutputCommitter", | |
| "spark.hadoop.mapreduce.outputcommitter.factory.scheme.s3a": "org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory", | |
| "spark.hadoop.fs.s3a.committer.name": "magic", | |
| "spark.hadoop.fs.s3a.committer.magic.enabled": "true", | |
| } |
OlderNewer