Guilherme Cardoso luminoso
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| vbell off | |
| term screen-256color | |
| hardstatus off | |
| hardstatus alwayslastline | |
| hardstatus alwayslastline '%{= G}[ %{G}%H %{g}][%= %{= w}%?%-Lw%?%{= R}%n*%f %t%?%{= R}(%u)%?%{= w}%+Lw%?%= %{= g}][ %{y}Load: %l %{g}][%{B}%Y-%m-%d %{W}%c:%s %{g}]' | |
| defnonblock on | |
| scrollback 15000 | |
| setenv LD_PRELOAD /usr/lib/libjemalloc.so |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import logging | |
| # level=logging._nameToLevel[verbosity_level] | |
| # level=logging.DEBUG | |
| # format="%(asctime)s [%(module)-12.12s] [%(levelname)-5.5s] %(message)s", | |
| logging.basicConfig( | |
| level=logging.DEBUG, | |
| format="%(asctime)s [%(levelname)-5.5s] %(message)s", |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # tests the avalability of cuda for pytorch | |
| import torch | |
| torch.cuda.current_device() | |
| torch.cuda.device(0) | |
| torch.cuda.device_count() | |
| torch.cuda.get_device_name(0) | |
| torch.cuda.is_available() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # trick to make iJulia work on CentOS 7 | |
| # CentOS looks like to have some kind of ABI incompatibility | |
| # ERROR: LoadError: InitError: could not load library "/home/centos/.julia/artifacts/8b67c92bc6eb60d96d3488bf4b48bd3a38f70c53/lib/libzmq.so" | |
| # /usr/bin/../lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /home/centos/.julia/artifacts/8b67c92bc6eb60d96d3488bf4b48bd3a38f70c53/lib/libzmq.so) | |
| ] | |
| add ZMQ@1.1.0 | |
| add iJulia | |
| using iJulia |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from pymongo import MongoClient | |
| # pprint library is used to make the output look more pretty | |
| from pprint import pprint | |
| import urllib.parse | |
| username = urllib.parse.quote_plus('xx') | |
| password = urllib.parse.quote_plus('yy') | |
| client = MongoClient(f'mongodb://{username}:{password}@172.31.8.226:27017/') |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # auto reload modules/files | |
| %load_ext autoreload | |
| %autoreload 2 | |
| # print execution times | |
| %load_ext autotime | |
| # interactive matplotlib outputs | |
| %matplotlib inline |
luminoso
/ serialize_jsolines_gz.py
Last active
March 13, 2023 09:47
jsonlines compressed serialization gz write file
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # reads and writes compressed jsonlines to a file | |
| # from bson import json_util | |
| # writeall_jsonl_gz('filename.jsonl.gz', objects, dumps=json_util.dumps) | |
| import gzip | |
| # !python -m pip install jsonlines | |
| from typing import List, Dict | |
| import jsonlines |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # list magics | |
| %lsmagic | |
| # | |
| # print all outputs of a cell | |
| # | |
| from IPython.core.interactiveshell import InteractiveShell | |
| InteractiveShell.ast_node_interactivity = "all" | |
| # |
luminoso
/ kafka_send_statistics.py
Last active
January 21, 2020 00:17
kafka producer send stats callback ensure message delivery
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| stats = { | |
| "errors": 0, | |
| "successes": 0, | |
| "min_offset": None, | |
| "max_offset": None, | |
| "errorTypes": {} | |
| } | |
| def errback_listener(error, **kargs): | |
| if type(error).__name__ not in kargs["stats"]["errorTypes"]: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ### Starting Zookeeper | |
| ./bin/zookeeper-server-start.sh config/zookeeper.properties | |
| ### Starting Kafka | |
| ./bin/kafka-server-start.sh config/server.properties | |
| ################################################# | |
| ### Topic creation | |
| ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic ipma.3days_day |