Skip to content

Instantly share code, notes, and snippets.

View AdityaSoni19031997's full-sized avatar
🎯
Focusing

Aditya Soni AdityaSoni19031997

🎯
Focusing
View GitHub Profile
@pvsune
pvsune / concurrent.kafka.consumer.py
Last active April 22, 2025 14:17
A multiprocess multithreaded Kafka consumer
#!/usr/bin/env python
import logging
import os
import threading
import time
from multiprocessing import Process
from queue import Queue
from confluent_kafka import Consumer
@AdityaSoni19031997
AdityaSoni19031997 / file_streaming_pytorch.py
Created April 21, 2020 02:19
In this gist i have tried to explain a very smart way of loading datasets by streaming them from bytes into PyTorch; It can be achieved in multiple ways, but here my focus was confined to David's idea of streaming records from a bytes file;
import torch
import io
import pandas as pd
import gc
import numpy as np
import transformers
'''
Original Code Author [@dlibenzi](https://github.com/dlibenzi)
from PIL import Image
import numpy as np
import hashlib
import os
import sys
import torch
import torch_xla.utils.tf_record_reader as tfrr
a = """
image/class/label tensor([82])
@ccj5351
ccj5351 / itera_dataloader_example.py
Created February 19, 2020 03:06
Modified code with the article "How to Build a Streaming DataLoader with PyTorch" at https://medium.com/speechmatics/how-to-build-a-streaming-dataloader-with-pytorch-a66dd891d9dd.
import random
from itertools import chain, cycle, islice
import torch.utils.data as data
import matplotlib.pyplot as plt
from matplotlib.patches import Rectangle
import time
import torch
import numpy as np
@ksopyla
ksopyla / polish_sentence_nltk_tokenizer.py
Last active September 28, 2024 13:09
A curated list of Polish abbreviations for NLTK sentence tokenizer based on Wikipedia text
import nltk
# interactive download
# nltk.download()
nltk.download('punkt')
extra_abbreviations = [
"ps",
"inc",
"corp",
from PIL import Image
import numpy as np
import sys
import torch
import torch_xla.utils.tf_record_reader as tfrr
a = """
image/class/label tensor([82])
image/class/synset n01796340
image/channels tensor([3])
@janpipek
janpipek / which.py
Last active January 26, 2020 02:59
How to implement `which` in pandas
import pandas as pd
def which(series):
# Error handling omitted
if not isinstance(series, pd.Series):
series = pd.Series(series)
return series[series.astype(bool) == True].index.tolist()
# Way to extend a Series
# See: https://pandas.pydata.org/pandas-docs/stable/development/extending.html#registering-custom-accessors
@abhishekkrthakur
abhishekkrthakur / slack_notifier.py
Created December 6, 2019 07:53
Slack notification from python
import os
import requests
import json
SLACK_WEBHOOK= os.environ.get("SLACK_WEBHOOK")
def send_message(messages, channel="abhishek", username="beast"):
"""
:param messages: list of texts
# coding=utf-8
# Copyright 2018 The HuggingFace Inc. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
@wvengen
wvengen / NOTES.md
Last active July 14, 2021 07:03
Semantic web of food notes