Skip to content

Instantly share code, notes, and snippets.

View mGalarnyk's full-sized avatar

Michael Galarnyk mGalarnyk

2k followers · 28 following
View GitHub Profile
@mGalarnyk
mGalarnyk / GetMore100TweetsTweepyOAuth2.py
Last active December 22, 2022 13:22
Get more than 100 Tweets at a time using paginator. If you need more than 100 Tweets, you will have to use paginator (https://docs.tweepy.org/en/latest/pagination.html). Replace the limit=1000 with the maximum number of tweets you want.
# OAuth2.0 Version
import tweepy
#Put your Bearer Token in the parenthesis below
client = tweepy.Client(bearer_token='Change this')
"""
If you don't understand search queries, there is an excellent introduction to it here:
https://github.com/twitterdev/getting-started-with-the-twitter-api-v2-for-academic-research/blob/main/modules/5-how-to-write-search-queries.md
"""
@mGalarnyk
mGalarnyk / SearchTweepyOAuth2.py
Last active October 18, 2022 21:54
Search for Tweets from the Last 7 Days using the tweepy API (OAuth2.0)
# OAuth2.0 Version
import tweepy
#Put your Bearer Token in the parenthesis below
client = tweepy.Client(bearer_token='Change this')
"""
If you don't understand search queries, there is an excellent introduction to it here:
https://github.com/twitterdev/getting-started-with-the-twitter-api-v2-for-academic-research/blob/main/modules/5-how-to-write-search-queries.md
"""
@mGalarnyk
mGalarnyk / changingTreeMethod.py
Last active June 3, 2024 19:45
Speeding up XGBoost model training by changing tree_method
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier
import time
# create synthetic dataset
X, y = make_classification(n_samples=100000, n_features=1000, n_informative=50, n_redundant=0, random_state=1)
# split data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.50, random_state=1)
@mGalarnyk
mGalarnyk / train.py
Last active February 4, 2022 19:18
Multi-node Training on a Ray Cluster. Gist modified from Introducing Ray Lightning: Multi-node PyTorch Lightning Training made easy: https://www.anyscale.com/blog/introducing-ray-lightning-multi-node-gpu-training-for-pytorch-lightning-made
from ray_lightning import RayPlugin
import pytorch_lightning as pl
import torch
import torch.nn.functional as F
from torch.utils.data import random_split, DataLoader
from torchvision.datasets import MNIST
from torchvision import transforms
class LightningMNISTClassifier(pl.LightningModule):
def __init__(self, config, data_dir=None):
@mGalarnyk
mGalarnyk / ParallelPyTorchLightningMNIST.py
Last active February 4, 2022 19:16
Gist originally from Introducing Ray Lightning: Multi-node PyTorch Lightning Training made easy: https://www.anyscale.com/blog/introducing-ray-lightning-multi-node-gpu-training-for-pytorch-lightning-made
from ray_lightning import RayPlugin
import pytorch_lightning as pl
import torch
import torch.nn.functional as F
from torch.utils.data import random_split, DataLoader
from torchvision.datasets import MNIST
from torchvision import transforms
class LightningMNISTClassifier(pl.LightningModule):
def __init__(self, config, data_dir=None):
@mGalarnyk
mGalarnyk / CreateTrainerStartTraining.py
Created September 8, 2021 19:04
Gist originally from Introducing Ray Lightning: Multi-node PyTorch Lightning Training made easy: https://www.anyscale.com/blog/introducing-ray-lightning-multi-node-gpu-training-for-pytorch-lightning-made
model = LightningMNISTClassifier(config, data_dir="./")
trainer = pl.Trainer( max_epochs=10)
trainer.fit(model)
@mGalarnyk
mGalarnyk / PyTorchLightningMNIST.py
Last active February 4, 2022 19:25
Gist originally from Introducing Ray Lightning: Multi-node PyTorch Lightning Training made easy: https://www.anyscale.com/blog/introducing-ray-lightning-multi-node-gpu-training-for-pytorch-lightning-made
import pytorch_lightning as pl
import torch
import torch.nn.functional as F
from torch.utils.data import random_split, DataLoader
from torchvision.datasets import MNIST
from torchvision import transforms
class LightningMNISTClassifier(pl.LightningModule):
def __init__(self, config, data_dir=None):
super(LightningMNISTClassifier, self).__init__()
@mGalarnyk
mGalarnyk / cluster.yaml
Created September 8, 2021 18:30
Gist originally from Introducing Ray Lightning: Multi-node PyTorch Lightning Training made easy: https://www.anyscale.com/blog/introducing-ray-lightning-multi-node-gpu-training-for-pytorch-lightning-made
cluster_name: ml
# Cloud-provider specific configuration.
provider:
type: aws
region: us-west-2
availability_zone: us-west-2a,us-west-2b
# How Ray will authenticate with newly launched nodes.
auth:
@mGalarnyk
mGalarnyk / ipythonParallel.py
Created September 1, 2021 20:32
import math
import numpy as np
from timebudget import timebudget
import ipyparallel as ipp
iterations_count = round(1e7)
def complex_operation(input_index):
print("Complex operation. Input index: {:2d}".format(input_index))
@mGalarnyk
mGalarnyk / specializedLibraries.py
Created September 1, 2021 20:25
import math
import numpy as np
from timebudget import timebudget
from multiprocessing import Pool
iterations_count = round(1e7)
def complex_operation(input_index):
print("Complex operation. Input index: {:2d}".format(input_index))