🤖
- people don't think it be like it is but it do
- in/mclark52
- https://confluence.woodside.com.au/display/~W43945
- http://github.com/wassname
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import pandas as pd | |
import os | |
from anycache import anycache | |
from loguru import logger | |
import time | |
def chunking_by_time(startTime: pd.Timestamp, endTime: pd.Timestamp, freqs=['YE', 'ME', 'D']): | |
""" | |
How do we cache timespans? We want to take year chunks, then for the remainder months, and so on. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from pathlib import Path | |
import pandas as pd | |
from tqdm.auto import tqdm | |
experiments = { | |
'exp1': dict(lr=1e-4), | |
] | |
cache_dir = Path('../data/30_processed/results_cache1') | |
cache_dir.mkdir(exists_ok=True) | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import pandas as pd | |
import numpy as np | |
def pandas_groupby_agg_mixed_dtype_df(df, groupby: list, agg='max', str_agg='first'): | |
""" | |
resample a df with numeric and strings | |
""" | |
numerics_cols = list(df.select_dtypes(include='number').columns)+groupby | |
other_cols = list(df.select_dtypes(exclude='number').columns)+groupby |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import pandas as pd | |
def chunking_by_time(startTime: pd.Timestamp, endTime: pd.Timestamp, freqs=['Y', 'M', 'D']): | |
""" | |
How do we cache timespans? We want to take year chunks, then for the remainder months, and so on. | |
That way old data is cached in big chunks, and new data is rechunked as needed | |
```py | |
startTime = pd.to_datetime('2023-01-02 01:01') |