Last active
October 23, 2019 20:36
-
-
Save JoaoCarabetta/1721743201957d4273d849b2bf5e57cc to your computer and use it in GitHub Desktop.
Splits stratified dataset into train and test using SQL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| /* Stratified splits dataset into training and test | |
| It guarantees that each group has the minimum size to be split. | |
| */ | |
| with ssize as ( | |
| select | |
| group | |
| from to_split_table | |
| group by group | |
| having count(*) >= {{ MINIMUM GROUP SIZE }}) -- {{ MINIMUM GROUP SIZE }} = 1 / {{ TEST_THRESHOLD }} | |
| select | |
| id, | |
| ts.group, | |
| case | |
| when | |
| cast(row_number() over (partition by ts.group order by rand()) as double) / cast(count() over (partition by ts.group) as double) | |
| < {{ TEST_THRESHOLD }} then 'test' | |
| else 'train' | |
| end as splitting | |
| from to_split_table ts | |
| join ssize | |
| on ts.group = ssize.group |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment