robandpdx

PyTorch workflow testing

In order to iterate faster on workflows it is necessary to setup an environment to run the test.

1. Launch a g5.4xlarge instance in aws with Deep Learning AMI GPU PyTorch 1.10.0 (Ubuntu 20.04) 20221003 ami-09b3fda578ad20138
2. Generate an ssh key pair and add it to your github account.
3. Clone pytorch/rl repo to /home/ec2-user/github/rl
4. Use the following script to start the container and get a shell...

#!/bin/bash

robandpdx

Splitting large commits that prevent push to GitHub

You have discovered some very large commits in a repo history that prevent pushing the repo to GitHub, even when attempting to do a chunked push. These commits are larger that the push limit of 2G.

There are a few options for dealing with these large commits to unblock pushing to GitHub.

1. delete the commit
2. split the commit

Both options will rewrite the repo history. Deleting the commit can be problematic if there are later commits that depend on the commit being deleted. Spliting the commit into smaller chunks is a better option. We have created the chunk commit script to help commit the files from the large c

robandpdx

Moving an SVN monorepo to git

git svn expects your repo to be in the following structure...

./bigsvn
├── branches
│   ├── project1
│   └── project3
├── tags
│   └── project2
└── trunk

robandpdx

Making a monorepo from multiple git repos using git subtree

You may have several git repos that you want to combine into a single git repo. Here is how you can accomplish this using git subtree...

1. Create a new empty git repo.

mkdir monorepo
cd monorepo
git init

robandpdx

Migrating git repos without metadata

Metadata, like pull requests and issues, are not part of a git repo. These features are provided by the platform that hosts your shared git repo. There are tools to migrate metadata from BitBucket, GitLab, Azure DevOps, GitHub Enterprise Server, and github.com. If tools don't exist to migrate the metadata from the platform you are using, your only option is to migrate only the git repo. To migrate a git repo without medatadata, follow the instructions below.

Pre-migration

Familiarize yourself with the pre-migration documentation here.
If you are migrating a repo that uses LFS, make sure you have git-lfs installed.

Migration

1. Clone the repo

robandpdx

LFS Migration

GitHub blocks files larger than 100 MiB. Therefore, if you are migrating a repository with objects larger than 100 Mib to Github, you'll need to migrate those large objects to LFS.

A note about OS

By default, windows and mac do not have a case sensitive filesystem. For this reason, I recommend using linux for lfs migration. Also, if the lfs migration seems to take a long time, this is often due to lots of disk I/O. To speed things up, use a cloud linux instance with max disk I/O.

Before migrating to LFS

Make sure you have git-lfs installed.

robandpdx

Documentation

GitHub CLI
GitHub Enterprise Importer CLI
GitHub Enterprise Importer (GEI)
GEI limitations

The following is a check-list for steps to take, before and after migrations.
Before Migration
Organization / Users:

• EMU organizations created (named differently than source organizations)