Configuring SSH to work with VSCode dev containers was not straight forward. Through much trial and error I was finally able to establish SSH connections within the container without having to copy the SSH keys manually. I first started unraveling everything after finding a link to Advanced Container Configuration.
The error I was seeing had the following in it after starting a dev container:
[error] connect ENOENT \.\pipe\openssh-ssh-agent: Error: connect ENOENT \.\pipe\openssh-ssh-agent
I could still navigate the environment and use the terminal, but I did not have access via SSH without manually copying my private and public SSH keys into the containerized environment. The following steps give your dev container access to your ssh keys without any extra work or modifications to your container build.