Migrate Archive Google Code SVN to Git

git-svn.md

Migrate Archive Google Code SVN to Git

Requirements

• git
• git-svn

Setup¹

$ sudo apt-get install git
$ sudo add-apt-repository ppa:git-core/ppa
$ sudo apt-get update
$ sudo apt-get install git-svn

¹ Ubuntu 14.04

How To

1. Download svn dump from Google archive.
2. Create a local svn repo by load the svn dump.
3. Start svn daemon.
4. Create authors.txt to map all svn users to git users.
5. Create a bare git repo.
6. git svn clone.
7. Add git remote then push.
8. Done.

$ wget https://storage.googleapis.com/google-code-archive-source/v2/code.google.com/your-google-code-project/repo.svndump.gz
$ gunzip repo.svndump.gz
$ svnadmin create /tmp/repo1
$ svnadmin load /tmp/repo1/ < repo.svndump
$ svnadmin --foreground -d

Start a new terminal Window

$ git svn --stdlayout -A authors.txt clone svn://localhost/tmp/repo1/
$ cd repo1
$ git remote add origin https://repo.git
$ git push --set-upstream origin master

authors.txt

Must map all svn users to git users.

user1 = user1 <[email protected]>
user2 = user2 <[email protected]>
(no author) = user3 <[email protected]>

Credits

• https://dominikdorn.com/2016/05/how-to-recover-a-google-code-svn-project-and-migrate-to-github/
• https://docs.gitlab.com/ee/workflow/importing/migrating_from_svn.html
• https://www.guyrutenberg.com/2011/11/09/author-no-author-not-defined-in-authors-file/

svnadmin --foreground -d

should be

svnserve --foreground -d

You can easily generate proper authors.txt file using this command:

svn log -q file:///tmp/repo1 | awk -F '|' '/^r/ {sub("^ ", "", $2); sub(" $", "", $2); print $2" = "$2" <"$2">"}' | sort -u > authors.txt

From Docker container in any host

• Added at https://gist.github.com/marcellodesales/2e87387cbd5866098766074895e7f3e9
• Run a container with a subversion server https://hub.docker.com/r/garethflowers/svn-server
• Can be executed in any host with Docker
• Note this approach uses a volume -v $(pwd):/dump \ in your host to prepare everything to the point you can add the github repo and push outside the container
  • You can add a -v to ~/.ssh:/root/.ssh if you want to push from inside the container and use your ssh keys

$ docker run \
    --name svn-server \
    --detach \
    -v $(pwd):/dump \
    garethflowers/svn-server
1f5ae9c2233a1fbf4d0b56dfe821ee8bb93c548d2a5dcc1e0eef465ce6810a68

$ docker ps
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS                            PORTS               NAMES
1f5ae9c2233a        garethflowers/svn-server   "/usr/bin/svnserve -…"   3 seconds ago       Up 3 seconds (health: starting)   3690/tcp            svn-server

• Go inside the container using sh

$ docker exec -ti svn-server sh
/var/opt/svn #

• Getting the dump from the project netbeams

/var/opt/svn # wget https://storage.googleapis.com/google-code-archive-source/v2/code.google.com/netbeams/repo.svndump.gz
--2020-09-06 05:56:49--  https://storage.googleapis.com/google-code-archive-source/v2/code.google.com/netbeams/repo.svndump.gz
Resolving storage.googleapis.com (storage.googleapis.com)... 172.217.172.208, 172.217.162.144, 172.217.28.16, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|172.217.172.208|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 235011376 (224M) [application/octet-stream]
Saving to: 'repo.svndump.gz'

repo.svndump.gz                        100%[=========================================================================>] 224.12M  13.7MB/s    in 17s

2020-09-06 05:57:08 (13.5 MB/s) - 'repo.svndump.gz' saved [235011376/235011376]

• Create and verify svn repository

/var/opt/svn # svnadmin create netbeams
/var/opt/svn # svn info svn://localhost:3690/netbeams
Path: netbeams
URL: svn://localhost/netbeams
Relative URL: ^/
Repository Root: svn://localhost/netbeams
Repository UUID: e474ffc5-7d29-46d1-bd38-add6906cf2a8
Revision: 0
Node Kind: directory
Last Changed Rev: 0
Last Changed Date: 2020-09-06 06:00:56 +0000 (Sun, 06 Sep 2020)

• Load the dump

/var/opt/svn # gunzip -c repo.svndump.gz  | svnadmin load netbeams
<<< Started new transaction, based on original revision 1
     * editing path : branches ... done.
     * editing path : tags ... done.
     * editing path : trunk ... done.

------- Committed revision 1 >>>

<<< Started new transaction, based on original revision 2
     * editing path : trunk/.classpath ... done.
     * editing path : trunk/.project ... done.
....
....

• Collect the authors where the name netbeams is the project

$ docker exec -ti my-svn-server sh
/var/opt/svn # svn log -q file:////var/opt/svn/netbeams | awk -F '|' '/^r/ {sub("^ ", "", $2); sub(" $", "", $2); print $2" = "$2" <"$2">"}' | sort -u > authors.txt
/var/opt/svn # ls -la authors.txt
total 12
-rw-r--r--    1 root     root           511 Sep  6 03:15 authors.txt
/var/opt/svn # cat authors.txt
(no author) = (no author) <(no author)>
[email protected] = [email protected] <[email protected]>
bhuynh78 = bhuynh78 <bhuynh78>
gamma.particle = gamma.particle <gamma.particle>
[email protected] = [email protected] <[email protected]>
kleberkls = kleberkls <kleberkls>
[email protected] = [email protected] <[email protected]>
marcello.sales = marcello.sales <marcello.sales>
[email protected] = [email protected] <[email protected]>
xiaoyudi84 = xiaoyudi84 <xiaoyudi84>

• Edit authors with vim, Still inside the docker container, install git and git-svn

/var/opt/svn # apk add vim git git-svn
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
(1/19) Installing ca-certificates (20191127-r4)
(2/19) Installing nghttp2-libs (1.41.0-r0)
(3/19) Installing libcurl (7.69.1-r0)
(4/19) Installing pcre2 (10.35-r0)
(5/19) Installing git (2.26.2-r0)
(6/19) Installing libbz2 (1.0.8-r1)
(7/19) Installing perl (5.30.3-r0)

• Edit vim as you wish and save it

/var/opt/svn # vim authors.txt

var/opt/svn # cat authors.txt
(no author) = Marcello de Sales <[email protected]>
[email protected] = [email protected] <[email protected]>
bhuynh78 = Bill Huynh <[email protected]>
gamma.particle = Teresa Johnson <[email protected]>
[email protected] = Teresa Johnson <[email protected]>
kleberkls = Kleber Sales <[email protected]>
[email protected] = Kleber Sales <[email protected]>
marcello.sales = Marcello de Sales <[email protected]>
[email protected] = Marcello de Sales <[email protected]>
xiaoyudi84 = xiaoyudi84 <xiaoyudi84>

• At this point you are cloning the master trunk of svn to git...
• If you have any branch specifically you want to clone, then specify in the command like explained in https://gist.github.com/trodrigues/1023167
• List everything you have from the svn server before you migrate

/var/opt/svn # svn ls svn://localhost:3690/netbeams
branches/
tags/
trunk/
wiki/

• This example uses the branch... The -T value must be the one that lists the contents of the repo to a directory called netbeams-marcello-git

/var/opt/svn # git svn --stdlayout -A authors.txt clone -T branches/marcello/persistence svn://localhost:3690/netbeams netbeams-marcello-git
	M	persistence/docs/marcello-dissertation/thesis/chapter8-experimental-results-correct-behavior-and-performance.tex
r718 = bdf4d4b181898643405706f5ea760a58c2d65a54 (refs/remotes/origin/marcello)
Checked out HEAD:
  svn://localhost/netbeams/branches/marcello/persistence r718
creating empty directory: versions/v2/persistence/data
creating empty directory: versions/v2/persistence/mongodb

• Move the directory netbeams-marcello-git to the dump directory where it is mapped to the host

/var/opt/svn # mv netbeams-marcello-git /dump

• Exit the container with exit and execute the steps in

$ cd netbeams-marcello-git
$ git remote add origin git:@github.com:marcellodesales/sfsu-cs-masters-thesis-netbeams.git
$ git push --set-upstream origin master

Wiki pages preserving history

• Verify if you have wiki pages

/var/opt/svn # svn ls svn://localhost:3690/netbeams/wiki
CreatingDSPBundles.wiki
DSPDataPersistence.wiki
DSPDataPersistenceOnMongoDB.wiki
DataSensorPlatform.wiki
ExternalResources.wiki
GumstixOESystem.wiki
Gumstix_3boards.jpg
Gumstix_components.jpg
Gumstix_fullconnect.jpg
InstallingDsp.wiki
MongoDBShardsClusterAndMapReduce.wiki
NetbeamsResearch.wiki
RunningDSPOnKnopflerfish_TextConsole.wiki
RunningDspOnKnopflerfish.wiki
RxtxSharedLibrary.wiki
Seabird_DataFlow.pdf
SystemRequirements.wiki

• Clone the wiki pages using the -T wiki to the dir netbeams-wiki

$ git svn --stdlayout -A authors.txt clone -T wiki svn://localhost:3690/netbeams netbeams.wiki
$ cd netbeams.wiki

• Convert all the old-style wiki pages to new ones (COPY ALL THE LINES ON THIS BLOCK AND PASTE IN THE TERMINAL TO EXECUTE THEM ALL)

find . -type f -iname '*.wiki' -exec sed -i.bak "s/^= /# /g" {} \;
find . -type f -iname '*.wiki' -exec sed -i.bak "s/^== /## /g" {} \;
find . -type f -iname '*.wiki' -exec sed -i.bak "s/^=== /### /g" {} \;
find . -type f -iname '*.wiki' -exec sed -i.bak "s/^==== /#### /g" {} \;
find . -type f -iname '*.wiki' -exec sed -i.bak "s/====$//g" {} \;
find . -type f -iname '*.wiki' -exec sed -i.bak "s/===$//g" {} \;
find . -type f -iname '*.wiki' -exec sed -i.bak "s/==$//g" {} \;
find . -type f -iname '*.wiki' -exec sed -i.bak "s/=$//g" {} \;
find . -type f -iname '*.wiki' -exec sed -i.bak "s/^{{{/\`\`\`/g" {} \;
find . -type f -iname '*.wiki' -exec sed -i.bak "s/^}}}/\`\`\`/g" {} \;
for e in *.wiki ; do sed -i '/^#summary/d' $e ; done
for e in *.wiki ; do sed -i '/^#labels/d' $e ; done
rm -f *.bak

• Make sure to examine the files

diff --git a/CreatingDSPBundles.wiki b/CreatingDSPBundles.wiki
index f760423..7d74edc 100644
--- a/CreatingDSPBundles.wiki
+++ b/CreatingDSPBundles.wiki
@@ -1,7 +1,7 @@
 #summary This page describes how to create a DSP component
 #labels osgi,Phase-QA,Phase-Support

-= DSP OSGi Component =
+# DSP OSGi Component

 A DSP component must have the following components:

@@ -13,7 +13,7 @@ A DSP component must have the following components:
  * A MANIFEST.MF file that describes the OSGi application;
  * A build.xml file that bundles all the previous artifacts into a Jar file.

-{{{
+```
 <?xml version="1.0" encoding="UTF-8"?>
 <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:mouseaction="http://dsp.netbeams.org/demo/mouseaction.xsd"
     xmlns:dspmessage="http://www.netbeams.dsp.org/message.xsd" xmlns:jaxb="http://java.sun.com/xml/ns/jaxb"
@@ -80,7 +80,7 @@ The only basic important step in defining the data schema is the root element mu
     </xs:element>

 </xs:schema>
-}}}
+```

 Please, refer to the XML SChema tutorials online in order to define XML Schemas, such as http://www.w3schools.com/Schema/default.asp.

@@ -90,11 +90,11 @@ http://netbeams.googlecode.com/svn-history/r190/trunk/versions/v2/apps/xml/jaxb/

 Since our DSP component data is defined to be transferred over the Internet, we suggest the usage of attributes for Metadata as much as possible, and us
e elements in other cases, such as collection holders, etc.

-== Generating the Java POJO classes from the XML Schema ==
+## Generating the Java POJO classes from the XML Schema

• `cp -R files-wiki /dump
• Exit the container
• Copy all the md files to the cloned wiki repo

$ for f in `find . -iname '*.wiki' -type f -print`;do  git mv "$f" ${f%.wiki}.md; done
$ git mv DataSensorPlatform.md Home.md
$ git status
On branch master
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
	renamed:    CreatingDSPBundles.wiki -> CreatingDSPBundles.md
	renamed:    DSPDataPersistence.wiki -> DSPDataPersistence.md
	renamed:    DSPDataPersistenceOnMongoDB.wiki -> DSPDataPersistenceOnMongoDB.md
	renamed:    ExternalResources.wiki -> ExternalResources.md
	renamed:    GumstixOESystem.wiki -> GumstixOESystem.md
	renamed:    DataSensorPlatform.wiki -> Home.md
	renamed:    InstallingDsp.wiki -> InstallingDsp.md
	renamed:    MongoDBShardsClusterAndMapReduce.wiki -> MongoDBShardsClusterAndMapReduce.md
	renamed:    NetbeamsResearch.wiki -> NetbeamsResearch.md
	renamed:    RunningDSPOnKnopflerfish_TextConsole.wiki -> RunningDSPOnKnopflerfish_TextConsole.md
	renamed:    RunningDspOnKnopflerfish.wiki -> RunningDspOnKnopflerfish.md
	renamed:    RxtxSharedLibrary.wiki -> RxtxSharedLibrary.md
	renamed:    SystemRequirements.wiki -> SystemRequirements.md

• Move to the dump

/var/opt/svn # cp -R netbeams-marcello-git.wiki /dump/

• On the host, make any other changes, git add and commit

$ git commit -a -m "Rename all wiki pages to md"

If you want, change the date according to the history

$ git commit --amend --no-edit --date="Thu Dec 17 21:22:53 2009 -0800"

• Go to the Github Repo page, Go to the Wiki section and click in Create Wiki
  • The repo is in the format [email protected]:ORG/REPO.wiki.git
  • But there's no need to clone the repo locally.
• Push with a forced update

$ git remote add origin [email protected]:marcellodesales/sfsu-cs-masters-thesis-netbeams.wiki.git
$ git push origin -f master
Counting objects: 100% (233/233), done.
Delta compression using up to 16 threads
Compressing objects: 100% (149/149), done.
Writing objects: 100% (233/233), 1.39 MiB | 856.00 KiB/s, done.
Total 233 (delta 164), reused 120 (delta 82)
remote: Resolving deltas: 100% (164/164), done.
To github.com:marcellodesales/sfsu-cs-masters-thesis-netbeams.wiki.git
 + acd4f74...aeceb1d master -> master (forced update)

No need to serve the svn repo, as git-svn clone supports file protocol.

So instead of

svnserve --foreground -d
git svn --stdlayout -A authors.txt clone svn://localhost/tmp/repo1/

just:

git svn --stdlayout -A authors.txt clone file:///tmp/repo1/

Also, could include the --no-metadata option as this is a one-shot import.

Reference:https://www.gitkraken.com/blog/migrating-git-svn

Update for copy/paste.

#!/bin/bash

set +x    # trace instructions

export GCPROJECT=fido2

cd /tmp
wget https://storage.googleapis.com/google-code-archive-source/v2/code.google.com/$GCPROJECT/repo.svndump.gz -O $GCPROJECT.svndump.gz
gunzip -v $GCPROJECT.svndump.gz

svnadmin create $GCPROJECT-svn
svnadmin load $GCPROJECT-svn --file=$GCPROJECT.svndump

svn log -q file:///tmp/$GCPROJECT-svn | awk -F '|' '/^r/ {sub("^ ", "", $2); sub(" $", "", $2); print $2" = "$2" <"$2">"}' | sort -u > $GCPROJECT-authors.txt

git svn clone --stdlayout -A $GCPROJECT-authors.txt file:///tmp/$GCPROJECT-svn $GCPROJECT
cd $GCPROJECT
echo $PWD