Migrate Archive Google Code SVN to Git using Docker

The source-code and wiki pages migrated using Docker, no need to install other software

Requirements

git
docker

Result of the manual steps

From Docker container in any host

Run a container with a subversion server https://hub.docker.com/r/garethflowers/svn-server
Can be executed in any host with Docker
Note this approach uses a volume -v $(pwd):/dump \ in your host to prepare everything to the point you can add the github repo and push outside the container
- You can add a -v to ~/.ssh:/root/.ssh if you want to push from inside the container and use your ssh keys

$ docker run \
    --name svn-server \
    --detach \
    -v $(pwd):/dump \
    garethflowers/svn-server
1f5ae9c2233a1fbf4d0b56dfe821ee8bb93c548d2a5dcc1e0eef465ce6810a68

$ docker ps
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS                            PORTS               NAMES
1f5ae9c2233a        garethflowers/svn-server   "/usr/bin/svnserve -…"   3 seconds ago       Up 3 seconds (health: starting)   3690/tcp            svn-server

Go inside the container using sh

$ docker exec -ti svn-server sh
/var/opt/svn #

Getting the dump from the project netbeams

/var/opt/svn # wget https://storage.googleapis.com/google-code-archive-source/v2/code.google.com/netbeams/repo.svndump.gz
--2020-09-06 05:56:49--  https://storage.googleapis.com/google-code-archive-source/v2/code.google.com/netbeams/repo.svndump.gz
Resolving storage.googleapis.com (storage.googleapis.com)... 172.217.172.208, 172.217.162.144, 172.217.28.16, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|172.217.172.208|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 235011376 (224M) [application/octet-stream]
Saving to: 'repo.svndump.gz'

repo.svndump.gz                        100%[=========================================================================>] 224.12M  13.7MB/s    in 17s

2020-09-06 05:57:08 (13.5 MB/s) - 'repo.svndump.gz' saved [235011376/235011376]

Create and verify svn repository

/var/opt/svn # svnadmin create netbeams
/var/opt/svn # svn info svn://localhost:3690/netbeams
Path: netbeams
URL: svn://localhost/netbeams
Relative URL: ^/
Repository Root: svn://localhost/netbeams
Repository UUID: e474ffc5-7d29-46d1-bd38-add6906cf2a8
Revision: 0
Node Kind: directory
Last Changed Rev: 0
Last Changed Date: 2020-09-06 06:00:56 +0000 (Sun, 06 Sep 2020)

Load the dump

/var/opt/svn # gunzip -c repo.svndump.gz  | svnadmin load netbeams
<<< Started new transaction, based on original revision 1
     * editing path : branches ... done.
     * editing path : tags ... done.
     * editing path : trunk ... done.

------- Committed revision 1 >>>

<<< Started new transaction, based on original revision 2
     * editing path : trunk/.classpath ... done.
     * editing path : trunk/.project ... done.
....
....

Collect the authors where the name netbeams is the project

$ docker exec -ti my-svn-server sh
/var/opt/svn # svn log -q file:////var/opt/svn/netbeams | awk -F '|' '/^r/ {sub("^ ", "", $2); sub(" $", "", $2); print $2" = "$2" <"$2">"}' | sort -u > authors.txt
/var/opt/svn # ls -la authors.txt
total 12
-rw-r--r--    1 root     root           511 Sep  6 03:15 authors.txt
/var/opt/svn # cat authors.txt
(no author) = (no author) <(no author)>
[email protected] = [email protected] <[email protected]>
bhuynh78 = bhuynh78 <bhuynh78>
gamma.particle = gamma.particle <gamma.particle>
[email protected] = [email protected] <[email protected]>
kleberkls = kleberkls <kleberkls>
[email protected] = [email protected] <[email protected]>
marcello.sales = marcello.sales <marcello.sales>
[email protected] = [email protected] <[email protected]>
xiaoyudi84 = xiaoyudi84 <xiaoyudi84>

Edit authors with vim, Still inside the docker container, install git and git-svn

/var/opt/svn # apk add vim git git-svn
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
(1/19) Installing ca-certificates (20191127-r4)
(2/19) Installing nghttp2-libs (1.41.0-r0)
(3/19) Installing libcurl (7.69.1-r0)
(4/19) Installing pcre2 (10.35-r0)
(5/19) Installing git (2.26.2-r0)
(6/19) Installing libbz2 (1.0.8-r1)
(7/19) Installing perl (5.30.3-r0)

Edit vim as you wish and save it

/var/opt/svn # vim authors.txt

var/opt/svn # cat authors.txt
(no author) = Marcello de Sales <[email protected]>
[email protected] = [email protected] <[email protected]>
bhuynh78 = Bill Huynh <[email protected]>
gamma.particle = Teresa Johnson <[email protected]>
[email protected] = Teresa Johnson <[email protected]>
kleberkls = Kleber Sales <[email protected]>
[email protected] = Kleber Sales <[email protected]>
marcello.sales = Marcello de Sales <[email protected]>
[email protected] = Marcello de Sales <[email protected]>
xiaoyudi84 = xiaoyudi84 <xiaoyudi84>

At this point you are cloning the master trunk of svn to git...
If you have any branch specifically you want to clone, then specify in the command like explained in https://gist.github.com/trodrigues/1023167
List everything you have from the svn server before you migrate

/var/opt/svn # svn ls svn://localhost:3690/netbeams
branches/
tags/
trunk/
wiki/

This example uses the branch... The -T value must be the one that lists the contents of the repo to a directory called netbeams-marcello-git

/var/opt/svn # git svn --stdlayout -A authors.txt clone -T branches/marcello/persistence svn://localhost:3690/netbeams netbeams-marcello-git
	M	persistence/docs/marcello-dissertation/thesis/chapter8-experimental-results-correct-behavior-and-performance.tex
r718 = bdf4d4b181898643405706f5ea760a58c2d65a54 (refs/remotes/origin/marcello)
Checked out HEAD:
  svn://localhost/netbeams/branches/marcello/persistence r718
creating empty directory: versions/v2/persistence/data
creating empty directory: versions/v2/persistence/mongodb

Move the directory netbeams-marcello-git to the dump directory where it is mapped to the host

/var/opt/svn # mv netbeams-marcello-git /dump

Exit the container with exit and execute the steps in

$ cd netbeams-marcello-git
$ git remote add origin [email protected]:marcellodesales/sfsu-cs-masters-thesis-netbeams.git
$ git push --set-upstream origin master

Wiki pages preserving history

Verify if you have wiki pages

/var/opt/svn # svn ls svn://localhost:3690/netbeams/wiki
CreatingDSPBundles.wiki
DSPDataPersistence.wiki
DSPDataPersistenceOnMongoDB.wiki
DataSensorPlatform.wiki
ExternalResources.wiki
GumstixOESystem.wiki
Gumstix_3boards.jpg
Gumstix_components.jpg
Gumstix_fullconnect.jpg
InstallingDsp.wiki
MongoDBShardsClusterAndMapReduce.wiki
NetbeamsResearch.wiki
RunningDSPOnKnopflerfish_TextConsole.wiki
RunningDspOnKnopflerfish.wiki
RxtxSharedLibrary.wiki
Seabird_DataFlow.pdf
SystemRequirements.wiki

Clone the wiki pages using the -T wiki to the dir netbeams-wiki

$ git svn --stdlayout -A authors.txt clone -T wiki svn://localhost:3690/netbeams netbeams.wiki
$ cd netbeams.wiki

Convert all the old-style wiki pages to new ones (COPY ALL THE LINES ON THIS BLOCK AND PASTE IN THE TERMINAL TO EXECUTE THEM ALL)

find . -type f -iname '*.wiki' -exec sed -i.bak "s/^= /# /g" {} \;
find . -type f -iname '*.wiki' -exec sed -i.bak "s/^== /## /g" {} \;
find . -type f -iname '*.wiki' -exec sed -i.bak "s/^=== /### /g" {} \;
find . -type f -iname '*.wiki' -exec sed -i.bak "s/^==== /#### /g" {} \;
find . -type f -iname '*.wiki' -exec sed -i.bak "s/====$//g" {} \;
find . -type f -iname '*.wiki' -exec sed -i.bak "s/===$//g" {} \;
find . -type f -iname '*.wiki' -exec sed -i.bak "s/==$//g" {} \;
find . -type f -iname '*.wiki' -exec sed -i.bak "s/=$//g" {} \;
find . -type f -iname '*.wiki' -exec sed -i.bak "s/^{{{/\`\`\`/g" {} \;
find . -type f -iname '*.wiki' -exec sed -i.bak "s/^}}}/\`\`\`/g" {} \;
for e in *.wiki ; do sed -i '/^#summary/d' $e ; done
for e in *.wiki ; do sed -i '/^#labels/d' $e ; done
rm -f *.bak

Make sure to examine the files

diff --git a/CreatingDSPBundles.wiki b/CreatingDSPBundles.wiki
index f760423..7d74edc 100644
--- a/CreatingDSPBundles.wiki
+++ b/CreatingDSPBundles.wiki
@@ -1,7 +1,7 @@
 #summary This page describes how to create a DSP component
 #labels osgi,Phase-QA,Phase-Support

-= DSP OSGi Component =
+# DSP OSGi Component

 A DSP component must have the following components:

@@ -13,7 +13,7 @@ A DSP component must have the following components:
  * A MANIFEST.MF file that describes the OSGi application;
  * A build.xml file that bundles all the previous artifacts into a Jar file.

-{{{
+```
 <?xml version="1.0" encoding="UTF-8"?>
 <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:mouseaction="http://dsp.netbeams.org/demo/mouseaction.xsd"
     xmlns:dspmessage="http://www.netbeams.dsp.org/message.xsd" xmlns:jaxb="http://java.sun.com/xml/ns/jaxb"
@@ -80,7 +80,7 @@ The only basic important step in defining the data schema is the root element mu
     </xs:element>

 </xs:schema>
-}}}
+```

 Please, refer to the XML SChema tutorials online in order to define XML Schemas, such as http://www.w3schools.com/Schema/default.asp.

@@ -90,11 +90,11 @@ http://netbeams.googlecode.com/svn-history/r190/trunk/versions/v2/apps/xml/jaxb/

 Since our DSP component data is defined to be transferred over the Internet, we suggest the usage of attributes for Metadata as much as possible, and us
e elements in other cases, such as collection holders, etc.

-== Generating the Java POJO classes from the XML Schema ==
+## Generating the Java POJO classes from the XML Schema

`cp -R files-wiki /dump
Exit the container
Copy all the md files to the cloned wiki repo

$ for f in `find . -iname '*.wiki' -type f -print`;do  git mv "$f" ${f%.wiki}.md; done
$ git mv DataSensorPlatform.md Home.md
$ git status
On branch master
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
	renamed:    CreatingDSPBundles.wiki -> CreatingDSPBundles.md
	renamed:    DSPDataPersistence.wiki -> DSPDataPersistence.md
	renamed:    DSPDataPersistenceOnMongoDB.wiki -> DSPDataPersistenceOnMongoDB.md
	renamed:    ExternalResources.wiki -> ExternalResources.md
	renamed:    GumstixOESystem.wiki -> GumstixOESystem.md
	renamed:    DataSensorPlatform.wiki -> Home.md
	renamed:    InstallingDsp.wiki -> InstallingDsp.md
	renamed:    MongoDBShardsClusterAndMapReduce.wiki -> MongoDBShardsClusterAndMapReduce.md
	renamed:    NetbeamsResearch.wiki -> NetbeamsResearch.md
	renamed:    RunningDSPOnKnopflerfish_TextConsole.wiki -> RunningDSPOnKnopflerfish_TextConsole.md
	renamed:    RunningDspOnKnopflerfish.wiki -> RunningDspOnKnopflerfish.md
	renamed:    RxtxSharedLibrary.wiki -> RxtxSharedLibrary.md
	renamed:    SystemRequirements.wiki -> SystemRequirements.md

Move to the dump

/var/opt/svn # cp -R netbeams-marcello-git.wiki /dump/

On the host, make any other changes, git add and commit

$ git commit -a -m "Rename all wiki pages to md"

If you want, change the date according to the history

$ git commit --amend --no-edit --date="Thu Dec 17 21:22:53 2009 -0800"

Go to the Github Repo page, Go to the Wiki section and click in Create Wiki
- The repo is in the format [email protected]:ORG/REPO.wiki.git
- But there's no need to clone the repo locally.
Push with a forced update

$ git remote add origin [email protected]:marcellodesales/sfsu-cs-masters-thesis-netbeams.wiki.git
$ git push origin -f master
Counting objects: 100% (233/233), done.
Delta compression using up to 16 threads
Compressing objects: 100% (149/149), done.
Writing objects: 100% (233/233), 1.39 MiB | 856.00 KiB/s, done.
Total 233 (delta 164), reused 120 (delta 82)
remote: Resolving deltas: 100% (164/164), done.
To github.com:marcellodesales/sfsu-cs-masters-thesis-netbeams.wiki.git
 + acd4f74...aeceb1d master -> master (forced update)

marcellodesales/migrate-googlecode-svn-repos-to-github-using-docker.md

Migrate Archive Google Code SVN to Git using Docker

Requirements

Result of the manual steps

From Docker container in any host

Wiki pages preserving history

Credits