$ mkdir -p src/contenary
$ cd src/contenary
$ git clone https://[email protected]/contenary/locus.git
The program 'git' is currently not installed. You can install it by typing:
sudo apt install git
$ sudo apt install git
[sudo] password for patrick:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
git-man liberror-perl
Suggested packages:
git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk gitweb git-arch git-cvs git-mediawiki git-svn
The following NEW packages will be installed
git git-man liberror-perl
0 to upgrade, 3 to newly install, 0 to remove and 6 not to upgrade.
Need to get 3,823 kB of archives.
After this operation, 25.6 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 liberror-perl all 0.17-1.2 [19.6 kB]
Get:2 http://gb.archive.ubuntu.com/ubuntu xenial-updates/main amd64 git-man all 1:2.7.4-0ubuntu1.1 [735 kB]
Get:3 http://gb.archive.ubuntu.com/ubuntu xenial-updates/main amd64 git amd64 1:2.7.4-0ubuntu1.1 [3,068 kB]
Fetched 3,823 kB in 1s (1,961 kB/s)
Selecting previously unselected package liberror-perl.
(Reading database ... 174030 files and directories currently installed.)
Preparing to unpack .../liberror-perl_0.17-1.2_all.deb ...
Unpacking liberror-perl (0.17-1.2) ...
Selecting previously unselected package git-man.
Preparing to unpack .../git-man_1%3a2.7.4-0ubuntu1.1_all.deb ...
Unpacking git-man (1:2.7.4-0ubuntu1.1) ...
Selecting previously unselected package git.
Preparing to unpack .../git_1%3a2.7.4-0ubuntu1.1_amd64.deb ...
Unpacking git (1:2.7.4-0ubuntu1.1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up liberror-perl (0.17-1.2) ...
Setting up git-man (1:2.7.4-0ubuntu1.1) ...
Setting up git (1:2.7.4-0ubuntu1.1) ...
$ git clone https://[email protected]/contenary/locus.git
Cloning into 'locus'...
Password for 'https://[email protected]':
remote: Counting objects: 150, done.
remote: Compressing objects: 100% (140/140), done.
remote: Total 150 (delta 59), reused 0 (delta 0)
Receiving objects: 100% (150/150), 984.37 KiB | 501.00 KiB/s, done.
Resolving deltas: 100% (59/59), done.
Checking connectivity... done.
$ sudo apt-get install docker.io
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
bridge-utils cgroupfs-mount containerd runc ubuntu-fan
Suggested packages:
aufs-tools btrfs-tools debootstrap docker-doc rinse zfs-fuse | zfsutils
The following NEW packages will be installed
bridge-utils cgroupfs-mount containerd docker.io runc ubuntu-fan
0 to upgrade, 6 to newly install, 0 to remove and 6 not to upgrade.
Need to get 16.4 MB of archives.
After this operation, 83.6 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 bridge-utils amd64 1.5-9ubuntu1 [28.6 kB]
Get:2 http://gb.archive.ubuntu.com/ubuntu xenial/universe amd64 cgroupfs-mount all 1.2 [4,970 B]
Get:3 http://gb.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 runc amd64 1.0.0~rc2-0ubuntu2~16.04.1 [1,476 kB]
Get:4 http://gb.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 containerd amd64 0.2.5-0ubuntu1~16.04.1 [4,041 kB]
Get:5 http://gb.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 docker.io amd64 1.12.6-0ubuntu1~16.04.1 [10.8 MB]
Get:6 http://gb.archive.ubuntu.com/ubuntu xenial-updates/main amd64 ubuntu-fan all 0.9.2 [30.7 kB]
Fetched 16.4 MB in 3s (4,612 kB/s)
Selecting previously unselected package bridge-utils.
(Reading database ... 174844 files and directories currently installed.)
Preparing to unpack .../bridge-utils_1.5-9ubuntu1_amd64.deb ...
Unpacking bridge-utils (1.5-9ubuntu1) ...
Selecting previously unselected package cgroupfs-mount.
Preparing to unpack .../cgroupfs-mount_1.2_all.deb ...
Unpacking cgroupfs-mount (1.2) ...
Selecting previously unselected package runc.
Preparing to unpack .../runc_1.0.0~rc2-0ubuntu2~16.04.1_amd64.deb ...
Unpacking runc (1.0.0~rc2-0ubuntu2~16.04.1) ...
Selecting previously unselected package containerd.
Preparing to unpack .../containerd_0.2.5-0ubuntu1~16.04.1_amd64.deb ...
Unpacking containerd (0.2.5-0ubuntu1~16.04.1) ...
Selecting previously unselected package docker.io.
Preparing to unpack .../docker.io_1.12.6-0ubuntu1~16.04.1_amd64.deb ...
Unpacking docker.io (1.12.6-0ubuntu1~16.04.1) ...
Selecting previously unselected package ubuntu-fan.
Preparing to unpack .../ubuntu-fan_0.9.2_all.deb ...
Unpacking ubuntu-fan (0.9.2) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for ureadahead (0.100.0-19) ...
ureadahead will be reprofiled on next reboot
Processing triggers for systemd (229-4ubuntu17) ...
Setting up bridge-utils (1.5-9ubuntu1) ...
Setting up cgroupfs-mount (1.2) ...
Setting up runc (1.0.0~rc2-0ubuntu2~16.04.1) ...
Setting up containerd (0.2.5-0ubuntu1~16.04.1) ...
Setting up docker.io (1.12.6-0ubuntu1~16.04.1) ...
Adding group `docker` (GID 129) ...
Done.
Setting up ubuntu-fan (0.9.2) ...
Processing triggers for systemd (229-4ubuntu17) ...
Processing triggers for ureadahead (0.100.0-19) ...
$ sudo apt-get install docker-compose
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
python-backports.ssl-match-hostname python-cached-property python-cffi-backend python-chardet python-cryptography python-docker
python-dockerpty python-docopt python-enum34 python-funcsigs python-functools32 python-idna python-ipaddress python-jsonschema
python-mock python-ndg-httpsclient python-openssl python-pbr python-pkg-resources python-pyasn1 python-requests python-six
python-texttable python-urllib3 python-websocket python-yaml
Suggested packages:
python-cryptography-doc python-cryptography-vectors python-enum34-doc python-funcsigs-doc python-mock-doc python-openssl-doc
python-openssl-dbg python-setuptools python-ntlm
The following NEW packages will be installed
docker-compose python-backports.ssl-match-hostname python-cached-property python-cffi-backend python-chardet python-cryptography
python-docker python-dockerpty python-docopt python-enum34 python-funcsigs python-functools32 python-idna python-ipaddress
python-jsonschema python-mock python-ndg-httpsclient python-openssl python-pbr python-pkg-resources python-pyasn1 python-requests
python-six python-texttable python-urllib3 python-websocket python-yaml
0 to upgrade, 27 to newly install, 0 to remove and 6 not to upgrade.
Need to get 1,438 kB of archives.
After this operation, 7,408 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 python-pkg-resources all 20.7.0-1 [108 kB]
Get:2 http://gb.archive.ubuntu.com/ubuntu xenial/universe amd64 python-cached-property all 1.3.0-1 [6,976 B]
Get:3 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 python-six all 1.10.0-3 [10.9 kB]
Get:4 http://gb.archive.ubuntu.com/ubuntu xenial-updates/main amd64 python-urllib3 all 1.13.1-2ubuntu0.16.04.1 [57.2 kB]
Get:5 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 python-chardet all 2.3.0-2 [96.3 kB]
Get:6 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 python-requests all 2.9.1-3 [55.6 kB]
Get:7 http://gb.archive.ubuntu.com/ubuntu xenial/universe amd64 python-backports.ssl-match-hostname all 3.4.0.2-1 [6,258 B]
Get:8 http://gb.archive.ubuntu.com/ubuntu xenial/universe amd64 python-websocket all 0.18.0-2 [165 kB]
Get:9 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 python-ipaddress all 1.0.16-1 [18.0 kB]
Get:10 http://gb.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 python-docker all 1.9.0-1~16.04.1 [29.9 kB]
Get:11 http://gb.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 python-dockerpty all 0.4.1-1~16.04.1 [10.9 kB]
Get:12 http://gb.archive.ubuntu.com/ubuntu xenial/universe amd64 python-docopt all 0.6.2-1build1 [25.6 kB]
Get:13 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 python-enum34 all 1.1.2-1 [35.8 kB]
Get:14 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 python-functools32 all 3.2.3.2-2 [10.7 kB]
Get:15 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 python-funcsigs all 0.4-2 [12.6 kB]
Get:16 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 python-pbr all 1.8.0-4ubuntu1 [46.6 kB]
Get:17 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 python-mock all 1.3.0-2.1ubuntu1 [46.5 kB]
Get:18 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 python-jsonschema all 2.5.1-4 [30.5 kB]
Get:19 http://gb.archive.ubuntu.com/ubuntu xenial/universe amd64 python-texttable all 0.8.1-1 [7,566 B]
Get:20 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 python-yaml amd64 3.11-3build1 [105 kB]
Get:21 http://gb.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 docker-compose all 1.8.0-2~16.04.1 [107 kB]
Get:22 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 python-cffi-backend amd64 1.5.2-1ubuntu1 [58.1 kB]
Get:23 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 python-idna all 2.0-3 [35.1 kB]
Get:24 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 python-pyasn1 all 0.1.9-1 [45.1 kB]
Get:25 http://gb.archive.ubuntu.com/ubuntu xenial-updates/main amd64 python-cryptography amd64 1.2.3-1ubuntu0.1 [199 kB]
Get:26 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 python-openssl all 0.15.1-2build1 [84.1 kB]
Get:27 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 python-ndg-httpsclient all 0.4.0-3 [25.1 kB]
Fetched 1,438 kB in 1s (1,322 kB/s)
Selecting previously unselected package python-pkg-resources.
(Reading database ... 175031 files and directories currently installed.)
Preparing to unpack .../python-pkg-resources_20.7.0-1_all.deb ...
Unpacking python-pkg-resources (20.7.0-1) ...
Selecting previously unselected package python-cached-property.
Preparing to unpack .../python-cached-property_1.3.0-1_all.deb ...
Unpacking python-cached-property (1.3.0-1) ...
Selecting previously unselected package python-six.
Preparing to unpack .../python-six_1.10.0-3_all.deb ...
Unpacking python-six (1.10.0-3) ...
Selecting previously unselected package python-urllib3.
Preparing to unpack .../python-urllib3_1.13.1-2ubuntu0.16.04.1_all.deb ...
Unpacking python-urllib3 (1.13.1-2ubuntu0.16.04.1) ...
Selecting previously unselected package python-chardet.
Preparing to unpack .../python-chardet_2.3.0-2_all.deb ...
Unpacking python-chardet (2.3.0-2) ...
Selecting previously unselected package python-requests.
Preparing to unpack .../python-requests_2.9.1-3_all.deb ...
Unpacking python-requests (2.9.1-3) ...
Selecting previously unselected package python-backports.ssl-match-hostname.
Preparing to unpack .../python-backports.ssl-match-hostname_3.4.0.2-1_all.deb ...
Unpacking python-backports.ssl-match-hostname (3.4.0.2-1) ...
Selecting previously unselected package python-websocket.
Preparing to unpack .../python-websocket_0.18.0-2_all.deb ...
Unpacking python-websocket (0.18.0-2) ...
Selecting previously unselected package python-ipaddress.
Preparing to unpack .../python-ipaddress_1.0.16-1_all.deb ...
Unpacking python-ipaddress (1.0.16-1) ...
Selecting previously unselected package python-docker.
Preparing to unpack .../python-docker_1.9.0-1~16.04.1_all.deb ...
Unpacking python-docker (1.9.0-1~16.04.1) ...
Selecting previously unselected package python-dockerpty.
Preparing to unpack .../python-dockerpty_0.4.1-1~16.04.1_all.deb ...
Unpacking python-dockerpty (0.4.1-1~16.04.1) ...
Selecting previously unselected package python-docopt.
Preparing to unpack .../python-docopt_0.6.2-1build1_all.deb ...
Unpacking python-docopt (0.6.2-1build1) ...
Selecting previously unselected package python-enum34.
Preparing to unpack .../python-enum34_1.1.2-1_all.deb ...
Unpacking python-enum34 (1.1.2-1) ...
Selecting previously unselected package python-functools32.
Preparing to unpack .../python-functools32_3.2.3.2-2_all.deb ...
Unpacking python-functools32 (3.2.3.2-2) ...
Selecting previously unselected package python-funcsigs.
Preparing to unpack .../python-funcsigs_0.4-2_all.deb ...
Unpacking python-funcsigs (0.4-2) ...
Selecting previously unselected package python-pbr.
Preparing to unpack .../python-pbr_1.8.0-4ubuntu1_all.deb ...
Unpacking python-pbr (1.8.0-4ubuntu1) ...
Selecting previously unselected package python-mock.
Preparing to unpack .../python-mock_1.3.0-2.1ubuntu1_all.deb ...
Unpacking python-mock (1.3.0-2.1ubuntu1) ...
Selecting previously unselected package python-jsonschema.
Preparing to unpack .../python-jsonschema_2.5.1-4_all.deb ...
Unpacking python-jsonschema (2.5.1-4) ...
Selecting previously unselected package python-texttable.
Preparing to unpack .../python-texttable_0.8.1-1_all.deb ...
Unpacking python-texttable (0.8.1-1) ...
Selecting previously unselected package python-yaml.
Preparing to unpack .../python-yaml_3.11-3build1_amd64.deb ...
Unpacking python-yaml (3.11-3build1) ...
Selecting previously unselected package docker-compose.
Preparing to unpack .../docker-compose_1.8.0-2~16.04.1_all.deb ...
Unpacking docker-compose (1.8.0-2~16.04.1) ...
Selecting previously unselected package python-cffi-backend.
Preparing to unpack .../python-cffi-backend_1.5.2-1ubuntu1_amd64.deb ...
Unpacking python-cffi-backend (1.5.2-1ubuntu1) ...
Selecting previously unselected package python-idna.
Preparing to unpack .../python-idna_2.0-3_all.deb ...
Unpacking python-idna (2.0-3) ...
Selecting previously unselected package python-pyasn1.
Preparing to unpack .../python-pyasn1_0.1.9-1_all.deb ...
Unpacking python-pyasn1 (0.1.9-1) ...
Selecting previously unselected package python-cryptography.
Preparing to unpack .../python-cryptography_1.2.3-1ubuntu0.1_amd64.deb ...
Unpacking python-cryptography (1.2.3-1ubuntu0.1) ...
Selecting previously unselected package python-openssl.
Preparing to unpack .../python-openssl_0.15.1-2build1_all.deb ...
Unpacking python-openssl (0.15.1-2build1) ...
Selecting previously unselected package python-ndg-httpsclient.
Preparing to unpack .../python-ndg-httpsclient_0.4.0-3_all.deb ...
Unpacking python-ndg-httpsclient (0.4.0-3) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for doc-base (0.10.7) ...
Processing 1 added doc-base file...
Setting up python-pkg-resources (20.7.0-1) ...
Setting up python-cached-property (1.3.0-1) ...
Setting up python-six (1.10.0-3) ...
Setting up python-urllib3 (1.13.1-2ubuntu0.16.04.1) ...
Setting up python-chardet (2.3.0-2) ...
Setting up python-requests (2.9.1-3) ...
Setting up python-backports.ssl-match-hostname (3.4.0.2-1) ...
Setting up python-websocket (0.18.0-2) ...
Setting up python-ipaddress (1.0.16-1) ...
Setting up python-docker (1.9.0-1~16.04.1) ...
Setting up python-dockerpty (0.4.1-1~16.04.1) ...
Setting up python-docopt (0.6.2-1build1) ...
Setting up python-enum34 (1.1.2-1) ...
Setting up python-functools32 (3.2.3.2-2) ...
Setting up python-funcsigs (0.4-2) ...
Setting up python-pbr (1.8.0-4ubuntu1) ...
update-alternatives: using /usr/bin/python2-pbr to provide /usr/bin/pbr (pbr) in auto mode
Setting up python-mock (1.3.0-2.1ubuntu1) ...
Setting up python-jsonschema (2.5.1-4) ...
update-alternatives: using /usr/bin/python2-jsonschema to provide /usr/bin/jsonschema (jsonschema) in auto mode
Setting up python-texttable (0.8.1-1) ...
Setting up python-yaml (3.11-3build1) ...
Setting up docker-compose (1.8.0-2~16.04.1) ...
Setting up python-cffi-backend (1.5.2-1ubuntu1) ...
Setting up python-idna (2.0-3) ...
Setting up python-pyasn1 (0.1.9-1) ...
Setting up python-cryptography (1.2.3-1ubuntu0.1) ...
Setting up python-openssl (0.15.1-2build1) ...
Setting up python-ndg-httpsclient (0.4.0-3) ...see instructions at https://bitbucket.org/contenary/locus
The referenced install link has changed a lot - TODO(PMM) check whether this is relevant group docker was already added
Adding group 'docker' (GID 129) ...
$ sudo gpasswd -a ${USER} docker
Adding user patrick to group docker
$ sudo service docker restart
$ groups
patrick adm cdrom sudo dip plugdev lpadmin sambashare
patrick@COMPAQ-UBUNTU:~/src/contenary/locus$ newgrp docker
patrick@COMPAQ-UBUNTU:~/src/contenary/locus$ groups
docker adm cdrom sudo dip plugdev lpadmin sambashare patrick
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
note: no need to exit shell at any point yet
lots of bandwidth is silently consumed at this point then quite a lot of CPU then some bandwidth ...
output is
make run
docker build --quiet=true --tag locus:latest .
sha256:919a1b0645113c85428f33c23d353e1c50c9f76ff6f4fdef368fa2b9904976b2
docker run --rm --tty --interactive --volume /home/patrick/src/contenary/locus:/opt/locus:rw locus:latest
Commands
build - run - bash - test
Docker commands
docker_test
$ make docker_test
docker build --quiet=true --tag locus:latest .
sha256:919a1b0645113c85428f33c23d353e1c50c9f76ff6f4fdef368fa2b9904976b2
docker run --rm --tty --interactive --volume /home/patrick/src/contenary/locus:/opt/locus:rw locus:latest test
find . -name 'flycheck*.py' -delete
nosetests -sv --with-coverage --cover-package=locus
test_field (test_etl.TestETL) ... ok
test_record (test_etl.TestETL) ... ok
test_clean_time (test_etl.TestExtractTS) ... ok
test_extract_tiploc (test_etl.TestExtractTS) ... ok
test_timetable_read_header (test_etl.TestTimetableReader) ... ok
test_timetable_read_small_timetable (test_etl.TestTimetableReader) ... ok
Name Stmts Miss Cover Missing
--------------------------------------------------------
locus.py 0 0 100%
locus/cli.py 0 0 100%
locus/data.py 0 0 100%
locus/data/controller.py 59 24 59% 30-43, 46-50, 53-57, 60-68, 71, 74-80, 111-114
locus/data/extract.py 40 5 88% 300-301, 308-310
locus/data/primitives.py 106 30 72% 27-33, 60, 63, 66-69, 79-82, 88, 104-107, 142-144, 158, 165-176, 189-190, 193
locus/data/visitor.py 7 0 100%
--------------------------------------------------------
TOTAL 212 59 72%
----------------------------------------------------------------------
Ran 6 tests in 0.557s
OK
sudo apt-get install python3-pip
https://gist.github.com/patrickmmartin/5b6b2ddecd29d6a1b2ffee2d8eea31ec
pip install --upgrade pip
pip install -r requirements.prod.txt
sudo -H pip install -r requirements.test.txt
$ ls -larth .coverage
-rw-r--r-- 1 root root 1.9K May 15 21:40 .coverage
$ sudo rm .coverage
$ make test
find . -name 'flycheck*.py' -delete
nosetests -sv --with-coverage --cover-package=locus
test_field (test_etl.TestETL) ... ok
test_record (test_etl.TestETL) ... ok
test_clean_time (test_etl.TestExtractTS) ... ok
test_extract_tiploc (test_etl.TestExtractTS) ... ok
test_timetable_read_header (test_etl.TestTimetableReader) ... ok
test_timetable_read_small_timetable (test_etl.TestTimetableReader) ... ok
Name Stmts Miss Cover Missing
--------------------------------------------------------
locus.py 0 0 100%
locus/cli.py 0 0 100%
locus/data.py 0 0 100%
locus/data/controller.py 59 24 59% 30-43, 46-50, 53-57, 60-68, 71, 74-80, 111-114
locus/data/extract.py 40 5 88% 300-301, 308-310
locus/data/primitives.py 106 30 72% 27-33, 60, 63, 66-69, 79-82, 88, 104-107, 142-144, 158, 165-176, 189-190, 193
locus/data/visitor.py 7 0 100%
--------------------------------------------------------
TOTAL 212 59 72%
----------------------------------------------------------------------
Ran 6 tests in 0.479s
OK