Skip to content

Instantly share code, notes, and snippets.

@calumroy
Last active January 5, 2022 01:58
Show Gist options
  • Save calumroy/ee7ea8768539ac2be0017fc158424763 to your computer and use it in GitHub Desktop.
Save calumroy/ee7ea8768539ac2be0017fc158424763 to your computer and use it in GitHub Desktop.
Docker containers

Docker Containers

Install docker from https://docs.docker.com/engine/installation/

Run the docker for Mac app. This must be started before you can call docker commands.

Docker containers are like a lightweight virtual machine.
They can be created from scratch or downloaded from the docker hub https://hub.docker.com/

An instance of an image is called a container. You have an image, which is a set of layers. If you start this image, you have a running container of this image. You can have many running containers of the same image. You can stop a container and you can run a stopped container and keep on doing things you where doing with it.

You can see all your images with docker images whereas you can see your running containers with docker ps (and you can see all containers with docker ps -a).

So a running instance of an image is a container.

Use docker pull container_name to download a container. Note: Some containers have multiple tags that are used to identify different versions. You may need to append the tag name onto the end of the container name e.g

docker pull container_name:1.2

To view all the docker images on the system

docker images -a

To delete a docker image

docker rmi Image Image

To view that status of any docker containers use:

docker ps

Docker containers are seperated from the OS. They have to be started and then logged into.

Docker containers will not save there state so if they are stopped and restarted all files that where saved in them are lost.

Mount a directory and save stuff in it to get around this issue. Alternatively you can save the state of a container with

docker ps

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS              NAMES
c3f279d17e0a        ubuntu:12.04        /bin/bash           7 days ago          Up 25 hours                            desperate_dubinsky
197387f1b436        ubuntu:12.04        /bin/bash           7 days ago          Up 25 hours                            focused_hamilton

$ docker commit c3f279d17e0a  svendowideit/testimage:version3

Note that this will create a new image, which will take a few GBs space on your disk.

Start a container

If my docker container is called gcr.io/tensorflow/tensorflow:latest-devel then start it with

docker run -it gcr.io/tensorflow/tensorflow:latest-devel

Here, ‘-it’ will allocate a terminal to created container.

You can log into the Docker Image using the root user (ID = 0) instead of the provided default user when you use the -u option. This will grant you sudo privleges.

docker run -u 0 -it mycontainer bash

Share folders between a container and the OS

To mount a folder so it can be accessed by both the OS and the docker container start docker container with the following command:

docker run -it -v $HOME/directory/:d_dir/ gcr.io/tensorflow/tensorflow:latest-devel

This starts our docker container and also mounts a directory from the OS named directory and
calls it dir in the container. This directory is now accessible from both the docker container and the OS.

Modifying an Existing Docker Image

To install a custom package or modify an existing docker image we need to

  1. run a docker a container from the image we wish to modify
  2. modify the docker container
  3. commit the changes to the container as a docker image
  4. test changes made to image

1.) Running a docker container from an image

The command to do this is,

docker run -it yhat/scienceops-python:0.0.2 /bin/bash
  • The -i tells docker to attach stdin to the container

  • The -t tells docker to give us a pseudo-terminal

  • /bin/bash will run a terminal process in your container

2.) Modify the docker container

Once we are in our container we can install package(s), and set environment variables

$ sudo apt-get install vim
$ export AWS_SECRET_KEY=mysecretkey123
$ export AWS_ACCESS_KEY=fooKey

Copy files/folders to the container or vice versa.

docker cp source_dir/. container_instance_name:/destination_dir/

Use the following to get the container instance name

docker ps --format "{{.Names}}"

When you are done modifying your container you must exit by running the exit command. Once we exit the container, we need to find the container ID by running

docker ps -a

3.) Commit the changes to the container as a new image

Copy the container ID for the container you just modified, and then run the docker commit command to commit changes to your container as an image.

docker commit [options] [container ID] [repository:tag]

An example docker commit command is the following.

docker commit e8f0671518a2 yhat/scienceops-python:0.0.2

Note here! You must commit the changes with the same tags as the scienceops image on your system. To see your new image run.

docker images

4.) Test changes made to image

To test your changes when adding an environment variable run the test command

$ docker run -it yhat/scienceops-python:0.0.2 echo $AWS_SECRET_KEY

Define a container with Dockerfile

Create an empty directory. Change directories (cd) into the new directory.
Create a file called Dockerfile with no name extension.
Copy-and-paste the following content into that file, and save it. Take note of the comments that explain each statement in your new Dockerfile.

# Use an official Python runtime as a parent image
FROM python:2.7-slim

# Set the working directory to /app
WORKDIR /app

# Copy the current directory contents into the container at /app
ADD . /app

# Install any needed packages specified in requirements.txt
RUN pip install --trusted-host pypi.python.org -r requirements.txt

# Make port 80 available to the world outside this container
EXPOSE 80

# Define environment variable
ENV NAME World

# Run app.py when the container launches
CMD ["python", "app.py"]
This Dockerfile refers to a couple of files we haven’t created yet, namely app.py and requirements.txt. Let’s create those next.

Building the image from the docker file

From inside the my_build directory (where the dockerfile is located), we’ll use the docker build command, passing the -t flag to “tag” the new image with a name, which in this case will be my_image. The . indicates that the Dockerfile is in the current directory, along with so-called “context” — that is, the rest of the files that may be in that location:

cd ~/my_build
docker build -t my_image .

Now once built we can use the following to view it along with all our other docker images.

docker images

Create a new image based off a downloaded image

  1. Create an empty directory and copy or create script.sh in it
  2. Create Dockerfile name it "Dockerfile" with following content:
FROM repo/image
WORKDIR $HOME

Where repo/image is the name of the downloaded image you want to base the new image on.

  1. Run the following command from the new directory.
docker build -t="new_image_name_with_tag" .
@calumroy
Copy link
Author

Docker commands

    attach    Attach to a running container
    build     Build an image from a Dockerfile
    commit    Create a new image from a container's changes
    cp        Copy files/folders between a container and the local filesystem
    create    Create a new container
    diff      Inspect changes on a container's filesystem
    events    Get real time events from the server
    exec      Run a command in a running container
    export    Export a container's filesystem as a tar archive
    history   Show the history of an image
    images    List images
    import    Import the contents from a tarball to create a filesystem image
    info      Display system-wide information
    inspect   Return low-level information on a container or image
    kill      Kill a running container
    load      Load an image from a tar archive or STDIN
    login     Log in to a Docker registry
    logout    Log out from a Docker registry
    logs      Fetch the logs of a container
    network   Manage Docker networks
    pause     Pause all processes within a container
    port      List port mappings or a specific mapping for the CONTAINER
    ps        List containers
    pull      Pull an image or a repository from a registry
    push      Push an image or a repository to a registry
    rename    Rename a container
    restart   Restart a container
    rm        Remove one or more containers
    rmi       Remove one or more images
    run       Run a command in a new container
    save      Save one or more images to a tar archive
    search    Search the Docker Hub for images
    start     Start one or more stopped containers
    stats     Display a live stream of container(s) resource usage statistics
    stop      Stop a running container
    tag       Tag an image into a repository
    top       Display the running processes of a container
    unpause   Unpause all processes within a container
    update    Update configuration of one or more containers
    version   Show the Docker version information
    volume    Manage Docker volumes
    wait      Block until a container stops, then print its exit code

@calumroy
Copy link
Author

calumroy commented May 6, 2018

Ubuntu Docker

Run docker commands without sudo

sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker

Start Docker with

sudo dockerd

@calumroy
Copy link
Author

calumroy commented May 6, 2018

Docker GUI with X11 forwarding and opengl

This docker container is a ubuntu 14.04 image with PyQt4 installed to run the HTM project.

MAC OSX

To start this container on OSX run this bash script:

#!/bin/bash

# this script starts a docker container and allows
# gui's for example pyqt4 to be used from the docker image.
# this script need Xquarts installed on mac osx.
# brew cask install xquartz
defaults write org.macosforge.xquartz.X11 enable_iglx -bool true
open -a XQuartz
ip=$(ifconfig en0 | grep inet | awk '$1=="inet" {print $2}')
xhost + $ip
docker run -e DISPLAY=$ip:0 -u 0 -it image_name bash

Linux

Make sure that you enable X11 forwarding with glx. Openai uses opengl.
This is done by editing /etc/X11/xorg.conf
Add the lines

Section "ServerFlags"  
    Option "AllowIndirectGLX" "on"  
    Option "IndirectGLX" "on"  
EndSection

Then run on the host

ip=$(ip route get 8.8.8.8 | awk '{print $NF; exit}')
echo $ip
xhost +

Note the xhost + command discards any protection a bettter option is xhost +local.
To re-enable protection after you have finished with the docker container use

xhost -

start the image with

sudo docker run -it -e DISPLAY=$ip:0    --env="DISPLAY"     --env="QT_X11_NO_MITSHM=1" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw"     --device /dev/dri     image_name     bash

@calumroy
Copy link
Author

calumroy commented May 9, 2018

Dockerfile

A Dockerfile is a simple text file that contains the commands a user could call to assemble an image.

Example, Dockerfile

FROM ubuntu:latest
MAINTAINER john doe 

RUN apt-get update
RUN apt-get install -y python python-pip wget
RUN pip install Flask

ADD hello.py /home/hello.py

WORKDIR /home

Docker Compose

Docker Compose is a tool for defining and running multi-container Docker applications.
Define the services that make up your app in docker-compose.yml so they can be run together in an isolated environment.
Get an app running in one command by just running

docker-compose up

Example, docker-compose.yml

version: '3'
services:
  web:
    build: .
    ports:
    - "5000:5000"
    volumes:
    - .:/code
    - logvolume01:/var/log
    links:
    - redis
  redis:
    image: redis
volumes:
  logvolume01: {}

Mount the current directory to the container

Using the keyword volumes in the docker compose file you can mount a directory into the container.
This allows for you to keep and edit projects on your host machine and share them to the container when you want to run them.
The above docker compose file mounts the current directory on the host to the containers /code directory.

 volumes:
    - .:/code

@calumroy
Copy link
Author

calumroy commented Apr 21, 2019

Stop a running docker image

open a new terminal that is not part of the running image you want to shutdown.
Run the command docker ps to view the running images. Copy the CONTAINER ID number and then run:

docker stop **CONTAINER ID**

How do I SSH into a running container

There is a docker exec command that can be used to connect to a container that is already running.

Use docker ps to get the name of the existing container
Use the command docker exec -it <container name> /bin/bash to get a bash shell in the container
If I want root privileges in the bash shell use:

docker exec -u root -t -i <container name> /bin/bash

Generically, use docker exec -it <container name> <command> to execute whatever command you specify in the container.

@calumroy
Copy link
Author

calumroy commented Apr 21, 2019

Images and Containers

Using an object-oriented programming analogy, the difference between a Docker image and a Docker container is the same as that of the difference between a class and an object. An object is the runtime instance of a class. Similarly, a container is the runtime instance of an image.

Delete an image or container

Image:

List images
docker images

Remove one image
docker rmi image_name

Force remove one image
docker rmi -f image_name

Container:

List all containers
docker ps -a

Remove one container
docker rm container_id

Force remove one container
docker rm -f container_id

@calumroy
Copy link
Author

Docker with GPU

Use Nvidia's docker version to easily allow the GPU to be used form docker containers.
You can use standard docker as well but more input parameters are required. See below.

For example using tensorflow with GPU in a docker container

Start the image with nvidia-docker-compose

This image can be easily started with the following docker-compose.yaml file in an empty directory.

version: '3'
services:
  tf:
    image: gcr.io/tensorflow/tensorflow:latest-gpu
    ports:
      - 8888:8888
    volumes:
      - .:/notebooks

To start the image use

nvidia-docker-compose up

or with the follwoing in .bashrc file

alias doc='nvidia-docker-compose'
alias docl='doc logs -f --tail=100'

then run

doc up

To run an interactive shell use

doc run tf

Alternatively

sudo nvidia-docker run -u 0 -it gcr.io/tensorflow/tensorflow:latest-gpu bash

To Run with standard docker run

docker run --rm --device /dev/nvidia0:/dev/nvidia0 --device /dev/nvidiactl:/dev/nvidiactl --device /dev/nvidia-uvm:/dev/nvidia-uvm -u 0 -it gcr

@calumroy
Copy link
Author

calumroy commented Jan 5, 2022

Use bash scripts to start and enter a running docker container

start_rtk_docker.sh

docker run -it -u rtkuser -e DISPLAY=$ip:0 \
					      --env="DISPLAY"     \
					      --env="QT_X11_NO_MITSHM=1" \
					      --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" \
					      --mount source=visualcode,target=/opt/visualcode \
					      -v $PWD/:/home/rtkuser/ \
					      -v /home/calum/Documents/logs_rosbag2/:/home/rtkuser/logs_rosbag2 \
                                              -v /home/calum/Documents/notebooks/:/home/rtkuser/notebooks \
					      --cap-add=SYS_PTRACE \
					      --security-opt seccomp=unconfined \
					      --network host \
					      --hostname aht002 \
					      --privileged \
					      rtk_dev:latest   \
					      /bin/bash -c "sudo dpkg -i /home/rtkuser/rti_deb/packages/rti-connext-dds_6.1.0-1_amd64.deb /home/rtkuser/rti_deb/packages/rti-connext-dds-dev_6.1.0-1_amd64.deb && /bin/bash"

# docker run -it -u rtkuser -e XDG_RUNTIME_DIR=/tmp \
# 				           -e WAYLAND_DISPLAY=$WAYLAND_DISPLAY \
# 				           -v $XDG_RUNTIME_DIR/$WAYLAND_DISPLAY:/tmp/$WAYLAND_DISPLAY  \
# 				           --user=$(id -u):$(id -g) \
# 				           --mount source=visualcode,target=/opt/visualcode -v $PWD/:/home/rtkuser/ --cap-add=SYS_PTRACE --security-opt seccomp=unconfined rtk_dev:latest   /bin/bash

To enter the same container run this bash script.

#!/bin/bash

# Enter the docker container that has already been started.
# This expects the container to have a certain tag name rtk
docker exec -it $(docker ps | grep 'rtk' | awk '{ print $1 }') /bin/bash 

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment