6626070
2997924

PL00, Docker

Back to the previous page｜page management
List of posts to read before reading this article

Contents

Introduction

“Big Data”




Basic command

URL
About image

$ docker search [image]
$ docker pull [image]
$ docker images      # same to below
$ docker image ls    # same to above
$ docker images -q
$ docker image rm [image]
$ docker commit [origin_container] [new_container]

$ docker run --help
$ docker run [image]
$ docker run [image] [command]
$ docker run -itd [image]
$ docker run --restart=always [image]
$ docker run --name "[container_name]" [image]

$ docker run -itd --restart=always --name "[container_name]" [image] [command]

About container

$ docker exec [container_name] [command]

$ docker ps                # same to below
$ docker container ls      # same to above
$ docker ps -q
$ docker stop [container_name]
$ docker ps -a
$ docker container ls -a
$ docker rename [old_container_name] [new_container_name]
$ docker rm [container_name]
$ docker start [container_name]
$ docker attach [container_name]
    # exit              : container be removed after exit
    # ctrl + d          : container be removed after exit
    # ctrl + p, g       : container be kept after exit(detach)
$ docker cp [container_name]:[path] [client_path]
$ docker cp [client_path] [container_name]:[path]
$ docker run -v [local_path]:[container_path]
    # docker run -v [local_path]:/local_host
$ docker container prune

About resource

$ docker --help
$ docker info
$ docker version
$ docker --version
$ docker-machine ls
$ docker system df




Configuration

/bin/docker_bash
step0] Create container

$ docker search [image_name]
$ docker pull [image_name]
$ docker run -itd --name [container_name(origin)] [image_name]
$ docker attach [container_name(origin)]


step1] Write shall script in origin

$ cd /bin/
$ vim docker_bash

#!/bin/sh
/etc/init.d/xinetd restart
export LANGUAGE=ko
LC_ALL=ko_KR.UTF-8 bash

ctrl + p, q


step2] Commit image file

$ docker commit [container_name(origin)] [image_name(new)]
$ docker images


step3] Remove origin image file

$ docker stop [container_name(origin)]
$ docker rm [container_name(origin)]


step4] Create new customized container

$ docker run -itd --restart=always --name [container_name(new)] [image_name(new)] /bin/docker_bash


step5] Use new customized container

$ docker attach [container_name(new)]




Infrastructure Limitations on Data

$ curl -sSL https://get.docker.com/ | sh
$ sudo usermod -aG docker [user]
$ sudo reboot

$ docker -v




the jupyter/scipy-notebook image

$ docker pull jupyter/scipy-notebook
$ docker run -p 8888:8888 jupyter/scipy-notebook

$ docker ps          # Monitor Running Docker Containers
$ docker stats       # Monitor Docker Memory Usage
$ docker restart [Container ID]
Container ID 
$ docker stats

image

Docker

Docker Is Not a Virtual Machine




Containerization




A Containerized Application




The Docker Container Ecosystem

image

$ snap install docker

# Display Docker Hosts Associated with the Running Attached Docker Engine
$ docker-machine ls

The Docker Client

The Host

The Docker Engine

The Docker Image and the Docker Container

The Docker Registry




Get Docker

Installing Docker on an Ubuntu System

URL

$ apt update
$ apt search docker | grep docker.io

# Display Meta-Information for docker.io Package
$ apt policy docker.io

# Remove Previous Installations of docker
$ apt remove docker

# Allow apt to Use a Repository Over HTTPS
$ apt install apt-transport-https ca-certificates curl software-properties.common

# Add Docker’s Official GPG Key
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add
$ apt-key fingerprint 0EBFCD88

# Add Your System’s Specific Docker Repository
$ add-apt-repository "deb [arch=$(dpkg --print-architecture)] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"




Configure Docker Repository

$ apt update
$ apt install -y docker-ce




Manage Docker as a Non-Root User

$ sudo groupadd docker
$ sudo usermod -aG docker $USER

Hello, Docker!

$ docker version
$ docker help
$ which docker
$ docker run hello-world
$ docker run -it ubuntu /bin/bash

Basic Networking in Docker

$ docker search httpd
$ docker pull httpd
$ docker run -d -p 80:80 httpd
$ docker ps
$ curl local host

# Launch the File Server Via the Docker Container
$ docker run -v ~:/home -p 5000:8000 python:2.7 python -m SimpleHTTPServer




Interactive Programming

Jupyter as Persistent Interactive Computing

How Not to Program Interactively




Setting Up a Minimal Computational Project

$ apt install tree
$ mkdir project project/bin project/src project/docker




Writing the Source Code for the Evaluation of a Bessel Function

$ vim bessel.c

// src/bessel.c
#include <stdio.h>
#include <gsl/gsl_sf_bessel.h>
int main(void)
{
 double x = 5.0;
 double y = gsl_sf_bessel_J0(x);
 printf("J0(%g) = %.18e\n", x, y);
 return 0;
}




Performing Your Calculation Using Docker

$ vim Dockerfile

FROM gcc
LABEL maintainer=@joshuacook
RUN apt-get update && \
 apt-get install -y \
 gsl-bin \
 libgsl0-dbg \
 libgsl0-dev \
 libgsl0ldbl

# Build the gsl Docker Image
$ docker build -t [Image_name] docker

# Display Local Docker Images
$ docker images




Compile Your Source Code

$ docker run -v 'pwd':/home gsl gcc -I /usr/include/ -L /usr/lib/ -lgsl -lgslcblas /home/src/bessel.c -o /home/bin/bessel




Execute Compiled Binary

$ docker run -v 'pwd':/home gsl_image /home/bin/bessel




How to Program Interactively

Launch IPython Using Docker

# Launch ipython Using the jupyter/scipy-notebook Image
$ docker run -it jupyter/scipy-notebook ipython




Persistence




Jupyter Notebooks




Port Connections




Data Persistence in Docker




Attach a Volume




The Docker Engine

Examining the Docker Workstation

image

# Display Docker Usage
$ docker

# Display Docker System Info
$ docker info

# Pull the alpine Docker Image
$ docker pull alpine

# Run an Interactive Shell to an alpine Container
$ docker run -it alpine /bin/sh

# Display All Docker Processes
$ docker ps -a




Hello, World in a Container

$ docker ps
$ docker images
$ docker run hello-world




Run Echo as a Service

# Echo “Hello, World!” as a Service
$ docker run alpine /bin/echo 'Hello, World!'

# Time the Execution of the Echo Service
$ time docker run alpine /bin/echo 'Hello World!'

# Time the Execution of echo Natively
$ time echo 'Hello World!'




Isolating the Bootstrap Time

# Time a Hard Sleep Natively
$ time sleep 2

# Time a Hard Sleep as a Service
$ time docker run alpine /bin/sleep 2




A Daemonized Hello World

# Listing 4-18. Display Currently Running Containers/Processes
$ docker run -d alpine /bin/sh -c "while true; do echo hello world; sleep 1; done"

# Display Currently Running Containers/Processes
$ docker ps

# Show Logs for a Detached Container
$ docker logs upbeat_easley

# Stop a Detached Container
$ docker stop upbeat_easley




The Dockerfile

Best Practices

Stateless Containers

While an application may run more than one process at a time, these processes should be stateless.


Single-Concern Containers

one process per container


Project: A Repo of Docker Images

Prepare for Local Development

# Prepare for Local Development

$ mkdir dockerfiles && cd dockerfiles
$ git init
$ touch README.md
$ git add README.md
$ git commit -m 'init'




Configure GitHub

On GitHub, create a repo called Dockerfiles.

$ git remote add origin git@github.com:<username>/dockerfiles.git
$ git push -u origin master




Building Images Using Dockerfiles

#Display docker build help
$ docker build




Dockerfile Syntax

# Comment
INSTRUCTION arguments




Designing the gsl Image

GNU Scientific Library (GSL) Create the gsl Source Directory

# Create a Directory for the gsl Image Containing an Empty Dockerfile
$ mkdir gsl
$ touch gsl/Dockerfile

# Run a Docker Build
$ docker build -t joshuacook/gsl gsl

Define the gsl Image

$ vim Dockerfile

FROM gcc
LABEL maintainer=@joshuacook
RUN apt-get update && \
 apt-get install -y \
 gsl-bin \
 libgsl0-dbg \
 libgsl0-dev \
 libgsl0ldbl

Build the gsl Image

$ docker build -t joshuacook/gsl gsl
Step Description
Step 1 : FROM gcc A valid Dockerfile must always begin with a FROM instruction.
Step 2 : LABEL maintainer @joshuacook The second layer uses the LABEL instruction to define metadata associated with your image. Each LABEL is a key-value pair.
Step 3 : RUN apt-get update && apt-get install -y The final layer of the image uses the RUN instruction, in this case to install the libraries necessary for using the GNU Scientific Library. 
$ docker inspect joshuacook/gsl

# Commit Changes and Push to GitHub
$ git add gsl/Dockerfile
$ git commit -m 'GSL IMAGE - initial build'
$ git push




The Docker Build Cache

Add an Additional Dependency Layer to joshuacook/gsl

RUN apt-get update && \
 apt-get install -y gdb

# Rerun the joshuacook/gsl Image Build
$ docker build -t joshuacook/gsl gsl

# Rerun the Identical Build Once More
$ docker build -t joshuacook/gsl gsl




Anaconda




Design the miniconda3 Image

Create the miniconda3 Source Directory

# Create a Directory for the miniconda3 Image Containing an Empty Dockerfile
$ mkdir miniconda3
$ touch miniconda3/Dockerfile
$ tree

Begin the Image with FROM, ARG, and MAINTAINER
miniconda3/Dockerfile

FROM debian

# Build the miniconda3 Image
$ docker build -t miniconda3 miniconda3

# Show Cached Images
$ docker images

Commit Changes to the Local Repository

$ git add Dockerfile
$ git commit -m 'MINICONDA3 IMAGE. Added FROM instruction.'

miniconda3/Dockerfile

FROM debian
LABEL maintainer=@joshuacook
ARG DEBIAN_FRONTEND=noninteractive

# An ARG Passed at Build Time
$ DEBIAN_FRONTEND=noninteractive docker build -t some_image .

Idempotently Run the Build

# Run the Build
$ docker build -t miniconda3 miniconda3

Commit Changes to the Local Repository

# Commit Changes
$ git add Dockerfile
$ git commit -m 'MINICONDA3 IMAGE. Added maintainer LABEL and ARG instruction'

Provision the miniconda3 Image
The RUN Instruction

RUN <command>

A RUN Instruction

RUN /bin/bash -c 'source $HOME/.bashrc ; echo $HOME'

Another RUN Instruction

RUN /bin/bash -c 'source $HOME/.bashrc ;\
echo $HOME'

miniconda3/Dockerfile

RUN apt-get update --fix-missing && \
 apt-get install -y \
 wget bzip2 ca-certificates \
 libglib2.0-0 libxext6 libsm6 libxrender1

Run the Build

# Run the Build
$ docker build -t miniconda3 miniconda3

# Idempotently Run the Build
$ docker build -t miniconda3 miniconda3

# Display Images in the Local Image Cache
$ docker images

Commit Changes to the Local Repository

# Commit Changes
$ git add Dockerfile
$ git commit -m 'MINICONDA3 IMAGE. OS provision statement.'

Install Miniconda
miniconda3/Dockerfile

RUN echo 'export PATH=/opt/conda/bin:$PATH' > /etc/profile.d/conda.sh && \
wget --quiet \
 https://repo.continuum.io/miniconda/Miniconda3-4.3.11-Linux-x86_64.sh -O
~/miniconda.sh && \
 /bin/bash ~/miniconda.sh -b -p /opt/conda && \
 rm ~/miniconda.sh

Run the Build

# Run the Build
$ docker build -t miniconda3 miniconda3

Commit the Changes to the Local Repository

# Commit Changes
$ git add Dockerfile
$ git commit -m 'MINICONDA3 IMAGE. Install miniconda3.'




tini

miniconda3/Dockerfile

RUN apt-get install -y curl grep sed dpkg && \
 TINI_VERSION=`curl https://github.com/krallin/tini/releases/latest |
grep -o "/v.*\"" | sed 's:^..\(.*\).$:\1:'` && \
 curl -L "https://github.com/krallin/tini/releases/download/v${TINI_
VERSION}/tini_${TINI_VERSION}.deb" > tini.deb && \
 dpkg -i tini.deb && \
 rm tini.deb && \
 apt-get clean

Run the Build

# Run the Build
$ docker build -t miniconda3 miniconda3

Commit the Changes to the Local Repository

# Commit Changes
$ git add Dockerfile
$ git commit -m 'MINICONDA3 IMAGE. Install tini.'

Configure the Environment Variable with ENV
miniconda3/Dockerfile

ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
ENV PATH /opt/conda/bin:$PATH




ENTRYPOINT

miniconda3/Dockerfile

ENTRYPOINT [ "/usr/bin/tini", "--" ]

Run the Build

# Run the Build




Design the ipython Image




Run the ipython Image as a New Container




Docker Hub

Docker Hub

Alternatives to Docker Hub




Docker ID and Namespaces




Image Repositories




Search for Existing Repositories




Tagged Images

Tags on the Python Image




Official Repositories




Pushing to Docker Hub

Create a New Repository




Push an Image




Pull the Image from Docker Hub




Tagged Image on Docker Hub




The Opinionated Jupyter Stacks

High-Level Overview

jupyter/base-notebook




Notebook Security




The Default Environment




Managing Python Versions




Extending the Jupyter Image Using conda Environments




Using joyvan to Install Libraries




Ephemeral Container Extension

Maintaining Semi-Persistent Changes to Images




The Data Stores

Serialization

Serialization Formats and Methods




Binary Encoding in Python




Redis

Pull the redis Image




Docker Data Volumes and Persistence

Create and View a New Data Volume




Launch Redis as a Persistent Service







Using Redis with Jupyter




A Simple Redis Example




Track an Iterative Process Across Notebooks




Pass a Dictionary via a JSON Dump




Pass a Numpy Array as a Bytestring




MongoDB

Set Up a New AWS t2.micro




Configure the New AWS t2.micro for Docker




Pull the mongo Image




Create and View a New Data Volume




Launch MongoDB as a Persistent Service




Verify MongoDB Installation




Using MongoDB with Jupyter




MongoDB Structure




pymongo




Mongo and Twitter




Obtain Twitter Credentials




Collect Tweets by Geolocation




Insert Tweets Into Mongo




PostgreSQL

Pull the postgres Image




Create New Data Volume




Launch PostgreSQL as a Persistent Service




Verify PostgreSQL Installation




Docker Container Networking




Minimally Verify the Jupyter-PostgreSQL Connection




Connnecting Containers by Name




Using PostgreSQL with Jupyter




Jupyter, PostgreSQL, Pandas, and psycopg2




Minimal Verification




Loading Data into PostgreSQL




PostgreSQL Binary Type and Numpy




Docker Compose

Install docker-compose




What Is docker-compose?

Docker Compose Versions




Build a Simple Docker Compose Application

Run Your Application with Compose




Jupyter and Mongo with Persistence

Specifying the Build Context




Specify the Environment File




Data Persistence




Build Your Application with Compose




Scaling an AWS Application via Instance Type




Restart Docker Compose Application




Complete the Computation

Encode Tweets as Document Vectors




Switch AWS Instance Type to t2.micro

Retrieve Tweets from MongoDB and Compare




Docker Compose Networks




Jupyter and Postgres with Persistence

Specifying the Build Context

Build and Run Your Application with Compose




Interactive Softwqre Development

A Quick Guide to Organizing Computational Biology Projects




A Project Framework for Interactive Development




Project Root Design Pattern




Initialize Project




Examine Database Requirements

Managing the Project via Git




Adding a Database to Your Application




Interactive Development

Create a Python Module Using Jupyter




Add Delayed Processing to Your Application




Extending the Postgres Module

Updating Your Python Module




List of posts followed by this article

Reference

OUTPUT