This chapter covers new Docker commands and details on how to share data between host system and containerized applications. Learn when and how to use persistent volumes and access host filesystems.

Intermediate Docker Commands

Help at a glance

Existing containers

Automatic container cleanup

Mounting the host filesystem

Passing a file to a running container

Gone, but not forgotten

Different methods, same result

Persistent volumes

Creating a container volume

Different containers, same volume

Commands and filesystems

Learn to expose networking services in Docker, create Docker networks, and use Dockerfiles to maintain network settings.

Networking refresher

What's my IP?

Valid ports

Making network services available in Docker

Mapping a port

Multiple containers, same host port?

Exposing ports with Dockerfiles

So many options

Mapping a container port

Using ephemeral mapping

Docker networks

Creating a docker network

Adding containers to a network

IP Ranges

Removing container networks

Networking

Learn to use Docker's advanced image management and build tools to create compact effective container images. Create multi-platform images capable of running on different processor architectures.

Optimizing Docker images

Contain your concerns

Docker image exploration

Determining the best image to use

Understanding layers

Layer cake

How many layers?

Multi-stage builds

Single-stage build

Multi-stage build

Understanding multi-stage builds

Multi-platform builds

Multi-platform steps

Multi-platform build

Building advanced container images

Combine the the volume, networking, and container handling capabilities of Docker into multi-container applications using Docker Compose. Update and manage application deployments via the docker-compose.yml file.

Introduction to Docker Compose

Managing an application with Docker Compose

Troubleshooting application startup

Creating compose.yaml files

Writing a compose.yaml file

Identifying sections

How many resources?

Dependencies and troubleshooting in Docker Compose

Resource startup order

Dependency calculation

Shutdown order

FROM python:3.12-alpine3.19
WORKDIR /code
RUN apk add --no-cache gcc musl-dev linux-headers
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
COPY . .
CMD ["python3", "app.py"]


import time
import redis

cache = redis.Redis(host='redis', port=6379, socket_timeout=1)

def get_hit_count():
  retries = 5
  while True:
    try:
      return cache.incr('hits')
    except redis.exceptions.ConnectionError as exc:
      if retries == 0:
        raise exc
      retries -= 1
      time.sleep(0.5)

def hello():
  count = get_hit_count()
  return f'Successfully communicated {count} times.\n'

for i in range(0, 5):
  print(f"{hello()}")


services:
  app:
    build: .
  redis:
    image: "redis:alpine3.20"
    depends_on:
      - app


services:
  app:
    build: .
    depends_on:
      - redis
  redis:
    image: "redis:alpine3.20"


- Run `docker compose up` or `docker compose up -d`
- If needed, use `docker compose logs` to view startup logs
- Update the compose.yaml to fix the dependencies
- Rerun `docker compose up` and verify you see "Successfully connected" messages (the number may vary)


Troubleshooting dependencies

Creating a data service within Docker

import duckdb
from flask import Flask

app = Flask(__name__)

@app.route("/tripcount")
def returndata():
  cursor = duckdb.connect()
  try:
    count = cursor.execute("select count(*) from read_parquet('/data/*.parquet')").fetchall()[0][0]
    return f"Total trips: {count}"
  except duckdb.duckdb.IOException:
    return f"Data files are not present", 503

@app.route("/first")
def returnfirst():
  cursor = duckdb.connect()
  try:
    first = cursor.execute("select min(tpep_pickup_datetime) from read_parquet('/data/*.parquet')").fetchall()[0][0]
    return f"First pickup: {first:%Y/%m/%d}" 
  except duckdb.duckdb.IOException:
    return f"Data files are not present", 503

@app.route("/last")
def returnlast():
  cursor = duckdb.connect()
  try:
    last = cursor.execute("select max(tpep_pickup_datetime) from read_parquet('/data/*.parquet')").fetchall()[0][0]
    return f"Last pickup: {last:%Y/%m/%d}" 
  except duckdb.duckdb.IOException:
    return f"Data files are not present", 503


FROM python:3.12-alpine3.19

#
# NOTE: Application reads contents of /data in container
# If not found, application returns errors.
# 

WORKDIR /app
ENV FLASK_RUN_HOST=0.0.0.0

COPY code/app.py .
COPY code/requirements.txt .

RUN pip install -r requirements.txt --root-user-action=ignore

EXPOSE 5000

CMD ["flask", "run"]


STATUS_CODE=$(curl --silent --head http://localhost:5000/tripcount | awk '/^HTTP/{print $2}')

if [ "$STATUS_CODE" == "503" ]; then
  echo "ERROR ACCESSING DATA SERVICE - PLEASE VERIFY DATA IS PRESENT"
  echo "Did you use a -v flag with your docker run command?"
elif [ "$STATUS_CODE" == "200" ]; then
  echo "SUCCESSFULLY ACCESSED DATA SERVICE!"
else
  echo "ERROR ACCESSING DATA SERVICE - PLEASE VERIFY SERVICE IS RUNNING"
  echo "Did you start your container with a -p flag?"
fi
  


There are several steps required to complete this exercise:

- Build the dataservice container using the following command in the terminal:

docker build . -t dataservice

- Run an instance of the Docker container as follows:

docker run -d -p5000:5000 -v ~/workspace/datasets:/data dataservice

- Verify that the service is working by running the ./app_test.sh on the terminal


Building a data service backend

Adding a data client

services:
  ____:
    image: dataservice
    volumes:
      - /home/repl/datasets:/data
    networks: 
      data_net:
        ipv4_address: 10.11.12.2

  dataclient:
    image: ____
    depends_on: 
      - ____
    networks:
      data_net:
        ipv4_address: 10.11.12.3

# Ignore everything below this line
networks:
  data_net:
    driver: macvlan
    ipam:
      config:
        - subnet: "10.11.12.0/24"
    


services:
  dataservice:
    image: dataservice
    volumes:
      - /home/repl/workspace/datasets:/data
    networks: 
      data_net:
        ipv4_address: 10.11.12.2

  dataclient:
    image: dataclient
    depends_on: 
      - dataservice
    networks:
      data_net:
        ipv4_address: 10.11.12.3

# Ignore everything below this line
networks:
  data_net:
    driver: macvlan
    ipam:
      config:
        - subnet: "10.11.12.0/24"
    


- Update the compose.yaml file as shown.
- Run the following command:

  docker compose up -d
  
- Run the command:

  docker compose logs dataclient


Using Docker Compose to define a full application

Congratulations!

Docker Compose

This course builds upon the foundations of learning Docker and containerization found in the Introduction to Docker course. We extend the concepts and tools covered in that course, adding the ideas of container image management and optimization, networking, file system communication, multi-platform and multi-container applications. When completed, you'll be able to:

- Create multi-stage builds to optimize the size, security, and reusability of containerized applications.
- Use Docker networking tools and concepts to add inter-container communications and provide networking services to the local network and the outside world.
- Share files and data between containers and the host machine using Docker file system mounts and volumes.
- Create mutli-container application deployments using Docker Compose, drastically simplifying the setup of complex tools common in data science and engineering environments.

You'll gain the most from this course if you have worked with and built some simple containers, but are looking to add to your development, testing, and distribution toolbox. We look forward to helping guide you during your journey of learning Docker.

This course expands on the knowledge gained from the Introduction to Docker course, delving deeper into containerization concepts and tools. We'll explore advanced topics such as container image management, networking, file system communication, and the development of multi-platform and multi-container applications.
<h2>Optimizing Containerized Applications</h2>
Learn to create multi-stage builds to enhance the size, security, and reusability of containerized applications. Through practical exercises, master techniques for optimizing container images, ensuring efficient deployment and management in diverse environments.
<h2>Enhancing Container Deployments with Docker Compose</h2>
Discover how to leverage Docker networking tools and concepts to facilitate inter-container communication and provide networking services to both local and external networks. Explore file system mounts and volumes for seamless data sharing between containers and the host machine. Additionally, gain proficiency in Docker Compose for orchestrating multi-container application deployments, simplifying the setup of complex tools prevalent in data science and engineering environments.
<h2>Empowering Development and Deployment</h2>
This course equips you with essential skills to enhance your development, testing, and distribution processes. While not essential, you will gain the most from this course if you have worked with and built some simple containers. By the course's end, you'll have the knowledge and tools to optimize containerized applications, streamline communication between containers, and deploy multi-container applications with ease, empowering you on your Docker learning journey.

Introduction to Docker

Master multi-stage builds, Docker networking tools, and Docker Compose for optimal containerized applications!

Adding a data client

Intermediate Docker

Hands-on interactive exercise