Efficient Local Full-Stack Development Using Docker/Podman Compose
Video
Introduction
For some aspects of building a multi-component or microservices application, it is very useful to have the entire suite of components running in your local development environment to ensure everything integrates as expected. Traditionally, developers may have created complex shell scripts, created Vagrant recipes, or used automation tools like Ansible to achieve this. In this Lab, we will demonstrate an approach which we feel is well aligned with Cloud Native Application Development, and that is the use of Compose files.
If you are experienced with Docker container engine, you may already be familiar with a tool called docker-compose. What you may not be aware of is that there is a similar tool for Podman and it is aptly called podman-compose. Our experience in using these tools has been that it is an efficient way to shorten feedback cycles while doing development by avoiding the delays of: git commits, pull requests, merges, CI/CD pipelines, and finally deployment to a Dev/Test environment. Those things are still needed and desireable, but shortening feedback cycles means we want other tools in our belts to further reduce overhead before we can confirm functionality and feature fit.
Pre-Requisites For This Lab
- EITHER
- Docker+Docker Compose
- On MacOS/Windows, ensure that your Docker Machine VM has 4 cores and >=6GB of RAM allocated
- IF you are using
docker
, you SHOULD ensure that your user is authorized to execute docker commands.- Be aware, giving unprivileged users permissions to the Docker Engine running as root opens a path to bypass OS security!
- Podman+Podman Compose
- MUST be properly configured for rootless
- Docker+Docker Compose
Installing podman-compose (OPTIONAL)
podman-compose
can be installed via some package managers on Linux (e.g. dnf install podman-compose). It can also be installed using pip. If you install it using pip
, you may want to install it in an isolated virtualenv.
Installing podman-compose
With pip
- OPTIONAL: Create a new virtualenv:
virtualenv compose
- Install
podman-compose
:pip install podman-compose
IF you use a virtual environment, you MUST ensure that you activate that virtualenv every time you want to use podman-compose
Our Example Stack
Getting Started
There is a comprehensive reference for compose
files. We will demonstrate how to craft a Compose file which can be used with both docker-compose
and podman-compose
. There are certain features of Docker Engine which are not compatible directly with Podman and the inverse is also true, and this lab will highlight those mismatches so they can be avoided.
The Basics Of A Compose File
- A Compose file starts like almost any other YAML file, with 3 hyphens (
---
). - We then specify the Compose file version.
- Next we start to define Services within the Compose file.yaml
--- version: "3.8" services: tododb: image: docker.io/postgres:latest keycloak: image: docker.io/jboss/keycloak:10.0.2 todoapi: image: docker.io/maven:3.6.3 todoui: image: node:lts oauth2-proxy: image: quay.io/oauth2-proxy/oauth2-proxy:latest
- Each service is keyed on it's name
- Each service specifies either the URI for a container image OR a
Dockerfile
to build- IF you build from a
Dockerfile
, by default the container image will be named the same as the Service
- IF you build from a
- Now, we need to add more information about how to configure those servicesyaml
# ...SNIP services: tododb: image: docker.io/postgres:latest environment: POSTGRES_USER: tododb POSTGRES_PASSWORD: tododb POSTGRES_DB: todos todoapi: image: docker.io/maven:3.6.3 environment: MAVEN_CONFIG: /tmp/.m2 command: - mvn - clean - compile - spring-boot:run - '-Dspring-boot.run.profiles=compose' - '-Dmaven.repo.local=/tmp/.m2/repository' - '-Dspring-boot.run.jvmArguments="-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=*:5005"' user: ${CONTAINER_UID:-1000} volumes: - ${TODO_API_SERVICE:-../todoapi}:/workspace${CONTAINER_VOLUME_OPTIONS} - ${HOME}/.m2/repository:/tmp/.m2/repository${CONTAINER_VOLUME_OPTIONS} working_dir: /workspace ports: - 9080:9080 ## HOST_PORT:CONTAINER_PORT - 5005:5005 ## HOST_PORT:CONTAINER_PORT # ...SNIP
- You will notice above that we have defined more of the configuration for both the database service and the API service
- In the data, we define environment variables to set the DB user/password/database name
- In the API service we configure:
- An environment variable to tell Maven where to cache JAR files
- The command and it's parameters to be run inside of the container
- The UID with which the container will run as (defaulting to 1000 if not set)
- Some volumes:
- We mount the source code for the API service inside the container as
/workspace
- We mount the Maven local repository inside of the container as
/tmp/.m2/repository
so that we keep the Maven artifacts cached between container restarts
- We mount the source code for the API service inside the container as
- The default working directory in which we will run commands inside of the container
- And finally the ports from the API Service we wish to expose to outside of our container
- Port 9080 for the HTTP interface to the API
- Port 5005 for the Java remote debugger
- Depending on the services you will use for your application, you will have to configure each service appropriately, but we recommend that you use environment variables as appropriate to keep this flexible for your fellow developersyaml
--- # The completed Compose file version: "3.8" services: tododb: image: docker.io/postgres:latest environment: POSTGRES_USER: tododb POSTGRES_PASSWORD: tododb POSTGRES_DB: todos todoapi: image: docker.io/maven:3.6.3 environment: MAVEN_CONFIG: /tmp/.m2 command: - mvn - clean - compile - spring-boot:run - '-Dspring-boot.run.profiles=compose' - '-Dmaven.repo.local=/tmp/.m2/repository' - '-Dspring-boot.run.jvmArguments="-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=*:5005"' user: ${CONTAINER_UID:-1000} volumes: - ${API_SERVICE:-../api}:/workspace - ${HOME}/.m2/repository:/tmp/.m2/repository working_dir: /workspace ports: - 9080:9080 ## HOST_PORT:CONTAINER_PORT - 5005:5005 ## HOST_PORT:CONTAINER_PORT keycloak: image: docker.io/jboss/keycloak:10.0.2 ports: - "8080:8080" ## HOST_PORT:CONTAINER_PORT command: - '-Dkeycloak.profile.feature.upload_scripts=enabled' - '-b' - '0.0.0.0' environment: JAVA_OPTS: '-Dorg.jboss.logmanager.nocolor=true' KEYCLOAK_USER: admin KEYCLOAK_PASSWORD: admin123 KEYCLOAK_IMPORT: /tmp/keycloak/realm.json DB_VENDOR: h2 volumes: - ./keycloak:/tmp/keycloak${CONTAINER_VOLUME_OPTIONS} todoui: build: context: ./ dockerfile: Dockerfile.npm_plus_java_jre image: npm_plus_jre command: - npm - run - compose user: ${CONTAINER_UID:-1000} volumes: - ${UI_SERVICE:-../ui}:/workspace${CONTAINER_VOLUME_OPTIONS} working_dir: /workspace ports: - "3000:3000" ## HOST_PORT:CONTAINER_PORT oauth2-proxy: # https://oauth2-proxy.github.io/oauth2-proxy/auth-configuration#keycloak-auth-provider image: quay.io/oauth2-proxy/oauth2-proxy:latest ports: - "4180:4180" ## HOST_PORT:CONTAINER_PORT restart: on-failure environment: OAUTH2_PROXY_PROVIDER: "oidc" OAUTH2_PROXY_OIDC_ISSUER_URL: "http://keycloak:8080/auth/realms/TodoApp" OAUTH2_PROXY_HTTP_ADDRESS: "0.0.0.0:4180" OAUTH2_PROXY_PASS_ACCESS_TOKEN: "false" OAUTH2_PROXY_PASS_AUTHORIZATION_HEADER: "true" OAUTH2_PROXY_SET_AUTHORIZATION_HEADER: "true" OAUTH2_PROXY_SSL_INSECURE_SKIP_VERIFY: "true" OAUTH2_PROXY_WHITELIST_DOMAIN: "*" OAUTH2_PROXY_COOKIE_DOMAINS: "*" OAUTH2_PROXY_COOKIE_SECURE: "false" OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL: "true" OAUTH2_PROXY_SKIP_PROVIDER_BUTTON: "true" OAUTH2_PROXY_UPSTREAMS: "http://todoui:3000/,http://todoapi:9080/api/v1/" OAUTH2_PROXY_COOKIE_SECRET: "fksdZJWUhzlfGw3Ve6POYc2jmjZogjCZ" OAUTH2_PROXY_CLIENT_ID: "oauth2-proxy" OAUTH2_PROXY_CLIENT_SECRET: "a21c6961-368c-46ad-9cb9-8a0fac72b308" OAUTH2_PROXY_EMAIL_DOMAINS: "*" OAUTH2_PROXY_REDIRECT_URL: "http://keycloak:4180/oauth2/callback" OAUTH2_PROXY_SKIP_AUTH_PREFLIGHT: "true" OAUTH2_PROXY_SKIP_AUTH_REGEX: "favicon.ico"
- There is a lot going on here, so let's talk through some of the new features we are seeing
- You'll notice that in the
oauth2-proxy
config, we have used a keyrestart: on-failure
. This ONLY works indocker-compose
for now and it tells Docker to restart that container automatically when it fails.- This container will often fail the first few times until Keycloak is fully running because it needs to request the OpenID Connect configuration from the Keycloak server
- Additionally, note that we are mounting a volume intended to provide a REALM configuration for Keycloak, then we are passing environment variables to tell Keycloak to load that REALM on startup
- You'll notice that in the
Scripting The Startup
In order to make this Compose file work consistently between docker-compose
and podman-compose
we need to set different values for some of the environment variables. This is due in large part to the ways that these two different tools work under Linux, Windows, and MacOS X. For example, when using podman-compose
you set the container UID to 0
because podman will use UID/GID mapping to map the root
UID to your user account's UID. Another example is that in podman
we may need to account for SELinux constraints, and so we set the CONTAINER_VOLUME_OPTIONS
to add :Z
to the end of the volume mount configuration to interface properly with SELinux. When running on Windows or MacOS (and to a lesser extent on Linux) the layers of abstraction for the filesystem in the container can cause performance problems for disk intensive activities, and for that reason we also use the ,delegated
option with our CONTAINER_VOLUME_OPTIONS
so that disk activity (Read and write) are cached inside of the container. Also for Windows/MacOS, when you have an application which uses natively compiled modules (like node-sass or GoLang or C/C++) then you need to separate your compiled objects for the Host OS from the compiled objects for the container because they are incompatible.
Let's look at an example script which would start this stack using the Compose file and environment variables.
#!/bin/bash
export CONTAINER_UID=${UID:-1000}
## These bind mount options improve performance for applications running
## inside of the container and accessing external files/directories
## Also, `:Z` makes everything work correctly with SELinux
export CONTAINER_VOLUME_OPTIONS=":Z,delegated"
## By default, use the `node_modules` directory directly from the frontend (The default for Linux hosts)
export CONTAINER_NODE_MODULES="${UI_SERVICE:-../ui}/node_modules"
## Detect which Host OS this is running on (If OSTYPE does not exist, we're likely on Windows)
export HOST_OS=${OSTYPE:-windows_nt}
## OS Customizations Via Environment Variables
if [[ "${HOST_OS}" = linux* ]]; then
## Linux host, or it should be!
printf "Detected LINUX host:\n"
fi
if [[ "${HOST_OS}" = windows* ]]; then
## Windows, we believe...
printf "Detected WINDOWS host:\n"
## Use a different directory for node_modules inside the container to avoid
## issues with modules compiled natively for Windows
export CONTAINER_NODE_MODULES="./container_node_modules"
fi
if [[ "${HOST_OS}" = darwin* ]]; then
## MacOS, or something is VERY weird...
printf "Detected DARWIN/MacOS host:\n"
## Use a different directory for node_modules inside the container to avoid
## issues with modules compiled natively for MacOS
export CONTAINER_NODE_MODULES="./container_node_modules"
fi
## Check for existing Maven .m2 directory and create it if it does not exist
## This allows for caching of Maven artifacts outside of the container
if ! [[ -d ${HOME}/.m2/repository ]]; then
mkdir -p ${HOME}/.m2/repository
fi
## Attempt to detect which container engine we should be using.
## By default, if docker is present it will be preferred
which docker-compose >> /dev/null
if [ $? -ne 0 ]; then
which podman-compose >> /dev/null
if [ $? -ne 0 ]; then
printf "Docker/Podman Compose is not currently installed or is not in your PATH. Go HERE to install docker compose: https://docs.docker.com/compose/install/ or here for podman compose: https://github.com/containers/podman-compose\n\n"
else
ENGINE=podman
fi
fi
printf "This script expects that you have the Todo API service and the React Todo UI checked out locally in the directory\n"
printf "just above this directory.\n\n"
printf "You can override the location where these source repositories are located using the following environment variables:\n"
printf " API_SERVICE\n"
printf " UI_SERVICE\n\n"
## Check to see if the UI and API source directories exist
if [ -e "${UI_SERVICE:-../ui}" ] && [ -e "${API_SERVICE:-../api}" ]; then
## Determine if we are using docker-compose or podman-compose
if [ "${ENGINE}" = "podman" ]; then
## Use UID==0 for Podman and relay on UID/GID mapping to keep permissions aligned with our local shell
export CONTAINER_UID=0
## Podman does not support the `delegated` bind mount option
## so it gets removed here
export CONTAINER_VOLUME_OPTIONS=":Z"
fi
## Setting EXTRA_COMPOSE_OPTIONS to `-d` will run the containers in the background and detach from the terminal
${ENGINE:-docker}-compose up${EXTRA_COMPOSE_OPTIONS:""}
fi
printf "In a few minutes, the following services will be available:\n"
printf "\thttp://localhost:9080/ - Todo API Without OAuth Proxy\n"
printf "\thttp://localhost:3000/ - React Todo UI Without OAuth Proxy\n"
printf "\thttp://localhost:4180/ - Todo API AND Todo UI WITH OAuth Proxy\n"
printf "\thttp://localhost:8080/ - KeyCloak with admin:admin123 and some example users\n"
printf "\tjdb://localhost:5005/ - Java Remote Debugger Port\n"
We believe you should be able to use this script as a starting point to start up any application stack using Compose files with either podman
or docker
. Take a few minutes to read through and make note of the in-line comments to better understand the logic in the script.
OS Compatibility
Different developers use different operating systems, and we are all for choice and preference. As such, we have tested this approach on Linux, MacOS (even Catalina), and Windows 10. This CAN work on all of these operating systems. You may find limitations to work around and we would love it if you provide us feedback should you run across an edge case we haven't accounted for.
It's Your Turn!
Let's take what you have learned so far and apply it to our example application stack.
- Change to a directory where you want to store the source code for the API service, the UI, and the Compose project.
- Clone the Todo API with the following command:
git clone https://github.com/redhat-appdev-practice/todo-api.git api;
- Clone the Todo UI with the following command:
git clone https://github.com/redhat-appdev-practice/todo-ui.git ui;
- Create the Compose project directory
mkdir bootstrap cd bootstrap
- Create the Compose file as
docker-compose.yml
in thebootstrap
directory - Create the launch script as
localdev.sh
in thebootstrap
directory - Make the launch script executable using
chmod 755 localdev.sh
- Create the
keycloak
directory underbootstrap
usingmkdir keycloak
- Download the REALM file into the
keycloak
directory - Run the launch script using
./localdev.sh
- NOTE: if you are using
podman
andpodman-compose
, you may need to manually start theoauth2-proxy
container once Keycloak is fully operationalpodman start bootstrap_oauth2-proxy_1
- NOTE: if you are using
- If everything went well, you should see the container images being pulled down locally and then the services will be launched.
- Now that everything is running, create an entry in your
/etc/hosts
file which points127.0.0.1
to the namekeycloak
so that the OAuth2 service names will align - Open a web browser and browse to http://keycloak:4180/
- You should be redirected to login via Keycloak, try the username and password of
jqconsultant
Compose Engine Feature Differences
There are some limitations which have to be considered if you wish to remain compatible with both docker-compose
and podman-compose
.
podman
andpodman-compose
do not supportrestart
policies YET- The current plan from the CRI-O team is that podman v2 will have a Docker compatible API which will then support almost all
docker-compose
features
- The current plan from the CRI-O team is that podman v2 will have a Docker compatible API which will then support almost all
- You cannot use any specialized networks in your Compose file
- You should use the newest version of
podman-compose
as possible for greatest compatibility podman-compose
starts all of the containers in a single Pod, which means that all of the containers use a shared network namespace (e.g. the same set of port bindings), butdocker-compose
does not and would allow you to use the same listening port over in different containerspodman
on Red Hat Enterprise Linux©, Fedora©, CentOS©, and perhaps other distributions of Linux use SELinux to limit privileges even to theroot
user, and thus you may need special volume mount options (e.g.:Z
) appended to make them work properlypodman-compose
does not support starting individual containers YET
Additional Operations With Compose Tools
- Restart a single container:
- Podman:
podman restart <container name>
- Docker:
docker-compose restart <service name>
- Podman:
- Stop all containers:
- Podman:
podman-compose [stop/down]
- Docker:
docker-compose [stop/down]
- Podman:
- Delete all Compose containers after stopping:
- Podman:
podman pod rm <pod name>
- Where the pod name is usually the name of the directory the Compose file was run from. - Docker:
docker-compose rm [-f]
- Podman:
- Start a single container which has exited/crashed
- Podman:
podman start <container name>
- Docker:
docker-compose start <service name>
- Podman: