DockerCon EU 2017

October 16: Full day workshop

Introduction to Docker for Enterprise Operations

Instructor: Bill Mills, Docker
Duration: Full Day  |  9:00 - 17:00
Experience Level: Working knowledge of Docker

In this introductory workshop, geared for administrators and operators in an enterprise, you will get an overview of the features of Docker Datacenter and perform common operational tasks including install, deploying applications, scaling, monitoring, image scanning, and using Docker Trusted Registry.  You will benefit by getting hands-on with Docker Datacenter and have a better understanding of the role of the Docker administrator in a large organization.

October 17: Day one

Keynote

Speaker: Several people from Docker, Google, Metlife, ... including Docker CEO and docker CTO/Co-founder

Notes: Big reveal of the day: Docker is going to support Kubernetes natively as one of the two (next to Swarm) orchestration providers. (https://www.docker.com/kubernetes)

Pictures:

img_1.jpg img_2.jpg img_3.jpg

Links:

What's New In Docker?

Speaker: Vivek Saraswat (PM at Docker)

Excerpt: It’s the first breakout after the keynote and you need to know more about all the latest and greatest Docker announcements. We've got you covered! In this session, the Docker team will go deeper, looking into what's new with Docker, demoing the latest features and answering your questions.

Pictures:

img_4.jpg img_5.jpg img_6.jpg img_7.jpg img_8.jpg img_9.jpg img_10.jpg

Docker? But I'm a sysadmin!

Speaker: Mike Coleman (Technology Evangelist, Docker)

Excerpt: Your developers just walked into your cube and said: "Here's the new app, I built it with Docker, and it's ready to go live." What do you do next? In this session, we'll talk about what containers are and what they are not. And we'll step through a series of considerations that need to be examined when deploying containerized workloads - VMs or Container? Bare Metal or Cloud? What about capacity planning? Security? Disaster Recovery? How do I even get started?

Notes:

  • Where to run containers? Cloud or on-prem? Bare metal or VM?
    • Depends on your app
    • Depends on your skillset, your infra, your requirements...
    • Performance is better on bare metal, VMs are more manageable
  • VMs are like houses
    • Seperate plumbing, electricity, ...
    • When I want a new room, I need to build a new foundation, with new plumbing, ...
  • Containers are like appartments
    • Share common plumbing
    • When I want a new container, I knock down a wall and hook into the existing plumbing
  • This means you can't treat VMs and containers the same!!! (cfr: monitoring)
  • Plan for higher density
    • Use bare metal
    • Use big VMs
    • More than 1 container per docker host
    • tune to optimize

Cloud Native Storage Patterns

Speaker: Alex Chircop

Excerpt: There is no such thing as a stateless architecture – containerized applications need to store data and state somewhere. We discuss what a storage platform should look like in cloud native architectures and what is needed to interface composable microservices with advanced containerized storage patterns. This session will also include a demo demonstrating how Docker Volume Plugins can manage a cluster wide storage pool and provide highly available persistent storage volumes to databases and applications. 3 things you will learn: 1. Move your databases, message buses and instrumentation to persistent storage volumes 2. Take advantage of advanced containerized storage patterns to improve high availability and security 3. Understand how to use Docker Volume Plugins in your environment

Notes:

cloud native?

  • horizontally scalable
  • no spof
  • resilient and survivable
  • minimal operator overhead (automation)
  • decoupled from udnerlying platform
  • app centric
  • platform agnostic (mix of # hardware, providers)
  • composable
  • api driven
  • performant
  • consistently available (netapp much?)
  • natively secure

StorageOS docker plugin: not that interesting for us...

Links:

Creating Effective Images

Speaker: Abby Fuller (Sr Technical Evangelist, AWS)

Excerpt: Sick of getting paged at 2am and wondering "where did all my disk space go?" This has actually happened to me, and you can learn from my mistakes! New Docker users often start with a stock image in order to get up and running quickly, but that isn't always the right answer. Creating efficient images is often overlooked, but important. Beyond saving resources, using minimal images also delivers important security benefits: include only what you need and not a whole runtime that might have security vulnerabilities. In this session, I'll talk about how to create effective images and lessons I've learned from running containers in production at a number of startups. I'll also cover topics like "how do layers work?" and some things you should think about when creating your images, such as; choosing or creating the right base image; the importance of caching; using RUN statements conservatively; cleaning up as you go. I'll also address best practices; both at a high level with multi-stage builds; and some language-specific best practices, for example, tips and tricks for creating containers for Node.js vs Go. To illustrate these points, we'll cover: * How layers work? * Choosing a base image vs. creating your own * The basics of building minimal images and the importance of choosing a base image vs. creating your own * The basics of building minimal images and the importance of caching * High level best practices for Linux containers (in general, and some language specific examples) * High level best practices for Windows container images * New and improved: multi-stage builds * Good vs. not so good Dockerfile examples * Docker Image Scanning, and other friends * What's up next? Looking to the future for more optimization.

Notes:

  • How do layers work?
    • Base image (stack of read-only container layers), can't change those
    • on top of that: thin read-write layer (data that I add when I run a container
  • Why do I care about layers?
    • more layers: larger images, larger attack surface, longer build/push/pull/...
  • how to reduce layers/size?
    • use shared base images where possible
    • chain RUN statements
    • prevent cache misses at build for as long as possible
      • stuff that stays the same: move it on the top (maintainer, label, ...)
      • add what changes at the bottom (RUN, ...)
      • rtfm the cache documentation
    • limit the data written to the container image
    • KISS: less layers is better, less stuff is better, only add what you need instead of what you might be needing in the future
  • some examples:
    • choose right base image: ubuntu (130mb) vs alpine (3mb) or build your own
    • only pick a full base os: for security, compliance, ease of development
    • move 'COPY|ADD' statements up since those files tend to stay te same -> leverage cache (sometimes also RUN)
    • chain RUN statements together: RUN apt-get update -y && apt-get install -y foobar -no-install-recommends
    • remove the contents o f/var/lib/apt/lists/* (or the yum cache)
    • switching USER adds layers (so group stuff together, or don't do it)
    • don't ADD large files, RUN curl is better
    • use two images: one to build/make (with all the fat make/openjdk/build deps) and one to run the actual app (just the compiled app, and openjre, minimal deps)
    • Java: multi-stage builds are your friend
      • FROM maven3.5-jdk-8 as BUILD
      • COPY --from=BUILD
    • mutli-stage jobs: build in stage 1, use the binary in stage 2
    • document all the steps!
  • cleaning up:
    • docker image prune -a (removes dangling images, that are not being used anymore)
    • docker system prune -a (will get rid of all unused images and volumes)
    • make sure k8s is garbage collecting !!! (there was a link in the slides, not sure where it is)
    • spotify-gc
  • cool new stuff
    • official images are multi-platform
    • new and better minimal images: rancherOS, alpine, ...

Making Monitoring a Cloud Native Platform Feature

Speaker: Alois Mayr (Tech Led, Dynatrace)

Excerpt: With the transformation towards Cloud Native platforms, we are seeing a paradigm shift in how self-sufficient teams develop and deploy applications. Microservice teams leverage built-in platform features to run, scale, rollback and upgrade their app deployments. Monitoring helps them to understand if their apps are performing properly and interacting correctly with other apps in production. But how do they know if poor cluster performance impacts application health, and how the overall performance of all apps deployed to a cluster look? In this session, Alois Mayr will explore how monitoring can be made a platform feature to help application teams monitor and maintain application and cluster health. Alois will explain why automatic discovery of dependencies across cluster nodes, containers and services is critical to pinpoint the root cause of degraded application response times and failure rates.

Notes:

  • container health != service health != application health
  • chaos monkey! containers dying should not result in app dying
  • pometheus/fluentd/... tools to monitor containers/microservices
  • container health: cpu/mem/disk => not very interesting
  • service health: starting to get interesting
  • app health: 500 return codes, page load times, javascript errors, where the actual value lies

October 18: Day two

Keynote

Speaker: Several people from Docker, IBM, ...

Notes: Big reveal of the day: IBM is joining the Docker MTA (Migrate Traditional Applications) programme as partner. Today's focus during the keynote was how MTA could help businesses save on costs by moving traditional applications to a container. This MTA program consists of Docker and/or it's partners to help a business by:

  • Selecting an application
  • Moving it into a container (creating a Dockerfile for it)
  • Helping to put this container in production (either on-prem or in one of the partner's clouds)
  • Doing all this within 5 days

Links: https://blog.scottlowe.org/2017/10/18/dockercon-eu-2017-day-2-keynote/

Tips and Tricks of the Docker Captains

Speaker: Adrian Mouat - Chief Scientist, Container Solutions

Excerpt: Docker Captain Adrian Mouat will present a grab bag of tips and tricks for getting the most out of Docker. These tips are aimed at avoiding common pitfalls, addressing common misunderstandings and making common operations easier. Topics covered will include: - Build Processes - Security - Volumes - Databases - Debugging and Maintenance - Calling Docker from Docker Whilst aimed primarily at new and intermediate users, even advanced users should pick up some new information. This talk will make your daily life with Docker easier!

Notes:

  • Daily Development
    • configure docker ps output (or docker container ls)
      • docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"
      • make it permanent by adding it to config.json (~/.docker/config.json)
        • "psFormat": "..."
    • file mounting gotcha
      • if you mount a single file as a volume, might now work as expected
      • if you then update it on the host, it is not changed in the container (volumes are mounted at the inode level) (if you edit a file, new inode you get)
      • solution: mount the dir, not the file | overwrite with 'echo "content" > $file'
    • cleaning up
      • delete dangling images => docker image prune
      • delete stopped containers => docker container prune
      • delete unused volumes => docker volume prune
      • delete unused networks => docker network prune
      • want it all? docker system prune
  • Building Images
    • See 'Effective Images' by Abby Fuller
    • docker build -t myimage .
      • that '.' is tarballed and sent to the docker daemon
      • don't run a build from ~/ or Downloads
      • use .dockerignore file to exclude large directories/files (aka: Jenkinsfile/Makefile)
    • beware of latest!
      • not guaranteeed to be "new"
      • not guaranteed to exist
    • use meaningful tags
      • semver
      • git hash
      • make it obvious what's running in prod!
    • use labels!!
      • docker build --label org.opencontainers.image.created="$(date --rfs-3339=s)" -t myimage .
      • docker inspect -f "{{json .ContainerConfig.Labels}} myimage
  • Container Lifecycle
    • do not require containers to start in a certain sequence
      • it should wait for that service (is there a db? no? wait 5 sec, rince repeat)
      • do this in app code or in a start-up script if you can't
      • see "12 fractured apps"
    • shutdown gracefully
      • docker stop => SIGTERM => wait 10 sec => SIGKILL
      • close network connectections, empty caches, flush to disk, write final logs
      • to make sure you receive the SIGTERM => run as PID 1 (or forward signals to your app) (see tini)
    • healtchecks
      • used by docker to see if container is healthy
      • HEALTHCHECK --interval=10s --timeout=3s \
      • CMD curl -f http://localhost/ || exit 1
      • docker knows it's running, but not if the actual app is healthy
      • docker swarm will only route to healthy containers
      • note that the command runs in the container (not on the host)
      • extra dependency, curl can be limiting, consider writing a bespoke tool
  • Security
    • use read only filesystem
      • docker run .... --read-only
      • poke holes using volumes
    • don't run as root
      • set a USER in Dockerfile
      • or use the nobody user
      • what if we really need root?
  • Other Stuff
    • Docker-in-docker?
      • Bad idea!
      • Issues with filesystems
      • Also caching, image stores
      • Jerome Petazzoni 'Do Not Use Dind For CI' https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/
      • Instead, mount the docker socket
        • less isolation :(
      • True DinD
        • use docker:dind image
        • docker run --link dind:docker docker docker run -d nginx
        • docker run --rm --link dind:docker docker docker ps

Monitoring Containers: Follow the Data

Speaker: Ilan Rabinovitch - Prodcut Director at Datadog

Excerpt: At Datadog we help thousands of organizations monitor their infrastructure and applications. In this session, we’ll dive deeper into the several hundred trillion data points we’ve gathered to extract information about the real-world use of containers and see trends in container use.

As we look at container use, we’ll also discuss the top applications being used in containers and, using the data, provide insight into which metrics you should watch and how to troubleshoot based on those metrics. In this session, we will also look at a framework for your metrics and how to use it to find solutions to the issues that come up.

We will cover the three types of monitoring data; what to collect; what should trigger an alert (avoiding an alert storm and pager fatigue); and how to follow the resources to find the root causes of problems.

Although the real-world container use data is derived from Datadog users, the focus of this session is not tool specific, so attendees will leave with strategies and frameworks they can implement in their container-based environments today regardless of the platforms and tools they use.

Notes:

  • Datadog: OSS Agent, Time series database (datadog is a SaaS though)
  • Monitoring in container world: care less about infra, care more about applications
  • 3 types of metrics:
    • work metrics: throughput, success, error, performance (very important, wake someone up!)
    • resource metrics: utilization, saturation, error, availability (less important, deal with it next business day)
    • events: code changes, alerts, scaling events, etc (cool to know, and correlate data with each other)
  • average: ~7 containers on the same host (more with kubernetes it seems, ci/cd even more)
  • average: container age ~2.5 vs VMs 23 days
  • ochestration adoption is rising faster than container adoption
  • docker adoption is rising faster in big enterprises than in small ones

Run your Docker apps in Production on Google Cloud with Kubernetes

Speaker: Mete Atamel - Developer Advocate, Google

Excerpt: Docker has fundamentally changed the way people run applications. Kubernetes offers rich primitives for deploying and managing distributed, containerized apps. It helps you reach new levels of availability and utilization, while lowering your ops burden. In this talk we'll explore some of the concepts in Kubernetes and take a look at how Kubernetes 1.6 advances the efforts of cloud native computing

Slides: https://drive.google.com/file/d/0B6j6Te0viCnHMjlsTEV5Y0ZCckE/view

Notes:

  • compute engine: vm
  • container engine: k8s as a service
  • app engine: give dockerfile, autoscale, traffic splitting over # versions
  • why google? container lives somewhere in google datacenter, when someone accesses it from around the world, connects to google access point, traffic routed over fast google network
  • k8s
    • containers are not enough
      • scale up and down?
      • redundancy ?
      • resiliency, sheduling, loadbalancing?
      • upgrading to new version? rolling upgrades?
      • config & secrets
    • enter k8s
      • pods/nodes/the usual k8s suspects
      • service: loadbalancer for pods to expose them to the work
  • gke (google container engine)
    • how we run k8s in google cloud
    • k8s as a service
  • google container registry
    • private docker registry hosted by google
  • google cloud container builder
    • docker builder in the sky, coupled with google container registry

Links: https://cloud.google.com/container-engine/ (free trial)

Troubleshooting Tips from a Docker Support Engineer

Speaker: Ada Mancini - Support Engineer, Docker

Excerpt: Docker makes everything easier. But even with the easiest platforms, sometimes you run into problems. In this session, you'll learn first-hand from someone whose job is helping customers fix these problems. Using Docker Swarm and Universal Control Plane, you can keep your apps running smoothly with minimal downtime. In this session, you'll learn how to: Ask better questions & identify the real problems you want to solve Use command line tools to inspect internal Swarm components Debug and troubleshoot interactions between Enterprise Edition components Apply these skills to troubleshooting Docker CE and EE

Notes:

  • Troubleshooting Basics
    • ask better questions
      • x/y problem (is asking about your attempted solution rather than your actual problem)
    • look at the logs
      • journald/json-file
      • docker logs | docker service logs
      • symlink stdout/stderr to where the app normally logs its files
      • create volume and mount it where the app writes logs, then use log collector on the host to do the research
  • Methods & tools used in testing hypothesis
    • netshoot container githob.com/nocolaka/netshoot
      • networking troubleshooting swiss army container
      • run it on a container network, or on the host network
      • more interesting for swarm it seems
    • linux cli toolbox
      • the usual suspects tcpdump/netcat/curl/jq/grep/awk/iptables/route/ip/socat/ipvsadm/...
      • docker api speaks http, use curl!
      • docker api returns json, use jq
    • If all else fails: Use the Source Luke!!!
      • search over the github repo/user/org docker/moby
  • How to get help:
    • more info is always better!
    • don't do "my dockerfile dont work"
    • better: "my dockerfile that looks like "FROM ubuntu" fails at 'RUN command'
    • best: "my dockerfile <> fails to build when it reaches 'RUN command', and this is the error message <>"
  • where to get docker help?

Links:

Kubernetes in Docker (known as Gordon's Secret Session)

Speakers: Alex Mavrogiannis, Daniel Hiltgen - Docker EE Engineering

Excerpt: What's this session about? Kubernetes on Docker!

Pictures:

img_11.jpg

Notes:

  • pause container is used to retain namespaces/cgroups
  • General CE/EE Architecture
    • see picture
    • win/mac: no swarm/k8s by default, spin it up when needed
    • docker swarm: imperative (do this once) vs k8s: get to this state
    • docker engine with swarm-mode enabled: desired state like k8s