Recollection

Test determinism

There were garbage collection tests that were automatically added to every subclass of test.helper.CPWebCase. They are non-deterministic and fail sporadically (there are other normal tests that have non-deterministic behaviour, see below). Because CPython’s reference counting garbage collector is deterministic, it’s a concurrency issue. What was really surprising for me hear was explanation like, okay, if it has failed run it once again until you’re sure that the failure persists. This test_gc supplementing is disabled now until everything else is resolved, and it is clear how to handle it properly.

Another thing that prevented certain classification to pass and fail and led to frequent freezes were request retries and absence of HTTP client’s socket timeout. I can only guess about the original reasons of such “carefulness” and not letting a test just fail. For instance, you can take some of concurrency tests that throw out up to a hundred client threads and imagine stability and predictability of the undertaking with 10 retires and no client timeout. These two were also fixed — test.webtest.openURL doesn’t retry on socket errors and sets 10-second timeout to its HTTP connections by default.

test.helper.CPWebCase starts full in-process CherryPy server and then uses real httplib.HTTPConnection to connect to it. Most of CherryPy tests are in fact either integration [3] or system tests [4].

Test dependency

How do you like tests named like test_0_check_something, test_1_check_something_else, and so on? And what if they were named this way intentionally, so they run in order, and fail otherwise because latter depend on former. Specifically, you can not run test individually. All occurrence I’ve found were renamed and freed from sibling shackles.

Test isolation

test.helper.CPWebCase‘s server startup routine was made in equivalent of unittest‘s setUpClass, so tests in a test case were not isolated from each other, which made it problematic to deal with the case of tests that interfere with internals as it was really hard to say whether such test has failed on its own or as a consequence. I’ve added per_test_setup attribute to CPWebCase to conditionally do the routine in setUp and accommodated tests’ code. I apprehended significant performance degradation, but it has just become around 20% slower, which was the reason for me to make it true by default. Not sure, maybe it should’t exist at all now.

Unfortunately, this doesn’t resolve all isolation issues because having to do a complete reset of in-process threaded server is a juggling whatsoever. When a tricky test fails badly, it still has a side-effect (see below), like this:

./cherrypy/process/wspbus.py:233: RuntimeWarning: The main thread is exiting, but the Bus is
in the states.STARTING state; shutting it down automatically now. You must either call
bus.block() after start(), or call bus.exit() before the main thread exits.

Testing progress

What has just been said about fixed and improved stuff is happening in my fork [5], it is not yet pushed upstream. More detailed description and discussion about current and above sections is in the thread in CherryPy user group [6]. Because it’s there until the moment Google decides to shutdown Google Groups because of low traffic, abuse or any other reason they use for such an announcement, I will summarise the achievements in improving CherryPy testing infrastructure.

1. Migrated tests to stdlib unittest, unitest2 as fallback for py26. Nose is unfriendly to py3 and probably was related to some locking issues.
2. Overall improvement in avoiding freezes in various test cases and testing environments. TTY in Docker, disabled interactive mode, various internals’ tests wait in a thread, et cetera.
3. Implemented “tagged” Tox 1.8+ configuration for matrix of various environments (see below).
4. Integrated with Drone.io [7] CI service and Codecov.io [8] code coverage tracking service.
5. Made various changes to allow parallel environment run with Detox to fit in 15 minutes of Drone.io free tier. It includes running tests in install directory [9], starting server each time on free port provided by OS, locking Memcached cases with an atomic operation and other.
6. Removed global and persistent configuration that prevented mixing HTTP and HTTPS cases.
7. Made it possible to work on the test suite in PyDev. When running tests in PyDev test runner, it adds another non-daemonic thread that tests don’t expect which leads to deadlocks or fails for 3 tests. They are now just skipped under PyDev runner.

Here’s Drone.io commands required to run the test suite. It uses Deadsnakes [10] to install old or not yet stable version of Python. Development versions are needed to build pyOpenSSL. The rest comes with Drone.io container out of the box.

echo 'debconf debconf/frontend select noninteractive' | sudo debconf-set-selections
sudo add-apt-repository ppa:fkrull/deadsnakes &> /dev/null
sudo apt-get update &> /dev/null
sudo apt-get -y install python2.6 python3.4 &> /dev/null
sudo apt-get -y install python2.6-dev python3.4-dev &> /dev/null

sudo pip install --quiet detox
detox
tox -e post

As you can see post environment closes the build. Unless the build was successful it won’t run and no coverage will be submitted or build artifacts become available. The build artifacts, what also needs to be listed in Drone.io, are several quality assurance reports that can be helpful for further improving the codebase:

coverage_report.txt
coverage_report.html.tgz
maintenance_index.txt
code_complexity.txt

For Codecov.io integration to work, environment variable CODECOV_TOKEN should be assigned.

Tagged Tox configuration

Final Tox configuration looks like the following. It emerged through series of rewrites that were led by tradeoffs between test duration, combined coverage report and need to test dependencies. I came to the last design after I realised importance of support of pyOpenSSL and thus the need to test it (see about SSL issues below). Then test sampling facility was easy to integrate to it.

[tox]
minversion = 1.8
envlist    = pre,docs,py{26-co,27-qa,33-nt,34-qa}{,-ssl,-ossl}

# run tests from install dir, not checkout dir
# http://tox.rtfd.org/en/latest/example/pytest.html#known-issues-and-limitations
[testenv]
changedir = {envsitepackagesdir}/cherrypy
setenv    =
    qa:   COVERAGE_FILE = {toxinidir}/.coverage.{envname}
    ssl:  CHERRYPY_TEST_SSL_MODULE = builtin
    ossl: CHERRYPY_TEST_SSL_MODULE = pyopenssl
    sa:   CHERRYPY_TEST_XML_REPORT_DIR = {toxinidir}/test-xml-data
deps =
    routes
    py26:      unittest2
    py{26,27}: python-memcached
    py{33,34}: python3-memcached
    qa:        coverage
    ossl:      pyopenssl
    sa:        unittest-xml-reporting
commands =
    python --version
    nt: python -m unittest {posargs: discover -v cherrypy.test}
    co: unit2 {posargs:discover -v cherrypy.test}
    qa: coverage run --branch --source="." --omit="t*/*" \
    qa: --module unittest {posargs: discover -v cherrypy.test}
    sa: python test/xmltestreport.py

[testenv:docs]
basepython = python2.7
changedir  = docs
commands   = sphinx-build -q -E -n -b html . build
deps       =
    sphinx < 1.3
    sphinx_rtd_theme

[testenv:pre]
changedir = {toxinidir}
deps      = coverage
commands  = coverage erase

# must be run separately after main envlist, because of detox
[testenv:post]
changedir = {toxinidir}
deps      =
    coverage
    codecov
    radon
commands  =
    bash -c 'echo -e "[paths]\nsource = \n  cherrypy" > .coveragerc'
    bash -c 'echo "  .tox/*/lib/*/site-packages/cherrypy" >> .coveragerc'
    coverage combine
    bash -c 'coverage report > coverage_report.txt'
    coverage html
    tar -czf coverage_report.html.tgz htmlcov
    - codecov
    bash -c 'radon mi -s -e "cherrypy/t*/*" cherrypy > maintenance_index.txt'
    bash -c 'radon cc -s -e "cherrypy/t*/*" cherrypy > code_complexity.txt'
whitelist_externals =
    /bin/tar
    /bin/bash

# parse and collect XML files of XML test runner
[testenv:report]
deps      =
changedir = {toxinidir}
commands  = python -m cherrypy.test.xmltestreport --parse

This design heavily relies on great multi-dimensional configuration that appeared in Tox 1.8 [11]. Given an environment, say py27-nt-ossl, Tox treats it like a set of tags, {'py27', 'nt', 'ossl'}, where each entry is matched individually. This way any environment, that makes sense though, with desired qualities can be constructed even if it isn’t listed in envlist. Here’s a table that explains the purpose of the tags.

Note {posargs: discover -v cherrypy.test} part. It makes possible to run a subset of the test suite in desired environments, e.g.:

tox -e "py{26-co-ssl,27-nt,33-qa,34-nt-ssl}" -- -v cherrypy.test.test_http \
  cherrypy.test.test_conn.TestLimitedRequestQueue \
  cherrypy.test.test_caching.TestCache.test_antistampede

Tackling entropy of threading and in-process server juggling

Even though there was a progress on making test suite more predictable, and actually runnable, it is still far from desired, and I was scratching my head trying to figure out how to gather test stats from all the environments. Solution was in the lighted place, right where I was looking for it. There’s a package unittest-xml-reporting, whose name is pretty descriptive. It’s a unittest runner that writes XML files likes TEST-test.test_auth_basic.TestBasicAuth-20150410005927.xml with easy-to-guess content. Then it was just a need to write some glue code for overnight runs and implement parsing and aggregation.

I’ve implemented and incorporated module named xmltestreport [12] into the test package. sa Tox tag corresponds to it. Running the following in terminal at evening, at morning can give you the following table (except N/A evironments, see below).

for i in {1..64}; do tox -e "py{26,27,33,34}-sa{,-ssl,-ossl}"; done
tox -e report

Actually, as you can see, it only had time for half of the iterations, and at the morning I interrupted it. Sylvain, you see, I mentioned those three tests in the group correctly. But there’re other candidates and this new arisen fragility of SSL sockets, if I understand it correctly. I mean all the tests with one fail or error. For example, test_session.TestSession.testFileConcurrency which is fine lock-wise but because it stresses networking layer with several dozen client threads, there’re two socket errors:

Exception in thread Thread-1098:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "py27-sa-ossl/lib/python2.7/site-packages/cherrypy/test/test_session.py", line 282, in request
    c.endheaders()
  File "/usr/lib64/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 805, in send
    self.sock.sendall(data)
  File "/usr/lib64/python2.7/ssl.py", line 293, in sendall
    v = self.send(data[count:])
  File "/usr/lib64/python2.7/ssl.py", line 262, in send
    v = self._sslobj.write(data)
error: [Errno 32] Broken pipe

Exception in thread Thread-1099:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "py27-sa-ossl/lib/python2.7/site-packages/cherrypy/test/test_session.py", line 282, in request
    c.endheaders()
  File "/usr/lib64/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 805, in send
    self.sock.sendall(data)
  File "/usr/lib64/python2.7/ssl.py", line 293, in sendall
    v = self.send(data[count:])
  File "/usr/lib64/python2.7/ssl.py", line 262, in send
    v = self._sslobj.write(data)
error: [Errno 104] Connection reset by peer

Other error details are available in XML reports which retain such details. You may think that I have found the reason for “carefulness” I questioned above, but I woulnd’t agree for several reasons. First, it looks like an SSL-only and also needs investigation — someone knows practical reasons for the SSL server and client to be more fragile? Second, if a test stresses server with a few dozens of client threads, regardless of SSL, it should be written in a way that tolerates failures for some of them rather than relying on magic recovery behind the scenes. And after all, all this integration testing is quite useless if we assume we’re fine having 9/10 of the client to fail in any test.

Status indication

Long story told short, I have changed my mind on status badges. After thinking a lot about organising project’s status indication and notification, where most important is regression notification when we have contributors who used to commit without running tests, I came to conclusion that status badges can do the job.

This way every contributor will know that an introduced regression will become a public knowledge in a matter of minutes. And that it won’t make other members of the community happy about it. Hopefully, this as a form of enforcement of standards and new testing tools will help us go smoother.

Another thing that made me thinking about status indication was March Google Code shutdown announcement [13]. For my Open Source code I was mostly using Google Code’s Mercurial. It was stable and minimal environment without distraction. Although it wasn’t a substantially maintained and over time it had become lacking several features, however it still has several exclusive ones. Essentially it was a project hosting in contrast to Bitbucket and Github which are a repository hosting. One could have as many repositories inside a project as needed. I know it is also feasible with Mercurial named branches, but it still feels a little denser than it should be. Another handy thing was that Google served raw content with appropriate MIME, e.g. JavaScript with text/javascript, thus allowed right-off-the-repo deployment for various testing purposes.

For status indication I used external links, the block in left side bar on main project page. You could say here is PyPi page, here’s CI, and here’s test coverage tracking. Because it was visible, in 1-click distance and parallel to project description it was sufficient.

Drawing the line

As far as I see, testing infrastructure is ready. I also made several related changes here and there. Now I think it is a good time to put it into upstream. First, of course for other contributors to make use of the new tools. Second, for the sake of granularity because there’s already a bunch of changes since January in my fork. And third, in coming months it is likely that I won’t have much time for contribution, thus it’s good time to discuss the changes if we need to.

Experiment

Recently there was a CherryPy StackOverflow question right about the topic [15]. The guy asked how to block SSL protocols in favour of TLS. It led me to coming to the following quick-and-dirty solution.

#!/usr/bin/env python
# -*- coding: utf-8 -*-


import os
import sys
import ssl

import cherrypy
from cherrypy.wsgiserver.ssl_builtin import BuiltinSSLAdapter
from cherrypy.wsgiserver.ssl_pyopenssl import pyOpenSSLAdapter

from cherrypy import wsgiserver
if sys.version_info < (3, 0):
  from cherrypy.wsgiserver.wsgiserver2 import ssl_adapters
else:
  from cherrypy.wsgiserver.wsgiserver3 import ssl_adapters

try:
  from OpenSSL import SSL
except ImportError:
  pass


ciphers = (
  'ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:'
  'DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:!aNULL:'
  '!eNULL:!MD5:!DSS:!RC4:!SSLv2'
)

bundle = os.path.join(os.path.dirname(cherrypy.__file__), 'test', 'test.pem')

config = {
  'global' : {
    'server.socket_host' : '127.0.0.1',
    'server.socket_port' : 8443,
    'server.thread_pool' : 8,

    'server.ssl_module'      : 'custom-pyopenssl',
    'server.ssl_certificate' : bundle,
    'server.ssl_private_key' : bundle,
  }
}


class BuiltinSsl(BuiltinSSLAdapter):
  '''Vulnerable, on py2 < 2.7.9, py3 < 3.3:
    * POODLE (SSLv3), adding ``!SSLv3`` to cipher list makes it very incompatible
    * can't disable TLS compression (CRIME)
    * supports Secure Client-Initiated Renegotiation (DOS)
    * no Forward Secrecy
  Also session caching doesn't work. Some tweaks are posslbe, but don't really
  change much. For example, it's possible to use ssl.PROTOCOL_TLSv1 instead of
  ssl.PROTOCOL_SSLv23 with little worse compatiblity.
  '''

  def wrap(self, sock):
    """Wrap and return the given socket, plus WSGI environ entries."""
    try:
      s = ssl.wrap_socket(
        sock,
        ciphers = ciphers, # the override is for this line
        do_handshake_on_connect = True,
        server_side = True,
        certfile = self.certificate,
        keyfile = self.private_key,
        ssl_version = ssl.PROTOCOL_SSLv23
      )
    except ssl.SSLError:
      e = sys.exc_info()[1]
      if e.errno == ssl.SSL_ERROR_EOF:
        # This is almost certainly due to the cherrypy engine
        # 'pinging' the socket to assert it's connectable;
        # the 'ping' isn't SSL.
        return None, {}
      elif e.errno == ssl.SSL_ERROR_SSL:
        if e.args[1].endswith('http request'):
          # The client is speaking HTTP to an HTTPS server.
          raise wsgiserver.NoSSLError
        elif e.args[1].endswith('unknown protocol'):
          # The client is speaking some non-HTTP protocol.
          # Drop the conn.
          return None, {}
      raise

    return s, self.get_environ(s)

ssl_adapters['custom-ssl'] = BuiltinSsl


class Pyopenssl(pyOpenSSLAdapter):
  '''Mostly fine, except:
    * Secure Client-Initiated Renegotiation
    * no Forward Secrecy, SSL.OP_SINGLE_DH_USE could have helped but it didn't
  '''

  def get_context(self):
    """Return an SSL.Context from self attributes."""
    c = SSL.Context(SSL.SSLv23_METHOD)

    # override:
    c.set_options(SSL.OP_NO_COMPRESSION | SSL.OP_SINGLE_DH_USE | SSL.OP_NO_SSLv2 | SSL.OP_NO_SSLv3)
    c.set_cipher_list(ciphers)

    c.use_privatekey_file(self.private_key)
    if self.certificate_chain:
        c.load_verify_locations(self.certificate_chain)
    c.use_certificate_file(self.certificate)
    return c

ssl_adapters['custom-pyopenssl'] = Pyopenssl


class App:

  @cherrypy.expose
  def index(self):
    return '<em>Is this secure?</em>'


if __name__ == '__main__':
  cherrypy.quickstart(App(), '/', config)

After I regenerated our test certificate to:

• use longer than 1024-bit key
• not use MD5 and SHA1 (also deprecated [26]) for signature
• use valid domain name for common name

Using the following versions, we have A- for the customised pyOpenSSL adapter on Qualys SSL Server Test [34]:

• Debian Wheezy (stable)
• Python 2.7.3-4+deb7u1, ssl.OPENSSL_VERSION == 'OpenSSL 1.0.1e 11 Feb 2013'
• OpenSSL 1.0.1e-2+deb7u16
• pyOpenSSL 0.14
• CherryPy from default branch (pre 3.7)

There’s no error, but the warnings:

• Secure Client-Initiated Renegotiation — Supported, DoS DANGER [16]
• Downgrade attack prevention — No, TLS_FALLBACK_SCSV not supported [17]
• Forward Secrecy — No, WEAK [18]

Here’s what Wikipedia says [25] about forward secrecy, to inform you that it is not something critical or required for immediate operation.

In cryptography, forward secrecy is a property of key-agreement protocols ensuring that a session key derived from a set of long-term keys cannot be compromised if one of the long-term keys is compromised in the future.

And here’s what can be squeezed out of built-in ssl, which is grievous.

Here goes the what-is-wrong list:

• POODLE (SSLv3) — Vulnerable, INSECURE [19]
• TLS compression — Yes, INSECURE [20]

There’s also half-implemented session resumption — No (IDs assigned but not accepted). And the same three warnings as for PyOpenSSL apply. ssl_builtin doesn’t handle underlying ssl module exceptions correctly, as the log was flooded by exceptions originating mostly from do_handshake(). PyOpenSSL’s log was the way more quiet with just a couple of page accesses.

[27/Mar/2015:14:07:49] ENGINE Error in HTTPServer.tick
Traceback (most recent call last):
  File "venv/local/lib/python2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 1968, in start
    self.tick()
  File "venv/local/lib/python2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 2035, in tick
    s, ssl_env = self.ssl_adapter.wrap(s)
  File "./test.py", line 53, in wrap
    ssl_version = ssl.PROTOCOL_SSLv23
  File "/usr/lib/python2.7/ssl.py", line 381, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib/python2.7/ssl.py", line 143, in __init__
    self.do_handshake()
  File "/usr/lib/python2.7/ssl.py", line 305, in do_handshake
    self._sslobj.do_handshake()

SSLError: [Errno 1] _ssl.c:504: error:1408A0C1:SSL routines:SSL3_GET_CLIENT_HELLO:no shared cipher
SSLError: [Errno 1] _ssl.c:504: error:14076102:SSL routines:SSL23_GET_CLIENT_HELLO:unsupported protocol
SSLError: [Errno 1] _ssl.c:504: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
SSLError: [Errno 1] _ssl.c:504: error:14094085:SSL routines:SSL3_READ_BYTES:ccs received early

But nothing is really that bad even with ssl_builtin. Let’s relax stability policy, make sure you understand the consequences if you’re going to do it on a production server, and add Debain Jessie’s (testing) source to /etc/apt/sources.list of the server (Debian Wheezy) and see what next Debain has in its briefcase:

• Python 2.7.9-1, ssl.OPENSSL_VERSION == 'OpenSSL 1.0.1k 8 Jan 2015'
• Python 3.4.2-2, ssl.OPENSSL_VERSION == 'OpenSSL 1.0.1k 8 Jan 2015'
• OpenSSL 1.0.1k-3
• pyOpenSSL 0.15.1 (installed via pip, probably has been updated since)

For you information, related excerpt from Python 2.7.9 release change log [21]. It was in fact a huge security update, not very well represented by the change of a patch version.

• The entirety of Python 3.4’s ssl module has been backported for Python 2.7.9. See PEP 466 [22] for justification.
• HTTPS certificate validation using the system’s certificate store is now enabled by default. See PEP 476 [23] for details.
• SSLv3 has been disabled by default in httplib and its reverse dependencies due to the POODLE attack.

All errors have gone, and in spite of the warning it supports forward secrecy for most clients. Here are just couple of warnings left:

• Secure Client-Initiated Renegotiation — Supported, DoS DANGER [16]
• Session resumption (caching) — No (IDs assigned but not accepted)

The result is pretty good. If you run it on plain ssl_builtin, is has B, because RC4 ciphers [24] aren’t excluded by default. Python 3.4.2 test on the customised ssl_builtin is obviously the same A- because both underlying libraries are the same. Customised ssl_pyopenssl on Python 2.7.9 gives same A- — again it has functional session resumption but doesn’t have forward secrecy for any client.

Problem

After the experiment has taken place, it is time to generalise the results. Here’s what is wrong with CherryPy’s SSL support now:

1. giving false sense of security
2. not flexible configuration: protocol, ciphers, options
3. not up to date with builit-in ssl (no SSL Context [27] support)
4. no support for pyOpenSSL for py3
5. no security assessment

Plan

I think that the problem is solvable with a reasonable effort. What has been said above may already have convinced you that no special cryptography knowledge is required. There’s a scalar letter grade for A to F and several design decisions aimed at flexibility and compatibility.

pip install -I pyOpenSSL

Nature

Basically Docker is like Vagrant + OpenVZ on vanilla kernel. The two are available for quite some time and have established use cases, reproducible development environment and lightweight, operating-system-level virtualisation [42], respectively. Although Docker combines the two qualities, and it’s tempting to (re)think that developers can build production-ready container images and deploy them through Docker magic powers, and that it’s what Docker is designated for, it would be a mistake to say so.

As Solomon says, Docker is mere a building block that comes in at the right time and provides high-level APIs for set of great low-level features modern Linux kernel has. There are several things I want to summarise from his latest interview given to FLOSS Weekly in episode 330 [43]:

• it is not a silver bullet, it is a building block
• it is used both ways: as VM and as “binary” (process-per-container); neither is the “true” one
• it is neither superior nor mutually exclusive with configuration management
• it is not a complete production solution; production is hard and is left to the reader
• it doesn’t primarily target homogenous applications; the platform’s tools may be sufficient

I would agree that except first one, one might find “however” to any of them. And especially, considering the pace of development, and promising trio of new experimental tools, that appeared to come along Docker for cluster orchestration, in Unix-way of loosely coupled components and responsibility separation: Machine, Swarm and Compose [44].

And I really like to think about Docker as a building block. No matter what the hype is, even though it entails imbalance in available information, there’re certain design decisions made that have certain implications. Taking them into account you decide whether Docker fits your certain task. For instance, there was a good comparison, LXC versus Docker, by Flockport [45], which emphasises the difference between the two as long as Docker goes with own libcontainer:

• performance penalty for a layered filesystem (e.g. AUFS)
• separate persistence management for immutable containers
• process-per-container design has issues with real world [46]:
  • most of existing software is written in expectation of init system
  • monitoring, cron jobs, system logging need a dedicated daemon and/or SSH daemon
  • system built of subsystems built of components is much harder to manage

As I said above both process-per-container and application-per-container both have merit. But because Docker originally was conceived with the former idea, most images in Docker Hub are process-per-container and people begin to fight the tool they have chosen by using Bash scripts, Monit or Supervisor to manage multiple processes.

Complete configuration management products, like Ansible, or even simpler Fabric-based approaches in conjunction with abundance of API for every single IaaS and PaaS provider made available infrastructure as code with any desired quality long before Docker appeared. Big companies were implementing their own infrastructure-as-code solutions on top of chosen service provider. Smaller companies mostly weren’t in the game. Docker, because of its momentum, can standardise service providers’ supply and popularise infrastructure as code in the industry which I see quite advantageous.

Another thing that shoundn’t fall out of your sight is that Docker is a new technology. It has known and significant risk for new security issues [47]. And even in its own realm it is not as nearly as perfect. Indeed Docker makes clearer responsibility separation between devs and ops. But there are cross-boundary configuration, like maximum open file handles or system networking settings which need to be set on host and have corresponding configuration in container, e.g. MySQL table cache. There are environment-specific configuration like load testing in QA and rate-limiting in production. There is a lot more of such things to consider in real system.

There is a lot of discussion of microservice architecture [48] today. One of its aspects is that what was inner-application complexity is now shifted into infrastructure. And the more you follow this route the more you immerse into distributed computing while distributed computing is still very hard. Docker guidelines go clearly along the route.

Lately I’ve seen a comprehensive and all around excellent presentation on distributed computing by Jonas Bonér [49]. Although it’s named “The Road to Akka Cluster and Beyond” the Akka, which is distributed application toolkit for JVM, part starts around 3/4 of the slides. The rest is overview, theoretical aspects and practical challenges of distributed computing.

Last thing to say in this section is about Docker’s competitors. There are obviously other fish in the sea. Besides already mentioned LXC [50], there are also newer container tools built on experience with existing tools, realising their shortcomings and addressing them. To name a few: Rocket [51], Vagga [52].

Application

With that being said I think thorough discussion of infrastructure and distributed application design within the scope of CherryPy project is not only a off-topic. It’s a hugely out of the context. However the topic itself is actual and interesting, but I’m sure everyone who is dealing or is willing to deal with an application at scale knows how to run pip install cherrypy.

What I already told to Sylvain, is on-topic and CherryPy QA can benefit from is Docker QA image with complete CherryPy test suite containing all environments and dependencies so every contributor can take it and run the test suite against introduced changes effortlessly. Here’s a dockerfile for the purpose.

FROM ubuntu:trusty

MAINTAINER CherryPy QA Team

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update
RUN apt-get --no-install-recommends -qy install python python-dev python3 python3-dev \
  python-pip python-virtualenv
RUN apt-get --no-install-recommends -qy install software-properties-common build-essential \
  libffi-dev mercurial ca-certificates memcached

RUN add-apt-repository ppa:fkrull/deadsnakes
RUN apt-get update
RUN apt-get --no-install-recommends -qy install python2.6 python2.6-dev python3.3 python3.3-dev

RUN pip install "tox < 2"
RUN pip install "detox < 0.10"

WORKDIR /root
RUN hg clone --branch default https://bitbucket.org/saaj/cherrypy

WORKDIR /root/cherrypy
RUN echo '#!/bin/bash\n \
  service memcached start; hg pull --branch default --update; tox "$@"' > serial.sh
RUN echo '#!/bin/bash\n \
  service memcached start; hg pull --branch default --update; detox "$@"' > parallel.sh
RUN chmod u+x serial.sh parallel.sh
ENTRYPOINT ["./serial.sh"]

hg clone line should be changed to the source of changes. Thus I think we should distribute it just as a dockerfile. Because of development dependencies it isn’t small — ~620 MB. I also made parallel Detox run possible, parallel.sh, but I think usually running an environment at a time is better to stress tests with more concurrency. What’s really nice about it is that it can be run just like a binary.

docker run -it cherrypy/qa -e "py{26-co-ssl,27-nt,33-qa,34-nt-ssl}" \
  -- -v cherrypy.test.test_http cherrypy.test.test_conn.TestLimitedRequestQueue