Recollection

Developer-friendly job search. The tooling

2018-04-15T00:00:00+02:00

It is quite interesting to me to see how software job market works in the Netherlands from an active candidate’s point of view. Unfortunately one great journey has almost ended and there is a firm incentive to find something at least as good. Having my current job through StackOverflow Jobs, I intended to keep using what proved to work, like marking the profile active, entering python -django -sysadmin into the search field, adding relevant filters and subscribing to the search. That, as such, of course works, but it looks it is not the most popular software talent recruitment service that Dutch companies use. And many companies are still quite happy to pay 20% of the gross year salary per new hire to the volume-oriented keyword-matching salesmen, colloquially referred as recruiters.

On the other side of price spectrum there are free (as a beer… on a degustation) services like Indeed and Glassdoor [1] where employers can place their vacancies. The two seem to me most relevant and complete, but because the latter also provides valuable insight into company state, salary range and interview process I will focus on it.

StackOverflow Jobs is in between. It’s definitely not free for employers. For our small company it costed around €2k for half-year access to active candidate database and one job posting. Most probably it’ll require a company to have a full-time HR employee to manage the interactions with candidates. But hiring 4-5 developers a year already justifies the investment, so Stack Exchange has growth potential here.

Glassdoor has poor search features, term-inclusion-only search and location, which makes it a chore to examine the proposals manually.

Contents

Problem
Storage
Retrieval
Glue
Listing relevant jobs
Wrap-up

Problem

StackOverflow Jobs search features include [2]:

tag inclusion and exclusion
full-text search on body, title and company
enumeration filters by contract type, industry and seniority
salary range
flag filters by remote option, visa and relocation sponsorship
posted time filtering

My job search criteria were the following:

Python (backend software development) job preferably in Amsterdam
The company develops own product, i.e. it’s not a consultancy
The job is available directly, i.e. it’s not an agency or any other kind of intermediary

At first glance it looks simple. What subset of the above list of feature is sufficient to cover these?

Location data should be good enough for equality comparison, but generally the country is small, very connected and geospatial search is not needed.
A Python job is a match against [python] minus some other tags (though only if carefully tagged), but Glassdoor doesn’t have tags and implementing NLP to recognise them was not feasible. Doing just Python will not only match all sorts of jobs with it in nice-to-have, but also match jobs that extensively use Python but are not about software development as such, e.g. devops, (data) science, test automation, screen scraping and probably others. Also don’t forget full-stack and junior/intern positions (no dedicated seniority field on Glassdoor either).
Solving consultancies and intermediaries should be possible with a blacklist, but needs full-text index on company name to mitigate dealing with optionally present suffixes like “B.V.” and “BV”, geographical suffixes and the like. Using NLP to break down jobs into direct and indirect would be an interesting research, but again it was not feasible.

Items 1 and 3 look clear. 2 needs more attention. It should also be possible to solve with smarter blacklisting. But let’s look at a sample of job titles, which match Python query:

Lead Data Scientist
DevOps Engineer (Linux, AWS)
Software Engineer Embedded Smart Systems (C/C++; C#; Python)
Agile Tester
Backend Software Engineer
Experienced Python/Django Developer in Zaandam (NL/ENG)
Java Developer
Junior Full Stack Engineer
Medior Front-end Developer
Bioinformatic Postdoctoral Fellow
C++ Software engineer Artificial intelligence DELFT 40k-70k
Portfolio Manager Quant Allocation
PhD Student “Modelling causal interactions between marine microbes”

Yes, all sorts of jobs as promised. The good thing is that indexing title and body separately can help a lot with blacklisting. Say, if the title includes Django it’s definitely a no-go, but if Django is mentioned in the body it depends and may be tolerable. The same way other terms, like Java or Embedded can be handled.

Storage

Thus on the level of storage/search it’s necessary to have:

primary key, original, if exposed, or hash of the whole record
title, full-text indexed
employer, full-text indexed
body, full-text indexed
JSON field or just blob for other properties
status, just integer for own convenience

What embedded database has full-text search support with document storage features and easy Python interface? The latter is about JSON-field support that PostgeSQL and MySQL have acquired. SQLite!

The version that I have from Ubuntu Xenial’s repository, at the time of writing libsqlite3-0 3.11.0-1ubuntu1, supports both:

$ echo "PRAGMA compile_options;" | sqlite3 | grep -E "FTS|JSON"
ENABLE_FTS3
ENABLE_FTS3_PARENTHESIS
ENABLE_FTS4
ENABLE_JSON1

And Python standard library sqlite3 [3] uses it! Now briefly about the extensions.

FTS3 and FTS4

The documentation says [4]:

FTS3 and FTS4 are SQLite virtual table modules that allow users to perform full-text searches on a set of documents… FTS3 and FTS4 are nearly identical… FTS4 is an enhancement to FTS3. FTS3 has been available since SQLite version 3.5.0 (2007-09-04). The enhancements for FTS4 were added with SQLite version 3.7.4 (2010-12-07).

A virtual table [5] in SQLite is:

…an object that is registered with an open SQLite database connection. From the perspective of an SQL statement, the virtual table object looks like any other table or view. But behind the scenes, queries and updates on a virtual table invoke callback methods of the virtual table object instead of reading and writing on the database file.

In case of FTS4 it looks like a view that behind the scenes operates 5 shadow tables [6] which are stored within the database file. And a note about ENABLE_FTS3_PARENTHESIS and query syntax:

New applications should also define the SQLITE_ENABLE_FTS3_PARENTHESIS macro to enable the enhanced query syntax [7].

It means that python -django won’t work. python AND NOT django (or python NOT django because AND is the implicit operator) should be used instead.

To better understand how full-text search is going to contribute to solving the problem, here is simplified code that represents the data structure [6], inverted index, that is used by the SQLite extension.

from collections import defaultdict


docs = [
  (100, 'alpha beta beta gamma gamma gamma'),
  (101, 'gamma delta'),
]
invertedIndex = defaultdict(list)
for docId, text in docs:
  docIndex = defaultdict(list)
  for offset, term in enumerate(text.lower().split()):
    docIndex[term].append(offset)
  for term, offsetList in docIndex.items():
    invertedIndex[term].append((docId, offsetList))

Result invertedIndex looks like:

{
  'alpha' : [(100, [0])],
  'beta'  : [(100, [1, 2])],
  'delta' : [(101, [1])],
  'gamma' : [(100, [3, 4, 5]), (101, [0])]
}

Using this structure FTS3 and FTS4 serve the queries. str.lower and str.split in the snippet represent tokenisation of the text. Here no stemming is needed because most of technical terms are present intact, so I’ll use unicode61 for the virtual table.

The “unicode61” tokeniser is available beginning with SQLite version 3.7.13 (2012-06-11). Unicode61 works very much like “simple” except that it does simple unicode case folding according to rules in Unicode Version 6.1 and it recognises unicode space and punctuation characters and uses those to separate tokens. The simple tokeniser only does case folding of ASCII characters and only recognises ASCII space and punctuation characters as token separators… By default, “unicode61” also removes all diacritics from Latin script characters.

One thing to point out is that “C++” and “C#” will be indistinguishable from “C” because non-alphanumeric characters will be treated as separators. SQLite doesn’t limit minimal term length or have built-in stop-word list. So to solve this tokenchars is enough:

…”tokenchars=” option may be used to specify one or more extra characters that should be treated as part of tokens instead of as separator characters.

I also want to exclude “.NET” and adding “.” to the tokenchars will cause issue with words on the end of a sentence, but it’s negligible for the use case. Alternatively LIKE can be safely used as a post-filter condition, or generally it’s possible to implement tokenisers in Python with sqlitefts [12].

JSON1

The documentation says [8]:

The json1 extension is a loadable extension that implements fifteen application-defined SQL functions and two table-valued functions that are useful for managing JSON content stored in an SQLite database… The json1 extension (currently) stores JSON as ordinary text… The json1 extension uses the sqlite3_value_subtype() and sqlite3_result_subtype() interfaces that were introduced with SQLite version 3.9.0 (2015-10-14).

Out of the 15 function I’ll use only json_extract [9]. Even though on storage level of JSON1 JSON values are just string blobs, on logical level given set of function extends relation model [10] of SQLite with document storage model [11]. This provides great flexibility in data modelling, where one can take the best out of the two models, and still use declarative query language, SQL.

Implementation

import json
import sqlite3


class JobStorage:

  _conn = None


  def __init__(self):
    self._conn = sqlite3.connect('db.sqlite', isolation_level = None)
    self._setup()

  def _setup(self):
    sql = '''
      CREATE VIRTUAL TABLE IF NOT EXISTS job USING fts4(
        properties, status, employer, title, body,
        notindexed=properties, notindexed=status,
        tokenize=unicode61 "tokenchars=.+#")
    '''
    self._conn.execute(sql)

  def save(self, job):
    sql = '''
      INSERT OR IGNORE INTO job(docid, properties, status, employer, title, body)
      VALUES(:docid, :properties, :status, :employer, :title, :body)
    '''

    title = job.pop('title')
    body = job.pop('body')
    self._conn.execute(sql, {
      'docid'      : job['dataset']['id'],
      'employer'   : job['employer'],
      'properties' : json.dumps(job, default = str),
      'status'     : 1,
      'title'      : title,
      'body'       : body
    })

isolation_level = None re-enables auto-commit (for simplicity, and generally to let transactions be an explicit measure, because explicit is better than implicit). job['dataset'] will be described in the next section.

Retrieval

Data retrieval in context of the web or how it’s called in less neutral terms — screen scraping — is usually implemented with HTTP client using a HTML parser that is capable of either XPath queries or CSS selectors. This time I wanted to try something new and the time was right. These days we see a new wave of headless browsers [13]. I’m not talking about PhantomJS or the like, but rather ubiquitous desktop browsers that have gained official automation APIs. For instance, Chromium since version 59 supports DevTools Protocol (a.k.a. Remote Debugging Protocol) [14] in Linux.

DevTools Protocol is a WebSocket-based protocol and Google maintains an official client for it, puppeteer [15], written in JavaScript. What’s better that there’s a port to Python 3.6+. It’s called pyppeteer [16].

Pyppeteer

Most things that you can do manually in the browser can be done using Puppeteer!

As it should be with Pyppeteer. Basically the protocol provides WebSocket-based RPC to BOM, DOM and user objects, and the implementation provides more-or-less transparent proxy object API. In case of Pyppeteer it’s asyncio API.

Here it a demonstration of the proxy behaviour. It’ll use pyppeteer.element_handle.ElementHandle [18] that represents DOM element. Here’s a JavaScript that one can run in Chromium’s Developer Tools.

var a = document.querySelector('div div.flexbox div a');
var hrefProperty = a.href;
var href = hrefProperty.valueOf();
var titleProperty = a.textContent;
var title = titleProperty.valueOf();
console.log(href, title);

And the same thing with Pyppeteer:

a = await page.querySelector('div div.flexbox div a')
hrefProperty = await a.getProperty('href')
href = await hrefProperty.jsonValue()
titleProperty = await a.getProperty('textContent')
title = await titleProperty.jsonValue()
print(href, title)

There’re two must-know things about Pyppeteer. The first is that every attribute and value access needs to be await‘ed. The second is that pyppeteer.page.Page.click and pyppeteer.page.Page.waitForNavigation have a race condition and should be awaited simultaneously [19], like:

await asyncio.gather(page.click('#HeroSearchButton'), page.waitForNavigation())

Towards the implementation, the idea for job retrieval API is to have an asynchronous iterator object implemented as asynchronous generator (one that mixes yield and await) as defined in PEP 525 [17]. It will encapsulate browser preparation, pagination, addtional page interactions, throttling sleeps and at the same time in the calling code there will be just an async for loop. I’ll be using:

Python 3.6.5
Pyppeteer 0.0.16
Chromium 64.0.3282.167

Iteration

import asyncio
import logging


class JobIter:

  _page = None

  _query    = None
  _location = None
  _days     = None

  _prepared = False


  def __init__(self, page, query, location, days):
    self._page = page

    self._query    = query
    self._location = location
    self._days     = days

  async def _prepare(self):
    logging.debug('Opening main page')
    await self._page.goto('https://www.glassdoor.nl/index.htm')

    logging.debug('Entering query')
    await self._page.type('#KeywordSearch', self._query)
    await self._page.evaluate(
      'function(id, loc) {document.getElementById(id).value = loc}',
      'LocationSearch', self._location)

    logging.debug('Opening result page')
    await asyncio.gather(
      self._page.click('#HeroSearchButton'),
      self._page.waitForNavigation()
    )

    logging.debug('Selecting posted date range')
    datePosted = '#DKFilters > div > div > div:nth-child(2)'
    await self._page.click(datePosted)

    daysAgo = '#DKFilters .filter.expanded ul li[value="{}"]'.format(self._days)
    await self._page.waitForSelector(daysAgo)
    await self._page.click(daysAgo)

    logging.debug('Waiting for list to update')
    await self._page.waitForSelector('.jlGrid.updating')
    await self._page.waitForSelector('.jlGrid.updating', hidden = True)

  async def _dismiss(self):
    button = await self._page.querySelector('button.mfp-close')
    if button:
        await button.click()

  async def __aiter__(self):
    if not self._prepared:
      await self._prepare()
      self._prepared = True

    while True:
      await self._dismiss()

      button = await self._page.querySelector('#FooterPageNav .page.current')
      p = await button.getProperty('textContent')
      current = (await p.jsonValue()).strip()
      logging.debug('Processing page %s', current)

      jobList = await self._page.querySelectorAll('ul.jlGrid > li')
      for j in jobList:
        parser = JobParser(j, self._page)
        job = await parser.parse()
        logging.debug('Processed job %s', job['title'])
        yield job
        await asyncio.sleep(0.5)

      button = await self._page.querySelector('#FooterPageNav .next')
      disabled = await button.querySelector('.disabled')
      if disabled:
        logging.debug('Last page has been reached')
        return

      await asyncio.gather(button.click(), self._page.waitForNavigation())
      await asyncio.sleep(2)

Parsing

Root element of a job listing is HTML list item tag which looks like:

<li class="jl selected"
  data-id="2618878823" data-emp-id="262109" data-is-organic-job="false"
  data-sgoc-id="-1" data-purchase-ad-order-id="0" data-is-easy-apply="false"
  data-njslv="false"/>

There’re several data attributes, notably id:

id is internal Glassdoor identifier of a job which will help deduplicate jobs on subsequent runs,
empId is an employer identifier, it’s is somewhat useful,
isOrganicJob, initially I thought it the flag to filter out intermediaries, but unfortunately I didn’t grasp how exactly it is set.

The attributes can be accessed as a dictionary under dataset property, see _getDataset below.

import datetime
from decimal import Decimal


class JobParser:

  _elementHandle = None
  _page = None


  def __init__(self, eh, page):
    self._elementHandle = eh
    self._page = page

  async def _getDataset(self):
    p = await self._elementHandle.getProperty('dataset')
    d = await p.getProperties()
    dataset = {}
    for k, jsh in d.items():
        dataset[k] = json.loads(await jsh.jsonValue())
    return dataset

  async def _getRating(self):
    span = await self._elementHandle.querySelector('.compactStars')
    if span:
      p = await span.getProperty('textContent')
      return Decimal((await p.jsonValue()).replace(',', '.'))

  async def _getTitleLink(self):
    a = await self._elementHandle.querySelector('div div.flexbox div a')
    p = await a.getProperty('href')
    href = await p.jsonValue()
    p = await a.getProperty('textContent')
    title = (await p.jsonValue()).strip()

    return title, href

  async def _getEmployer(self):
    div = await self._elementHandle.querySelector('.empLoc div:nth-child(1)')
    firstNode = await div.getProperty('firstChild')
    p = await firstNode.getProperty('nodeValue')
    return (await p.jsonValue()).strip().strip('–').strip()

  async def _getCity(self):
    span = await self._elementHandle.querySelector('.subtle.loc')
    p = await span.getProperty('textContent')
    return (await p.jsonValue()).strip()

  async def _getDate(self):
    div = await self._elementHandle.querySelector('.hotListing')
    ts = datetime.datetime.now()
    if div:
      return ts

    span = await self._elementHandle.querySelector('.showHH span')
    p = await span.getProperty('textContent')
    value, unit = (await p.jsonValue()).strip().split(maxsplit = 1)
    value = int(value)
    if unit == 'd':
      ts -= datetime.timedelta(days = value)
    elif unit == 'u':
      ts -= datetime.timedelta(hours = value)
    elif unit == 'min':
      ts -= datetime.timedelta(minutes = value)
    else:
      raise ValueError('Unsupported unit {}'.format(unit))

    return ts

  async def _getBody(self):
    p = await self._elementHandle.getProperty('classList')
    await self._elementHandle.click()
    spinner = await self._page.querySelector('.jobDetails .ajaxSpinner')
    if spinner:
      spinner = await self._page.waitForSelector('.jobDetails .ajaxSpinner', hidden = True)

    div = await self._page.querySelector('.jobDetails .jobDescriptionContent')
    p = await div.getProperty('textContent')
    return (await p.jsonValue()).strip()

  async def parse(self):
    title, url = await self._getTitleLink()
    dataset, rating, employer, city, date = await asyncio.gather(
      self._getDataset(), self._getRating(), self._getEmployer(), self._getCity(), self._getDate())
    body = await self._getBody()

    return {
      'dataset'  : dataset,
      'rating'   : rating,
      'url'      : url,
      'employer' : employer,
      'city'     : city,
      'date'     : date,
      'title'    : title,
      'body'     : body
    }

Glue

By default on first run Pyppeteer downloads Chromium build that is guaranteed to work. But because I already had compatible version pre-installed I explicitly provided executablePath which disables the behaviour. headless = False is useful for debugging.

from pyppeteer import launch


async def main(query, location, days):
  browser = await launch(executablePath = '/usr/bin/chromium-browser', headless = True)
  page = await browser.newPage()
  storage = JobStorage()
  async for job in JobIter(page, query, location, days):
    storage.save(job)
  await browser.close()


if __name__ == '__main__':
  logging.basicConfig(
    level = logging.DEBUG, format = '%(asctime)s %(levelname)s %(name)s %(message)s')
  for l in ('websockets', 'pyppeteer'):
    logging.getLogger(l).setLevel(logging.WARNING)

  asyncio.get_event_loop().run_until_complete(main('Python', 'Nederland', 7))

Listing relevant jobs

And finally after executing the above code, in several minutes there’s db.sqlite. I used sqliteman [20] (available from Ubuntu Xenial’s repository) to query and explore it, because it correctly works with JSON attributes and clipboard (surprisingly SQLiteStudio, which I usually open SQLite databases with, has issues with both).

A sample record looks like:

{
  "dataset": {
    "id"                : 2622677334,
    "empId"             : 823453,
    "isOrganicJob"      : true,
    "sgocId"            : -1,
    "purchaseAdOrderId" : 0,
    "isEasyApply"       : false,
    "njslv"             : false
  },
  "rating"   : "2.9",
  "url"      : "https://www.glassdoor.nl/partner/jobListing.htm?...",
  "employer" : "Takeaway.com",
  "city"     : "Amsterdam",
  "date"     : "2018-03-23 16:10:59.413456"
}

Here follows the query I used. Yours most probably will be different.

SELECT
  title,
  status,
  round(julianday('now') - julianday(json_extract(properties, '$.date'))) ago,
  json_extract(properties, '$.employer') employer,
  json_extract(properties, '$.dataset.empId') eid,
  json_extract(properties, '$.rating') rating,
  json_extract(properties, '$.city') city,
  json_extract(properties, '$.url') url
FROM job
WHERE job MATCH '
  python
  NOT (django OR php OR java OR ruby OR perl OR C# OR .NET)
  NOT (title:devops OR title:analyst OR title:analist OR title:support OR title:graduation OR
    title:director OR title:phd OR title:researcher OR title:biologist OR
    title:test OR title:tester OR title: "quality assurance" OR title:hacker OR
    title:intern OR title:internship OR title:junior OR title:young OR
    title: "full stack" OR title:fullstack OR title: "front end" OR
    title:embedded OR title:firmware OR title:fpga OR title:C++ OR
    title:science OR title:scientist OR title:consultant OR title:coach OR
    title:marketing OR title:network OR title:operations OR title:student OR title:linux OR
    title:architect OR title:android OR title:automation OR title: "ops engineer" OR
    title:verification OR title:validation OR title:manager OR title:expert OR
    title:azure OR title:vmware OR title: "net developer" OR title:trader)
  NOT (employer:CareerValue OR employer: "Bright Cubes" OR employer:Deloitte OR
    employer:Cegeka OR employer: "Star Apple" OR employer:HUMAN-CAPITAL OR employer:Bonque OR
    employer:Sogeti OR employer:Yacht OR employer:iSense OR employer: "Talent Relations" OR
    employer:Place-IT OR employer:Professionals OR employer:ISAAC OR
    employer:Trinamics OR employer:CINQ OR employer:Teqoia OR employer:JouwICTvacature OR
    employer:Gazelle OR employer:Consulting OR employer:Topicus OR employer: "Blue Lynx" OR
    employer:Nobru OR employer:Qualogy OR employer:YER OR employer: "IT Human Resources" OR
    employer:Montash OR employer:CodeGuild OR employer:recruitment OR employer: "hot item" OR
    employer:MobGen OR employer:Mobiquity OR employer: "Orange Quarter" OR
    employer: "You Get" OR employer:ITHR)
'
ORDER BY employer, date(json_extract(properties, '$.date')) DESC

Yes, it’s pretty restrictive and thus gives relevant results, although I’m not very happy with constant manual blacklisting (there’s really whole lot of intermediaries). Maybe next time I’ll be looking for a job I employ some NLP or other machine learning. But now now this is it.

Wrap-up

SQLite a Swiss army knife of an ad-hoc storage and search,
Ubiquitous browsers have become automatable out of the box,
Even though some vacancy search platforms provide decent search features and other try to automate job collection, like Indeed with its XML job feed format [21], it looks like the effort is not enough and penetration of developer-friendly automated job search tools can be significantly improved,
Python for the win!

P.S. The new journey has already been scheduled :-)

[1]	https://www.glassdoor.nl/

[2]	https://stackoverflow.com/help/advanced-search-parameters-jobs

[3]	https://docs.python.org/3/library/sqlite3.html

[4]	https://sqlite.org/fts3.html

[5]	https://sqlite.org/vtab.html

[6]	(1, 2) https://sqlite.org/fts3.html#data_structures

[7]	https://sqlite.org/fts3.html#_set_operations_using_the_enhanced_query_syntax

[8]	https://sqlite.org/json1.html

[9]	https://sqlite.org/json1.html#jex

[10]	https://en.wikipedia.org/wiki/Relational_database

[11]	https://en.wikipedia.org/wiki/Document-oriented_database

[12]	https://pypi.org/project/sqlitefts/

[13]	https://en.wikipedia.org/wiki/Headless_browser

[14]	https://github.com/ChromeDevTools/devtools-protocol

[15]	https://github.com/GoogleChrome/puppeteer

[16]	https://pypi.org/project/pyppeteer/

[17]	https://www.python.org/dev/peps/pep-0525/

[18]	https://miyakogi.github.io/pyppeteer/reference.html#elementhandle-class

[19]	https://miyakogi.github.io/pyppeteer/reference.html#pyppeteer.page.Page.click

[20]	https://github.com/pvanek/sqliteman

[21]	https://www.indeed.com/intl/en/xmlinfo.html

CherryPy questions: testing, SSL and Docker

2015-04-23T00:00:00+02:00

Basically this is a CherryPy work-in-progress article. However, it aims to draw the line with testing infrastructure I was working on, as it seems to be completed. The article covers what has been done and what issues I met along the way of reviving the test suite to serve its purpose. The way is not traversed fully of course and there’s a lot left to do. It also seeks to justify concerns about CherryPy SSL functionality’s correctness and viability. As well, it tries to answer appeared questions about official Docker image and its range of application.

Contents

Testing
SSL
- Experiment
- Problem
- Plan
Docker
- Nature
- Application
Questions

Testing

As was discovered in Future of CherryPy quality assurance was a glaring problem. After I volunteered and started working on putting tests back in the right way, I’ve discovered several testing methodological errors that were made along in years of CherryPy existence. Also, back in the day CherryPy had its own test runner and later survived migration to Nose, but lost SSL tests. Not that the tests were purged away from the codebase, but werern’t possible to be run automatically and required a tester to change code to run tests with SSL. Obviously no one did it, and it led to several issues that rendered SSL functionality broken. Actually it still is, as at the moment of writing in latest CherryPy release, 3.6, SSL doesn’t work.

Besides testing issues, there were issues with CherryPy functionality as such. For instance, session file locking didn’t seem to be tested on machines with more than one execution unit. This way session locks didn’t work all the time on machines with a multi-core CPU.

You may be confused to think that as long as there is GIL [1], you shouldn’t care about number of CPUs. Unfortunately, serial execution doesn’t mean execution in one certain order. Thus on different number of execution units and on different GIL implementations (Python 3.2 has new GIL [2]) a threaded Python program will likely behave differently.

I’ve fixed file locking by postponing lock file removal to a cleanup process. Briefly covering issues with other session backends is to say Postgres session backend was decided to be removed because it was broken, untested and mostly undocumented. Memcached session backend was mostly fine, except some minor str/bytes issue in py3 and the fact it isn’t very useful in its current condition. The problem is that it doesn’t implement distributed locking, so once you have two or more CherryPy instances that use a Memcached server as a session backend and your users are not distributed consistently across the instances, all sorts of concurrent update issues may raise.

Substantial methodological errors in the test suite were:

test determinism
test dependency
test isolation

Test determinism

There were garbage collection tests that were automatically added to every subclass of test.helper.CPWebCase. They are non-deterministic and fail sporadically (there are other normal tests that have non-deterministic behaviour, see below). Because CPython’s reference counting garbage collector is deterministic, it’s a concurrency issue. What was really surprising for me hear was explanation like, okay, if it has failed run it once again until you’re sure that the failure persists. This test_gc supplementing is disabled now until everything else is resolved, and it is clear how to handle it properly.

Another thing that prevented certain classification to pass and fail and led to frequent freezes were request retries and absence of HTTP client’s socket timeout. I can only guess about the original reasons of such “carefulness” and not letting a test just fail. For instance, you can take some of concurrency tests that throw out up to a hundred client threads and imagine stability and predictability of the undertaking with 10 retires and no client timeout. These two were also fixed — test.webtest.openURL doesn’t retry on socket errors and sets 10-second timeout to its HTTP connections by default.

test.helper.CPWebCase starts full in-process CherryPy server and then uses real httplib.HTTPConnection to connect to it. Most of CherryPy tests are in fact either integration [3] or system tests [4].

Test dependency

How do you like tests named like test_0_check_something, test_1_check_something_else, and so on? And what if they were named this way intentionally, so they run in order, and fail otherwise because latter depend on former. Specifically, you can not run test individually. All occurrence I’ve found were renamed and freed from sibling shackles.

Test isolation

test.helper.CPWebCase‘s server startup routine was made in equivalent of unittest‘s setUpClass, so tests in a test case were not isolated from each other, which made it problematic to deal with the case of tests that interfere with internals as it was really hard to say whether such test has failed on its own or as a consequence. I’ve added per_test_setup attribute to CPWebCase to conditionally do the routine in setUp and accommodated tests’ code. I apprehended significant performance degradation, but it has just become around 20% slower, which was the reason for me to make it true by default. Not sure, maybe it should’t exist at all now.

Unfortunately, this doesn’t resolve all isolation issues because having to do a complete reset of in-process threaded server is a juggling whatsoever. When a tricky test fails badly, it still has a side-effect (see below), like this:

./cherrypy/process/wspbus.py:233: RuntimeWarning: The main thread is exiting, but the Bus is
in the states.STARTING state; shutting it down automatically now. You must either call
bus.block() after start(), or call bus.exit() before the main thread exits.

Testing progress

What has just been said about fixed and improved stuff is happening in my fork [5], it is not yet pushed upstream. More detailed description and discussion about current and above sections is in the thread in CherryPy user group [6]. Because it’s there until the moment Google decides to shutdown Google Groups because of low traffic, abuse or any other reason they use for such an announcement, I will summarise the achievements in improving CherryPy testing infrastructure.

Migrated tests to stdlib unittest, unitest2 as fallback for py26. Nose is unfriendly to py3 and probably was related to some locking issues.
Overall improvement in avoiding freezes in various test cases and testing environments. TTY in Docker, disabled interactive mode, various internals’ tests wait in a thread, et cetera.
Implemented “tagged” Tox 1.8+ configuration for matrix of various environments (see below).
Integrated with Drone.io [7] CI service and Codecov.io [8] code coverage tracking service.
Made various changes to allow parallel environment run with Detox to fit in 15 minutes of Drone.io free tier. It includes running tests in install directory [9], starting server each time on free port provided by OS, locking Memcached cases with an atomic operation and other.
Removed global and persistent configuration that prevented mixing HTTP and HTTPS cases.
Made it possible to work on the test suite in PyDev. When running tests in PyDev test runner, it adds another non-daemonic thread that tests don’t expect which leads to deadlocks or fails for 3 tests. They are now just skipped under PyDev runner.

Here’s Drone.io commands required to run the test suite. It uses Deadsnakes [10] to install old or not yet stable version of Python. Development versions are needed to build pyOpenSSL. The rest comes with Drone.io container out of the box.

echo 'debconf debconf/frontend select noninteractive' | sudo debconf-set-selections
sudo add-apt-repository ppa:fkrull/deadsnakes &> /dev/null
sudo apt-get update &> /dev/null
sudo apt-get -y install python2.6 python3.4 &> /dev/null
sudo apt-get -y install python2.6-dev python3.4-dev &> /dev/null

sudo pip install --quiet detox
detox
tox -e post

As you can see post environment closes the build. Unless the build was successful it won’t run and no coverage will be submitted or build artifacts become available. The build artifacts, what also needs to be listed in Drone.io, are several quality assurance reports that can be helpful for further improving the codebase:

coverage_report.txt
coverage_report.html.tgz
maintenance_index.txt
code_complexity.txt

For Codecov.io integration to work, environment variable CODECOV_TOKEN should be assigned.

Tagged Tox configuration

Final Tox configuration looks like the following. It emerged through series of rewrites that were led by tradeoffs between test duration, combined coverage report and need to test dependencies. I came to the last design after I realised importance of support of pyOpenSSL and thus the need to test it (see about SSL issues below). Then test sampling facility was easy to integrate to it.

[tox]
minversion = 1.8
envlist    = pre,docs,py{26-co,27-qa,33-nt,34-qa}{,-ssl,-ossl}

# run tests from install dir, not checkout dir
# http://tox.rtfd.org/en/latest/example/pytest.html#known-issues-and-limitations
[testenv]
changedir = {envsitepackagesdir}/cherrypy
setenv    =
    qa:   COVERAGE_FILE = {toxinidir}/.coverage.{envname}
    ssl:  CHERRYPY_TEST_SSL_MODULE = builtin
    ossl: CHERRYPY_TEST_SSL_MODULE = pyopenssl
    sa:   CHERRYPY_TEST_XML_REPORT_DIR = {toxinidir}/test-xml-data
deps =
    routes
    py26:      unittest2
    py{26,27}: python-memcached
    py{33,34}: python3-memcached
    qa:        coverage
    ossl:      pyopenssl
    sa:        unittest-xml-reporting
commands =
    python --version
    nt: python -m unittest {posargs: discover -v cherrypy.test}
    co: unit2 {posargs:discover -v cherrypy.test}
    qa: coverage run --branch --source="." --omit="t*/*" \
    qa: --module unittest {posargs: discover -v cherrypy.test}
    sa: python test/xmltestreport.py

[testenv:docs]
basepython = python2.7
changedir  = docs
commands   = sphinx-build -q -E -n -b html . build
deps       =
    sphinx < 1.3
    sphinx_rtd_theme

[testenv:pre]
changedir = {toxinidir}
deps      = coverage
commands  = coverage erase

# must be run separately after main envlist, because of detox
[testenv:post]
changedir = {toxinidir}
deps      =
    coverage
    codecov
    radon
commands  =
    bash -c 'echo -e "[paths]\nsource = \n  cherrypy" > .coveragerc'
    bash -c 'echo "  .tox/*/lib/*/site-packages/cherrypy" >> .coveragerc'
    coverage combine
    bash -c 'coverage report > coverage_report.txt'
    coverage html
    tar -czf coverage_report.html.tgz htmlcov
    - codecov
    bash -c 'radon mi -s -e "cherrypy/t*/*" cherrypy > maintenance_index.txt'
    bash -c 'radon cc -s -e "cherrypy/t*/*" cherrypy > code_complexity.txt'
whitelist_externals =
    /bin/tar
    /bin/bash

# parse and collect XML files of XML test runner
[testenv:report]
deps      =
changedir = {toxinidir}
commands  = python -m cherrypy.test.xmltestreport --parse

This design heavily relies on great multi-dimensional configuration that appeared in Tox 1.8 [11]. Given an environment, say py27-nt-ossl, Tox treats it like a set of tags, {'py27', 'nt', 'ossl'}, where each entry is matched individually. This way any environment, that makes sense though, with desired qualities can be constructed even if it isn’t listed in envlist. Here’s a table that explains the purpose of the tags.

tag	meaning
nt	normal test run via `unittest`
co	compatibility run via `unittest2`
qa	quality assurance run via `coverage`
sa	test sampling run via `xmlrunner`
ssl	run using Python bulitin `ssl` module
ossl	run using PyOpenSSL

Note {posargs: discover -v cherrypy.test} part. It makes possible to run a subset of the test suite in desired environments, e.g.:

tox -e "py{26-co-ssl,27-nt,33-qa,34-nt-ssl}" -- -v cherrypy.test.test_http \
  cherrypy.test.test_conn.TestLimitedRequestQueue \
  cherrypy.test.test_caching.TestCache.test_antistampede

Tackling entropy of threading and in-process server juggling

Even though there was a progress on making test suite more predictable, and actually runnable, it is still far from desired, and I was scratching my head trying to figure out how to gather test stats from all the environments. Solution was in the lighted place, right where I was looking for it. There’s a package unittest-xml-reporting, whose name is pretty descriptive. It’s a unittest runner that writes XML files likes TEST-test.test_auth_basic.TestBasicAuth-20150410005927.xml with easy-to-guess content. Then it was just a need to write some glue code for overnight runs and implement parsing and aggregation.

I’ve implemented and incorporated module named xmltestreport [12] into the test package. sa Tox tag corresponds to it. Running the following in terminal at evening, at morning can give you the following table (except N/A evironments, see below).

for i in {1..64}; do tox -e "py{26,27,33,34}-sa{,-ssl,-ossl}"; done
tox -e report

py26-sa	F	E	S
OK
py26-sa-ossl	F	E	S
test_conn.TestLimitedRequestQueue.test_queue_full	27	0	0
test_request_obj.TestRequestObject.testParamErrors	1	0	26
test_static.TestStatic.test_file_stream	27	0	0
py26-sa-ssl	F	E	S
test_caching.TestCache.test_antistampede	0	33	0
test_config_server.TestServerConfig.testMaxRequestSize	0	1	32
test_conn.TestLimitedRequestQueue.test_queue_full	33	0	0
test_static.TestStatic.test_file_stream	33	0	0
py27-sa	F	E	S
OK
py27-sa-ossl	F	E	S
test_conn.TestLimitedRequestQueue.test_queue_full	33	0	0
test_session.TestSession.testFileConcurrency	1	0	32
test_states.TestWait.test_wait_for_occupied_port_INADDR_ANY	1	0	32
test_static.TestStatic.test_file_stream	33	0	0
test_tools.TestTool.testCombinedTools	2	0	31
py27-sa-ssl	F	E	S
test_caching.TestCache.test_antistampede	0	33	0
test_config_server.TestServerConfig.testMaxRequestSize	0	4	29
test_conn.TestLimitedRequestQueue.test_queue_full	33	0	0
test_static.TestStatic.test_file_stream	33	0	0
test_tools.TestTool.testCombinedTools	2	0	31
py33-sa	F	E	S
test_conn.TestPipeline.test_HTTP11_pipelining	0	16	17
py33-sa-ossl	F	E	S
N/A
py33-sa-ssl	F	E	S
test_caching.TestCache.test_antistampede	0	33	0
test_static.TestStatic.test_file_stream	33	0	0
test_tools.TestTool.testCombinedTools	3	0	30
py34-sa	F	E	S
test_conn.TestLimitedRequestQueue.test_queue_full	0	1	32
test_conn.TestPipeline.test_HTTP11_pipelining	0	14	19
py34-sa-ossl	F	E	S
N/A
py34-sa-ssl	F	E	S
test_caching.TestCache.test_antistampede	0	32	0
test_config_server.TestServerConfig.testMaxRequestSize	0	1	31
test_static.TestStatic.test_file_stream	32	0	0
test_tools.TestTool.testCombinedTools	3	0	29

Actually, as you can see, it only had time for half of the iterations, and at the morning I interrupted it. Sylvain, you see, I mentioned those three tests in the group correctly. But there’re other candidates and this new arisen fragility of SSL sockets, if I understand it correctly. I mean all the tests with one fail or error. For example, test_session.TestSession.testFileConcurrency which is fine lock-wise but because it stresses networking layer with several dozen client threads, there’re two socket errors:

Exception in thread Thread-1098:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "py27-sa-ossl/lib/python2.7/site-packages/cherrypy/test/test_session.py", line 282, in request
    c.endheaders()
  File "/usr/lib64/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 805, in send
    self.sock.sendall(data)
  File "/usr/lib64/python2.7/ssl.py", line 293, in sendall
    v = self.send(data[count:])
  File "/usr/lib64/python2.7/ssl.py", line 262, in send
    v = self._sslobj.write(data)
error: [Errno 32] Broken pipe

Exception in thread Thread-1099:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "py27-sa-ossl/lib/python2.7/site-packages/cherrypy/test/test_session.py", line 282, in request
    c.endheaders()
  File "/usr/lib64/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 805, in send
    self.sock.sendall(data)
  File "/usr/lib64/python2.7/ssl.py", line 293, in sendall
    v = self.send(data[count:])
  File "/usr/lib64/python2.7/ssl.py", line 262, in send
    v = self._sslobj.write(data)
error: [Errno 104] Connection reset by peer

Other error details are available in XML reports which retain such details. You may think that I have found the reason for “carefulness” I questioned above, but I woulnd’t agree for several reasons. First, it looks like an SSL-only and also needs investigation — someone knows practical reasons for the SSL server and client to be more fragile? Second, if a test stresses server with a few dozens of client threads, regardless of SSL, it should be written in a way that tolerates failures for some of them rather than relying on magic recovery behind the scenes. And after all, all this integration testing is quite useless if we assume we’re fine having 9/10 of the client to fail in any test.

Status indication

Long story told short, I have changed my mind on status badges. After thinking a lot about organising project’s status indication and notification, where most important is regression notification when we have contributors who used to commit without running tests, I came to conclusion that status badges can do the job.

This way every contributor will know that an introduced regression will become a public knowledge in a matter of minutes. And that it won’t make other members of the community happy about it. Hopefully, this as a form of enforcement of standards and new testing tools will help us go smoother.

Another thing that made me thinking about status indication was March Google Code shutdown announcement [13]. For my Open Source code I was mostly using Google Code’s Mercurial. It was stable and minimal environment without distraction. Although it wasn’t a substantially maintained and over time it had become lacking several features, however it still has several exclusive ones. Essentially it was a project hosting in contrast to Bitbucket and Github which are a repository hosting. One could have as many repositories inside a project as needed. I know it is also feasible with Mercurial named branches, but it still feels a little denser than it should be. Another handy thing was that Google served raw content with appropriate MIME, e.g. JavaScript with text/javascript, thus allowed right-off-the-repo deployment for various testing purposes.

For status indication I used external links, the block in left side bar on main project page. You could say here is PyPi page, here’s CI, and here’s test coverage tracking. Because it was visible, in 1-click distance and parallel to project description it was sufficient.

Drawing the line

As far as I see, testing infrastructure is ready. I also made several related changes here and there. Now I think it is a good time to put it into upstream. First, of course for other contributors to make use of the new tools. Second, for the sake of granularity because there’s already a bunch of changes since January in my fork. And third, in coming months it is likely that I won’t have much time for contribution, thus it’s good time to discuss the changes if we need to.

SSL

As a part of discussion of design decisions made in the test suite [6], Joseph Tate, another CherryPy contributor expressed opinion that SSL support in CherryPy should be deprecated. Basically the message is about how can we guarantee security of the implementation if we can’t even guarantee its functionality.

Later in discussion with Joseph when I stood on fact that CherryPy’s SSL adapters are a wrapper on wrapper, Python ssl or pyOpenSSL on OpenSSL, and we don’t need a professional security expert in the community or special maintenance, he relaxed his opinion.

Distros have patched OpenSSL to remove insecure stuff, so maybe this is not as big a concern as I’m making it out to be, but the lack of an expert on this code is hindering its viability.

Also I want to inform Joseph, that his knowledge of CherryPy support for intermediate certificates isn’t up to date. cherrypy.server.ssl_certificate_chain is documented [14] and was there at least since 2011. I’ve verified it works in default branch.

Anyway, I took Joseph’s questions seriously and I’m sure we should not in any way give our users false sense of security. Security here is in theoretical sense as the state of art. I’m neither about zero-day breaches in algorithms and implementations, nor about fundamental problem of CA trust, no-such-agencies and generally post-Snowden world we live in.

On the other hand I believe that having pure Python functional (possibly insecure in theoretical sense) HTTPS server is beneficial for many practical use cases. There’re various development tools that are HTTPS-only unless you fight them. There’re environments where you have Python but can’t install other software. There’re use cases then any SSL, as an obfuscation, is fairly enough secure.

CherryPy is described and known as a complete web server and such reduction would be a big loss. And as I will show you below, although there’re several issues it is possible to workaround them and configure CherryPy to serve an application over HTTPS securely even today.

Experiment

Recently there was a CherryPy StackOverflow question right about the topic [15]. The guy asked how to block SSL protocols in favour of TLS. It led me to coming to the following quick-and-dirty solution.

#!/usr/bin/env python
# -*- coding: utf-8 -*-


import os
import sys
import ssl

import cherrypy
from cherrypy.wsgiserver.ssl_builtin import BuiltinSSLAdapter
from cherrypy.wsgiserver.ssl_pyopenssl import pyOpenSSLAdapter

from cherrypy import wsgiserver
if sys.version_info < (3, 0):
  from cherrypy.wsgiserver.wsgiserver2 import ssl_adapters
else:
  from cherrypy.wsgiserver.wsgiserver3 import ssl_adapters

try:
  from OpenSSL import SSL
except ImportError:
  pass


ciphers = (
  'ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:'
  'DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:!aNULL:'
  '!eNULL:!MD5:!DSS:!RC4:!SSLv2'
)

bundle = os.path.join(os.path.dirname(cherrypy.__file__), 'test', 'test.pem')

config = {
  'global' : {
    'server.socket_host' : '127.0.0.1',
    'server.socket_port' : 8443,
    'server.thread_pool' : 8,

    'server.ssl_module'      : 'custom-pyopenssl',
    'server.ssl_certificate' : bundle,
    'server.ssl_private_key' : bundle,
  }
}


class BuiltinSsl(BuiltinSSLAdapter):
  '''Vulnerable, on py2 < 2.7.9, py3 < 3.3:
    * POODLE (SSLv3), adding ``!SSLv3`` to cipher list makes it very incompatible
    * can't disable TLS compression (CRIME)
    * supports Secure Client-Initiated Renegotiation (DOS)
    * no Forward Secrecy
  Also session caching doesn't work. Some tweaks are posslbe, but don't really
  change much. For example, it's possible to use ssl.PROTOCOL_TLSv1 instead of
  ssl.PROTOCOL_SSLv23 with little worse compatiblity.
  '''

  def wrap(self, sock):
    """Wrap and return the given socket, plus WSGI environ entries."""
    try:
      s = ssl.wrap_socket(
        sock,
        ciphers = ciphers, # the override is for this line
        do_handshake_on_connect = True,
        server_side = True,
        certfile = self.certificate,
        keyfile = self.private_key,
        ssl_version = ssl.PROTOCOL_SSLv23
      )
    except ssl.SSLError:
      e = sys.exc_info()[1]
      if e.errno == ssl.SSL_ERROR_EOF:
        # This is almost certainly due to the cherrypy engine
        # 'pinging' the socket to assert it's connectable;
        # the 'ping' isn't SSL.
        return None, {}
      elif e.errno == ssl.SSL_ERROR_SSL:
        if e.args[1].endswith('http request'):
          # The client is speaking HTTP to an HTTPS server.
          raise wsgiserver.NoSSLError
        elif e.args[1].endswith('unknown protocol'):
          # The client is speaking some non-HTTP protocol.
          # Drop the conn.
          return None, {}
      raise

    return s, self.get_environ(s)

ssl_adapters['custom-ssl'] = BuiltinSsl


class Pyopenssl(pyOpenSSLAdapter):
  '''Mostly fine, except:
    * Secure Client-Initiated Renegotiation
    * no Forward Secrecy, SSL.OP_SINGLE_DH_USE could have helped but it didn't
  '''

  def get_context(self):
    """Return an SSL.Context from self attributes."""
    c = SSL.Context(SSL.SSLv23_METHOD)

    # override:
    c.set_options(SSL.OP_NO_COMPRESSION | SSL.OP_SINGLE_DH_USE | SSL.OP_NO_SSLv2 | SSL.OP_NO_SSLv3)
    c.set_cipher_list(ciphers)

    c.use_privatekey_file(self.private_key)
    if self.certificate_chain:
        c.load_verify_locations(self.certificate_chain)
    c.use_certificate_file(self.certificate)
    return c

ssl_adapters['custom-pyopenssl'] = Pyopenssl


class App:

  @cherrypy.expose
  def index(self):
    return '<em>Is this secure?</em>'


if __name__ == '__main__':
  cherrypy.quickstart(App(), '/', config)

After I regenerated our test certificate to:

use longer than 1024-bit key
not use MD5 and SHA1 (also deprecated [26]) for signature
use valid domain name for common name

Using the following versions, we have A- for the customised pyOpenSSL adapter on Qualys SSL Server Test [34]:

Debian Wheezy (stable)
Python 2.7.3-4+deb7u1, ssl.OPENSSL_VERSION == 'OpenSSL 1.0.1e 11 Feb 2013'
OpenSSL 1.0.1e-2+deb7u16
pyOpenSSL 0.14
CherryPy from default branch (pre 3.7)

There’s no error, but the warnings:

Secure Client-Initiated Renegotiation — Supported, DoS DANGER [16]
Downgrade attack prevention — No, TLS_FALLBACK_SCSV not supported [17]
Forward Secrecy — No, WEAK [18]

Here’s what Wikipedia says [25] about forward secrecy, to inform you that it is not something critical or required for immediate operation.

In cryptography, forward secrecy is a property of key-agreement protocols ensuring that a session key derived from a set of long-term keys cannot be compromised if one of the long-term keys is compromised in the future.

And here’s what can be squeezed out of built-in ssl, which is grievous.

Here goes the what-is-wrong list:

POODLE (SSLv3) — Vulnerable, INSECURE [19]
TLS compression — Yes, INSECURE [20]

There’s also half-implemented session resumption — No (IDs assigned but not accepted). And the same three warnings as for PyOpenSSL apply. ssl_builtin doesn’t handle underlying ssl module exceptions correctly, as the log was flooded by exceptions originating mostly from do_handshake(). PyOpenSSL’s log was the way more quiet with just a couple of page accesses.

[27/Mar/2015:14:07:49] ENGINE Error in HTTPServer.tick
Traceback (most recent call last):
  File "venv/local/lib/python2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 1968, in start
    self.tick()
  File "venv/local/lib/python2.7/site-packages/cherrypy/wsgiserver/wsgiserver2.py", line 2035, in tick
    s, ssl_env = self.ssl_adapter.wrap(s)
  File "./test.py", line 53, in wrap
    ssl_version = ssl.PROTOCOL_SSLv23
  File "/usr/lib/python2.7/ssl.py", line 381, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib/python2.7/ssl.py", line 143, in __init__
    self.do_handshake()
  File "/usr/lib/python2.7/ssl.py", line 305, in do_handshake
    self._sslobj.do_handshake()

SSLError: [Errno 1] _ssl.c:504: error:1408A0C1:SSL routines:SSL3_GET_CLIENT_HELLO:no shared cipher
SSLError: [Errno 1] _ssl.c:504: error:14076102:SSL routines:SSL23_GET_CLIENT_HELLO:unsupported protocol
SSLError: [Errno 1] _ssl.c:504: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
SSLError: [Errno 1] _ssl.c:504: error:14094085:SSL routines:SSL3_READ_BYTES:ccs received early

But nothing is really that bad even with ssl_builtin. Let’s relax stability policy, make sure you understand the consequences if you’re going to do it on a production server, and add Debain Jessie’s (testing) source to /etc/apt/sources.list of the server (Debian Wheezy) and see what next Debain has in its briefcase:

Python 2.7.9-1, ssl.OPENSSL_VERSION == 'OpenSSL 1.0.1k 8 Jan 2015'
Python 3.4.2-2, ssl.OPENSSL_VERSION == 'OpenSSL 1.0.1k 8 Jan 2015'
OpenSSL 1.0.1k-3
pyOpenSSL 0.15.1 (installed via pip, probably has been updated since)

For you information, related excerpt from Python 2.7.9 release change log [21]. It was in fact a huge security update, not very well represented by the change of a patch version.

The entirety of Python 3.4’s ssl module has been backported for Python 2.7.9. See PEP 466 [22] for justification.

HTTPS certificate validation using the system’s certificate store is now enabled by default. See PEP 476 [23] for details.

SSLv3 has been disabled by default in httplib and its reverse dependencies due to the POODLE attack.

All errors have gone, and in spite of the warning it supports forward secrecy for most clients. Here are just couple of warnings left:

Secure Client-Initiated Renegotiation — Supported, DoS DANGER [16]
Session resumption (caching) — No (IDs assigned but not accepted)

The result is pretty good. If you run it on plain ssl_builtin, is has B, because RC4 ciphers [24] aren’t excluded by default. Python 3.4.2 test on the customised ssl_builtin is obviously the same A- because both underlying libraries are the same. Customised ssl_pyopenssl on Python 2.7.9 gives same A- — again it has functional session resumption but doesn’t have forward secrecy for any client.

Problem

After the experiment has taken place, it is time to generalise the results. Here’s what is wrong with CherryPy’s SSL support now:

giving false sense of security
not flexible configuration: protocol, ciphers, options
not up to date with builit-in ssl (no SSL Context [27] support)
no support for pyOpenSSL for py3
no security assessment

Plan

I think that the problem is solvable with a reasonable effort. What has been said above may already have convinced you that no special cryptography knowledge is required. There’s a scalar letter grade for A to F and several design decisions aimed at flexibility and compatibility.

Outright state of affairs

Now we have the following note in SSL support section [14] in Deploy documentation page.

You may want to test your server for SSL using the services from Qualys, Inc.

To me after doing this research it doesn’t even look like a half-true thing expressed modestly. It looks we lie to users, giving them false sense of security. It may look like we’re so sure that it’s okay that we let them do a voluntary task of checking it once again, just in case. But in fact it is to deploy their py2 application and see it’s vulnerable (I assume Python 2.7.9 isn’t going to updated to soon for various reasons). This is absolutely redundant and we can tell that Python < 2.7.9 is a insecure platform right away. There should be a warning in documentation, like “Please update to Python 2.7.9, install pyOpenSSL or start off with Python 3.4+. Otherwise it will only obfuscate your channel”.

Python < 2.7.9 should be considered an insecure platform with appropriate Python warning, see below.

Configuration

cherrypy.server in addition to ssl_certificate, ssl_certificate_chain, ssl_private_key should support:

ssl_cipher_list available for all Python versions and both adapters. Mostly needed for Python < 2.7.9.
ssl_context available for Python 2.7.9+ and 3.3+. pyOpenSSL is almost there. SSL Context [27] allows user to set: protocol, ciphers, options.

SSL Context [27] when not provided from user should be created with ssl.create_default_context which takes care of security defaults. It is available for Python 2.7.9+ and 3.4+. For Python 3.3 context should be configured at CherryPy side (the function is two dozen of lines [29]).

ssl_adapters should be available at cherrypy.wsgiserver for user be able to set custom adapter (see the code above). Already implemented in my fork.

Built-in SSL adapter update

The adapter should make use of PEP 466 [22]. This way, inside it may be two adapters. One for Python 2.7.9+ with SSL Context [27] support and one for Python < 2.7.9. The latter should be explicitly documented as vulnerable and emit warning when used. Documentation and warning text can be taken from urllib3‘s InsecurePlatformWarning [30].

The adapter should handle SSL exceptions according to the protocol, not flooding the log with them.

Python 3 support for pyOpenSSL adapter

Purpose of pyOpenSSL adapter should be rethought as Python < 2.6 is no longer supported and Python 2.6+ has ssl module built in. Thus the purpose is not to provide a fallback. The purpose is to provide more flexibility for users to manage their own security. Python binary that comes from a disto’s repository have their own version of OpenSSL. When an OpenSSL breach is discovered it is reasonable to expect patched OpenSSL package to be shipped first. This gives an user ability to rebuild pyOpenSSL with latest OpenSSL as soon as possible in matter of doing:

pip install -I pyOpenSSL

Embracing this purpose Python 3 support should be provided. Currently pyOpenSSLAdapter.makefile uses wsgiserver.CP_fileobject directly and uses wsgiserver.ssl_pyopenssl.SSL_fileobject which inherits from wsgiserver.CP_fileobject. wsgiserver.CP_fileobject is available only in wsgiserver.wsgiserver2. Therefore pyOpenSSLAdapter.makefile should use wsgiserver.CP_makefile in py3 just like BuiltinSSLAdapter does. Here an advice or involvement of original authors is likely to be needed because it’s unclear in what circumstances pyOpenSSLAdapter.makefile should return non-SSL socket.

Automated security assessment

Debian package of OpenSSL is described as:

[When] you need it to perform certain cryptographic actions like… SSL/TLS client and server tests.

There are also things like sslyze [31], which is described as fast and full-featured SSL scanner. I think at some extent this approach will give automated security test, but it’s an rough road. Basically, the quality of results will depend on our proficiency and maintenance of the chosen tool. Much easier way is to you a service.

Luckily, in January Qualys announced SSL Labs API beta [32]. At the dedicated page [33] there’re links to official command like client and protocol manual. It makes sense to expect the same behaviour from the API as the normal tester [34] has. If so it won’t work on bare IP address and on non 443 port.

To externalise this idea we have some prerequisite. If there’s real CherryPy instance running behind Nginx on cherrypy.org at WebFaction, then do we have access to it? Can we run several additional instances there? If not, can we run them somewhere else?

If think of instances for the following environments:

py2-ssl-insecure.ssl.cherrypy.org with Python < 2.7.9 that will probably be stable version in distos’ repositories for quite some time
py2-ssl-secure.ssl.cherrypy.org with Python 2.7.9+
py2-pyopenssl-secure.ssl.cherrypy.org with any py2 with pyOpenSSL
py3-ssl-secure.ssl.cherrypy.org with Python 3.4+
py3-pyopenssl-secure.ssl.cherrypy.org with any py3 with pyOpenSSL once it’s available

Then we need to multiplex them to outer port 443 because likely it’ll be running on single server. Because Nginx doesn’t seem to be able passthrough upstream SSL connections [35] we need to employ something else, maybe HAProxy [36] [37].

What’s left to do thereafter is to implement SSL Labs API, run in some schedule and we’re done. It is able to give us detailed and actual reports on every environment that can be exposed to users to make their decisions.

Docker

People are running in circles, screaming “DevOps”. Besides over half of them also yell “Docker” at odd turns.

Scene of the day; Baron “Xaprb” Schwartz authored an article about DevOps identity crisis [38] which makes it all feel this disordered. Or from the other end, at LinuxCon 2014 talk founder and now CTO of Docker, Inc. Solomon Hykes presented [39]:

I know 2 things about Docker: it uses Linux containers, and the Internet won’t shut up about it.

And all you want in the end of the day to all the these people really shut up and stop touting you their magic medicine. But as long as Sylvain started dedicated topic for collecting feedback about CherryPy’s official Docker image [40], I was one who unintentionally started this discussion (we talked about Drone.io test environment which is a Docker), and wanted to reply anyway, I started investigating what the thing that causes itch not only to all the cool kids is really about, except it eliminates ops as a class and cures indigestion by the way.

I have to admit that as soon as I have gotten to the original source of information, have listened to a couple of Solomon’s interviews and talks and seen how he was involved into Docker reputation management by replying to various critical articles, I became more well-disposed toward the project. Solomon himself looks like a positive and nice kind of guy who knows what he is doing. Not as nice, actually, as if he hadn’t rewritten DotCloud’s Python code to next Google’s attempt at language engineering, though reasonably nice anyway. You may often see him saying something like this [41]:

I know that Docker is pretty hyped right now — I personally think it’s a mixed blessing for exactly this reason: unexperienced people are bound to talk about Docker, and say stupid things about what Docker can do and how to use it. I wish I could magically prevent this.

So I ought to limit myself in saying stupid things ;-)

Nature

Basically Docker is like Vagrant + OpenVZ on vanilla kernel. The two are available for quite some time and have established use cases, reproducible development environment and lightweight, operating-system-level virtualisation [42], respectively. Although Docker combines the two qualities, and it’s tempting to (re)think that developers can build production-ready container images and deploy them through Docker magic powers, and that it’s what Docker is designated for, it would be a mistake to say so.

As Solomon says, Docker is mere a building block that comes in at the right time and provides high-level APIs for set of great low-level features modern Linux kernel has. There are several things I want to summarise from his latest interview given to FLOSS Weekly in episode 330 [43]:

it is not a silver bullet, it is a building block
it is used both ways: as VM and as “binary” (process-per-container); neither is the “true” one
it is neither superior nor mutually exclusive with configuration management
it is not a complete production solution; production is hard and is left to the reader
it doesn’t primarily target homogenous applications; the platform’s tools may be sufficient

I would agree that except first one, one might find “however” to any of them. And especially, considering the pace of development, and promising trio of new experimental tools, that appeared to come along Docker for cluster orchestration, in Unix-way of loosely coupled components and responsibility separation: Machine, Swarm and Compose [44].

And I really like to think about Docker as a building block. No matter what the hype is, even though it entails imbalance in available information, there’re certain design decisions made that have certain implications. Taking them into account you decide whether Docker fits your certain task. For instance, there was a good comparison, LXC versus Docker, by Flockport [45], which emphasises the difference between the two as long as Docker goes with own libcontainer:

performance penalty for a layered filesystem (e.g. AUFS)
separate persistence management for immutable containers
process-per-container design has issues with real world [46]:
- most of existing software is written in expectation of init system
- monitoring, cron jobs, system logging need a dedicated daemon and/or SSH daemon
- system built of subsystems built of components is much harder to manage

As I said above both process-per-container and application-per-container both have merit. But because Docker originally was conceived with the former idea, most images in Docker Hub are process-per-container and people begin to fight the tool they have chosen by using Bash scripts, Monit or Supervisor to manage multiple processes.

Complete configuration management products, like Ansible, or even simpler Fabric-based approaches in conjunction with abundance of API for every single IaaS and PaaS provider made available infrastructure as code with any desired quality long before Docker appeared. Big companies were implementing their own infrastructure-as-code solutions on top of chosen service provider. Smaller companies mostly weren’t in the game. Docker, because of its momentum, can standardise service providers’ supply and popularise infrastructure as code in the industry which I see quite advantageous.

Another thing that shoundn’t fall out of your sight is that Docker is a new technology. It has known and significant risk for new security issues [47]. And even in its own realm it is not as nearly as perfect. Indeed Docker makes clearer responsibility separation between devs and ops. But there are cross-boundary configuration, like maximum open file handles or system networking settings which need to be set on host and have corresponding configuration in container, e.g. MySQL table cache. There are environment-specific configuration like load testing in QA and rate-limiting in production. There is a lot more of such things to consider in real system.

There is a lot of discussion of microservice architecture [48] today. One of its aspects is that what was inner-application complexity is now shifted into infrastructure. And the more you follow this route the more you immerse into distributed computing while distributed computing is still very hard. Docker guidelines go clearly along the route.

Lately I’ve seen a comprehensive and all around excellent presentation on distributed computing by Jonas Bonér [49]. Although it’s named “The Road to Akka Cluster and Beyond” the Akka, which is distributed application toolkit for JVM, part starts around 3/4 of the slides. The rest is overview, theoretical aspects and practical challenges of distributed computing.

Last thing to say in this section is about Docker’s competitors. There are obviously other fish in the sea. Besides already mentioned LXC [50], there are also newer container tools built on experience with existing tools, realising their shortcomings and addressing them. To name a few: Rocket [51], Vagga [52].

Application

With that being said I think thorough discussion of infrastructure and distributed application design within the scope of CherryPy project is not only a off-topic. It’s a hugely out of the context. However the topic itself is actual and interesting, but I’m sure everyone who is dealing or is willing to deal with an application at scale knows how to run pip install cherrypy.

What I already told to Sylvain, is on-topic and CherryPy QA can benefit from is Docker QA image with complete CherryPy test suite containing all environments and dependencies so every contributor can take it and run the test suite against introduced changes effortlessly. Here’s a dockerfile for the purpose.

FROM ubuntu:trusty

MAINTAINER CherryPy QA Team

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update
RUN apt-get --no-install-recommends -qy install python python-dev python3 python3-dev \
  python-pip python-virtualenv
RUN apt-get --no-install-recommends -qy install software-properties-common build-essential \
  libffi-dev mercurial ca-certificates memcached

RUN add-apt-repository ppa:fkrull/deadsnakes
RUN apt-get update
RUN apt-get --no-install-recommends -qy install python2.6 python2.6-dev python3.3 python3.3-dev

RUN pip install "tox < 2"
RUN pip install "detox < 0.10"

WORKDIR /root
RUN hg clone --branch default https://bitbucket.org/saaj/cherrypy

WORKDIR /root/cherrypy
RUN echo '#!/bin/bash\n \
  service memcached start; hg pull --branch default --update; tox "$@"' > serial.sh
RUN echo '#!/bin/bash\n \
  service memcached start; hg pull --branch default --update; detox "$@"' > parallel.sh
RUN chmod u+x serial.sh parallel.sh
ENTRYPOINT ["./serial.sh"]

hg clone line should be changed to the source of changes. Thus I think we should distribute it just as a dockerfile. Because of development dependencies it isn’t small — ~620 MB. I also made parallel Detox run possible, parallel.sh, but I think usually running an environment at a time is better to stress tests with more concurrency. What’s really nice about it is that it can be run just like a binary.

docker run -it cherrypy/qa -e "py{26-co-ssl,27-nt,33-qa,34-nt-ssl}" \
  -- -v cherrypy.test.test_http cherrypy.test.test_conn.TestLimitedRequestQueue

Side-project

As I said, I think that certain answers to questions of infrastructure, orchestration or application design are off-topic inside the project. CherryPy was never meant to answer them. But it’s really nice to have all this information available nearby. Like the series of posts Sylvain wrote in his blog [53].

Sylvain, I can give you a starting point if you wish. I have a CherryPy project, cherrypy-webapp-skeleton [54], which is a complete, traditional CherryPy deployment on Debain. I had a fabfile [55] there which I used test the tutorial against a fresh Debain virtual machine. What I did is I basically translated in into a dockerfile [56]. It’s a virtual machine style, application-per-container thing, and it’s good for testing. It is also one of possible ways of deployment, but it obviously doesn’t provide means to scaling. So if you’re interested in splitting it to separate containers and providing configuration for containers and orchestration for such an example I would surely accept your changes. You’re also free to point me if you see something is wrong there — it’s all discussable.

In case you’re interested I’ll give an overview of the project and at the same time address some of your questions:

components as-is: CherryPy, Nginx, MySQL, Monit
containers to-be: 2 x CherryPy, Nginx, MySQL
it runs through init script which calls cherryd-like daemon
it is installed into a virtualenv but it probably doesn’t make sense in a container
we outght to adhere to distro standard directory layout (/var/www, /var/logs, etc.), but the extent is in question

If we will present it as an example we must also address init system and other issues [46], in some way, I wrote above.

Questions

For these I can expect the answers only from Sylvain, but others’ ideas are also welcomed.

Will you review my changes in the fork [5]? Or it’s easier if I’ve created a pull request right away?
Can you update documentation to explicitly express the state of our SSL support? I think it is needed as soon as possible.
Do we have any progress on bugtracker?
As you have already mentioned a CherryPy roadmap, I think, it’s getting a direction, like: 3.7 for QA, 3.8 for SSL.
I still want to hear about your ideas about test_gc.
We still have this outdated and poitless wiki home as landing page for the repo.
Recent coverage report [8] uncovered that we have a dozen with zero-coverage modules which are probably obsolete.

[1]	https://wiki.python.org/moin/GlobalInterpreterLock

[2]	http://www.dabeaz.com/python/NewGIL.pdf

[3]	http://en.wikipedia.org/wiki/Integration_testing

[4]	http://en.wikipedia.org/wiki/System_testing

[5]	(1, 2) https://bitbucket.org/saaj/cherrypy

[6]	(1, 2) https://groups.google.com/forum/#!topic/cherrypy-users/Qm-uLscS4z4

[7]	https://drone.io/bitbucket.org/saaj/cherrypy

[8]	(1, 2) https://codecov.io/bitbucket/saaj/cherrypy?ref=default

[9]	http://tox.rtfd.org/en/latest/example/pytest.html#known-issues-and-limitations

[10]	https://launchpad.net/~fkrull/+archive/ubuntu/deadsnakes

[11]	http://tox.readthedocs.org/en/latest/config-v2.html

[12]	https://bitbucket.org/saaj/cherrypy/src/default/cherrypy/test/xmltestreport.py

[13]	http://google-opensource.blogspot.com/2015/03/farewell-to-google-code.html

[14]	(1, 2) http://cherrypy.readthedocs.org/en/latest/deploy.html#ssl-support

[15]	http://stackoverflow.com/q/29260947/2072035

[16]	(1, 2) https://community.qualys.com/blogs/securitylabs/2011/10/31/tls-renegotiation-and-denial-of-service-attacks

[17]	https://datatracker.ietf.org/doc/draft-ietf-tls-downgrade-scsv/

[18]	https://community.qualys.com/blogs/securitylabs/2013/06/25/ssl-labs-deploying-forward-secrecy

[19]	https://community.qualys.com/blogs/securitylabs/2014/10/15/ssl-3-is-dead-killed-by-the-poodle-attack

[20]	https://community.qualys.com/blogs/securitylabs/2012/09/14/crime-information-leakage-attack-against-ssltls

[21]	https://www.python.org/downloads/release/python-279/

[22]	(1, 2) https://www.python.org/dev/peps/pep-0466/

[23]	https://www.python.org/dev/peps/pep-0476/

[24]	https://community.qualys.com/blogs/securitylabs/2013/03/19/rc4-in-tls-is-broken-now-what

[25]	http://en.wikipedia.org/wiki/Forward_secrecy

[26]	https://community.qualys.com/blogs/securitylabs/2014/09/09/sha1-deprecation-what-you-need-to-know

[27]	(1, 2, 3, 4) https://docs.python.org/2/library/ssl.html#ssl-contexts

[28]	https://docs.python.org/2/library/ssl.html#ssl.create_default_context

[29]	https://hg.python.org/cpython/file/5b5a22b9327b/Lib/ssl.py#l395

[30]	https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning

[31]	https://github.com/nabla-c0d3/sslyze

[32]	https://community.qualys.com/blogs/securitylabs/2015/01/22/ssl-labs-apis-now-available-in-beta

[33]	https://www.ssllabs.com/projects/ssllabs-apis/

[34]	(1, 2) https://www.ssllabs.com/ssltest/

[35]	http://forum.nginx.org/read.php?2,234641,234641

[36]	https://serversforhackers.com/using-ssl-certificates-with-haproxy

[37]	https://scriptthe.net/2015/02/08/pass-through-ssl-with-haproxy/

[38]	http://www.xaprb.com/blog/2015/02/07/devops-identity-crisis/

[39]	http://www.youtube.com/watch?v=UP6HxoC66nw

[40]	https://groups.google.com/forum/#!topic/cherrypy-users/WywUwX8dGOM

[41]	http://www.krisbuytaert.be/blog/docker-vs-reality-0-1#comment-5449

[42]	http://en.wikipedia.org/wiki/Operating-system-level_virtualization

[43]	http://twit.tv/show/floss-weekly/330

[44]	http://blog.docker.com/2015/02/orchestrating-docker-with-machine-swarm-and-compose/

[45]	http://www.flockport.com/lxc-vs-docker/

[46]	(1, 2) http://phusion.github.io/baseimage-docker/

[47]	https://zeltser.com/security-risks-and-benefits-of-docker-application/

[48]	http://martinfowler.com/articles/microservices.html

[49]	http://www.slideshare.net/jboner/the-road-to-akka-cluster-and-beyond

[50]	https://linuxcontainers.org/

[51]	https://coreos.com/blog/rocket/

[52]	http://vagga.readthedocs.org/en/latest/what_is_vagga.html

[53]	http://www.defuze.org/

[54]	https://bitbucket.org/saaj/cherrypy-webapp-skeleton

[55]	https://bitbucket.org/saaj/cherrypy-webapp-skeleton/src/backend/fabfile.py

[56]	https://bitbucket.org/saaj/cherrypy-webapp-skeleton/src/system/docker/Dockerfile

Future of CherryPy: bright and shiny?

2015-01-25T00:00:00+01:00

First off, I really hope so, tough at the same time I see the reasons for it. Moreover I have been trying to contribute back to the community by giving knowledge that could hopefully cut the rough edges that I’ve met on my way of learning and employing CherryPy [1]. And when CherryPy is calling, I have something to say.

This article is my consideration in reply to the holiday post [2] by Sylvain Hellegouarch in CherryPy user group about its current state and future, which recalls to older big discussion [3] about the status of development of CherryPy.

Acquisition

My CherryPy experience started in beginning of 2011. I had a chance to jump ship to Python web application development and I didn’t lose it. At that moment I had years of web development experience, so I was looking for the certain piece of software. If I would start this search today, most of the front page titles are the same. The same you don’t have to be the best to be popular still applies to the very domain of Python application.

I’ve spent a couple of weeks reading the web: reviews, comparisons, benchmarks [4] and StackOverflow [5]. And I have to admit that there are paths for newcomers to CherryPy, who look for stable, fast, well-designed and pythonic HTTP framework with its own Zen [6] (like Python itself [7]), that doesn’t dictate application design, nor what tools to use, but rather provides structure, testability and primitives that help code scale.

Scope

Where CherryPy stands? I think it perfectly stands as a Python HTTP application server. It allows a developer to write a genuine Python code, use the best from the Cheese Shop, design applications in whatever desired way (but with restrictions of threaded execution model).

Self-contained packages

I agree that stable pure-Python HTTP server is one of major features of CherryPy. Because basically, in most cases one can just forget about WSGI, and use good old HTTP — bare CherryPy or behind reverse-proxy like nginx. Less intermediaries means less problems as it simplifies development and deployment. However, I don’t think it’s an additive feature. If you extract the web-server into a stand-alone package, be it named Cheroot or something else, it won’t be as valuable for application development. If there are other frameworks’ users who host their WSGI applications with CherryPy, it’s their problem of deployment and of these framework’s authors. This usage is ultimately a byproduct of CherryPy’s WSGI compliance and stability, and let it stay so.

On the other hand, as the questions risen in the post are mostly about the cost of CherryPy maintenance, if the increased modularity will help to reduce it, then it surely makes sense. But the extraction also has its cost, and if the code of target functionality is tightly coupled with the rest of the codebase, then benefits of extraction should clearly outweigh its cost. At least inner-package refactoring towards more encapsulated components is possible with almost the same benefits.

Implementation support

Regarding supported Python implementation official website says: Python 2.5+, 3.1+, PyPy, Jython and Android. Documentation says: CPython, IronPython, Jython and PyPy and CherryPy supports Python 2.3 through to 3.4.. There’s no need to say about official distribution, CPython, but its versions are in question. There’s a lot of compatibility code for old Python versions in CherryPy, which can be shrunk by lifting the bar closer to actual ones. I don’t think the move is for 3.x series, but for CherryPy 4, CPython 2.7 and 3.3+ are fine. This is a real way to make the codebase cleaner, smaller and fun to deal with again.

PyPy is constantly taking more ground and is supported by many libraries that also grow in number. However I can’t get many real use-cases out of my head (if we don’t actually include the case of a long task which should be run in background, but is run synchronously as an effort tradeoff and assuming it can benefit from JIT), nor do I know if there’s CPU-bound hot CherryPy code. Mostly I feel it’s a cool-factor, but with possible real use-cases. So I think it’s a good idea to keep it. The problem with supporting PyPy is its garbage collectors. They aren’t based on reference counting, thus files, sockets and everything that was able to close itself passing out of the execution scope with zero references on CPython, won’t deterministically keep doing it on PyPy. It makes more strict demands on code to keep it memory-leak free. Python3’s ResourceWarning can help with it. I left a bug report [8] about the subject once when I was porting a CherryPy-based library to Python3.

The rest is more questionable. CherryPy running on Android is clearly a cool-factor. SL4A with Python isn’t straightforward to install, and even though there’s complete CPython 2.6, it barely has a single real use-case. There’s file API, SQLite and the rest important things we use server-side for, for PhoneGap and the like, for ones who write HTML5 application. There’s Kivy with established deployment. The same way I feel of Jython and IronPython support. I have no idea if anyone is actually using CherryPy on them, but having them officially supported complicates things. Testing, debugging, developing the framework. I think these highly specific environments should be covered by enthusiasts, if there’re any out there, who deal with them. If there’s a bug, say for Jython, it should be treated as enhancement and fixed only if the fix doesn’t impede design, performance and maintainability of primary implementations’ code. I think guarantees and according measures are needed, so it’s clear what is officially supported by CherryPy, and what it may run on.

To depend or not to

For an application it is perfectly fine to depend on as many libraries to delegate as most out-of-domain functionality away from its codebase. On the other hand for a library it’s not as good idea. There’s a functional difference between a library and a framework, but for current form of CherryPy these terms feel interchangeable.

For a library having no dependencies is a feature when general less moving parts, less issues applies. It may relate to broken, abandoned, license-changed 3rd-party dependencies. It may be incompatible changes, or used C-modules which complicate installation and use in PyPy and other implementations and platforms. Thus as CherryPy has grown its own pure-Python internals which are fast and stable, I think there’s no point in abandoning them.

Hype, fuss and choices

Last, but not least thing I want to note about the scope of CherryPy is its technology choice. WSGI is good example here. From inception it likely was a cool thing in Python. WSGI here and WSGI there, standards, uniformity and stuff. And then, a little later… oops, a synchronous protocol. No WebSockets, no SSE, no other close to real-time techniques. This choice affects, for example things like django-sse [9] and its users [10] (though this community is strongest in seeing everything as a nail). But also PaaS providers like OpenShift [11], and probably others.

That is to say it’s okay to be compliant (e.g. WSGI-compliant), but it’s not okay to be a pliable bigot, who praises next cool X at every turn. In that sense I like how CherryPy stands. Simple is simple in a threaded server. Real-time stuff needed? Sylvain’s ws4py [12] to the rescue. Simple background processing? There you go with cherrypy.process.plugins.BackgroundTask. So I want to keep seeing CherryPy making conscious and pragmatic technology choices.

2014 has seen new breed of asynchronous libraries within Python 3.4+ batteries, asyncio. It is nice to see co-operative execution in Python finds more expressive language constructs and tooling, but apart from its niche uses I suspect it’ll be next asynchronicity-everywhere craze, which will lead to rewrites, new cool stuff and more “pressure” on users of “old” software. At least, there’s no evidence that it can be faster on generic load, as CherryPy is neck and neck with Tornado [4]. There’s good socket performance notice [13] in Python documentation which also applies generally to such design choice.

There’s no question that the fastest sockets code uses non-blocking sockets and select to multiplex them. You can put together something that will saturate a LAN connection without putting any strain on the CPU. The trouble is that an app written this way can’t do much of anything else - it needs to be ready to shuffle bytes around at all times.

…

Finally, remember that even though blocking sockets are somewhat slower than non-blocking, in many cases they are the “right” solution. After all, if your app is driven by the data it receives over a socket, there’s not much sense in complicating the logic just so your app can wait on select instead of recv.

Audience

Naturally appreciation comes from the target audience. For a newbie in HTTP (which is a complex protocol by the way) and application development in general, Django may be a blessed gift because it helps to meet her goals by answering as many development questions as possible, and at the same time for an experienced developer it may seem a worthless bulk of bloatware that solves all its tasks equally poor (like all generic solutions, though). The same newbie won’t be able to effectively use CherryPy because it won’t answer the several dozen of questions for her.

If CherryPy stays CherryPy there can’t and shoundn’t be answers for many of them. How to design my application? What application directory layout to use? What persistance layer and storage to employ? How do I render a template in CherryPy? It’s all up to you, developer. From my experience, reading others’ opinions on the web, and particularly in the user group, there is an unanimously opinion that CherryPy’s barrier to entry is high, but once you wrapped your head around its concepts you’re enlightened and empowered.

These points lead to two measures of increasing CherryPy user base, and hopefully contributors at some proportional rate. First is targeting more specific audience. Examining the main page [1] it is at least not bad. It shows that simple is easy, and gives clues that complex is possible. It refers to any other object-oriented Python program, which indicates good API. But it’s biased opinion, because I look at things in reverse. Is it really that comprehensible for one who is making decision at the moment? Is there some real-world CherryPy code that is widely considered worth exposing, that is done “right” way and promotes best CherryPy practices? Second is means of lowering and alleviating barrier to entry, see Improvement section.

A note about code hosting that is supposed to affect the audience. I agree with Konstantin Molchanov, that magic pixies… wait no, the best friend of Ruby and Rails, to the backbone awesome, the octocat won’t make CherryPy popular by the wave of its tentacle. There’s no outreach program nor there’s traffic charity for newcomers on Github. It’s a popular code hosting which makes it easy to contribute to a Git repository, in case there’re ones who are willing to. But if there are no new contributors, I don’t see the way it can help.

Though it may not work in opposite direction, because if there’s a Python developer willing to contribute to a Python project, the developer may expect a Python tool for it, which Git is not. I think it’s a disillusioned and naive idea to turn away from Mercurial to get more Python users. At the same time, I cannot disagree with Nic Young that there’re a few developer services (mostly for quality assurance) which integrate only with Github, which are free for open source projects and may be useful for CherryPy (e.g. TravisCI is not Git-only, but Github-only because it’s tied to its API), albeit none of them lack alternatives.

Part taken

A year after I commenced my CherryPy affair one web application project was in production and I possessed certain portion of experience. Specifically saying, deployment struggle experience. Having HTTP-only deployment was very helpful, because I could reuse the knowledge I had, but Linux part was at most scarce, poorly covered and I was groping for valuable pieces of knowledge all over the web. Having mint Debian box how do I make my application continuously deployable? How do I make it first-class daemon and run it as www-data? How do I put it behind nginx? How do I monitor its memory and CPU usage? How to restart it if it has crashed? How do I rotate its logs?

That time I already had the answers, and they were put together and published as tutorial/skeleton project [14]. I felt it was necessary contribution to make. Obviously, it hasn’t received high volume traffic, but qualitatively there’re constant visits from all over the world (and least from the countries where people do IT), which indicates that CherryPy still attracts new users.

In mid-2013 I’ve written another tutorial/skeleton project [15], and even though its goal is maintainable JavaScript website-like application design, its server-side is CherryPy application with real-world template layout for Jina2 and according CherryPy tool. I’ve found time to write documentation for it only in the end of 2014, so it is also an on topic thing.

In the spring of 2014 I started to participate on StackOverflow’s CherryPy tag [5]. It was another pay-off kind of thing, because the site was helpful to me, and its CherryPy part specifically. But the latter sadly had a lot of unanswered question making it feel a little abandoned. In fact many questions are quite interesting dealing with performance, design, execution model of CherryPy, so it requires an answerer to have a good grasp of documentation and codebase. Besides making other users’ CherryPy experience better, it is fun and cognitive activity. It sheds the light on how people really use CherryPy, how they start, where are remaining rough edges and what is the room for improvement.

Improvement

Inherently I’m an application guy. I pick out cog-wheels, tune them and make spin together. I don’t work on a library oftentimes, nor do I have much time for it, but because I really value CherryPy I want to have a way to do it, in case I have time and idea for improvement or fix. For instance, I want to add privilege drop plugin to cherryd and make other minor improvement to it.

When you file a bug report, you generally expect someone, a core contributor or so, to tell you whether it is generally valid or not. Then it may be re-prioritized, assigned to someone or put on a shelf to wait its day. Without this initial input it’s a little confusing. “Okay, no one replied. It should be working other way”, — you may think, at best.

Understanding the flow of one’s patch to CherryPy codebase is also important. Official site says fork CherryPy on BitBucket here and submit pull-request with your modifications. When it’s a bug either when it’s an enhancement? Who and where to ask a flow question, say about branching, or about eligibility of a change. There’s CONTRIBUTING.txt which didn’t appear long ago and it points to Jason Coombs’ post [17] about writing a perfect pull request. It is at least something but I think it can be more clear and CherryPy-specific. Also I think it should in the documentation, linked from the website’s main page.

Quality assurance

I think things like bug #1298 [16] should never happen to a project which is maintained in a responsible way and puts effort in quality assurance. Okay, I see tox.ini in root of the project. I clone it, and run tox. Duh, all environments have failed. Besides, not all the claimed environments are in the file. Moreover, it’s some kind of back in Soviet Russia story. Why the hell unit tests ask me questions!? Are they examining me instead? More surprises? ShiningPanda is dead [18], the same way as the link to it from the CherryPy website under online tests section (continuous integration is kind of standard term for the thing, as online tests refer more to IQ tests, psychology, et cetera).

To name it honestly, this is a reckless way of maintaining a project. Reckless to users whose applications will break with next release published this way. And interpretation result is as much time-consuming as to see whether the build page is green or red. No, I’m not promoting these cool badges one can see all around. I’m talking about the responsive flow where every contributor runs tests on all supported Python implementations (with Tox) before pushing her changes. If one is lazy to care and to set up all implementations, but has itching fingers to push some changes anyway, having the notification like: Hey, you’ve just broken CherryPy. Please fix ASAP would be helpful. I think for everyone who has write permission to the repository it makes sense to receive such notification. At very least the one who publishes CherryPy to the Cheese Shop should have rigid requirement to check CI page.

The same concern, with lesser strain, applies to the ReadTheDocs builds, which all have failed for last 3 months [19].

This is the problem which needs to be solved as soon as possible. In 2014 I tried the CI service called Drone.io, which supports the three major code hosing services. Well, it turned out to be usable (build page example [20]). I can help assist or set it up for CherryPy, once flow and role questions are answered.

Barrier to entry

Hopefully, one of biggest barriers I was facing myself seems to have passed. I mean that documentation link erosion where almost every single CherryPy-related link on the web leaded to 404 page. I think ReadTheDocs adoption is huge improvement, especially considering that new documentation, being clearly a better one, lacks some details that were covered in version 3.3. Having persistent, versioned and indexable documentation is very helpful.

I think, it should be stated more or less directly that if one wants a framework to make application design decisions it’s better off using something else, not CherryPy. Okay, here’s Joe who knows enough about Python. He’s conscious and is willing to make application design decisions and take responsibility for them. But he also has a deadline. Joe doesn’t expect his first real-world CherryPy application to be shiny from experienced CherryPy developer point of view, albeit functional from end-user’s. What’s the best way to help Joe create functional prototype quickly, and let him gradually improve the design and codebase of his application thereafter, along with CherryPy knowledge?

Let’s look at CherryPy knowledge as a tree. Each branch represents a functionality. Say a CherryPy tool branch. As it goes from the root, there’re nodes like Copy-paste this decorator to turn your handler result into JSON, What other tools are capable of?, What is a CherryPy tool, and how to write one?, Tool hook-points, types, priorities, etc.. The same is for other branches — from basic to advanced. And what will Joe benefit from the most is breadth-first traversing on the tree. Better if some of a few first levels are illustrated with runnable snippets. This way it’s possible to give overview, tunable blocks of code and directions for improvement. For the later is important to spur creativity, but not enforcing design decisions, in the way: start with this, improve by yourself.

I’ve just glanced over the latest documentation over again, and surprise, it narrates in very similar fashion! Sylvain has done a great job. But what is catching my eye is that there’s two separate tutorial series now: one in the source code [21], another in the documentation [22]. Even though both serve the same purpose, the former is much easier to maintain, covering with tests which leads to the bonus of better regression test suite. If the later’s textual flexibility can be sacrificed for less maintenance cost, I think, it’s possible to generate tutorial section from module docstrings and code, and merge the two.

However, both series deal with simplest cases and there’s a big gap between them and real-world application requirements. Here I also a snippet to base on directly or get ideas from to get a prototype faster, but with real-world tools. Like safe database access in threaded environment, URL routing and generation, multi-allplication design and so on. Sylvain’s CherryPy recipes [23] referred in the documentation several times may answer this to some extent, but in general I think it’s still in question. But also whether it should maintained in centralized fashion within CherryPy. Or if it’s more suitable for external source, like post on somebody’s blog or StackOverflow.

Contribution packages

I think Eric Larson’s suggestion about cherrypy-* pageages makes sense. For example there’s a package called cherrys [24] which is the good example. It’s a Redis adapter for CherryPy session handling. What’s important it has clearly defined scope and doesn’t affect an application design. This way it might have been named cherrypy-redis-session.

A package that is an implementaion of an CherryPy interface must be appropriate for packaging and sharing. Sharing a CherryPy application with clearly defined scope, like Dowser [25], is also beneficial.

On the contrary specific tools or plugins that affect application design shouldn’t be packaged. For instance, looking at Sylvain’s recipes they are all better to say recipes. Let’s look at Jinja2 recipe [26]. Besides having a class per file isn’t pythonic, and splitting a thing into plugin and tool isn’t always rewarding, it makes certian assumtion on how template file names are defined for the URL. Jina2 is a flexible template engine with notion of template inheritance. Real-world projects have dozens of templates and tend to orgianize them by role and use. Having some convention-over-configuration rule for template name is very handy. So I want to say it’s usually better to have a small and effiective application-specific CherryPy tool or plugin rather than trying to invent something overly broad and generic one.

Other changes

Bitbucket pages should have either actual supplementing information or no supplementing information. Menu community link [27] leads to wiki home page where version section has latest version of 3.2.5. Menu development link [28], which points to the repository overview page refers to Python 2.3, where the site’s version starts with 2.5+. Referring python setup.py install as installation command, which requires a manual download, ought to be changed to normal pip install cherrypy.

I think, minor changes on the website are needed:

no need in download link in the menu,
add install section after features which says pip install cherrypy,
community link is better to refer to the user group,
rename online tests, change the service link, use better sentence which will emphasize reliability.

About _cp module prefix. There’re some public classes like cherrypy._cptools.HandlerTool, which may be used in user code directly or as super classes. This isn’t correct naming, according to the underscore convention. So it’s a good idea to change it some day. Though as a PyDev user I can’t recall to any issues with it.

Wrap-up

I can conclude that there’re two things that I’m seeing problematic: quality assurance for various claimed enviroments CherryPy to support and lack of contribution guiadance, specifically in the form of bug tracker supervisor. The rest feels normal, if that makes sense and as it is usually the case needs small constant improment and polishing.

[1]	(1, 2) http://cherrypy.org/

[2]	https://groups.google.com/forum/#!topic/cherrypy-users/lT1cxovGyy8

[3]	https://groups.google.com/forum/#!topic/cherrypy-users/MjrIZ_jljRQ

[4]	(1, 2) http://nichol.as/benchmark-of-python-web-servers

[5]	(1, 2) http://stackoverflow.com/questions/tagged/cherrypy

[6]	https://bitbucket.org/cherrypy/cherrypy/wiki/ZenOfCherryPy

[7]	https://www.python.org/dev/peps/pep-0020/

[8]	https://bitbucket.org/cherrypy/cherrypy/issue/1331

[9]	https://github.com/niwibe/django-sse

[10]	https://lincolnloop.com/blog/architecting-realtime-applications/

[11]	http://stackoverflow.com/q/25845161/2072035

[12]	https://ws4py.readthedocs.org/

[13]	https://docs.python.org/2/howto/sockets.html#performance

[14]	https://bitbucket.org/saaj/cherrypy-webapp-skeleton

[15]	https://bitbucket.org/saaj/qooxdoo-website-skeleton

[16]	https://bitbucket.org/cherrypy/cherrypy/issue/1298

[17]	http://blog.jaraco.com/2014/04/how-to-write-perfect-pull-request.html

[18]	http://shiningpanda.com/shiningpanda-ci-clap-de-fin.html

[19]	https://readthedocs.org/builds/cherrypy

[20]	https://drone.io/saaj/qooxdoo-cherrypy-json-rpc

[21]	https://bitbucket.org/cherrypy/cherrypy/src/tip/cherrypy/tutorial/?at=default

[22]	http://cherrypy.readthedocs.org/en/latest/tutorials.html

[23]	https://bitbucket.org/Lawouach/cherrypy-recipes

[24]	https://pypi.python.org/pypi/cherrys

[25]	http://www.aminus.net/wiki/Dowser

[26]	https://bitbucket.org/Lawouach/cherrypy-recipes/src/tip/web/templating/jinja2_templating/?at=default

[27]	https://bitbucket.org/cherrypy/cherrypy/wiki/Home

[28]	https://bitbucket.org/cherrypy/cherrypy/overview

Qooxdoo “next” and targeting website development

2014-12-08T00:00:00+01:00

Initially, this post was meant to be an open letter to Martin Wittemann, an urge to do not break the only robust and consistent JavaScript development environment, in sense of bug #8618 [1]. How robust and consistent? Most importantly it is about:

Code structure: modern feature-rich object model with properties, events, interfaces, mixins and more, namespace tree, data-binding,
Dependency management: automatic and transparent dependency resolution, component packaging,
Deployment: code optimisation, single- and multi-part builds,
Standard library: i18n, CLDR, high level BOM API, normalised events, normalised ECMAScript, test infrastructure, API documentation, feature detection.

All this comes out-of-the-box, builds layer on layer, is seamlessly integrated and handed to an end developer on a silver platter. The codebase is one of the best quality I have ever seen. It is readable, comprehensible, possible to extend, and is a huge a knowledge resource per se. And note, I am talking about tools and environment, not UI. Like former Qooxdoo developer, Fabian Jakobs, stated in one of his presentations [2]:

Don’t just look at the widgets, look at the development tools as well.

Luckily, as was recently stated in Website fundamentals: next challenges [3], the “next” (branch and then a separate repo [4]; later referenced Next) turned out to be a experimental website-oriented fork of Qooxdoo, which doesn’t mean to be merged back to Qooxdoo. At least any time soon. Panic is soothed, impending doom doesn’t overhang the good times. It’s time to dissect and discuss.

The bug report

What was meant to break? The bug report mandated to rewrite Qooxdoo test infrastructure with callback spaghetti of mocha.js. Callback spaghetti?! When most code structure is a callback in callback in callback and so on, with the state expressed in free closure variables spanning across nesting levels, it is a callback spaghetti. And for average JavaScript code out there this structure practice is very common, which is nothing good though, but for Qooxdoo it might have been an awful decision, replacing clean, object-oriented code with that. But since Next is not Qooxdoo it may depend on some factors.

In the report Martin told:

There has been some hard decisions for me to take lately but I’m still convinced that not trying to be the expert for everything gives us the opportunity to focus on our strengths.

And listed some features that supposedly impressed him:

pretty printing,
scaffolding over to XML handling,
Levenshtein distance calculation,
GUI toolkits.

First is simple, second is needed only for recorded tests (e.g. Selenium), third is already there with qx.util.EditDistance. And forth is quite a thing Qooxdoo was made for. So these features are kind of false argument.

Moreover, mocha.js is 5k SLOC, qx.dev.unit is 4.4k SLOC, testrunner is 3.9k SLOC. Including these numbers and taking into account more expressive Qooxdoo code, either Martin is trading more for less, or it is just an expensive experiment.

Anyway framework consistency is absolutely a strength. Test code is a canonical source of thruth about an API. Test code is required for maintaining a complex application (which most websites are not though). And test infrastructure should not be compared to a disposal widget like qx.ui.embed.HtmlArea which wasn’t quality in first place, required special knowledge to maintain, and most importantly it isn’t hard create a wrapper for TinyMCE implementing qx.ui.form.IStringForm and qx.ui.form.IForm. Since that moment it is transparent and seamlessly integrated. Making test code an alien in a framework, makes it seamful, inconsistent, and what is worst can discourage developers from writing tests. Well, of course unless all the rest code is the spaghetti.

Last note here is how framework consistency from language perspective changes in the course of recent time. Qooxdoo branch_3_0_x roughly has nothing but Qooxdoo JavaScript code for a browser and Python code for generator. Next has Qooxdoo JavaScript code and mocha.js callback spaghetti test code for a browser, CommonJS code for generator that varies being object-oriented prototypal code, procedural code, and Qooxdoo object-oriented code, which also doesn’t exclude some Python code still necessary for new generator to function. So what’s clear, having hard time maintaining this is one of likely outcomes.

Goals

Andreas Ecker wrote in the post [3], that they have been looking into the challenges of a more lightweight, website-oriented approach to web applications. Sounds quite ambiguous and curious at the same time. Lightweight in byte size? Website-oriented in a non-widget UI? At the same time, it sounds as odd as arithmetic approach to differential equations. Then he expanded:

Reasons are two-fold: primarily, we are involved in company-internal projects in that very domain. Secondly, by touching core components of qooxdoo we can experiment with benefits of web technologies and concepts available today.

First thing, considering 1&1’s “very domain”, is probably something like Amazon AWS management UI. A web-application which spans multiple pages, where each page has some HTML from server and complementing JavaScript code. So it may be looked at as a “website-oriented approach”. Second thing again points to an experiment.

qxWeb

Qooxdoo targets websites since 2.0 which was released in June 2012 [5]. Initially I was enthusiastic about qxWeb, I thought it’ll be another virtue of the framework, and I still think the idea to target websites is right as even at mid-scale they benefit from most of Qooxdoo non-UI features. But the design and development of qxWeb seems to me misguided and mostly pointless now.

A year after qxWeb release, the time I was looking at it from various angles, trying to find its strength and way to integrate it in my flow to make it handle infrastructure task I used to handle with Qooxdoo in case of web application, I submitted the bug report [6] that proposes design that enables website development with Qooxdoo (native application type [14]), points qxWeb shortcomings and its possible shortcut-purpose, and the problems to be solved for normal operation. The biggest problem with qxWeb is that it wasn’t designed to be a Qooxdoo thing, but rather be jQuery with API done right. Long story told short: it’s not for writing an application, it’s for scribbling ad hoc per page scripts. And I would prefer qxWeb over jQuery for writing one-off script, but in general jQuery is de facto standard in DOM patching and I don’t see any significant niche for qxWeb as yet another DOM patcher plus this and that. Moreover even for shortcut-purpose in a Qooxdoo application, qxWeb is still an alien, because it needs manual dependency declaration for its “modules”. Qooxdoo way of DOM patching was qx.bom.Collection [7], removed in version 3.0. I think it was a mistake.

Starting with Qooxdoo 3.5 [8], there was an addition to qxWeb. Since then it supported low-level widgets. Such widget expects certain HTML structure and classes, which later is animated on widget instance construction. Basically, qxWeb started to mimic jQueryUI as well, not just jQuery. Not that I want to say it’s useless, but rather it’s reinvention of the wheel, reimplementation of a task that was already implemented long ago, and all who wanted such functionality already have it at hand. What’s worse the reimplementation is being done with outdated technology. DOM binding, MVVM [9], libraries emerged in 2009-2010, and at the moment of Qooxdoo 3.5 release, late 2013, were a commodity, as well as understanding that a lot of manual DOM patching is a chore, that can be automated with a view model. Bug #8101 [10] lead me to working KnockoutJS [11] adapter for Qooxdoo, but at the framework side it didn’t gain traction, except that Martin confirmed importance of the topic, and that also it’s the topic of his master thesis [12].

I want to summarise this section quoting Martin’s focus on our strengths. How the whole thing looks to me now, is that in area of website development Qooxdoo team tries to avoid its strengths at all costs. No build tools, no view model, no application design having great tools, data layer and application design experience. Instead Qooxdoo team is toying a jQuery/jQueryUI.

A Qooxdoo website today

Even though initial changes in Next doesn’t seem anything promising, I’m interested what the experiment may yield. But to address the need of a robust development environment for a website today I want to point again to the bug #7481 [6] and its proof of concept project [13].

Eventually all blocking bugs for #7481 were resolved and a Qooxdoo native application [14] since 3.5.1 was already fine as a basis for a website codebase. From my side I’ve been experimenting with the approach on a few real websites and result are good. When all website’s JavaScript code is centralised, it really starts to feel like an application, thusly significantly improving maintainability. Late autumn 2014 I’ve written documentation for the project, backported minor improvements and made it standalone.

qooxdoo-website-skeleton [13] is a mixture of a tutorial and skeleton. Because initial configuration is tricky, when starting from a scratch with native application, it’s easier to fork and remove a few unused files. Here I would like to tell about two aspects that Andreas pointed to (see above), and for the rest you can follow the link. First, a website approach to UI here is to provide a choice. One can use qxWeb DOM patching facilities or qxWeb widgets. One can use KnockoutJS adapter and use Qooxdoo data models as view models (MVVM). There’s also no restriction for using jQuery or KnockoutJS as is or whatever it makes sense to use.

Second, is being lightweight. Without external libraries Qooxdoo native application build with basic qxWeb modules is over 100 kB gzipped. For example boot part for the project is 361/110 kB (optimised/gzip -6). I would agree it is a little too much for a website yesterday, but today I don’t see this concern reasonable. Here are three hello pages for popular social websites requested from desktop Firefox.

website	size, kB	requests
twitter.com	156	3
linkedin.com	180	16
facebook.com	615	16

And here are home pages for big e-commerce websites.

website	size, kB	requests
ebay.com	114	2
alibaba.com	143	7
amazon.com	560	30

These show that having roughly over 100 kB is a norm today. And if further optimisation of Next will go as feature-size trade-off, then it wouldn’t probably make sense.

Epilogue

Before official announcement Next’s bug #8618 was in line with other dubious decisions taken since Qooxdoo 3.5. Before 3.5 I could agree to virtually every decision taken by the team, as there was clear reason, argument and pragmatic approach overall.

Now it’s mixed. There’s still fundamental work like migration to pointer events, but also several other topics that look like a waste of time. Like what? Like rewriting the generator to NodeJS and GruntJS [15]. Surprise! There’s ongoing shift to gulp.js and GruntJS is no longer cool JavaScript thiggie. And I’m curious to see how Qooxdoo team will handle such situations in general. Fancy toys change in the public sandbox fast [17]. Will you try to catch up every new cool replacement or will maintain an unpopular one?

Like, unconditional RESOLVED NEVER [16] set to all old bug. No prioritization, there is no past, the future’s so bright. The team briefly state something about turning to an agile development, and happily turned to developing new features. And of course, no doubt developing features is fun, PR is easier with less open bugs, and resources are always limited. But when new developers stumble the same bugs again and re-report them, it is a failure. I don’t really understand it, so since that change I’m just discouraged to file new bug and I don’t. If no one fixes them, and it takes time to analyse, make it reproducible and propose a solution, I probably shouldn’t, right?

I can whine more, but I want to point to two origins of priorities. It’s understanding the scope and understanding the users. First, is a universal JavaScript framework. How universal? And what is JavaScript here? JavaScript like in Netscape? Or like ECMAScript in NodeJS? Should we expect Qooxdoo approach to URL routing or database connection pooling? In second, I see opposite sides. One commenters in various places complain that Qooxdoo doesn’t play well with JavaScript ecosystem. I don’t think they are Qooxdoo users at all, but ones who couldn’t handle it. You know, you can’t do your Qooxdoo work copy-pasting from StackOverflow, you can do jQuery work this way. On the other side are users, including me, who say:

Dear Qooxdoo, we don’t really give a damn about JavaScript ecosystem, you are our only concern. We come from various established application development platforms, in sense “Swing/qt/Cocoa for the web”. We value design and maintainability. Please, keep on avoiding us from it.

That is to say. I’m an optimist, and there’s nothing parallel to Qooxdoo. So I hope the team won’t forget its strengths and its users, and will keep on the path of solving real problems.

[1]	http://bugzilla.qooxdoo.org/show_bug.cgi?id=8618

[2]	http://www.slideshare.net/fjakobs/masterin-large-scale-java-script-applications

[3]	(1, 2) http://news.qooxdoo.org/website-fundamentals-next-challenges

[4]	http://github.com/qooxdoo/next

[5]	http://news.qooxdoo.org/qooxdoo-2-0-released

[6]	(1, 2) http://bugzilla.qooxdoo.org/show_bug.cgi?id=7481

[7]	http://demo.qooxdoo.org/2.0.4/apiviewer/#qx.bom.Collection

[8]	http://news.qooxdoo.org/qooxdoo-3-5-released

[9]	http://en.wikipedia.org/wiki/Model_View_ViewModel

[10]	http://bugzilla.qooxdoo.org/show_bug.cgi?id=8101

[11]	http://knockoutjs.com/

[12]	http://www.slideshare.net/wittemann/data-binding-in-qooxdoo

[13]	(1, 2) https://bitbucket.org/saaj/qooxdoo-website-skeleton

[14]	(1, 2) http://manual.qooxdoo.org/3.5.1/pages/development/skeletons.html#native

[15]	http://news.qooxdoo.org/tool-chain-the-next-step

[16]	http://www.eclipsezone.com/eclipse/forums/t83053.html

[17]	http://readwrite.com/2014/12/04/node-js-fork-io-js