Improving our integration tests

In the last weeks I focussed on our integration test environment. For opsi 4.1 we are increasing the amount of tests we run because in addition to the tests descriped here we added automated migration tests. All these tests should not only be reliable but they also should be fast so that it is possible to run them often.

One downside of our test implementation was that it spent a lot of time in sleep. If we know a reboot would happen then we inserted a sleep 120 to wait 120 seconds and then hopefully the machine was up. Two minutes are a lot of time but sometimes during heavy load that wasn't enough for the machine to be up and accessible. So in some places this got increased to 180 to make really, really sure that the machine could be reached. This made tests a little bit more reliable but also slower. The overall runtime for one of these tests is usually somewhere between two and four hours. Three minutes do not seem like much but there is usually more than one sleep in each test.

So I went on to replace all the sleeps with something better. As mentioned before the sleeps are usually inserted when we wait for a port being reachable. This usually is SSH or opsiclientd which we then use for further work.

My attempt was to write a small Python script that does the waiting for us. Connection attempts are made until the port becomes reachable with a short delay inbetween checks. Once a port is reachable it exits. If a connection can not be made after five minutes it is assumed that something went wrong and the script ends with an non-zero exitcode. The exitcode helps in using this script inside our jenkins stages as this marks the step as failed.

The script does have some tricks though. If a port is reachable right away it will exit right away. The script also understands different kinds of waiting. It may wait for a port to come up, a port to come down or a reboot to happen (port goes down and then comes up again).

With the script in use we managed to reduce the overall runtime by some minutes. Not totally overwhelming but in the end every minute counts. In addition this also improves the reliability of our tests. There is another neat advantage: most of the tests are implemented as functions and the changes made for the tests of opsi 4.1 also effect the runtime of our opsi 4.0 tests.

Using jq to work with JSON

I often see processing of JSON on the shell through tools like grep and awk. Sure, this works but usually it depends on the output having only one item per line and not having the keywords you are grep'ing for in other places.

So I was very delighted to learn that there is a better way through the commandline tool jq.

With jq you can filter, slice and alter JSON. And you get some highlighting of the output which makes it nicer to read than plain text!

Here are two small examples I tried out today to process output from opsi-admin:

opsi-admin -d method host_getHashes | jq '.[] | .id + ": " + .lastSeen'

This will list all your clients along with the time they have been last seen. The output on my test machine looks like this

"vtest16r.uib.local: 2013-09-08 21:28:50"
"vtest18-w2k-r.uib.local: 2008-05-24 17:58:35"
"zedach.uib.local: 2011-01-19 15:47:18"

And to see what installation status the products on your clients have you can use this snippet:

opsi-admin -d method productOnClient_getHashes | jq '.[] | "On " + .clientId + " the product " + .productId + " is " + .installationStatus'

Again with some example output:

"On pcbon4.uib.local the product xnview is installed"
"On fscnoteb1.uib.local the product xpknife is unknown"
"On vtest16r.uib.local the product xpknife is installed"
"On fscnoteb1.uib.local the product yed is installed"
"On hpnoteb1.uib.local the product yed is installed"
"On pcbon4.uib.local the product yed is installed"

Creating new JSON data is also possible:

opsi-admin -d method productOnClient_getHashes | jq '.[] | {hostId: .clientId, productId: .productId, status: .installationStatus}'

This return something like the following (look ahead: no JSON list but multiple standalone dictionaries instead):

  "hostId": "vmex12w10x64c.uib.local",
  "productId": "winscp",
  "status": "installed"
  "hostId": "vmex12w10x86.uib.local",
  "productId": "winscp",
  "status": "installed"

If you want to know more I'd suggest to start with the documentation. If you do not want to install anything on your machine there even is an online version to play with.

Smoother transition to opsi 4.1

Work towards a public opsi 4.1 release is making progress. Today's release has brought us a small step closer to a seamless migration from 4.0 to 4.1.

The release includes an opsi-atftpd which provides opsi-tftpd and the updated opsipxeconfd, opsi-depotserver and opsi4ucs all require this. This makes it easier to switch to a different tftpd without requiring any further user interaction.

opsi 4.1 will be released in a separate repository. Our current migration path is adding the new repository and then run apt, yum or zypper with their upgrade options to migrate to the new version. To finish the migration you will have to run opsi-setup with the update parameter for the backends you use and this is it.

Reproducable Debian Packages

If you are following tech news you probably read that this week Debians efforts in having reproducible builds resulted in making it a policy that packages should be reproducible. In short this means that given the source could anyone should be able to reproduce the resulting packages that are distributed through the Debian repositories.

I think this is very good!

Over at the GitHub page you can have a look inside the repositories and you will find all parts that are needed for building. Whenever we release a new version the commit used for that package receives a tag. This is useful not only in pinning down changes between different versions but also if we want to provide patches for specific versions.

Regarding git we still have some repositories left to migrate from Subversion but I am confident to get this tackled until the end of the year. Then everything will be accessible at one location - again.

News from the Machine Room - Summer Season 2017

This is just a small update from the machine room.

Summer season has hit the office and many of our co-workers enjoy their holidays. The developer team is helping out in the support to not have any shortcomings for our customers.

Other than that we are in preparation of a new release for opsi 4.0 that will bring support to some newer distributions - Debian 9, openSUSE Leap 42.3 and Univention Corporate Server 4.2.

Unfortunately this delays the release of opsi 4.1. The upside is: we can have support for these distributions right at the release. While some preparations still need to be done we are making good progress.

I hope to be back soon with longer posts. Until then: enjoy your summer!