Using jq to work with JSON

I often see processing of JSON on the shell through tools like grep and awk. Sure, this works but usually it depends on the output having only one item per line and not having the keywords you are grep'ing for in other places.

So I was very delighted to learn that there is a better way through the commandline tool jq.

With jq you can filter, slice and alter JSON. And you get some highlighting of the output which makes it nicer to read than plain text!

Here are two small examples I tried out today to process output from opsi-admin:

opsi-admin -d method host_getHashes | jq '.[] | .id + ": " + .lastSeen'

This will list all your clients along with the time they have been last seen. The output on my test machine looks like this

...snip...
"vtest16r.uib.local: 2013-09-08 21:28:50"
"vtest18-w2k-r.uib.local: 2008-05-24 17:58:35"
"zedach.uib.local: 2011-01-19 15:47:18"
...snip...

And to see what installation status the products on your clients have you can use this snippet:

opsi-admin -d method productOnClient_getHashes | jq '.[] | "On " + .clientId + " the product " + .productId + " is " + .installationStatus'

Again with some example output:

...snip...
"On pcbon4.uib.local the product xnview is installed"
"On fscnoteb1.uib.local the product xpknife is unknown"
"On vtest16r.uib.local the product xpknife is installed"
"On fscnoteb1.uib.local the product yed is installed"
"On hpnoteb1.uib.local the product yed is installed"
"On pcbon4.uib.local the product yed is installed"
...snip...

Creating new JSON data is also possible:

opsi-admin -d method productOnClient_getHashes | jq '.[] | {hostId: .clientId, productId: .productId, status: .installationStatus}'

This return something like the following (look ahead: no JSON list but multiple standalone dictionaries instead):

...snip...
{
  "hostId": "vmex12w10x64c.uib.local",
  "productId": "winscp",
  "status": "installed"
}
{
  "hostId": "vmex12w10x86.uib.local",
  "productId": "winscp",
  "status": "installed"
}
...snip...

If you want to know more I'd suggest to start with the documentation. If you do not want to install anything on your machine there even is an online version to play with.

Smoother transition to opsi 4.1

Work towards a public opsi 4.1 release is making progress. Today's release has brought us a small step closer to a seamless migration from 4.0 to 4.1.

The release includes an opsi-atftpd which provides opsi-tftpd and the updated opsipxeconfd, opsi-depotserver and opsi4ucs all require this. This makes it easier to switch to a different tftpd without requiring any further user interaction.

opsi 4.1 will be released in a separate repository. Our current migration path is adding the new repository and then run apt, yum or zypper with their upgrade options to migrate to the new version. To finish the migration you will have to run opsi-setup with the update parameter for the backends you use and this is it.

Reproducable Debian Packages

If you are following tech news you probably read that this week Debians efforts in having reproducible builds resulted in making it a policy that packages should be reproducible. In short this means that given the source could anyone should be able to reproduce the resulting packages that are distributed through the Debian repositories.

I think this is very good!

Over at the opsi.org GitHub page you can have a look inside the repositories and you will find all parts that are needed for building. Whenever we release a new version the commit used for that package receives a tag. This is useful not only in pinning down changes between different versions but also if we want to provide patches for specific versions.

Regarding git we still have some repositories left to migrate from Subversion but I am confident to get this tackled until the end of the year. Then everything will be accessible at one location - again.

News from the Machine Room - Summer Season 2017

This is just a small update from the machine room.

Summer season has hit the office and many of our co-workers enjoy their holidays. The developer team is helping out in the support to not have any shortcomings for our customers.

Other than that we are in preparation of a new release for opsi 4.0 that will bring support to some newer distributions - Debian 9, openSUSE Leap 42.3 and Univention Corporate Server 4.2.

Unfortunately this delays the release of opsi 4.1. The upside is: we can have support for these distributions right at the release. While some preparations still need to be done we are making good progress.

I hope to be back soon with longer posts. Until then: enjoy your summer!

opsi4instituts conference

On 18th-19th May 2017 the first opsi4instituts conference took place at the Georg Eckard Institute. This conference had a focus on opsi packaging and was a great meet & greet with opsi users from institutes taking part in the German research network Deutsches Forschungsnetzwerk. The response to the first event was amazing. A few days after announcement all seats were already taken.

We want to give special thanks to Detlef Krummel from the project leading group of opsi4instituts and the Georg Eckard Institute. They organized and financed this conference. The range of participants was wide. It varied from people interested in opsi up to long time opsi users.

The program was great and had bits for everyone. At first Detlef talked about security in client infrastructures and how opsi can help with it. Thomas Besser held a talk about the experiences and requirements in the packaging work from opsi4instituts. I myself presented the releasemanagement of opsi, quality assurance, the new opsi 4.x test environment, what is new in opsi 4.0 and what will come with opsi 4.1. In addition I also had a small track on the second conference day. This track focused on packaging best practices and made a live coding session with the webservice and methods of opsi. uib's technical partners c.a.p.e. IT GmbH and Univention GmbH also participated in the conference. Rico Barth (c.a.p.e. IT GmbH) presented the opsi-inventory integration in kixdesk. Ben Haberhauer (Univention GmbH) talked about opsi integration on Univention Corporate Server (UCS).

We have had two large open discusions. In both all of the participants had the chance to discuss most of the questions directly with the project leaders of opsi4instituts and the vendor of opsi. Many experiences were shared at this conference. After the first day a social event was held in the old town of Braunschweig. The city is great and I can recommend everyone to plan a visit there. Also the "Source Hollandaise" was great ;)

It was also amazong to see what GEI has done. We had a tour through the library of GEI. There we learned what the GEI does in regard to worldwide schoolbook research. It was very interesting to see how different the world is regarding this topic. In my personal opinion this is a very important research. Thank you for this valuable insights.

Many ideas were born at this conference and we will try to meet again in 2018. We will announce more information ASAP.

opsi4instituts conference participants

Cross-distribution packaging: Providing a systemd service

We use the upcoming opsi 4.1 to remove obsolete raiminigs that accumulated over time. This is also a good time to get the packaging up to date.

On systems running systemd we used some hacks with opsi 4.0 that allowed us to treat support for systemd as an optional feature. Now we want to make systemd first class, remove hacks and use the standards that the various distributions recommend.

I want to show how we end up packaging opsiconfd to work with systems using Debians .deb package format and those using .rpm. .deb files work the same across all major distributions and we serve Debian, Ubuntu and Univention Corporate Server (UCS) with the same setup. On the RPM-side we focus on the distributions provided by SUSE (SLES, openSUSE), Red Hat (RHEL) and the open source variant CentOS.

Read more…

ARMed

We have new stuff to play with! Since last week we have two ARM boards. The official openSuse Build Service can't make Debian and Ubuntu packages for the ARM architecture. The OBS lacks the ARM repositories within the Debian and Ubuntu projects from where the packages are downloaded. Luckily we have our own internal instance. Let's see where this will get our beloved opsi.

BananaPi M2 Ultra & Odroid-Xu4