First service endpoint running with Python 3

I am really happy with the progress we are making with supporting Python 3. Our internal build server already serves the first packages that are running on Python 3 only. There are still errors popping up where we need to port parts but this is what I expected earlier.

The previous weeks saw some feedback-based improvements to the opsi monitoring connector in opsi 4.1. Because I was already working in the codebase and it is comparably small I decided the first endpoint I want to port to Python 3 will be /monitoring.

Read more…

New appliance based on Ubuntu 18.04

For a very long time we have been providing a preconfigured opsi server (often referred as opsivm) in the form of a virtual machine, which can be integrated under various virtualisation systems including Virtualbox and VMWare. After the server has been started for the first time, a script is executed to setup the last configuration settings. These are, among other things name, domain, IP of the server, the standard gateway, the DNS server and also the passwords for the already created users. The opsi server is then ready for use and only needs to be filled up with the opsi standard packages.

We always try to keep the opsi server up to date, both in terms of the opsi version and the Linux distribution used. The latest version has been changed from being based on Ubuntu 16.04 to Ubuntu 18.04.

Network configuration

Since Ubuntu 18.04, netplan has been used instead of ifconfig as the default network configuration. After a first attempt to disable netplan and keep the configuration procedure using ifconfig as before, this attempt has been discarded because this solution would require installing additional programs as well as making more adjustments. So the decision was made to use the netplan configuration.

The configuration for netplan can be found in /etc/netplan/config.yaml:

network:
  version: 2
  renderer: networkd
  ethernets:
    enp3s0:
      addresses:
        - 10.10.10.2/24
      gateway4: 10.10.10.1
      nameservers:
          search: [mydomain, otherdomain]
          addresses: [10.10.10.1, 1.1.1.1]

By making adjustments to our start script we are able to use a template based on the default configuration. This template is patched with the values as retrieved during the first startup.

network:
  version: 2
  renderer: networkd
  ethernets:
    enp3s0:
      addresses:
        - @ip@/@cidrnetmask@
      gateway4: @gateway@
      nameservers:
          search: [@domain@]
          addresses: [@dns@]

A particular challenge was the network mask, which is required to be in format 255.255.255.0, but in config.yaml must be given as in CIDR notation including the network address as 10.10.10.1/24.

Once the template is filled the following call activates the network settings from config.yaml:

netplan apply

Deploying a root shell

Another change was the way we made the use of a terminal with root privileges available via a desktop icon.

Most of the tasks on an opsi server can be performed by a dedicated user called adminuser. If system administrator rights are necessary, there is a desktop call to open a shell with root rights. So far this has been done using the gksu tool which is considered obsolete and is therefore no longer supported in the latest versions of Debian and Ubuntu.

For this reason we now start our root shell with the following call:

lxterminal --title "root shell" --command "sudo -s"

Testing the new opsi-appliance

Are you curious? Download the latest version here and then follow the Getting Started.

Have fun

Making parts of opsi-package-updater re-usable

This weeks release of opsi-utils 4.1.1.26 consists mostly of internal changes. We sometimes do this to keep up with current developments but this time it is a little bit different. This change lead to shrinking the size of opsi-package-updater by a huge margin.

How did this come you might ask. Well, it all started a while ago when a customer got in contact through the support because he wanted to use some functionality from opsi-package-updater with his own script. Discussing the options it came clear that the best way to achieve this was to make sure that the functionality of opsi-package-updater becomes available for importing in another Python module. Luckily the customer was up for this route even though this would not be the fastest option but the one that provided the best long-term support for such an solution.

To achieve this we added the module OPSI.Util.Task.UpdatePackages to python-opsi. This now includes the majority of code that is opsi-package-updater. Besides becoming reusable we now also can provider proper unittests for the different components. Working on the module I have split some components up to achieve looser coupling. I'd like to refactor some internal workings of this in the future but right now my focus is on opsi supporting Python 3.

This brings us to the other changes that happend in opsi-utils. The small changes that have been made are to achieve better compatibility with Python 3. For most tools it will now be enough to change the shebang line to use Python 3 to work with python3-opsi. Some more work will have to be done there but we can already cross more things of our todo list.

opsi-package-updater: Failing on locked products

With an software as old as opsi it is sometimes inevitable to stumble over something and then be left scratching your head.

One of this cases I recently discovered in opsi-package-updater. Whenever an package is installed it is made in a way that ignores the lock of the product on the depot.

The lock is in place to avoid problems where multiple tools would (un)install the same product. This could result in problems which leave the system in an undefined state regarding the product installation - something that should be avoided!

With opsi-utils 4.1.1.24 we do not ignore the lock anymore. If a locked product is detected no installation will be made but we will abort with an error instead. This may lead to problems coming to the surface which may have been hidden before, especially when having a heavily automated setup, but we think not knowing about the problem is far worse. The problems probably have been there before but were usually not in sight.

If you run into such an problem now you should take a look at the logs and find out what is causing your problem. Once you figured this out and resolved it you can use opsi-package-manager --install --force /var/lib/opsi/repository/<packagefile>.opsi to make install the package in way that ignores the locked state of the product. If everything works you can run opsi-package-updater again to finish what was started.

Improved backup handling

Small things do make a difference. This weeks release brought some small changes to opsi-backup that should make the handling of opsi backups easier in day-to-day life.

The first change improved automatic detection of restorable data. There is no need anymore to pass what backends should be restored as the new mechanism will by default restore all backend data that is found in your backup file. This boils a restore for most cases down to opsi-backup restore <backupfile>.

The second change is that if any combination of options is given that would lead to not performing a restore the process will be aborted which is much easier to spot.

And the last change is that it is now possible to get a short listing to see what data (backendtype and if there is configuration data) is in a backup with the command opsi-backup list <filenames>

Go give it a try and send us some feedback! All you need is opsi-utils 4.1.1.22 or newer which can currently be found in the testing branch.

API: What is the signature of a method?

The opsi API does not get changed often but it happens. Additions happen as part of the normal development cycle but removals usually only appear as part of larger releases like the release of opsi 4.1. There is also the chance of a changed method signature. The same rules as above apply to the method signatures.

If one of the changes affects a method you use there are different possibilities to handle them.

Checking the version

If you know what version a change appeared in you can use this knowledge to check for a specific version. Reading the changelog of python-opsi is usually a good starting point to find out what changed. This is the module that contains the business logic and API.

The API method backend_info will return a JSON object that contains opsiVersion which is basically the version of python-opsi.

You can easily try it for yourself:

curl -X POST --user youruser --data '{"params": [], "id": 1, "method": "backend_info"}' https://localhost:4447/rpc

This approach works very well for the object-oriented API methods. These have the form objectType_action.

Checking method signature

In opsi we also have dynamically loaded backend extensions. The official extensions are distributed through the package python-opsi but nothing keeps you from using extensions from different versions. There could also be custom extensions that are not distributed through python-opsi. Methods defined through extensions usually don't follow the naming convention of objectType_action but there is no enforcing of this. As you see checking just the version may not work when the method originates from an backend extension.

Lucky for us it is possible to get a description of the available API methods and their signatures through another API call: backend_getInterface.

curl -X POST --user youruser --data '{"params": [], "id": 1, "method": "backend_getInterface"}' https://localhost:4447/rpc

This lists all available methods with their name, params and defaults among others. The values listed in params show how many parameters a method accepts and what their names are. If a parameter has one leading asterik (i.e. *attributes as a parameter of host_getObjects) then this indicates that this is an optional parameter. If a parameter has two leading asteriks (i.e. **filter as a parameter of host_getObjects) then this indicates that the value is optional and a JSON dictionary is expected. If defaults are listed then it contains the default values of optional parameters excluding those with two asteriks.

Summary

Handling multiple API version with opsi isn't too hard but it all depends on what was changed.

With backend_info you can easily check the version.

The powerful backend_getInterface will list the methods exposed through the API. This can be used to handled changed methods but it can also be useful to see what methods are available on the server.

opsi-linux-bootimage and new Dell devices

Within the last week we encountered a problem with our opsi-linux-bootimage in combination with new Dell devices. This problem results in a black screen and a seemingly dead machine whenever a Windows or 32bit Linux netboot installations runs. Furthermore an analysis is hard as the machine runs in a black screen of death (BSOD) before even properly starting the opsi-linux-bootimage kernel itself. However we have a workaround for this issue: use the 64Bit bootimage.

By default setting a windows netboot product on setup, triggers the opsipxeconfd, which uses a template to generate a specific PXE pipe. This file is within the opsi tftpboot directory. This directory is /tftpboot/linux on most opsi supported distributions (or /var/lib/tftpboot/opsi on SUSE based distributions). Within this tftpboot directory the PXE templates are found within the pxelinux.cfg directory. The specific template is the file install. This file has to be modified to use the 64bit kernel and miniroot of the opsi-linux-bootimage. The template should look like this after the modification:

default opsi-install

label opsi-install
  kernel install-x64
  append initrd=miniroot-x64.bz2 video=vesa:ywrap,mtrr vga=791 quiet splash --no-log console=tty1 console=ttyS0

The downside of this modification is that after an update one has to modify the template again. We are currently working on a permanent solution.

opsiconf 2018

It is a week already since the first ever opsiconf took place in Mainz, Germany.

I may be a bit biased as I was involved in some smaller parts of the preparation but my focus before the conference was to have a stable release for opsi 4.1 out. However I don't think that you had to be involved to have a good time.

The thing I enjoyed the most were the discussions with various opsi users. I got a lot of input and was able to help users with my knowledge. There was a great feeling of community - it didn't matter if you are an opsi veteran or just starting with opsi - everyone was welcome.

It was great for me to hear how many people actually use the API to automate their workflows with opsi. There have been lighting talks from users showing how they integrated opsi into their system which I found very inspiring!

It sure boosts my wish to improve the API documentation to make the usage simpler - especially for newcomers. If you have any questions now don't hesitate to post them in the fresh opsi development subforum.

The conference was also used to announce the public repositories for opsi on ARM devices. Head over to the announcement in the forums to read more.

I am really happy with how everything went. There is still a bit of room for improvements but it was a good debut in my eyes. If there will be another conference (I hope there will be) I'd love to see more talks given by users next time.

PS: This wasn't the first opsi-centered conference. There hase been the opsi4instituts conference in 2017. Our community sure is busy :)