So, if you have been looking around my site, you will have noticed that I am big fan of and a heavy user of Ansible. Indeed, as a computer engineer (or "software systems engineer", depending on which way you look at the coin), I view it as one of the essential tools in the DevOps/SysAdmin/Developer space to extend the concept of "DRY" (as in "Don't Repeat Yourself") out of the world of writing software and into the larger space of working with computers as a whole. After all, while it is one thing for some simple task such as make clean ; make build && sudo make install, even that is pushing it if you really get down to it. It is far simpler to just do your commits for a change you have done your minimal test build and unit test run, push the changes up to the main repository, and let something like Jenkins or some other CI/CD pipeline to run the real tests, produce any reports such as for coverage, and push to the test server, or even production, based on the branch. The reason I bring this up is that the couple of weeks, I have been working on a Ansible playbook to take a bare minimal RHEL based install (in this case, an actual RHEL 9 machine, thanks to my free Red Hat Developer Subscription) to replace what my supervisor at work calls a "snowflake" (because every machine is unique) with a very reproducible machine.
While the current machine did start out as a machine I installed using a minimal ISL (Initial System Load) and an Ansible playbook when I built it two years ago, there are a few things I did manually the first time around, or did differently (such as installing a non-modular version of PHP), or even wrote smaller playbooks, like one to just install Emacs 29. But the original playbook never got updated. And so, just like when you walk through the snow, you end up with snow sticking to you and your boots... the time has come for me to stomp my feet to dislodge that snow. Add the fact that I wanted to actually use one of my RHEL 9 licenses, since I wanted to do some stuff with Red Hat's Ansible Platform, and not just the free version of Ansible which anyone can use. Only one problem with that. As I recall, while I have a free license for it, unlike RHEL itself, where I can have up to 16 nodes, with Ansible Platform, I can only manage 10 nodes, and I am nearing 32 active nodes (Yes... 32. And this is not counting things like the game consoles and PCs which my daughter and her husband have, or my switches/routers, our phones, my UPSes, the IPMI management modules in my servers, or my laser printer.)
One other thing heavily influencing this is that I have built up a large collection of custom roles, some of which have been becoming more complex as I have learned quite a bit more about Ansible. For example, two years ago, I was using a plugin which was far from ideal for merging a bunch of variables into a single value, such as to create a list or SSL certificates to distribute to a given target, and another for services which need to be restarted While I could have maintained a list in each target's host_vars data, in some cases, I have certificates associated with a group of machines, such as the certificates associated with my LDAP servers, and it seemed a better idea to list that in a group file and merge. Now, instead of using that plugin and having issues with it, I am doing things like explicitly looking up variable names using a pattern from ansible.builtin.varnames, using query() to get the value from ansible.builtin.vars and using filters like union() or community.general.mergeby() to combine them together. So, the end result is a playbook which is entirely different than the old one, which used to have a bunch of tasks which have been replaced by reusable roles. An example of this is that I now have a role which I use, which instead of having to use geerlingguy.repo-epel and geerlingguy.repo-remi directly, after pushing the authorized_keys file, making sure pip is installed if needed, while adding DNF repos and associated RPM signing keys for other repositories, like Emacs 29, because of variables associated with groups, it does the same thing where EPEL and Remi's PHP repository are concerned. All driven by the groups groups to which I have added the target host.
The playbook I have been working with installs the nginx web server. And to do this, I have been using geerlingguy.nginx role from Ansible Galaxy, instead of the "official" nginxinc.nginx and nginxinc.nginx_config roles. This is because the latter roles are far more restrictive in what they handle in configuring say a virtual host. Only one problem... while doing the configuration using Jinja2 templates, it uses the command nginx -t -c %s to validate the configuration before copying it into place (the "%s" allows the task to substitute the new file by its temporary path). And, for some reason, nginx creates the PID file used to track the running service, but with an incorrect SELinux file context. This means that when the role has found the new configuration to be valid, has copied it into place, and tries to start the nginx process, nginx gets a permission error due to SELinux seeing the existing file having the wrong context, and thus denying nginx from accessing the PID file for write. But just to make things real interesting, that attempt at running nginx using the systemctl command cleans up the file, so that when a second attempt is made to start it, things work. And so, I did what any good developer does in IT... I opened a bug on the role, and submitted a pull request (aka a PR) with a proposed fix. But one thing I am still learning is how the CI/CD tests for roles work. And so, I missed the fact that it tests to see if that PID file still existed when it expected it to, and the tests failed. I started a conversation on the PR, and got to wondering "Why did the tests not pick up on the problem I saw in the first place???!!!" And last night, I did some digging. Having that background, now to the real reason for the post. I found what is to me a glaring vulnerability in the way in which Ansible CI/CD testing works. Not a vulnerability like the ones which the Black Hat's use to gain access to a system, but one where you still run the risk of not having security. You see, for their CI/CD pipelines, your tests run either on Ubuntu Linux, Mac OS/X or Windows. And because it is Ubuntu Linux, which almost certainly has not gone through the added steps to switch to using SELinux instead of AppArmor, even if using a RHEL based docker container, such as the Rocky 9 Linux container used by Jeff Geerling, [1] for testing his roles, which are used EXTENSIVELY by the Ansible community[2] it does not and cannot properly test interactions with SELinux. And thus, he never saw the problem I encountered.
And it goes beyond just this role. Not only is SELinux far superior to AppArmor IMNSHO when it comes to protecting a system from security intrusions [1] and not only is Jeff a major contributor of roles, but he is the author of one of the better books I have found on Ansible. If you do a Google search for "books on ansible", his Ansible for DevOps is the first one mentioned, followed by Ansible: Up and Running by Hochstein, et.al. And in Chapter 13, Jeff talks about using Molecule and Docker for testing roles. And, it is not something which we would normally think about. And so, it is probably at least a good item for Jeff to note in his book, so that others are aware of the limitation/volunerability. And maybe, just possibly, we as a community might be able to convince GitHub to add virtual machines to their mix where SELinux is enabled, rather than having to farm out that testing to our own virtual machines running on AWS or our personal networks. And the result would be a far more secure world.
Footnotes:
[1] aka @geerlingguy on GitHub, as well as on Ansible Galaxy.
[2] His geerlingguy.docker role alone has over 29 million downloads, while his geerlingguy.nginx has over 15 million downloads,
[3]This is due in part because SELinux labels the file metadata for a extremely granular approach which stays with the file, even if it is moved/renamed, and its ability to define roles specific to a given process, allowing you to only grant access to certain files/directories and ports to a given service daemon. Sadly, too few take the time to learn about SELinux, and the non RHEL based distros like Debian and Ubuntu go with AppArmor as their default, which is simpler to configure.
Categories