Frelling Dingleberry Recruiters

This is a full blown core dump, and not the sort which just got you dropped back at the command prompt, but one which in the old days would have the printer console go nuts with the BRRZT…BRRZT…BRRZT… as line after line of information on registers, stack and all filled a page or two of paper. And here is what prompted it:

Hi Douglas

This is Afria from *REDACTED*, I came across your resume in my database and wanted to share the latest positions with you. Please read the Job details below & get back to me if you are interested in this position.

Position: Applications Programmer (Senior Java/Angular Developer).
Location: 750 East Pratt Street, 6th Floor, Baltimore, MD
Duration: 4 Years.
Complete Description:
MHBE seeks Applications Programmer (Senior Java/Angular Developers) to support the MHBE. The Applications Programmer will be responsible for understanding, defining, analyzing, coding, evaluating, testing, debugging, documenting, and implementing complex software applications for the development and maintenance of the Maryland Health Benefit Exchange (HBX) and related systems.
Design, develop, maintain and support enterprise applications in Java/J2EE under the HBX technological platform utilizing open-source driven Java EE architecture, Angular JS, HTML, Struts, Spring, Hibernate, RESTful API, and JBoss Enterprise Platform.
Interface with business and IT teams to understand and translate business requirements into formal technical requirements and application code.
Develop and execute quality unit test and integration plans/scripts to validate that application changes meet technical specifications.
Work with testing team and business teams to complete acceptance testing and participate in integration testing.
Participate in technical reviews at appropriate stages of software and application development.
Adhere to all security, Project Management Office (PMO) change control, work management and service delivery policies, processes, tools and methodologies.
Note: The candidate must have the flexibility to work overtime, as needed, to include weekends, holidays, and off-hours.
Minimum Qualifications:
Ability to create efficient and effective Nintex & SP Designer Workflows for very large libraries 7 Year’s Experience Required.
Experience storing many Terabytes of data in SharePoint 7 Year’s Experience Required.
Style, Create, Configure and update out of the box SharePoint sites, pages, lists & libraries, content types and via Web UI & SP Designer 7 Year’s Experience Required.
Ability to query SharePoint with CSOM and Rest services from Client-Side Code (JSOM/CSOM/JavaScript) 7 Year’s Experience Required.
Experience with CMIS (Content Management Interoperability Services) and SharePoint. Experience with InputAccel 7 Year’s Experience Required.
Experience defining/maintaining SharePoint information architecture, managed metadata, term sets, content type hub and site security with AD groups 7 Year’s Experience Required.
Preferred Qualifications:
A minimum of ten (10) years of experience in in Java/J2EE, JavaScript, JSP, Servlets, Struts 2.0, Spring, Hibernate and Web Services.
Experience in application security scanner software like Veracode, AppScan and Fortify.
Proven experience in build and deployment processes and tools such as Maven, ANT and Jenkins.
Experience working with PostgreSQL database and SQL Stored Procedures.
Strong knowledge and hands-on experience of a minimum of two (2) years of experience in Angular JS.
Knowledge of Micro Service Architecture.
Proven ability to work with and build and maintain strong relationships with technical teams.
Working knowledge of organizational change management principles, methodologies and tools. 
Familiarity with various traditional and innovative project management approaches, tools and phases of the project lifecycle.
Knowledge and/or experience with Agile software development practices.
Experience with the State Based Marketplace solutions or Federally Facilitated Marketplace (FFM) solutions.
Experience with managing multiple project priorities under tight deadlines.
Experience working with the Project Management Office (PMO) processes, policies and procedures.

Looking forward to hearing you.
Thank You.
Arifa *redacted*
Staffing Manager
(remainder redacted)

Generally, these just get flagged as spam, if they make it through. But TWO back to back from the same “individual”, for the same exact position… and then that position is for SharePoint (.NET) and Java… TWO SKILLS WHICH ARE NOWHERE ON MY RESUME!!! I’m sorry, but that is walking up to the hornet’s nest in your underwear, grabbing it off the branch, tossing it to the ground and then stomping on it in your bare feet. Or for those of you who enjoy hockey… that is going up to the ref and high sticking them across the throat, followed with a stick to the groin. Not only do you get your entire agency’s domain reported for spam, you get called out damn near 100% publically. Such messages leave me wondering things like…

  • Did you even bother to read my resume?
  • Do you even know computers in a technical way, or are you barely able to turn one on?

This actually appears to be an instance where some recruiting firm subcontracted out to a firm, likely in a location such as Mumbai or Bengaluru but with an “office” in some complex here in the states, to go through job sites harvesting names and sending them out, trying to get people to respond… in effect firing a shotgun as the wall of a barn to try to hit the few horseflies (and you run the risk of hitting any livestock or the odd hornets nest in the vicinity). And often, these are the same firms which call me, are sometimes very high pressure, and where I can barely understand them. Overall, the quality of their work reflects poorly upon the companies which use them to reach out, as well as upon the companies which ultimately have the job which needs to be filled. It reflects a “cheapness” which has made entire nations a laughing stock over the course of my lifetime. But, lest we forget, it also reflects a philosophy which has been taught to countless MBAs over the past three to four decades… the idea of Stockholder Supremacy… where if you can cut corners, quality and the like to give a few more pennies per share to the stockholder, you must do it. Its the same ideology which has companies wanting people with 20 years of experience at the entry level compensation of say Raleigh-Durham, but in DC or NYC…. you are not going to get a developer who has 20 years experience and knows what they are doing for $54K in those places, so why bother???!!!

And so, this place gets 20 minutes of my time, and rather than my just clicking the “Spam” button, I also add a filter that says that any email from their entire domain is spam, along with me sending a message along to the folks who maintain the spam lists… because when I was joining the effort to build what we today call “the Internet”, putting hosts to bring the total count closer to 1000… this is not what we had in mind, where more and more would believe things like the world being flat. ***SIGH***

PHP Upgrades (aka the joys of running a LTS operating system)

Well, earlier today, I got a reminder that I had not upgraded PHP. Indeed, unlike most of my installs, the virtual host running my WordPress sites was installed from a Live CD, and was running the dated PHP 5.4 version which CentOS/RHEL 7 comes with as a part of their base. It is a joy of running an operating system which comes with “long term support”, aka LTS. When an OS such as CentOS/RHEL, or Ubuntu’s LTS releases is going through the release process, the out-of-the-box repositories result in configurations are pretty much set in stone as to what versions of given software packages are included, and they don’t always take into account things like how much longer software package X will be supported. So when the process started for RHEL 7.0 (from which CentOS 7 is compiled) in late 2013 for the July 2014 release, they packaged things like PHP 5.4.16, Python 2.7.5 and other old packages into the release, and at a point in the release cycle, even if there is a minor version upgrade (from say 5.4.15 to 5.4.16), they do not pick up the new version, because of all the testing which would need to be done to guarantee stability. They might backport certain security fixes, but no more until the next release. And then, through the entire 7.x lifecycle (or the lifecycle of say Ubuntu 10.04LTX), it is pretty much a given that PHP would remain a 5.4.x release. And for RHEL, this cutoff date was actually such that PHP 5.5, which was released June 2013, much less PHP 5.6, which was released August 2014 have never made it into the core CentOS/RHEL repositories, and are installed by default when you install the package called “php”. The result is that RHEL (and thus CentOS) were running with versions which were no-longer supported… indeed, 5.6 patches were no longer being released to be backported either, since support for 5.6 ended this past December. And for Python, it is much the same story, with releases through 2.7.16 now being available.

Why are LTS releases out there? Because sometimes, even the changes going from say 2.7.5 to 2.7.6 can cause issues for software vendors trying to support their software, and when you start talking about going from say a 2.6.x to 2.7.x release, or worse, a 2.y.x to a 3.y.x release, the odds of that happening increase, sometimes significantly. Indeed, changes like that often result in the downline vendors having to go through their own release cycles, which can be quite expensive. And ultimately, you have a battle with multiple sides trying to come to an accord which balances things like finances, security, new features and more, and where the costs and risks can easily run $100K up to values in the millions, depending on application, the number of installs, etc. (When I was at Bell Labs Messaging, a simple patch to the OS for a OS bug might start with $5K or more of testing by myself, before it even hit our QA team, where bundled with other software patches, a testing cycle might run another $100K easily, all for a new release of Audix or Conversant… and until then, it was only installed manually on very select customers who had run into the problem and could not wait).

Now, depending on the operating system, there are options to help with this for those who are willing to expend some additional effort on the upgrades, and any testing of their environments. For PHP, this involves either installing from either the IUS Repository (“IUS” = “Inline with Upstream Stable), or Remi’s RPM Repository (run by the Remi Collet, who is a PHP contributor who also maintains many of the RPM packages for the Fedora/RHEL/CentOS distros). But these two repositories take slightly different approaches, and different versions of PHP could not traditionally be installed side-by-side.

For myself… I actually used the IUS repository… I dislike how the Remi versions of the RPMs install everything under /opt/remi/... instead of /usr/... And while it does not have PHP 7.3 yet, it does have PHP 7.2. And thankfully, the upgrade appears to have gone relatively smoothly. I tend to also prefer everything being installed via just via RPMs… why should I have to keep track of what was installed via RPM, as well as via PHP’s pear/pecl, or Python’s pip utilities. I am coming to use those utilities more, as so much is not available as RPMs… but it is a layer of nuisance I would rather not have to deal with. Unfortunately, the PHP ssh2 module required me to install it via pecl, which meant additional development packages needing to be installed right now. Sometime halfway soon, I hope to instead start looking at re-packaging some of these into RPMs myself. I would far rather have my own repo (I actually have two per distribution/release which I use, one for packages I figure to share, another for packages which contain things I consider to be security sensitive and will not). But for now, things seem to be good. 🙂

Will I ever stop using a LTS distro/release? No… I consider Fedora’s release cycle to have been enough of a pain in my ass, that outside of my workstation and perhaps a development VM, all of my servers will be of the LTS variety. After all, with changes which occured with Fedora 20’s installer, I had my workstation remaining at FC19 until just a few months ago, and here in a month or so, I will likely just say “reinstall my workstation” to cobbler, reboot the workstation, and get up the next morning to find it all shiny and new. And when RHEL 8 is released and CentOS 8 comes out, I likely will do the same with many of my servers, as I am currently doing some testing of the RHEL 8 beta release they made awhile back. Now, if only WinBlows were as easy…

@#$%@ Ansible

While the title may indicate that this is a core dump post, I won’t quite say that it clears that hurdle… quite… But it is definitely a frustration which has raised its head a few times, and over the past 24 hours, went from a minor nuisance to a major frustration.

The problem, and the solution in theory

For those of you not familiar with Ansible, it is used for performing tasks controlled from a central host, and doing so using things called “roles” and “playbooks” (which you write once and reuse, like any good developer, DevOps member, etc.). And for your inventory, you can have variables associated with a given host, or for the groups to which it belongs. But, by default, Ansible overwrites the variables of the same name, based on a prioritized hierarchy. For example, let us suppose we have a variable listing users we want to add to a machine if they are not already there, which we will call provisioned_users. And depending on which group of machines, such as development servers, testing servers, or web servers (with say names of dev_servers, test_servers, and web_servers), there are a set of users which we want to be on the machine. But what if we have a machine which belongs to multiple groups, or there is a really special user, such as an application developer who helps the normal testers and DevOps folks at all stages in the life cycle. Normally, Ansible would then require you to go to all sorts of hassles for that. But I found a post by the folks at Leapfrog which talks about a plugin which solves this problem, and even shows some good examples to understand the problem better. And they even share their plugin for folks to use!

Now… for me, the problem came up with my letsencrypt-certs role, which I have been using to push my SSL certificates from a central administration host to the various web servers I have for both internal and external use (i.e. this one), along with the SSL certificates for my LDAP server, and more. This has meant that I have run into the collision on variables between groups and the host, and last night, I got a warning from my Nagios installation on a couple of those certificates needing “renewed”, which was mainly just my needing to push the certificates into place. And given that one of the certificates is a part of the collision… time to address the issue, and I might as well do it right. 🙂

The solution in reality

Well, in reality, I found a couple of gotchas… the first is that the instructions seems to imply that with a playbook in /etc/ansible/playbooks that the folder in installation step 4 would be /etc/ansible/action_plugins/… but this is not the case. Indeed, I got the following (with the verbosity cranked a bit):

# ansible-playbook -vvvvvvvvv update-certificates.yml
ansible-playbook 2.6.3
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 2.7.5 (default, Jul 13 2018, 13:06:57) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
Parsed /etc/ansible/hosts inventory source with yaml plugin
ERROR! no action detected in task. This often indicates a misspelled module name, or incorrect module path.

The error appears to have been in '/home/cinnion/git/ansible-roles/letsencrypt-certs/tasks/main.yml': line 3, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

# tasks file for letsencrypt-certs
- name: Merge certificates
  ^ here

And just putting it in the modules directory does not help. There, it complains that it does not start with the interpreter line (e.g. #!/usr/bin/env python or some equivalent). Instead, for a playbook in that location which uses it, the location would be one of these locations:

  • /etc/ansible/playbooks/action_plugins
  • /usr/share/ansible/plugins/action (the default, which is mentioned in the ansible.cfg file, and can be overridden there.)

or, in a directory named action_plugins directly under the role itself (e.g. parallel to the handlers, tasks, and similar subdirectories. Since I will be using this for multiple playbooks/roles (Keep things DRY!!!), and really dislike putting things in a directory like the default ansible uses (if it included /usr/local, I would have less hesitation, though spreading customizations out is still sub-optimal in my book), I created /etc/ansible/plugins/action, placed the file in that directory, and changed the config file to include that location.

A second issue comes up in that is a minor pain is that instead of being able to have a role which conditionally executes from the playbook based off the variable being defined, the task file in the role needs to do this condition handling. It is not a huge deal… it just means refactoring the file. But that is probably a good move in the long run anyways.

A third issue, which may just be caused by my using fact caching, is that even without specifying that the variables which are the result of the merge are to be cached, I am getting warnings about overwriting the fact. That will take some looking into at some point.

Now, to see how things work in real usage for a week or two.