Gonzo’s Puppet Journey: Updating an Existing Package on Solaris 10 Using Puppet 2.7

Introduction

A high-level picture of how Puppet applies resources during a run. From Puppet Labs.

I’m willing to bet that most of us have built a server or two in our careers.  If you’re lucky, you have a long list of cryptic instructions from someone who left the company three years ago that tells you what software is required for your new server.  Are you absolutely sure you followed every instruction to the letter?  What if you skipped a step – how would you know?  Also, how long did it take you to build your maybe-I-sure-hope-I-got-this-right server?  An hour?  Two?  I’ve often heard the term “work of art” used as an analogy for building a server.  No matter how hard you try, they never come out the same – each server is its own unique and beautiful snowflake.

Enter tools like Puppet or Chef (and the concept of “infrastructure as code”).  If you’re not familiar with Puppet, it’s a fairly popular tool for helping to maintain your infrastructure.  The basic idea is automating the creation of your servers.  That way, your servers are the same in every environment – from development to staging to production.  Puppet thinks of programs, files, users, etc. as resources.  You can write scripts (called manifests, which are organized into modules) using Puppet’s DSL (domain-specific language) to interact with these resources and set them up correctly on your new server.  After you’ve created your Puppet manifests and modules, you can have Puppet apply them to any number of servers.  In effect, you’ve “scripted out” your server configuration and you can run it at will (more or less).  Using Puppet, you can set up a new environment in a few minutes instead of a few days.  Also, you’ve automated creating your servers – no more following incomprehensible instructions.  Puppet will build your servers the same way, every time.

Bonus:  once your Puppet manifests and modules are all squared away, you can check them into source control.  Now, your server configuration is versioned!  If you need to make a change to a Puppet manifest or module, it’s just like anything else – you check out the file(s) from source control, make your change, and check it back in.

Disclaimer

Statler and Waldorf are not amused.

I am by no means a Puppet expert. This post represents my particular approach to solving the issue of updating existing packages on Solaris. Puppet veterans probably have a much slicker way to deal with this particular issue. By all means, I encourage you to follow their advice. Were I using this post as a presentation, I bet Statler and Waldorf would have quite a few things to say.

Also:  hang in there with me folks – I don’t know how far I’m gonna be able to take this Puppet/Muppet thing.

Scenario/Use Case

Like our bare-bones Solaris machines, Gonzo is all dressed up with nowhere to go.

Puppet is our weapon of choice for automating the building up of new machines.  Here’s the scenario:

• We’re handed a group of bare-bones Solaris 10 servers.
• These servers all have Java 6 update 21 installed.
• We want our Solaris 10 servers to have Java 6 update 43 installed.
• We want to use Puppet to make sure any Solaris 10 servers we create are in a consistent state with a common set of programs (and versions of those programs) installed.
• After all the common programs have been installed (java, wget, etc.), we can then use Puppet to install “role-specific” software on the server, for example:
  • If the server we’re provisioning is destined to be a web server, we can use Puppet to install Apache or Tomcat or your favorite web server.
  • If the server we’re provisioning is to become a database server, we can instruct Puppet to  install MySQL.

Issues/Challenges

Even Gonzo checks the cannon for any potential issues/pitfalls. Safety first, kids.

Have you – or someone you know – ever wanted to be fired out of a cannon?  Granted, travel by cannon might not be the safest way to travel, but it’s certainly one of the flashiest.  I happen to know of a professional cannoneer (and personal hero) – Gonzo!  I consider Gonzo the Great to be one of the foremost authorities on all things cannon.  However, Gonzo knows that being fired from a cannon is no trivial feat.  There are all sorts of challenges to overcome: trajectory, powder charge, fire safety, concussive force, blunt trauma, the list goes on.  Now, I seriously doubt that Gonzo considered any of those things, but he probably should’ve.  So, I’m going to use Gonzo as an example of what not to do and examine some of the challenges we’ll try to face while resolving our use case.

Package Versioning

• What’s the problem? Unlike some other providers, Puppet’s sun provider (the default provider used by Puppet when installing software on Solaris) doesn’t support checking a package’s version number.
• What does that mean? Well, that means that we can’t differentiate between versions of the same package.  If we say “Hey Puppet, I want you to install the SUNPigsInSpace package”, here’s what happens (you can see this yourself if you run sudo puppet agent --test --debug):
  • Puppet will run pkginfo -l SUNPigsInSpace, looking to see if SUNPigsInSpace has been installed already.
  • If pkginfo -l finds the package, Puppet will skip the installation, having determined that the package we care about is already installed. Even if the existing package is an older version.
  • If pkginfo -l doesn’t find an existing instance of SUNPigsInSpace, then Puppet will call pkgadd to install SUNPigsInSpace.
  • If you use ensure => latest instead of ensure => installed, Puppet will execute pkginfo -l -d [whatever your source attribute is pointed to] SUNPigsInSpace to look for the existence of the package and its version.
• What can we do? It seems like the best course of action is to remove/uninstall the existing version of the package and install the version of the package we care about.
  • More specifically, we can use Solaris’s package management commands (pkginfo and pkgrm) to explicitly find and uninstall the package before re-installing it.
  • We can ask Puppet to execute those commands for us during a run, using the exec type.

NOTE:  The concept of server roles, environments etc. is well beyond the scope of this article.  If you’re curious, I encourage you to read this article.  Then go through this slide deck.

Resource Declaration

• What’s the problem? When creating a Puppet manifest, a resource can only appear once in that manifest.
• What does that mean? Considering the problem defined above is a core concept for how Puppet works, it’s a bit tough to call this a “problem”.  In the context of our issue, this means we can’t declare our packages with an ensure => absent attribute then come along later in our manifest and declare our package(s) with an ensure => installed attribute.  For example, we can’t do this:

  class install_pigs_in_space {
  	package { "SUNWPigsInSpace":
  		ensure	=> absent,
  	}
  
  	# do some other stuff...
  
  	package { "SUNWPigsInSpace":
  		ensure	=> installed, 
  		source	=> "/opt/tmp/SUNWPigsInSpace",
  	}
  }

  That’s a bit of a bummer – it’d be nice to treat a resource like a Java or C# object. Y’know, something I can change after I create an instance of it. Honestly, this was one of my biggest stumbling blocks to overcome while learning Puppet. It seems like I should be able to change a resource’s properties at run-time. Nope – not so much. Well, not “easily” anyway:

  • There might be a way to work around this problem using Puppet’s virtual resources.  I played around with this for a bit and wasn’t successful, so I abandoned it in favor of what we’re going to see later.
  • As far as changing the values of a resource during a run, you canchange the value of an existing resource in a couple ways:
    • Declare global-level defaults for your resource: You can define some default values for your resources and then override them in your resource later on.
    • Sub-classing/extending your resource:From what I understand, it’s also possible to change a value of a resource’s attribute(s) by means of subclassing the resource. This seems clunky and convoluted. Furthermore, sub-classing/extending/inheriting classes is frowned upon (according to the Puppet folks).

• What can we do? We have to make sure that our “uninstall package” resources have a different name than our “install package” resources. We can use some string manipulation to make sure our “uninstall” resources look different than our “install” resources.

Solaris 10 Tools

• What’s the problem? The Solaris package management tools (e.g. pkginfo and pkgrm) don’t fail “silently” – they will throw errors if they don’t find the package they’re looking for. For example, if we use pkginfo to look for a package that isn’t installed on our Solaris machine, here’s what happens:

  vagrant@puppet-master:[~] $ pkginfo -l SUNWPigsInSpace
  ERROR: information for "SUNWPigsInSpace" was not found

  Similarly, if we try to arbitrarily remove a package that doesn’t exist, here’s what we end up with:

  vagrant@puppet-master:[~] $ sudo pkgrm -n SUNWPigsInSpace
  pkgrm: ERROR: no package associated with

  It’d be nice if the tools didn’t return anything, or maybe wrote a message to stdout instead of stderr, but we have to work with what we’ve got.

  • Now that I think about it, I probably could’ve looked into redirecting stderr to stdout, but… Well… I can’t think of everything, y’know?
• What does that mean? If we want to use Puppet’s exec type, we’re going to have to be careful. If we want to use the uninstall/re-install strategy for upgrading our packages, we need to make sure the package exists first.
• What can we do? Fortunately, this problem’s pretty easy to tackle. Puppet’s exec type comes with an onlyif attribute. onlyif allows us to make our exec resource conditional. That is, we can tell puppet “only run pkgrm if you find the package we’re looking for by using pkginfo.

Solution

Answerville, here we come!

Well, we’ve taken a look at some of the challenges our cannon trip will face. Pack your bags, get your helmet, check your cape, because the time for whining about issues is OVER! We’re loading up our cannon for a one-way trip to Answerville!

To clarify, here’s our plan to update an existing package on a Solaris 10 machine:

• We’re going to use pkgrmto remove an existing package.
  • We’re only going to remove that package if the package already exists.
• We’re going to install the package again, using a more current version of the package.

define utils::remove_solaris_package($pkg_admin_file) {
	exec { "uninstall_${name}":
		command 	=> "pkgrm -n -v -a ${pkg_admin_file} ${name}",
		logoutput	=> on_failure,
		onlyif		=> "test `pkginfo -x | grep -w ${name} | awk 'END{print NR \"\"}'` -eq 1",
		path		=> "/usr/bin:/usr/sbin:/bin:/sbin:/usr/local/bin:/usr/local/sbin:/opt/csw/bin:/opt/csw/sbin",
		require		=> File["${pkg_admin_file}"],
	}
}

jdk_solaris.pp – a Module to Install the Java JDK

Moving on, jdk_solaris is a Puppet class that attempts to replicate Oracle’s instructions to install Java on Solaris, which are paraphrased below:

1. Get the desired version of the Java JDK.
2. Extract the contents to some directory.  This gives us the packages we need to install.
   1. There are 4 packages that matter:  SUNWj6cfg, SUNWj6dev, SUNWj6man, SUNWj6rt
   2. If you want to install the Japanese man pages (which this class will do), the SUNWj6jmp package contains them.
3. Remove the previous version of the JDK.
4. Install the required packages using pkgadd.
5. Clean up the download directory when we’re finished.

/etc/puppet/modules/java/manifests/jdk_solaris.pp:

class java::jdk_solaris ($version,
                         $update
) {
	$working_dir		= "/opt/tmp"
	$zip_file_name_32 	= "jdk-${java::jdk_solaris::version}u${java::jdk_solaris::update}-solaris-i586.tar.Z"
    $admin_file 		= "java_admin"
    $java_packages 		= [ "SUNWj6cfg", "SUNWj6dev", "SUNWj6jmp", "SUNWj6man", "SUNWj6rt" ]

	file { 
		"${java::jdk_solaris::working_dir}":
   		ensure 	=> directory;

		"${java::jdk_solaris::working_dir}/${java::jdk_solaris::zip_file_name_32}":
    	ensure 	=> file,
    	source	=> "puppet:///modules/java/${java::jdk_solaris::zip_file_name_32}";

    	"${java::jdk_solaris::working_dir}/${java::jdk_solaris::admin_file}":
		ensure 	=> file,
		source 	=> "puppet:///modules/java/${java::jdk_solaris::admin_file}";
   	}

	exec { "zcat_32":
		cwd		=> "${java::jdk_solaris::working_dir}",
		command	=> "zcat ${java::jdk_solaris::zip_file_name_32} | tar -xf -",
		creates	=> [
			"${java::jdk_solaris::working_dir}/SUNWj6cfg",
			"${java::jdk_solaris::working_dir}/SUNWj6dev", 
			"${java::jdk_solaris::working_dir}/SUNWj6jmp", 
			"${java::jdk_solaris::working_dir}/SUNWj6man",
			"${java::jdk_solaris::working_dir}/SUNWj6rt" 
		],
		path	=> "/usr/bin:/usr/sbin:/bin:/sbin:/usr/local/bin:/usr/local/sbin:/opt/csw/bin:/opt/csw/sbin",
		require => File[
				"${java::jdk_solaris::working_dir}",
				"${java::jdk_solaris::working_dir}/${java::jdk_solaris::zip_file_name_32}"
			],
	}				   

#------------------------------------------------------------------------------------------------------------------------
# INSTALL JDK
#------------------------------------------------------------------------------------------------------------------------
        # Use our "remove_solaris_package" defined type to un-install the Java packages we want to update.
	utils::remove_solaris_package { $java_packages: 
		pkg_admin_file 	=> "${java::jdk_solaris::working_dir}/${java::jdk_solaris::admin_file}",
		require			=> Exec["zcat_32"],
	} 
	->
	package { $java_packages:
		ensure 		=> latest,
		adminfile 	=> "${java::jdk_solaris::working_dir}/${java::jdk_solaris::admin_file}",
		source		=> "${java::jdk_solaris::working_dir}",
	}

#------------------------------------------------------------------------------------------------------------------------
# CLEAN UP AFTER INSTALLATION
#------------------------------------------------------------------------------------------------------------------------

	exec { "jdk_solaris_cleanup":
		cwd 	=> "${java::jdk_solaris::working_dir}",
		command => "rm -r ${java::jdk_solaris::admin_file} SUNWj* THIRDPARTYLICENSEREADME.txt COPYRIGHT LICENSE README.html ${java::jdk_solaris::zip_file_name_32}",
		path	=> "/usr/bin:/usr/sbin:/bin:/sbin:/usr/local/bin:/usr/local/sbin:/opt/csw/bin:/opt/csw/sbin",
		require => Package[$java_packages],
	}
}

Here’s what’s going on:

• On line 4, we’re including our utils::solaris class, which we use to safely uninstall any pre-existing versions of our package.
• On lines 11-22, we have an array of fileresources.  We’re making sure that all of the files we’ll need are present before we ask Puppet to start the installation process.  Those files are:
  • The /opt/tmp directory – this will be our working directory where we put the .tar.Z file and our admin file.’
  • The .tar.Z file that contains the packages we need to install java.
  • the java_admin admin file – this file provides input for the pkgrm command.
• On line 24, we’re extracting the contents of our .tar.Z file and making sure that directories for the packages we care about are being created.
  • Before we run the zcat command, we’re switching to the /opt/tmp directory.
• On line 45, we’re actually leveraging our remove_solaris_packageutility class to find and uninstall a pre-existing package.
  • Note the -> symbol?  That’s a chaining arrow. The chaining arrow tells Puppet that the remove_solaris_package resource has to be execute prior to the package resources being installed.
• On line 50, were installing our Java packages.
• On line 60, we’re removing all our temporary files that we created during this installation.

# == Class: java
#
# This class will install the java openjdk and java openjdk-devel on a CentOS server.
# The yum provider will be used to handle the installation.
#
# === Parameters
#
# [*distribution*]
#	The type of package to install, either 'jdk' or 'jre'.
#
# [*version*]
#	The single-digit version of Java (e.g. 6 or 7) that should be installed.
#
# [*update*]
#	The update version of the JDK or JRE that should be installed.
#
# === Example
#
# Shown below is an example of how to use the java class to install the JDK and JDK-devel
# packages on a node called "activemq.example.com":
#
# node "activemq.example.com" {
#        class  { 'java':
#					distribution	=> 'jdk',
#                	version 		=> '6',
#					update			=> '25',
#        }
# }
class java ($distribution, 
			$version, 
			$update
) {
	$class_prefix = $distribution ? { 
		jdk => 'java::jdk',
		jre => 'java::jre',
	 }

	case $::operatingsystem {
		'RedHat', 'CentOS': {
        	class { "${java::class_prefix}_redhat": 
        		version => "${java::version}" 
        	} 
        	->
        	Class["java"]
    	}

    	'Solaris': {

    		class { "${java::class_prefix}_solaris": 
    			version	=> "${java::version}", 
    			update	=> "${java::update}",
    		} 
            ->
            Class["java"]
    	}
	}
}