pkgsrc-2014Q4: LTS, signed packages, and more
The latest quarterly release of our binary package sets for SmartOS and illumos introduces a number of new features that I’m excited to announce.
Long Term Support
We have produced quarterly releases of pkgsrc for a number of years, and since
pkgsrc-2013Q2 release have built every package available (10,000+), but
until now have not formalised our support for them.
This has meant that when serious security issues such as Heartbleed are disclosed, we are obliged to backport these fixes to every branch we have ever produced. Despite all our efforts on performance improvements this is still a large effort and takes a long time on older branches where we do not have huge resources available and backports can be tricker due to the differences involved.
We’ve tried as best we can to keep these older branches updated, but as we’ve added new branches each quarter the load increases further, and we cannot keep doing this forever.
pkgsrc-2014Q4 (SmartOS 14.4.x images) we are introducing a new
yearly Long Term Support (LTS) model, which can be summarised as:
pkgsrc-2015Q4, …) will be an LTS release, and will receive suitable backports for 3 years from the time it is made available.
We will continue to produce the other quarterly releases (SmartOS 15.1.x images and onwards), so that users can get the latest packages available, but each of those releases will be closed for updates as soon as the next one is available.
What is a “suitable” backport? Anything which is a security or build fix, and
which does not affect API or ABI compatibility. For example, we would not
introduce a new major version of OpenSSL or PHP into an LTS release, but we
would update OpenSSL from
1.0.1k or PHP from
as they are minor releases which only introduce fixes. We may also introduce
new leaf packages (i.e. those with no dependencies), for example new releases
Who is the target market for each type of release?
LTS is primarily useful for people who have a very static set of requirements, do not like changes, and are primarily interested in ensuring that the software they run does not have active vulnerabilities.
Latest quarterly releases are for everyone else, users who want the latest stuff (and the latest security fixes), and are happy to reprovision their applications onto the newest images at regular intervals.
We hope the introduction of LTS releases satisfies both types of users, and that by freeing up our resources spent on maintaining our legacy branches, we can invest more time into ensuring the stability and security of the LTS releases.
For SmartOS users, the LTS releases will receive an additional
-lts suffix on
the image name to make it even easier to identify which are LTS.
Image Name Changes
Related to LTS, we are also slightly changing our naming scheme for the base
images. This is to accommodate the new
-lts suffix, and also to allow us to
introduce a new “minimal” image, and make it clear which is which.
The current base image names are:
With the introduction of the “minimal” image, the new names will be:
And for LTS releases:
What’s the new “minimal” image? It’s effectively a stripped down “base”, with only the pkgsrc bootstrap and a couple of packages installed which are required for the zone to boot correctly. This will be of primary interest to users who have custom requirements for their zones and/or produce their own images, and want to ensure they are building on the smallest possible foundation.
As a quick comparison:
|image size (compressed)
The “minimal” images are fully functional and use the same package set, the only difference is fewer packages are installed by default.
Updated SmartOS Build Hosts
Up until now we have built all our package sets on an old SDC 6.5 install, to ensure that the packages we built can run across all hosts in the Joyent Public Cloud. Building on the lowest common denominator is great for compatibility, but has meant we are running on a limited number of older machines, and each quarterly release added yet more strain to already overloadeded systems.
Starting with 2014Q4 LTS we have moved to newer build hosts, running
joyent_20141030T081701Z. This will soon be the most common platform
available in the Joyent cloud, and ensures we aren’t tied to a legacy release
for another 3 years. The next 3 quarterly releases (2015Q1-3) will also be
produced on this platform, and we will then evaluate which platform to choose
for the next LTS in 2015Q4.
This may mean incompatibilities if you are either running an older SmartOS release, or if you are running a different illumos distribution which does not have some of the newer SmartOS features. You are most likely to see issues where packages have picked up support for newer interfaces such as epoll or inotify, which have been introduced as part of the LX brand work.
Please feel free to raise a GitHub issue if this is causing problems for you. We are happy to turn off support for newer features if it improves compatibility, as often these features are picked up by autoconf checks but either aren’t used correctly or should be using different interfaces on illumos platforms anyway.
One of the primary concerns in recent times is provenance, and ensuring that what you are receiving hasn’t been tampered with in any way. Until now our packages have been protected by checksums, so that it is difficult for an attacker to modify packages in-flight and deliver something we did not provide.
However, it isn’t impossible, and to further ensure that what you are installing came from Joyent we have implemented signed packages for 2014Q4 onwards. Here’s how it works:
- During the package build process, a detached package hash file is created which contains a SHA512 checksum of the package.
- This hash file is then signed with our PGP key.
- All three files are bundled into an
ar(1)archive and delivered as the
- At package install time, the archive is unpacked, the hash file is verified against our public PGP key, and then the package is checksummed against the recorded checksum in the hash file. If all these checks pass, the package is installed, otherwise the installation is aborted.
Let’s take a look at a package file (digest) to see in more detail:
+PKG_HASH file contains all the details about the actual digest package
which is stored in the archive.
We can verify that the checksum is correct.
So we know that the
+PKG_HASH file matches the
file. However, how do we know that both haven’t been tampered with? That’s
+PKG_GPG_SIGNATURE file comes in. It is a detached signature of
+PKG_HASH file, signed with the Joyent key, so that if a malicious user
has tampered with the package file and generated a new checksum, the
+PKG_HASH file will no longer be verified and we know that it isn’t what was
We can verify that on the command line with GPG, as long as you have imported the public key for that package set:
A quick note about the warnings shown above. This is where the PGP web of
trust comes in. We know that the files were signed with the
but how do we know that the key belongs to
email@example.com? It essentially
comes down to trust, and whether you believe this is really our key, or whether
someone has tricked you to believe that when it’s not. We can help persuade
you in a few ways:
- The bootstrap packages and our SmartOS images will come by default with that
key installed in
/opt/local/etc/gnupg/pkgsrc.gpg. This is required so that you can start installing signed packages out of the box with no setup necessary.
- The public keys will be published to PGP key servers, and I will sign them
with my key (
D532A578). My key in turn is signed by a number of other people, so that you can verify whether you believe I am who I say I am.
- We will publish the keys in a couple of other places, certainly on the main http://pkgsrc.joyent.com/ site.
So, if you are a diligent user who checks all of these sources, an attacker would need to infiltrate every single one of them simultaneously to have a chance of delivering you a malicious packages which bypasses all of the checks. Hopefully you are convinced that this would be extremely difficult.
Finally, how is all of this used in practise? We’ve worked hard to make this
as transparent as possible, including integration of Alistair Crooks’ excellent
pkg_install, so as a user you should never be aware of any of it
unless there is a problem (a core Unix philosophy):
As mentioned above, the PGP key is distributed by default, so you don’t need to
import keys or anything to get started. We have added the following to
GPG_KEYRING_VERIFY is set to our public key, and
VERIFIED_INSTALLATION=trusted means that a signature is required, and if one
isn’t available then you are prompted for how to proceed. Trying to install
the package file from our
ar(1) archive example above shows what happens:
And if we try to install a package with an incorrect signature/hash:
If you build your own packages then you’re going to want to handle this
properly. The simplest option is to use a custom
installing your own packages, for example:
The alternative is to sign your own packages. This is reasonably straight-forward:
- Install GPG, create a signing key, and then configure
With those additions, pkgsrc will prompt you for your PGP passphrase at package
time, and then sign the package with the key you have configured. You can use
gpg-agent to automate this in a controlled environment.
Closely related to package signing, now that we have infrastructure support for
verification in our bootstrap packages, we’ve also enabled easy verification of
pkg-vulnerabilities file. For those who aren’t aware, there is a team of
volunteers for pkgsrc who maintain a list of security vulnerabilities, which
can be checked against the list of installed packages and show you which ones
are currently vulnerable.
This provides you as the administrator with the information necessary to decide whether the current vulnerabilities are acceptable in your environment.
However, there is a missing piece. As you can see above, the vulnerabilities
file is signed. This is important as an attacker with access to modify this
file could hide vulnerabilities from you and leave your system exposed. With
the verification infrastructure now in place, we can now provide the
pkgsrc-security PGP key for you to easily verify that the
file is as expected.
First we need to install the
firstname.lastname@example.org PGP key on the
system. As this key changes quite frequently, we cannot include it directly in
the bootstrap tarball as we have done with the package signing key, as it will
eventually be out of date. So we instead provide a new
package which includes it, bundle that in the bootstrap, and we can then
distribute updates to this package as normal via
We then add that file to our
/opt/local/etc/pkg_install.conf file with:
To verify the pkg-vulnerabilities file, use
Add these checks to your automated reports to ensure you aren’t being lied to about possible vulnerabilities.
Reduced Package REQUIRES
Each package lists the libraries that it requires, and those are checked prior
to installation to ensure the package will work correctly on the target host.
Recently we’ve seen a few issues where some illumos distributions have moved
platform libraries to a different location (but still in the default search
path), which means the
REQUIRES no longer match and the package won’t
From 2014Q4 we have reduced the way that
REQUIRES are computed. Previously
every library that was pulled in was recorded, essentially using the output of
ldd, so for example with the
libpcap package you end up with:
In 2014Q4 we have stopped using
ldd to resolve the library dependencies, and
elfdump to only look at the
NEEDED entries that are recorded in
SHT_DYNAMIC section for each executable. This results in a much simpler
and direct list:
due to excluding all of
/lib/libdlpi.so.1’s dependencies, and increases the
portability of our packages across illumos distributions.
We also get a side benefit of being able to easily identify packages which are
incorrectly linking against system versions of e.g.
libxml2.so.2 when they
should instead be using the pkgsrc version.
There is the usual grab bag of updates in 2014Q4/14.4.x:
go-1.4.2now includes Keith Wesolowski’s patches to add support for cgo. This brings Go for illumos up to feature parity with other operating systems and increases the amount of Go software that will build and run.
SmartOS 14.4.x images now deliver an SSL bundle in
/etcwhich makes Go work correctly, and we also ensure that
/usr/bin/curlhas access to certificates.
libgohas been removed from the
gcc47-libspackage. It is unused, and doing this saves 40MB from the bootstrap kits and images.
We now build with cwrappers, as detailed in my performance post. This speeds up the builds a lot, so in the event of another Heartbleed we should be able to deliver updated packages a lot faster.
pkginis now at version 0.8.0 including support for the new
preferred.conf, plus a number of important bug fixes.
A number of small internal improvements to the build infrastructure. As a user you shouldn’t notice any changes, if you do please let us know!
Plus all the usual upstream pkgsrc changes as announced here.
As always, please raise a GitHub issue if you run into any problems or have any suggestions on ways we can improve any of this stuff.
- 16 Jul 2015 » Reducing RAM usage in pkgin
- 03 Mar 2015 » pkgsrc-2014Q4: LTS, signed packages, and more
- 06 Oct 2014 » Building packages at scale
- 04 Dec 2013 » A node.js-powered 8-bit CPU - part four
- 03 Dec 2013 » A node.js-powered 8-bit CPU - part three
- 02 Dec 2013 » A node.js-powered 8-bit CPU - part two
- 01 Dec 2013 » A node.js-powered 8-bit CPU - part one
- 21 Nov 2013 » MDB support for Go
- 30 Jul 2013 » What's new in pkgsrc-2013Q2
- 24 Jul 2013 » Distributed chrooted pkgsrc bulk builds
- 07 Jun 2013 » pkgsrc on SmartOS - creating new packages
- 15 Apr 2013 » What's new in pkgsrc-2013Q1
- 19 Mar 2013 » Installing SVR4 packages on SmartOS
- 27 Feb 2013 » SmartOS is Not GNU/Linux
- 18 Feb 2013 » SmartOS development preview dataset
- 17 Jan 2013 » pkgsrc on SmartOS - fixing broken builds
- 15 Jan 2013 » pkgsrc on SmartOS - zone creation and basic builds
- 10 Jan 2013 » Multi-architecture package support in SmartOS
- 09 Jan 2013 » Solaris portability - cfmakeraw()
- 08 Jan 2013 » Solaris portability - flock()
- 06 Jan 2013 » pkgsrc-2012Q4 illumos packages now available
- 23 Nov 2012 » SmartOS and the global zone
- 24 Oct 2012 » Setting up Samba on SmartOS
- 10 Oct 2012 » pkgsrc-2012Q3 packages for illumos
- 23 Aug 2012 » Creating local SmartOS packages
- 10 Jul 2012 » 7,000 binary packages for OSX Lion
- 09 Jul 2012 » 9,000 packages for SmartOS and illumos
- 07 May 2012 » Goodbye Oracle, Hello Joyent!
- 13 Apr 2012 » SmartOS global zone tweaks
- 12 Apr 2012 » Automated VirtualBox SmartOS installs
- 30 Mar 2012 » iptables script for Debian / Ubuntu
- 20 Feb 2012 » New site design
- 11 Jan 2012 » Set up anonymous FTP upload on Oracle Linux
- 09 Jan 2012 » Kickstart Oracle Linux in VirtualBox
- 09 Jan 2012 » Kickstart Oracle Linux from Ubuntu
- 22 Dec 2011 » Last day at MySQL
- 15 Dec 2011 » Installing OpenBSD with softraid
- 21 Sep 2011 » Create VirtualBox VM from the command line
- 14 Sep 2011 » Creating chroots for fun and MySQL testing
- 30 Jun 2011 » Graphing memory usage during an MTR run
- 29 Jun 2011 » Fix input box keybindings in Firefox
- 24 Jun 2011 » How to lose weight
- 23 Jun 2011 » How to fix stdio buffering
- 13 Jun 2011 » Serving multiple DNS search domains in IOS DHCP
- 13 Jun 2011 » Fix Firefox URL double click behaviour
- 20 Apr 2011 » SSH via HTTP proxy in OSX
- 09 Nov 2010 » How to build MySQL releases
- 29 Apr 2010 » 'apt-get' and 5,000 packages for Solaris10/x86
- 16 Sep 2009 » ZFS and NFS vs OSX
- 12 Sep 2009 » pkgsrc on Solaris
- 09 Dec 2008 » Jumpstart from OSX
- 31 Dec 2007 » Set up local caching DNS server on OSX 10.4