The RPM Package Manager (RPM) has for years stood as one of the core components of many popular Linux Distrobutions. RedHat Enterprise Linux, The Fedora Project, SuSE Linux Enterprise, openSUSE, and a slew of others have long since standardized on using RPM to manage software on the system. To list a few of the key benefits of RPM and other package management systems is not difficult:
Why then, I continually ask, does packaging our own RPMs feel like such a daunting task? As a System Administrator for myself, as well as at work I can admit that I've been in those shoes many times where I have given into the temptation to take the easy route. The "easy route" being to download the source tarball, and run those 3 simple commands:
[root@linuxbox ~]# ./configure && make && make install
In nearly 90% of all cases I have encountered this *does* work just fine, no questions asked. The software runs, and the customer is happy. However, How do we manage to properly uninstall said software without leaving hundreds if not thousands of useless, unwanted files lingering on the system? What happens when we overwrite/upgrade software that has previously been installed by RPM? Simple, the RPM database is wrong... and doesn't know that it is wrong. What happens when up2date thinks packageA is at version X.X.y, and relies on X.X.y, but in reality version X.Y.z was compiled and installed from source. Now we potentially are running the risk that not only is our system dirty and not being managed properly, now we have broken our nightly up2date system that keeps us patched and regular.
I could try not to, but am content in saying that compiling and installing software from source on an RPM based distribution is just plain dirty. That's it... tainted, no longer pure.... dirty.
From the other SysAdmins that I have worked with and others I have stood by (and sighed), I would put $5 on the table that says we would rather install from source than have to sit down and learn how to build an RPM Package. And why is that? I wonder what it is that keeps us from wanting to step over that line from the meadows of rpm and into the thick of rpmbuild.
Other than the sheer fact that installing from source is often just so quick and easy, I found out for myself that the documentation on packaging RPMs is not 100%. Don't get me wrong, Max-RPM is a fantastic resource for RPM in general and even more so does a great job of covering the actual build process as well. However, being written in more than 6 years ago leaves it slightly behind the times, as well as severely lacking explanations for techniques I've seen many RPM Developers use... wondering all the while "what-in-the-world does that mean/do"? More than a number of times I found myself in a flood of obscenities, cursing those responsible for documenting the process of packaging software via RPM. For those I have disowned, please forgive me... for it is now that I have found my way. And in return... in my appreciation to you will do my best to enlighten the Linux world of the news: Rolling your own RPMs is not difficult.... what-so-ever! Take 5 minutes to read this, or take 30 to try it out... either way, I am confident that anyone can successfully handle building and/or rebuilding RPM Packages with ease.
In general terms (using RedHat Enterprise Linux as my reference) the following are the main areas on the filesystem to be familiar with:
Most of what your time building RPMs will be spent in the default location of /usr/src/redhat. The following explains each directory in a bit of detail:
The spec file is made up of several sections or blocks of code. For us to take a look at this, lets grab a source RPM and install it:
root@linuxbox ~/# wget http://rpms.5dollarwhitebox.org/clean/pLsearch/pLsearch-0.1.7-1.bjd.noarch/SRPMS/pLsearch-0.1.7-1.bjd.src.rpm root@linuxbox ~/# rpm -Uvh pLsearch-0.1.7-1.bjd.src.rpm 1: pLsearch ############################# [100%] root@linuxbox ~/# cd /usr/src/redhat/SPECS root@linuxbox SPECS/# ls pLsearch.bjd.spec
Now using your favorite editor (I use vim) open it up. The first section is known as the Preamble, and looks something like this:
Summary: pLsearch was created to assist System Administrators in their quest for grep'ing logs and other files for specific content
Name: pLsearch
Version: 0.1.7
Release: 1.bjd
Packager: BJ Dierkes \<wdierkes@5dollarwhitebox.org\>
Vendor: BJ Dierkes \<wdierkes@5dollarwhitebox.org\>
Copyright: GPL
Group: Utilities/System
Source0: http://superb-east.dl.sourceforge.net/sourceforge/plsearch/pLsearch-%{version}.tar.gz
Buildroot: %{_tmppath}/%{name}-%{version}-root
BuildRequires: perl >= 5.8
Requires: perl >= 5.8
Requires: perl(File::Locate)
Requires: perl(Getopt::Long)
Requires: perl(PerlIO::gzip)
Patch0: pLsearch-0.1.7-sample.patchThis is all very straight forward. Some are optional (such as the Buildarch) and some are required. Allow me to explain a bit about each:
A few things should be touched on. First, either the Version or Release need to be incremented whenever the Spec file changes. Secondly, it is very important that you are accurate with the BuildRequires/Requires directives as this will help guarantee that the software will Build/Install properly on other systems.
After the preamble we move on to the next couple of sections. The following is a snippet out of the Spec file for pLsearch:
%prep
%setup -q
%patch0 -p1 -b .sample
%build
# Nothing to do here, but if it was a C program we would do the usual
# ./configure --with-option
# make DESTDIR=$RPM_BUILD_ROOT
#
%install
[ "$RPM_BUILD_ROOT" != "/" ] && rm -rf $RPM_BUILD_ROOT
# Depending on the software, we might do make install here
mkdir -p ${RPM_BUILD_ROOT}%{_bindir}
install -o root -m 755 pLsearch ${RPM_BUILD_ROOT}%{_bindir}/pLsearch
%clean
[ "$RPM_BUILD_ROOT" != "/" ] && rm -rf $RPM_BUILD_ROOTThe %prep section allows us to 'Prepare' for building. This will be anything from patching, to setting environment variables, to running a ./configure command. As this is a Spec for a Perl app there is no './configure' line or 'make' statements.
The %setup section is part of the %prep section. This is a basic example above where the source code is extracted to the BUILD directory, and then rpmbuild changes into the source directory (/usr/src/redhat/BUILD/pLsearch-0.1.7).
The macro '%patch0 -p1 -b .sample' is equivalent to the command 'cat patch-0.1.7-sample.patch | patch -p1 -b .sample'.
The %build section is where we build any necessary source code. As this is a Spec for a Perl app there is no './configure' line or 'make' statements.
Once the software is built we install any necessary files using the %install block. To keep things clean, the first thing we do is to 'rm -rf' the $RPM_BUILD_ROOT. Once that is done, we create any directories in the Build Root, and then install files to them. Here we are introduced to the use of some RPM macros such as %{_bindir}. The line '${RPM_BUILD_ROOT}%{_bindir}/pLsearch' equates to '/var/tmp/pLsearch-0.1.7/usr/bin/pLsearch'. We only install files to a temporary location when building RPMs. In the actual RPM, the $RPM_BUILD_ROOT prefix is removed and files are actually installed in the proper places.
Finally we want to clean up our mess with the %clean section. Here we generally just remove the $RPM_BUILD_ROOT.
On to the last group of sections:
%files
%{_bindir}/pLsearch
%doc README CHANGELOG
%changelog
* Thu Jan 18 2007 BJ Dierkes <wdierkes@5dollarwhitebox.org> - 0.1.7-1
- Beginning of spec buildThe %files section is used to specify the files that are packaged with the RPM. Any directories listed under a %files block will recursively include everything within the directory. However, if you just want to list the directory, use the '%dir' directive. NOTE: The most important thing is to only list directories and files that are installed by this package. Imagine the issues you might run into if you list '%dir %{_bindir}'... This package would try to claim '/usr/bin' on the target system, and I don't think I need to explain why that would be a bad thing.
The %doc directive installs the listed files to '%{_docdir}/%{name}-%{version}', or equally for this example '/usr/share/doc/pLsearch-0.1.7'.
One of the ultimately most important sections is the %changelog section. Any and all changes should be documented here. This section is often overlooked, however please make every effort to document your changes properly. Additionally, you will note that the date must be a specific format... any other format and you will receive errors.
So, just to bring everything together... the following is the complete Spec file for the pLsearch package:
Summary: pLsearch was created to assist System Administrators in their quest for grep'ing logs and other files for specific content
Name: pLsearch
Version: 0.1.7
Release: 1.bjd
Packager: BJ Dierkes <wdierkes@5dollarwhitebox.org>
Vendor: BJ Dierkes <wdierkes@5dollarwhitebox.org>
Copyright: GPL
Group: Utilities/System
Source0: http://superb-east.dl.sourceforge.net/sourceforge/plsearch/pLsearch-%{version}.tar.gz
Buildroot: %{_tmppath}/%{name}-%{version}-root
Buildarch: noarch
BuildRequires: perl >= 5.8
Requires: perl >= 5.8
Requires: perl(File::Locate)
Requires: perl(Getopt::Long)
Requires: perl(PerlIO::gzip)
Patch0: pLsearch-0.1.7-sample.patch
%description
pLsearch was written to assist SysAdmin's in grep'ing logs and other files for specific content, and then spitting it out in a pretty format which is generally easy on the eyes.
pLsearch uses the (s)locate database to search for specific files that the administrator wants to work with. It then accepts a query, and/or subquery, and/or inverted query which will then be used to grep through the located files for content. It is therefore a two way search. This is meant to be extremely useful for System Administration tasks such as attempting to track down exploitable FormMail applications that were being used to Send Spam as an open relay, or possibly used to search common files for a known exploits (such as "urldecode" in all files named "viewtopic.php" which is a well known phpBB exploit signature), etc.
%prep
%setup -q
%patch0 -p1 -b .sample
%build
# Nothing to do here, but if it was a C program we would do the usual
# ./configure --with-option
# make DESTDIR=$RPM_BUILD_ROOT
#
%install
[ "$RPM_BUILD_ROOT" != "/" ] && rm -rf $RPM_BUILD_ROOT
# Depending on the software, we might do make install here
mkdir -p ${RPM_BUILD_ROOT}%{_bindir}
install -o root -m 755 pLsearch ${RPM_BUILD_ROOT}%{_bindir}/pLsearch
%clean
[ "$RPM_BUILD_ROOT" != "/" ] && rm -rf $RPM_BUILD_ROOT
%files
%{_bindir}/pLsearch
%doc README CHANGELOG
%changelog
* Thu Jan 18 2007 BJ Dierkes <wdierkes@5dollarwhitebox.org> - 0.1.7-1
- Beginning of spec buildNow that we have an understanding of our Spec file, lets try and build the package. The following commands assume that we are in the '/usr/src/redhat' directory:
[root@linuxbox redhat/]# rpmbuild -ba SPECS/pLsearch.bjd.spec Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.69853 + umask 022 + cd /usr/src/redhat/BUILD + cd /usr/src/redhat/BUILD + rm -rf pLsearch-0.1.7 + tar -xf - + /bin/gzip -dc /usr/src/redhat/SOURCES/pLsearch-0.1.7.tar.gz + STATUS=0 + '[' 0 -ne 0 ']' + cd pLsearch-0.1.7 ++ /usr/bin/id -u + '[' 0 = 0 ']' + /bin/chown -Rhf root . ++ /usr/bin/id -u + '[' 0 = 0 ']' + /bin/chgrp -Rhf root . + /bin/chmod -Rf a+rX,u+w,g-w,o-w . + echo 'Patch #0 (pLsearch-0.1.7-sample.patch):' Patch #0 (pLsearch-0.1.7-sample.patch): + patch -p1 -b --suffix .sample -s + exit 0 Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.69853 + umask 022 + cd /usr/src/redhat/BUILD + cd pLsearch-0.1.7 + exit 0 Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.69853 + umask 022 + cd /usr/src/redhat/BUILD + cd pLsearch-0.1.7 + '[' /var/tmp/pLsearch-0.1.7-root '!=' / ']' + rm -rf /var/tmp/pLsearch-0.1.7-root + mkdir -p /var/tmp/pLsearch-0.1.7-root/usr/bin + install -o root -m 755 pLsearch /var/tmp/pLsearch-0.1.7-root/usr/bin/pLsearch + /usr/lib/rpm/brp-compress + /usr/lib/rpm/brp-strip + /usr/lib/rpm/brp-strip-static-archive + /usr/lib/rpm/brp-strip-comment-note Processing files: pLsearch-0.1.7-1.bjd Executing(%doc): /bin/sh -e /var/tmp/rpm-tmp.2435 + umask 022 + cd /usr/src/redhat/BUILD + cd pLsearch-0.1.7 + DOCDIR=/var/tmp/pLsearch-0.1.7-root/usr/share/doc/pLsearch-0.1.7 + export DOCDIR + rm -rf /var/tmp/pLsearch-0.1.7-root/usr/share/doc/pLsearch-0.1.7 + /bin/mkdir -p /var/tmp/pLsearch-0.1.7-root/usr/share/doc/pLsearch-0.1.7 + cp -pr README CHANGELOG /var/tmp/pLsearch-0.1.7-root/usr/share/doc/pLsearch-0.1.7 + exit 0 Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 Requires: /usr/bin/perl perl >= 5.8 perl(File::Locate) perl(Getopt::Long) perl(PerlIO::gzip) perl(strict) Checking for unpackaged file(s): /usr/lib/rpm/check-files /var/tmp/pLsearch-0.1.7-root Wrote: /usr/src/redhat/SRPMS/pLsearch-0.1.7-1.bjd.src.rpm Wrote: /usr/src/redhat/RPMS/noarch/pLsearch-0.1.7-1.bjd.noarch.rpm Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.2435 + umask 022 + cd /usr/src/redhat/BUILD + cd pLsearch-0.1.7 + '[' /var/tmp/pLsearch-0.1.7-root '!=' / ']' + rm -rf /var/tmp/pLsearch-0.1.7-root + exit 0
What we are always hoping for is of course 'exit 0'. This means the package built without error, and successfully wrote the RPM packages as expected. You will see everything that rpmbuild is doing (thanks to the lovely 'set -x' in Bash). Take a few moment to examine how rpmbuild read our instructions and used them to build the package.
You will note towards the end that our files are written:
Wrote: /usr/src/redhat/SRPMS/pLsearch-0.1.7-1.bjd.src.rpm Wrote: /usr/src/redhat/RPMS/noarch/pLsearch-0.1.7-1.bjd.noarch.rpm
So, we now have our RPM... but what is it going to install, what does it need to install, and what do I get from it:
[root@linuxbox SPECS/]# rpm -qlp RPMS/noarch/pLsearch-0.1.7-1.bjd.noarch.rpm /usr/bin/pLsearch /usr/share/doc/pLsearch-0.1.7 /usr/share/doc/pLsearch-0.1.7/CHANGELOG /usr/share/doc/pLsearch-0.1.7/README [root@linuxbox redhat/]# rpm -qp --requires RPMS/noarch/pLsearch-0.1.7-1.bjd.noarch.rpm /usr/bin/perl perl >= 5.8 perl(File::Locate) perl(Getopt::Long) perl(PerlIO::gzip) perl(strict) rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 [root@linuxbox redhat/]# rpm -qp --provides RPMS/noarch/pLsearch-0.1.7-1.bjd.noarch.rpm pLsearch = 0.1.7-1.bjd
One of the most important things to become familiar with are the RPM macros available. The default macros are set at '/usr/lib/rpm/macros' on RedHat systems. You can override the defaults with your own macros file in your home directory. Simply create the file '~/.rpmmacros' and add the macro changes there.
For the purpose of interest, the following is a snippet from '/usr/lib/rpm/macros':
%_usr /usr
%_usrsrc %{_usr}/src
%_var /var
%__awk gawk
%__bzip2 /usr/bin/bzip2
%__cat /bin/cat
%__chgrp /bin/chgrp
%__chmod /bin/chmod
%__chown /bin/chown
%__cp /bin/cp
%__cpio /bin/cpio
%__file /usr/bin/file
%__gpg /usr/bin/gpg
%__grep /bin/grep
%__gzip /bin/gzip
%__id /usr/bin/id
%__install /usr/bin/install
%__ln_s ln -s
%__make /usr/bin/make
%__mkdir /bin/mkdir
%__mkdir_p /bin/mkdir -p
%__mv /bin/mvYou will notice that directories generally have a single '_' and commands have a double '__'. You should adapt this schema when creating your own macros. To define a macro, or more commonly a variable in the Spec itself you simply define it:
%define myvar value
You would then use the variable as '%{myvar}' which of course equals 'value'. You can also define variables at command line. Take for example, you may want to add an OS release tag to the RPMs (such as .rhel4). Add the following to the top of the Spec where the Release directive is:
Release: 1.bjd.%{os_ver_tag}Now from command line, you would do the following:
<b>[root@linuxbox SPECS/]#</b> rpmbuild -ba \ SPECS/pLsearch.bjd.spec \ --define="os_ver_tag rhel4"
Note: The backslashes delimite the line breaks. This is actually a single command.
The result would be an RPM looking like 'pLsearch-0.1.7-1.bjd.rhel4.noarch.rpm'. In a future section will will talk about 'If Conditions and the Bit Operator' where I will explain a cleaner way to add OS Version Tags like this.
Well this concludes the first section. At this point we are building RPMs from scratch, and we should be damn proud of it. Before you move on to the next sections, feel free to take a peak at some other Spec files. I ensure you that it will help you understand that there is MUCH more that you can do, and I will do my best to explain that in future sections. Here are some recommended Specs to look at:
Please note that any stupid issues with these may be related to me not knowing what I was doing when I first built them, or modified them. As I am in an RPM Development roll at work now, due to conflict of interest I can no longer feed the RPMs I build up to my personal webserver anymore. So I haven't been adding new stuff in a while (sorry).
The RPM Spec file provides a number of ways to deal with differing conditions that alter the way the package should build. I myself use condition statements heavily in order to use a single spec file that builds properly on multiple Operating Systems (Redhat EL3, EL4, etc).
Consider the following situation:
When building a spec for PHP, one of the requirements for php-imap is '/usr/include/imap/c-client.h'. Now, on a Redhat EL3 box this file is provided by the package 'imap-devel'. However, on Redhat EL4 the packages changed and this particular file is now provided by 'libc-client-devel'.
The reason Redhat doesn't maintain a single spec is because the version of software on Rhel3 is never the same as the versions of software on Rhel4 because they lock in version numbers before official release. However, for me I want to build the latest version of PHP for both Rhel3 and Rhel4 the same. Therefore, it wouldn't really make any sense to maintain two different spec files. If I did, you would have two entries like so:
Rhel3:
BuildRequires: imap-devel
Rhel4:
BuildRequires: libc-client-devel
Now, you decide "he's right... If I can do this all in a single spec, why not?". Exactly. So now, lets make some changes to our spec and that allow for this. At the top of your spec, add the following:
%define _with_rhel3 1
%define rhel3 %{?_with_rhel3:1}%{!?_with_rhel3:0}
%define rhel4 %{?_with_rhel4:1}%{!?_with_rhel4:0}
%define os_ver_tag %{?_with_rhel3:.rhel3}%{?_with_rhel4:.rhel4}
The first line defines the variable '_with_rhel3' and sets it to 1. If you think of this logically, we are saying 'This spec is for a Rhel3 box'. The second and third lines read that definition and set the appropriate variables to '1' as well.
Now this might seem redundant, but there is a reason for this. In the future it should be clear, however the simple reason is that you can't call a '%if' statement on a variable that hasn't been define, however with the bit operator '%{?}' you can. For our example above, we would not be able to do the following:
%if _with_rhel4 ...do something %endif
We can't do it, because rpmbuild will complain that '_with_rhel4' has not been defined. Now, what we can do is this:
%if rhel4 ...do something for rhel4 %endif
The difference is that we have defined 'rhe4' to equal zero, but at least it is define. Back to our example, lets pick apart the following line to better understand it:
%define rhel3 %{?_with_rhel3:1}%{!?_with_rhel3:0}
Broken up in sections:
You can see that we are defining the variable 'rhel3'. The second section displays our 'bit operator'. The bit operator works with '1' and '0' values. On of Off. The following statements are equivalent:
%{?_with_rhel3:1}If the variable '_with_rhel3' is '1' then output the value after the colon ':' (in this case is 1 also... which is confusing I know).
I understand the use of 1s and 0s in this example is confusing, but it is something you will see. Lets take our os_ver_tag line for a better example:
%define os_ver_tag %{?_with_rhel3:.rhel3}%{?_with_rhel4:.rhel4}
This is saying:
So, again the following are the same:
%define _with_rhel3 1
%define rhel3 %{?_with_rhel3:1}%{!?_with_rhel3:0}
%define rhel4 %{?_with_rhel4:1}%{!?_with_rhel4:0}
%define os_ver_tag %{?_with_rhel3:.rhel3}%{?_with_rhel4:.rhel4}%define _with_rhel3 1 %define rhel3 1 %define os_ver_tag .rhel3
%define _with_rhel4 1 %define rhel4 1 %define os_ver_tag .rhel4
You can see how we can hard code definitions, or using our first example have a dynamic configuration that changes with a single definition. Now you can add '%{os_ver_tag}' to you Release line like this:
Release: 1%{os_ver_tag}
The output will be:
Release: 1.rhel3
This is helpful when you are building the same software for different OS versions. Now back to our Requires definition where we use our changes. With the OS definitions in place, we can now have the following replace our two separate files:
%if %{rhel3}
BuildRequires: imap-devel
%elseif %{rhel4}
BuildRequires: libc-client-devel
%endif
Here we see the use of our if/elseif/else standard condition statement. This literally says:
You could also change '%elseif %{rhel4}' to '%else' if you want a generic setting if nothing else matches in the condition statement previously.
Note: I have found that the if/elseif/else condition statement does not always act as expected when you go above 2 elseif/else conditions. Remember that if you have no way of explaining why your condition isn't working the way it logically should.
You have now seen the condition and bit operators in action a bit. Take some time to look through this PHP spec file to see how it is also used for adding optional additional modules (pay close attention to the configure lines). Searching for 'mhash' and 'mcrypt' will be helpful to find the conditions in action:
One more thing I want to go over is passing command line options. A simple example is defining a variable at command line like this:
rpmbuild -ba example.spec --define="_installpath /usr/local/mypath"
This is essentially the same as having the following definition in the spec file itself:
%define _installpath /usr/local/mypath
This can be handy for making changes for one-off build. I often use this when building in a non-standard directory like this:
rpmbuild -ba example.spec --define="_topdir /home/wdierkes/usr/src/redhat"
You can define anything this way that you can in the spec itself.
Another command line option is the '--with' and '--without' flags. These flags are used to enable or disable configure options for the build. Say I wanted to remove the following line from my example above:
%define _with_rhel3 1
If I want to specify 'rhel3' or 'rhel4' add build time, I can simply pass in that option like this:
rpmbuild -ba example.spec --with rhel3
Which is the same as defining it in the spec file:
%define _with_rhel3 1
Additionally, I could disable the option with '--without rhel3'. Now, that doesn't make much sense for our os_ver_tag example, but it does for optional additional modules in a package such as 'mhash' or 'mcrypt' in the PHP spec linked above.
This is a rough draft for this section, so I will hopefully be cleaning it up soon. Please send any comments to wdierkes [at] 5dollarwhitebox [dot] org.
I will be adding more sections soon to hopefully cover the following:
Take care!
RPM:
External Package Builders
Misc Reference