Tuesday, 21 October 2014

Why fake RPM's ?

In IT companies across the globe, infrastructure and servers are provisioned at the click of a button - this is the world the Devops engineer is creating.  The deployment tool we used at this particular organisation is called ansible.  Ansible allows the programmatic install of RPM's and config files on target servers - our whole estate was built off the back of ansible scripts.

We wanted to develop the ansible scripts using TDD (Test Driven Development), so that we could put the whole regression test and deployment into a pipeline.

Importantly for TDD to work you get yourself into a feedback loop:

1) Write specs
2) Update ansible scripts
3) Run specs  (this will start with an empty base server build then apply the ansible scripts)
4) Repeat 2 and 3 until specs pass

The problem we had was that running the specs could take up an hour.  The feedback loop was way too long!  It turned out that some of the RPM's which the ansible scripts install took the best part of 30 mins to complete.

So we made a compromise - we would install fake rpms for the longest running offenders (luckily these were "bloated 3rd party monitoring tools" so not critical to the servers functionality)


FPM is a ruby gem to create .rpm's (or .debs if running on ubuntu).  To install you will need ruby:

curl -sSL https://get.rvm.io | bash -s stable
rvm install 1.9.3
rvm use 1.9.3 --default
gem install fpm

Fake it

The idea, is to add the fake rpm to the yum repository (alongside the original), but give the fake a higher version number, so it gets picked up by the yum install command.

What we found was that in some of the in house rpm's, the epoch had been set to the unix epoch time and in fact whenever the rpm was regenerated the epoch would increase.  In this scenario, the highest epoch wins, irrespective of the version number.

So the strategy for creating the winning the version, is very much dependent on how the original RPM was versioned:
mkdir -p /tmp/fake_package
fpm -s dir -t rpm -n sysmontool -v 999999999 --epoch 2000000000 -f -a noarch -C /tmp/fake_package ./
fpm -s dir -t rpm -n patchmontool -v 0.0.1 --epoch 2000000000 --iteration 999 -f -C /tmp/fake_package ./

Strangely, you can not just set the epoch to 9999999999,   We actually set the epoch to `date +%s`


The fpm command will create a .rpm.  Copy this into your yum repository and do a yum update.  You make need to do a yum clean all if its not getting picked up.

From experience, for server builds which are automated, it might be a good idea to set the yum,conf metadata_expire field to a low value.  This avoids the confusing scenario of thinking that something has installed/updated when it hasn't.

Wednesday, 15 October 2014

How to pair using TMUX


Tmux provides a way to remotely pair using a shared terminal.  This is great as it means you can pair when working from another office or from home.  It's ideal if you are comfortable using a command line, and enjoy memorizing keyboard shortcuts.


Re-sizing windows - when one side of the pair has a different resolution or differently sized terminal window, if you're not careful, you will see screen sheering.  This was completely solved by using putty or cygwin and ensuring that both clients use the same font settings.  Once its working, it works really well:

Weird control characters - If you see weird characters on the screen then set the window translation -> character set translation to UTF-8.  This seems to be only a problem in putty.

Git bash - Tmux does not work nicely with git bash

Server Setup

Assuming you are using vagrant on your local PC, create a vagrant server on, using the following vagrant snippet when setting up the network:

config.vm.define "a" do |server|
  ## usual config
  ## ...
  server.vm.network "private_network", ip: ""
  server.vm.network "forwarded_port", guest: 2222, host: 22

What's important here is that you forward 2222 to the ssh port 22.

ssh onto your vagrant instance and install and run tmux:
yum install -y tmux

Now use the ip of your physical box to ssh in:

Client Setup

Install cygwin or putty.  From cygwin remotely attach to the tmux session:

ssh vagrant@<ip>  
tmux attach  (this will attach to the session started above)

Remote Connectivity

  • Use a wired connection.  Once I plugged in directly to the home router a lot of the connectivity issues went away (despite having a 5 bar wireless connection).
  • If using vmware or virtual box to host vagrant instance, ensure you use Bridged Mode.  If you use NAT mode, we found the connection would occasionally drop.  
  • If you have both vmware and virtual box running, then it may not be possible to enable bridged mode.  Uninstall one of them!


We tried several voip clients, including skype and googlr hangouts... The most stable turned out to be Viber.  I installed viber as a client on the remote PC.  Combined with a wired connection, it made the voip experience rock solid.
  • Use viber with a hands free headset.  I used a cheap bluetooth headset from amazon.
  • Be warned that voip eats bandwidth - try and get a wifi connection.


Tmux takes a bit of getting used to - you may find the default bind key is a bit of a stretch (ctrl-b).  I changed it to (ctrl-a).  Use this Tmux cheat sheet to quickly get up and running.  I've put together a set of dotfiles to make the whole tmux experience a lot nicer: https://github.com/coder36/dotfiles

Back button bindings

Sometimes you need to change what happens when the browser back button is pressed - for example you might want to disable the back button, or as I did, bind it to a hidden button.

The simplest way is as follows (assuming you are using jquery):


$( function() {
  history.pushState("back", null, null);
  if (typeof history.pushState === "function") {
    history.pushState("back", null, null);
    window.onpopstate = function (evt)
      history.pushState('back', null, null);
      alert( "Back button pressed!" )


$( () ->
  history.pushState("back", null, null);
  if typeof history.pushState == "function"
    history.pushState("back", null, null);
    window.onpopstate = (evt) ->
      history.pushState('back', null, null);
      $( "#back_a_page" ).click()

The coffeescript example will result in the button with id="back_a_page" getting clicked, whenever the back button is pressed.  

In chrome, to quickly test, this, go to  github  (which uses jquery) and hit the F12 button, then select console.  Paste in the javascript code and hit enter.  Now hit the back button :)