Coder36

Monday, 12 November 2012

How to set up a Jboss 7.1 domain

With the new JBOSS 7.1 domain controller, a cluster of jboss instances can be administered via a single console. In this tutorial we will step through the simple process of setting up a domain controller, and 3 nodes.

Target Architecture:

Simplistically we are aiming for the following architecture:

I've assumed IP addresses, passwords and names to keep thing simple.

Later on we will need base64 version of the identity. I used this website to generate the following base64 strings:

Identity Base64
node1id bm9kZTFpZA==
node2id bm9kZTJpZA==
node3id bm9kZTNpZA==

1) Configure the ManagementRealm

The nodes authenticates with the domain controller (DC) by passing it's node name and password. The DC does a name/password lookup against the ManagementRealm.

so... the first step is to configure the ManagementRealm.

On the domain controller run:
add-user node1 (when prompted set password to node1id)
add-user node2 (when prompted set password to node1id)
add-user root (when prompted set password to password)

(The root user is used for logging into the jboss console.)

Start up the domain controller:
domain -b 192.168.0.24 -bmanagement 192.168.0.24

2) Configure Node 1

Open host.xml (under jboss-as-7.1.1.Final\domain\configuration) and edit as follows:

a) Update <host tag to read:

b) Add server-identities section to <security-realm name="ManagementRealm">

<security-realm name="ManagementRealm">
<server-identities>
<secret value="bm9kZTFpZA=="/>
</server-identities>

<authentication>
...
</authentication>
</security-realm

The secret value is the base64 encoded version of the Node password.

c) Edit the <domain-controller> section to read:

<domain-controller>

<remote host="${jboss.domain.master.address:192.168.0.24}"
port="${jboss.domain.master.port:9999}"
security-realm="ManagementRealm"/>
</domain-controller>

Don't forget to add the the security-realm attribute otherwise you will get messages like:

[Host Controller] 16:26:06,454 ERROR [org.jboss.remoting.remote.connection] (Rem
oting "host1:MANAGEMENT" read-1) JBREM000200: Remote connection failed: javax.se
curity.sasl.SaslException: Authentication failed: all available authentication m
echanisms failed
[Host Controller] 16:26:06,469 ERROR [org.jboss.as.host.controller] (Controller
Boot Thread) JBAS010901: Could not connect to master. Aborting. Error was: java.
lang.IllegalStateException: JBAS010942: Unable to connect due to authentication
failure.

d) Start up the node
domain.bat -b 192.168.0.25 -bmanagment 192.168.0.25

3) Configure Node 2

Configure as above, setting <host name="node2" and <secret value="bm9kZTJpZA=="

4) Configure Node 3

Configure as above, setting <host name="node3" and <secret value="bm9kZTNpZA=="

Testing

If everything is working, then open a browser and navigate to: http://192.168.0.24:9990/console using root/password when prompted. On the left hand side, there should be a list of nodes.

Friday, 9 November 2012

How to enable security logging in Jboss 7

To view what's happening within the jboss 7 security subsystem, you need to enable logging of org.jboss.security.

Assuming standalone.xml, update the <subsystem xmlns="urn:jboss:domain:logging:1.1"> section, adding a new console-handler which deals with TRACE level messages only:

<console-handler name="CONSOLE_TRC">
<level name="TRACE"/>
<formatter>
<pattern-formatter pattern="%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n"/>
</formatter>
</console-handler>

and link it up to a new org.jboss.securty logger:

</handlers>

</logger>

Finally, restart the jboss server :)

Thursday, 8 November 2012

Securing JBoss 7 management console using SSL

This tutorial describes how to secure the JBOSS 7 web management console and the CLI management interface. For clarity, it assumes you're running a stand alone jboss server.

Create a java keystore

Start by creating a java keystore with a self signed certificate:
c:
mkdir ssl
cd ssl
keytool -genkey -keyalg RSA -alias mycert -keystore keystore.jks -storepass password -keypass password -validity 360 -keysize 2048 -dname "cn=coder36.com, O=Coder36 L=Newcastle, S=England, C=GB"

The keystore password is: password and the private key password is: password

View the certificate using:
keytool -list -v -keystore keystore.jks -alias mycert -storepass password

c:/ssl/keystore.jks will now contain your private key and a self signed certificate.

Enable SSL

The next step is to enable SSL on the management interfaces. Edit standalone.xml. Search for the <management> tags and update. I've highlighted what's changed from out of the box:

<management>
<security-realms>
<security-realm name="ManagementRealm">
<server-identities>
<ssl protocol="TLSv1">
<keystore path="C:/ssl/keystore.jks" password="password"/>
</ssl>
</server-identities>
<authentication>
<properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>
</authentication>
</security-realm>
<security-realm name="ApplicationRealm">
<authentication>
<properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
</authentication>
</security-realm>
</security-realms>
<management-interfaces>
<native-interface security-realm="ManagementRealm">
<socket-binding native="management-native"/>
</native-interface>
<http-interface security-realm="ManagementRealm">
<socket-binding https="management-https"/>
</http-interface>
</management-interfaces>
</management>

Testing

Restart the JBoss standalone server and test:
Open a web and navigate to: https://localhost:9443/console. You will be presented with the usual warnings about using untrusted certificates Interestingly, the https console does not work with Google Chrome, but works fine with Internet Explorer, and what ever is built into eclipse.

Test the CLI using:
jboss-cli.sh -c

Using this solution we can prove that the jboss server we are calling is who we think we are calling. It's also more secure in that it encrypts traffic to and from the server.

Comments

With a few changes, it straight forward to get the management console protected with SSL.

I'm of the opinion that java key stores are a bit clunky and unfriendly. I much prefer the .pem format with openssl etc. From my commercial experience, it's generally a bad idea to use java to do the SSL work. Architecturally it would be better to offload the SSL termination to dedicated hardware (CISCO provides ACE modules to do this kind of work), or dedicated software like apache - see one of my earlier posts for how this could be done.

Saturday, 3 November 2012

TVHeadend on the Raspberry PI

Tvheadend

Tvheadend is a linux service which converts a dvb-t feed into a network stream. This enables you to stream live tv feed from your raspberry pi to any device on your LAN. The aim of this tutorial to set up a Tvheadend streaming server on the Raspberry PI, then watch it via an XBMC client on the Nexus 7.

Pre-requisites - Hardware

1) Raspberry PI (I've got the newer 512MB version)
2) Powered USB hub (New Link 4 Port USB Hub with Mains Adaptor from modmypi )
3) USB DVB-T Tuner (Hauppauge WinTV Nova-TD)
4) USB WIFI adapter (Edimax EW-7811UN Wireless)
5) Memory card (Lexar class 10 32GB micro SD inside an SD card adapter)
6) Keuboard and mouse (I've got a wireless "MINI ULTRA SLIM DESIGN LAPTOP USB KEYBOARD MOUSE COMBO" from here)

Pre-requisistes OS

The operating system of choice is Raspbian (Wheezy), available from here. Expand the root partition to fill up the SD card, otherwise you''ll run out of space.

--------------

so lets begin....

sudo apt-get update

sudo apt-get install libcurl4-openssl-dev git

Set up the NOVA-TD DVB-T TV tuner

Plug in the tv-tuner then run
dmesg

You will see something along the lines of:

[ 1138.657011] usb 1-1.3.2: new high-speed USB device number 7 using dwc_otg
[ 1138.758550] usb 1-1.3.2: New USB device found, idVendor=2040, idProduct=9580
[ 1138.758581] usb 1-1.3.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 1138.758599] usb 1-1.3.2: Product: NovaT 500Stick
[ 1138.758611] usb 1-1.3.2: Manufacturer: Hauppauge
[ 1138.758624] usb 1-1.3.2: SerialNumber: 4027809413
[ 1138.865068] IR NEC protocol handler initialized
[ 1138.896509] IR RC5(x) protocol handler initialized
[ 1138.929688] IR RC6 protocol handler initialized
[ 1138.957795] IR JVC protocol handler initialized
[ 1138.968073] dib0700: loaded with support for 21 different device-types
[ 1138.968456] dvb-usb: found a 'Hauppauge Nova-TD Stick/Elgato Eye-TV Diversity' in cold state, will try to load a firmware
[ 1138.991871] IR Sony protocol handler initialized
[ 1139.025138] IR MCE Keyboard/mouse protocol handler initialized
[ 1139.026512] dvb-usb: did not find the firmware file. (dvb-usb-dib0700-1.20.fw) Please see linux/Documentation/dvb/ for more details on firmware-problems. (-2)
[ 1139.026723] usbcore: registered new interface driver dvb_usb_dib0700
[ 1139.057895] lirc_dev: IR Remote Control driver registered, major 251
[ 1139.062582] IR LIRC bridge handler initialized

The problem here is that the dvb-usb module failed to find the nova-td firmware file (dvb-usb-dib0700-1.20.fw) A quick google reveals that the best place to find tv tuner firmware for linux is here

cd ~
wget http://linuxtv.org/downloads/firmware/dvb-usb-dib0700-1.20.fw
sudo cp dvb-usb-dib0700-1.20.fw /lib/firmware

or more elegantly ...

sudo apt-get install firmware-linux-nonfree

Pull out the tv-tuner, and push back in.

demesg

[ 1660.721801] usb 1-1.3.2: USB disconnect, device number 7

[ 1663.006468] usb 1-1.3.2: new high-speed USB device number 8 using dwc_otg

[ 1663.107981] usb 1-1.3.2: New USB device found, idVendor=2040, idProduct=9580

[ 1663.108013] usb 1-1.3.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3

[ 1663.108029] usb 1-1.3.2: Product: NovaT 500Stick

[ 1663.108042] usb 1-1.3.2: Manufacturer: Hauppauge

[ 1663.108054] usb 1-1.3.2: SerialNumber: 4027809413

[ 1663.118347] dvb-usb: found a 'Hauppauge Nova-TD Stick/Elgato Eye-TV Diversity' in cold state, will try to load a firmware

[ 1663.173435] dvb-usb: downloading firmware from file 'dvb-usb-dib0700-1.20.fw'

[ 1663.378063] dib0700: firmware started successfully.

[ 1663.886683] dvb-usb: found a 'Hauppauge Nova-TD Stick/Elgato Eye-TV Diversity' in warm state.

[ 1663.889868] dvb-usb: will pass the complete MPEG2 transport stream to the software demuxer.

[ 1663.890139] DVB: registering new adapter (Hauppauge Nova-TD Stick/Elgato Eye-TV Diversity)

[ 1664.159908] DVB: registering adapter 0 frontend 0 (DiBcom 7000PC)...

[ 1664.201931] MT2266: successfully identified

[ 1664.373180] dvb-usb: will pass the complete MPEG2 transport stream to the software demuxer.

[ 1664.373438] DVB: registering new adapter (Hauppauge Nova-TD Stick/Elgato Eye-TV Diversity)

[ 1664.552728] DVB: registering adapter 1 frontend 0 (DiBcom 7000PC)...

[ 1664.557289] MT2266: successfully identified

[ 1664.766474] Registered IR keymap rc-dib0700-rc5

[ 1664.767378] input: IR-receiver inside an USB DVB receiver as /devices/platform/bcm2708_usb/usb1/1-1/1-1.3/1-1.3.2/rc/rc0/input3

[ 1664.767916] rc0: IR-receiver inside an USB DVB receiver as /devices/platform/bcm2708_usb/usb1/1-1/1-1.3/1-1.3.2/rc/rc0

[ 1664.769460] dvb-usb: schedule remote query interval to 50 msecs.

[ 1664.769497] dvb-usb: Hauppauge Nova-TD Stick/Elgato Eye-TV Diversity successfully initialized and connected.

That was easy enough! This could easily have been a nightmare, but by the looks of it Raspbian has the correct kernel modules loaded. Now that the tv tuner has been detected and initialised we can proceed with building tvheadend.

Building tvheadend from source

cd ~
git clone -b release/3.2 https://github.com/andoma/tvheadend.git
cd tvheadend
./configure
make
sudo make install

This will install the tvheadend binary: /usr/local/bin/tvheadend
This targets release 3.2, but by leaving off -b release/3.2 you can target the bleeding edge!

Make it auto start at boot

Create tvheadend user home and group

sudo groupadd tvheadend
sudo useradd -g tvheadend -G video -m tvheadend

Create file named /etc/init.d/tvheadend

#!/bin/bash

TVHNAME="tvheadend"
TVHBIN="/usr/local/bin/tvheadend"
TVHUSER="tvheadend"
TVHGROUP="tvheadend"

case "$1" in
start)
echo "Starting tvheadend"
start-stop-daemon --start --user ${TVHUSER} --exec ${TVHBIN} -- \
-u ${TVHUSER} -g ${TVHGROUP} -f -C
;;
stop)
echo "Stopping tvheadend"
start-stop-daemon --stop --quiet --name ${TVHNAME} --signal 2
;;
restart)
echo "Restarting tvheadend"

start-stop-daemon --stop --quiet --name ${TVHNAME} --signal 2

start-stop-daemon --start --user ${TVHUSER} --exec ${TVHBIN} -- \
-u ${TVHUSER} -g ${TVHGROUP} -f -C

;;
*)
echo "Usage: tvheadend {start|stop|restart}"
exit 1
esac
exit 0

Set the permissions to make it runnable:

sudo chmod 755 /etc/init.d/tvheadend

Enable start during boot process

sudo update-rc.d tvheadend defaults

Finally start it manually so we can configure it:

sudo /etc/init.d/tvheadend start

Configuration

Navigate to http://<rasppi_ip>:9981

Configuration -> General : Set the language (and hit 'Save Changes')

Configuration->TV Adapters: Select the Adapter

Hit Add DVB Network by location and select Generic->auto_Default

Now wait while the Muxes have been scanned for services. This will take a long time. The right hand panel will update as channels (services) are discovered.

Hit Map DVB services to channels (once it becomes enable) and wait. In the meantime hit the services tab, to see what's been found. If everything is working as it should, there should be a list of TV and radio channels.

Select Configuration->Channels to view mapped channels

XBMC On the Nexus 7

All that's left to do is to install XBMC on the nexus 7. The easiest way to do this, is by installing the 'nightlies' from the XDA forums (for the nexus 7 it's the non-neon build you need).

To connect to TVHeadend, add the "Tvheadend HTSP Client" Addon:

System -> Settings -> Add-ons -> Search

Type: Tvheadend

Enable and Configureas follows:

Tvheadend hostname or IP address : <RaspPI+IP>

HTTP port: 9981

HTSP port: 9982

System -> Settings -> Live TV -> General : set to Enabled

That's it... Go to Live-TV -> TV Channels

The XBMC tvheadend client is not the finished article. The EPG looks a bit screwed up, as do some of the channel names. Also not all the channels appear to be present. Apart from the above niggles, the XBMC client looks amazing and seems to fit the bill.

TVHGuide

Available from here. It's not in the google play store, so you need to download it and install it by hand. In comparison to XBMC, this seems to be the more finished product although the interface could do with some polish - its fairly basic! The EPG works, although is a bit industrial. The configuration is intuitive, and as a bonus it can use an external player like the amazing MXPlayer

Until the bugs in XBMC can be resolved, I'll be sticking with TVHGuide.

Thursday, 27 September 2012

Apache SSL reverse proxy tutorial

In this tutorial, we will take a high level requirement, work out a design, then prototype a solution.

High level Requirement

All communication must be encrypted and trusted

Analysis

In terms of our java application which calls a remote web service, this means:

We need to authenticate the remote web service - they are who they say they are
The remote web service needs to authenticate us - we are who we say we are
http calls to be encrypted

A good technology fit for this problem is SSL with mutual authentication. We will use apache as an SSL reverse proxy (this will forward our plain http requests to the remote web service, applying SSL). Will use certificate based authentication to prove the authenticity of the server and client.

What is a Certificate trust chain?

Certificate chains are all about trust. In the diagram below the certificate authority ca.zaltek has signed the certificates for "Bob Ltd" and "Fred Ltd". The certificate authority is saying "I trust Bob Ltd and Fred Ltd".

Now the leap - If I trust that certificate authority (ca.zaltek) will only sign certificates for legitimate organizations, then I can trust any organization who presents me with a certificate signed by ca.zaltek.

That's SSL authentication in a nutshell

Target Architecture

In the next section we will create 2 certificates and get them signed by a certificate authority. We are going to use the same certificate authority for both, but equally we could used different ones - as long as we trust them. We will then set up an apache ssl reverse proxy and emulate a remote SSL web servers.

Generate certificates

Will generate private keys and certificates for both the client and the server using openssl and tinyca:

mkdir c:/ssl
mkdir c:/ssl/client
mkdir c:/ssl/server
mkdir c:/ssl/cacerts
mkdir c:/ssl/apache

# Generate private keys and remove password protection
cd c:/ssl/client
openssl genrsa -des3 -out key_.pem 512 (use password)
openssl rsa -in key_.pem -out key.pem
rm key_.pem
# Generate CSR (Certificate Signing Request)
openssl req -out csr.pem -key key.pem -new (cn=client.coder36)

cd ../server
openssl genrsa -des3 -out key_.pem 512 (use password)
openssl rsa -in key_.pem -out key.pem
rm key_.pem
# Generate CSR (Certificate Signing Request)
openssl req -out csr.pem -key key.pem -new (cn=server.coder36)

The next step is to get the CSR's signed. Openssl can do this, however I prefer tinyca See here for a quick guide to using tinyca. Copy the signed certificates into their respective folders ssl/client/cert.pem and ssl/server/cert.pem. Copy the ca certificate into ssl/cacerts/ca.pem.

Certificates can be viewed using:
openssl x509 -in cert.pem -noout -text

Checking certificates and keys are all working:

The easiest way to confirm that the keys and certificates are playing nicely is to use openssl as a server.
cd c:/ssl/server
openssl s_server -accept 443 -www -key key.pem -cert cert.pem -CAfile ../cacerts/ca.pem -Verify 1

This will set the server listening on port 443 (the SSL port)

In another session, start the client:

openssl s_client -connect localhost:443 -verify 1 -CAfile ../cacerts/ca.pem -cert cert.pem -key key.pem
get /

If everything is as it should be, the client will perform the SSL handshake without any errors as per the above picture.

Okay, so this is the base line. We know the certificates work. The next step is to get the certificates working with apache.

Setting up apache

Download the latest version of apache (2.4) from here. Extract to c:/

Combine server certificate and key into single file
cd c:/ssl/server
cat cert.pem > certandkey.pem
cat key.pem >> certandkey.pem

Create apache config c:/ssl/apache/proxy.conf :

LoadModule authz_core_module modules/mod_authz_core.so
LoadModule logio_module modules/mod_logio.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule ssl_module modules/mod_ssl.so

LogFormat "%h %l %u %t \"%r\" %>s %b" common
CustomLog "c:/ssl/apache/access.log" common
ErrorLog "c:/ssl/apache/error.log"
LogLevel warn

Listen 80

<VirtualHost localhost:80>
SSLProxyEngine on
SSLProxyCACertificateFile "c:/ssl/cacerts/ca.pem"
SSLProxyVerify require
SSLProxyVerifyDepth 2
SSLProxyMachineCertificateFile "c:/ssl/server/certandkey.pem"
<Proxy *>
</Proxy>

<Location />
ProxyPass https://server.coder36:443/
ProxyPassReverse https://server.coder36:443/
</Location>
</VirtualHost>

Update hosts file:
127.0.0.1 server.coder36
127.0.0.1 client.coder36

Testing

Start up apache:
cd c:/apache24/bin
httpd -f c:/ssl/apache/proxy.conf

In another session, start the openssl server:

cd c:/ssl/server
openssl s_server -accept 443 -www -key key.pem -cert cert.pem -CAfile ../cacerts/ca.pem -Verify 1

Test 1

http https
Browser -------> Apache -------> Openssl Server

Fire up a browser and go to: http://server.coder36. If everything is working you should be presented with the openssl server diagnostic page.

Test 2
http https
Java App -------> Apache -------> Openssl Server

To test from a java app use the following code:

import java.io.*;
import java.net.*;

public class Client {
public static void main( String [] args ) {
try {
URL url = new URL( args[0] );
URLConnection con = url.openConnection();
InputStream is = con.getInputStream();
BufferedReader br = new BufferedReader( new InputStreamReader( is ));
String s = "";
while ( (s = br.readLine()) != null ) {
System.out.println( s );
}
is.close();
}
catch( Exception e ) {
e.printStackTrace();
}
}
}

To run::
java Client http://server.coder36

Resources

CA Certificate:
-----BEGIN CERTIFICATE-----
MIIEGzCCA4SgAwIBAgIJALf/i8hTdAG8MA0GCSqGSIb3DQEBBQUAMIGPMQswCQYD
VQQGEwJVSzEQMA4GA1UECBMHRW5nbGFuZDEcMBoGA1UEBxMTTmV3Y2FzdGxlLXVw
b24tdHluZTETMBEGA1UEChMKWmFsdGVrIEx0ZDESMBAGA1UEAxMJY2EuemFsdGVr
MScwJQYJKoZIhvcNAQkBFhhtYXJreW1pZGRsZXRvbkBnbWFpbC5jb20wHhcNMTIw
NDA1MDI0MzQ2WhcNMjIwNDAzMDI0MzQ2WjCBjzELMAkGA1UEBhMCVUsxEDAOBgNV
BAgTB0VuZ2xhbmQxHDAaBgNVBAcTE05ld2Nhc3RsZS11cG9uLXR5bmUxEzARBgNV
BAoTClphbHRlayBMdGQxEjAQBgNVBAMTCWNhLnphbHRlazEnMCUGCSqGSIb3DQEJ
ARYYbWFya3ltaWRkbGV0b25AZ21haWwuY29tMIGfMA0GCSqGSIb3DQEBAQUAA4GN
ADCBiQKBgQCv/X87xEwkmpoKMQE+KA5BHhCnF70OOcvn9sO/FONvNIX7DA7HUe6Y
WrrE8nKHLWt8XTDvbJKzbq2DyxOhPJZ759IbDBlmUpHQDfPexBL5Rg7AJR41TMAw
SQAqjGk03hPckptWxK1mEzib9X8XO2dkUe9KUuSqoYiht6ScxBXsNwIDAQABo4IB
ezCCAXcwHQYDVR0OBBYEFEpA7/Lo5wBFi7t725ViT6+8hp5SMIHEBgNVHSMEgbww
gbmAFEpA7/Lo5wBFi7t725ViT6+8hp5SoYGVpIGSMIGPMQswCQYDVQQGEwJVSzEQ
MA4GA1UECBMHRW5nbGFuZDEcMBoGA1UEBxMTTmV3Y2FzdGxlLXVwb24tdHluZTET
MBEGA1UEChMKWmFsdGVrIEx0ZDESMBAGA1UEAxMJY2EuemFsdGVrMScwJQYJKoZI
hvcNAQkBFhhtYXJreW1pZGRsZXRvbkBnbWFpbC5jb22CCQC3/4vIU3QBvDAPBgNV
HRMBAf8EBTADAQH/MBEGCWCGSAGG+EIBAQQEAwIBBjAJBgNVHRIEAjAAMCsGCWCG
SAGG+EIBDQQeFhxUaW55Q0EgR2VuZXJhdGVkIENlcnRpZmljYXRlMCMGA1UdEQQc
MBqBGG1hcmt5bWlkZGxldG9uQGdtYWlsLmNvbTAOBgNVHQ8BAf8EBAMCAQYwDQYJ
KoZIhvcNAQEFBQADgYEACVt3BMuusRAOW3G6w4T11FZFbJnKn5/GY5KFZQtn5hFV
AheWyWMrDPcHsxTW8EiGakjqbIjepOuHHFS21NkTWnwaLcQ2WtxwZyuEbee4kUWr
yRxjzsPWfShkAPE+ky6rvPmpORXXjq7ryEJdtdVdW9oobpHqA51j/K1r3wsmwpM=
-----END CERTIFICATE-----

Client Certificate
-----BEGIN CERTIFICATE-----
MIID5TCCA06gAwIBAgIBATANBgkqhkiG9w0BAQUFADCBjzELMAkGA1UEBhMCVUsx
EDAOBgNVBAgTB0VuZ2xhbmQxHDAaBgNVBAcTE05ld2Nhc3RsZS11cG9uLXR5bmUx
EzARBgNVBAoTClphbHRlayBMdGQxEjAQBgNVBAMTCWNhLnphbHRlazEnMCUGCSqG
SIb3DQEJARYYbWFya3ltaWRkbGV0b25AZ21haWwuY29tMB4XDTEyMDQwNTAyNTIy
NVoXDTE3MDQwNDAyNTIyNVowgZQxCzAJBgNVBAYTAlVLMRAwDgYDVQQIEwdFbmds
YW5kMRwwGgYDVQQHExNOZXdjYXN0bGUtdXBvbi10eW5lMRMwEQYDVQQKEwpaYWx0
ZWsgTHRkMRcwFQYDVQQDEw5jbGllbnQuY29kZXIzNjEnMCUGCSqGSIb3DQEJARYY
bWFya3ltaWRkbGV0b25AZ21haWwuY29tMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJB
AKmb4tqyOly9t934sK7MNytO1rmJiNi/VOC0DoNImzxArVV78LznE5oqCjVxcZ5T
8Z44HMt9a5SfshJ4YH2961cCAwEAAaOCAYwwggGIMAkGA1UdEwQCMAAwEQYJYIZI
AYb4QgEBBAQDAgSwMCsGCWCGSAGG+EIBDQQeFhxUaW55Q0EgR2VuZXJhdGVkIENl
cnRpZmljYXRlMB0GA1UdDgQWBBTpnreeFat6HsREismDmymwrnGgpzCBxAYDVR0j
BIG8MIG5gBRKQO/y6OcARYu7e9uVYk+vvIaeUqGBlaSBkjCBjzELMAkGA1UEBhMC
VUsxEDAOBgNVBAgTB0VuZ2xhbmQxHDAaBgNVBAcTE05ld2Nhc3RsZS11cG9uLXR5
bmUxEzARBgNVBAoTClphbHRlayBMdGQxEjAQBgNVBAMTCWNhLnphbHRlazEnMCUG
CSqGSIb3DQEJARYYbWFya3ltaWRkbGV0b25AZ21haWwuY29tggkAt/+LyFN0Abww
IwYDVR0SBBwwGoEYbWFya3ltaWRkbGV0b25AZ21haWwuY29tMCMGA1UdEQQcMBqB
GG1hcmt5bWlkZGxldG9uQGdtYWlsLmNvbTALBgNVHQ8EBAMCBaAwDQYJKoZIhvcN
AQEFBQADgYEAIgOaDKqrgdw3CEojzb6ZRWGI7x/W8pMy9VtSlYzpoDOeQl9GMTZ/
uyLKCA6Xm6rLP81G0aZ/u1KndL9Y5aTFQjPhd1ackNGymT/LuOK4pJlq0DJJuH4q
9JD9b2NwwjTtZ4UHx/W5Q5uq03kFIKIPwgqFnitPJ9xx18kyhDcZx9c=
-----END CERTIFICATE-----

Client Private Key
-----BEGIN RSA PRIVATE KEY-----
MIIBOwIBAAJBAKmb4tqyOly9t934sK7MNytO1rmJiNi/VOC0DoNImzxArVV78Lzn
E5oqCjVxcZ5T8Z44HMt9a5SfshJ4YH2961cCAwEAAQJACz4B820M4UjR/yyUyYFP
xtca9IFhSLyl658Dt0AAyiWDqb0oOiEN6KA4nNjxe2bFmii10y37V6b8FrknWhie
AQIhAN4N7ubIgwOKsE24o4c4zz36ocgNkh8x2hTkfdzY7vhhAiEAw4mBm60gK8b7
UDnEHQ7BddLVCRRxR1bZMdKtx9AvHrcCIQC2oQUE4JMIfa+FLL+qPmoMkq8uhyLP
70Nq/YYa9nyM4QIgA8PsGQFWc0FbqYnTOvQDya8Lpycbn1IO0KTL7+MUZTUCIQDa
AsruwTONhb1A96ljtFci82oER9tkwTdvKsBBvBvJjA==
-----END RSA PRIVATE KEY-----

Server Certificate
-----BEGIN CERTIFICATE-----
MIID2DCCA0GgAwIBAgIBAjANBgkqhkiG9w0BAQUFADCBjzELMAkGA1UEBhMCVUsx
EDAOBgNVBAgTB0VuZ2xhbmQxHDAaBgNVBAcTE05ld2Nhc3RsZS11cG9uLXR5bmUx
EzARBgNVBAoTClphbHRlayBMdGQxEjAQBgNVBAMTCWNhLnphbHRlazEnMCUGCSqG
SIb3DQEJARYYbWFya3ltaWRkbGV0b25AZ21haWwuY29tMB4XDTEyMDQwNTAyNTgy
NloXDTE3MDQwNDAyNTgyNlowgZQxCzAJBgNVBAYTAlVLMRAwDgYDVQQIEwdFbmds
YW5kMRwwGgYDVQQHExNOZXdjYXN0bGUtdXBvbi1UeW5lMRMwEQYDVQQKEwpaYWx0
ZWsgTHRkMRcwFQYDVQQDEw5zZXJ2ZXIuY29kZXIzNjEnMCUGCSqGSIb3DQEJARYY
bWFya3ltaWRkbGV0b25AZ21haWwuY29tMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJB
AN+SXjkPPUjpdglZ1VeoTGjmW+YgWzkrr3a0ZzLRiV1WKi/UzbtTv37PWWDBuRLB
KYxLyfbGLOiBwwurJXMtFtUCAwEAAaOCAX8wggF7MAkGA1UdEwQCMAAwEQYJYIZI
AYb4QgEBBAQDAgZAMCsGCWCGSAGG+EIBDQQeFhxUaW55Q0EgR2VuZXJhdGVkIENl
cnRpZmljYXRlMB0GA1UdDgQWBBTZgIKxgIeIxlvxIovV1fKyvTL8ejCBxAYDVR0j
BIG8MIG5gBRKQO/y6OcARYu7e9uVYk+vvIaeUqGBlaSBkjCBjzELMAkGA1UEBhMC
VUsxEDAOBgNVBAgTB0VuZ2xhbmQxHDAaBgNVBAcTE05ld2Nhc3RsZS11cG9uLXR5
bmUxEzARBgNVBAoTClphbHRlayBMdGQxEjAQBgNVBAMTCWNhLnphbHRlazEnMCUG
CSqGSIb3DQEJARYYbWFya3ltaWRkbGV0b25AZ21haWwuY29tggkAt/+LyFN0Abww
IwYDVR0SBBwwGoEYbWFya3ltaWRkbGV0b25AZ21haWwuY29tMCMGA1UdEQQcMBqB
GG1hcmt5bWlkZGxldG9uQGdtYWlsLmNvbTANBgkqhkiG9w0BAQUFAAOBgQAYjpjF
W4S40VJtqDJ5jciXlzrCs2VmIC3MG22iRKdUzrASH3kphpqFSsV1KXnaZvrRCxZu
+2ADrTFGAzVkS80Zqwhk7+Snq1MANEtt4qCKaBfaKMvaVKDH6JD3BF55M1Rh0fiM
Ey2Qq/X86qjJ4zRlfa8EBEDCLAeg+XoTcOXNBw==
-----END CERTIFICATE-----

Server Key
-----BEGIN RSA PRIVATE KEY-----
MIIBOgIBAAJBAN+SXjkPPUjpdglZ1VeoTGjmW+YgWzkrr3a0ZzLRiV1WKi/UzbtT
v37PWWDBuRLBKYxLyfbGLOiBwwurJXMtFtUCAwEAAQJAH+EeIGLE6V2o4CJ4Ily0
MI0Qc5sPhvJsva9xu7RXMwCvAzpN/7bijBXzanSuveKuVoNQyrv5rQgfP86/s5lg
YQIhAP0lhGFxXepBEZmGEChyQyI3a7i1HtOUXyYQo+/21IEtAiEA4heCcFe+KRiU
svC+TEMz+cQbEQIv+wmEwmc1WruD5UkCIBK17y70nihuXsqmKhOTPnsM9Z1VZc3l
rZK2sCdQayyRAiBGPReA30Fscv5/ONtlxdr31TyzZAKT8rkTzAx/S5IGQQIhAOi+
6gm1IHJagF7n3BJj4K5Sqx2ca4Bo/sn+5ogBuoOp
-----END RSA PRIVATE KEY-----

Certificate Trust chain

Wednesday, 26 September 2012

Spring batch with Hibernate Part 1

Hibernate is a great ORM for web applications (it's traditional usage pattern), but it can also be very good at batch. A word of warning, though, you need to be very careful as your HQL query can result in multiple SQL calls, which would be a disaster for batch.

This post describes how hibernate can be configured to operate in a batch mode to minimizes the number of SQL calls to the underlying database.

Enabling batch mode

Configure hibernate with the following properties (see improving hibernate performacne):

hibernate.jdbc.batch_size=1000
hibernate.default_batch_fetch_size=1000

eg.

<bean id="sessionFactory" class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
<property name="dataSource" ref="dataSource"/>
<property name="packagesToScan">
<list>
<value>coder36.sbent.sample.domain</value>
<value>coder36.sbent.sample.domain.stage</value>
</list>
</property>
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.MySQLDialect</prop>
<prop key="hibernate.jdbc.batch_size">1000</prop>
<prop key="hibernate.default_batch_fetch_size">1000</prop>
</props>
</property>
</bean

A good tip, is to also enable hibernate SQL logging, so you can see exactly what hibernate is doing behind the scenes (see here for the types of logging output) :

<prop key="hibernate.show_sql">true</prop>
<prop key="hibernate.format_sql">true</prop>
<prop key="hibernate.use_sql_comments">true</prop>

How does hibernate batch mode work?

The best way to describe it is via an example taken from sbent. Have the following entities:

@Entity
public class Transaction {
...

@ManyToOne(fetch=FetchType.LAZY)
Customer customer;

public Customer getCustomer() {
return customer;
}
}

and

@Entity
public class Customer {
...

@Column( nullable=false, length=60 )
private String name;

public String getName() {
return name;
}

}

To demonstrate what happens under the covers, load some data, then retrieve it with the following code:

List<Transaction> trans =
session.createQuery( "select t from Transaction t" ).list();
for ( Transaction t: trans ) {
System.out.println( "getting customer");
Customer c = t.getCustomer();
System.out.println( "name: " + c.getName() );
}

With hibernate SQL logging turmed on, the following is sent to stdout:

getting list of Transaction's
Hibernate:
/* select t from Transaction t */
select
transactio0_.id as id6_,
transactio0_.amount as amount6_,
transactio0_.bank_id as bank4_6_,
transactio0_.customer_id as customer5_6_,
transactio0_.version as version6_
from
Transaction transactio0_
getting customer
Hibernate:
/* load coder36.sbent.sample.domain.Customer */
select
customer0_.id as id1_0_,
customer0_.name as name1_0_,
customer0_.nino as nino1_0_,
customer0_.version as version1_0_
from
Customer customer0_
where
customer0_.id in ( ?, ?, ? )
name: Mark Middleton
getting customer
name: John Smith
getting customer
name: Fred Dibnah

What's happening here?

The Transaction entity is configured to lazily load its Customer. So when the customer is first accessed, (c.getName()), the lazy load is triggered. As batch mode is enabled, hibernate also triggers the lazy load of all Transaction.customer's for all Transaction entities in the cache - hence the customer0_.id in ( ?,?,? ) bit.

Summarize

Enable hibernate batch mode by setting hibernate.jdbc.batch_size and hibernate.default_batch_fetch_size
Use lazy loading - ensure hibernate entities are correctly configured with fetch=FetchType.LAZY
Enable SQL logging during unit testing, to prove hibernate is behaving as expected.

Tuesday, 25 September 2012

Spring Batch in the Enterprise

Large organizations by their nature are reluctant to move wholesale over to new technologies and this can result in a mishmash of new, legacy and bespoke components, all of which want a piece of the action. They also stick with what they know works. I'm always looking for software solutions which are a good fit, and sometimes I'm not so sure when it comes to spring batch and the enterprise.

Why Spring batch?

The great thing about spring batch, is that it provides the majority of what you need in terms of the framework. This allows you to invest your effort and time writing business code (this is the genius of spring). What it doesn't do so well, is to bridge onto legacy components, bespoke components and working patterns that prevail in the enterprise.

How can spring batch be made a better fit for the enterprise?

1) Work out batch patterns which fit with the enterprise. i.e. How does hibernate fit with batch?
2) Bridge onto enterprise components (i.e. the enterprise scheduler)
3) Work with enterprise data strategies (i.e. stream from database, rather than a file)
4) Monitoring tools (i.e. need to know if errors have occured)
5) Testing tools - testing to align with system testers. In the enterprise it's all about integration testing.

What is SBENT?

Spring Batch Extensions for the Enterprise (SBENT) is a thin software bridge built to span 'out of the box' spring batch and the corporate enterprise. It's a set of Readers, Writers, batch admin extensions, XML processing tasklets and design patterns.

SBENT is released under Apache License Version 2.0.

The sample batch project

The best way to describe SBENT is via a reference batch project. spent-sample makes use of the various

ItemWriters, ItemReaders and so on provided by SBENT.

The Batch scenario

1) Bank to push a return in the form of XML onto an AMQP Queue (RabbitMQ). The return contains all transactions its customers have made across a given period:

<bankName>Brookside Bank PLC</bankName>

</header>

<name>Mark Middleton</name>

</customer>

<name>John Smith</name>

</customer>

</return>

2) Pick up the XML submission, and save it to the database.

3) Parse the XML submission and save into staging tables

4) Process the staged data, saving it into the "domain model"

5) Generate "ticker" reporting customers overall transaction balance (a person can be a customer in many banks)

6) Publish ticker to AMQP Queue.