Coder36: September 2012

Thursday, 27 September 2012

Apache SSL reverse proxy tutorial

In this tutorial, we will take a high level requirement, work out a design, then prototype a solution.

High level Requirement

All communication must be encrypted and trusted

Analysis

In terms of our java application which calls a remote web service, this means:

We need to authenticate the remote web service - they are who they say they are
The remote web service needs to authenticate us - we are who we say we are
http calls to be encrypted

A good technology fit for this problem is SSL with mutual authentication. We will use apache as an SSL reverse proxy (this will forward our plain http requests to the remote web service, applying SSL). Will use certificate based authentication to prove the authenticity of the server and client.

What is a Certificate trust chain?

Certificate chains are all about trust. In the diagram below the certificate authority ca.zaltek has signed the certificates for "Bob Ltd" and "Fred Ltd". The certificate authority is saying "I trust Bob Ltd and Fred Ltd".

Now the leap - If I trust that certificate authority (ca.zaltek) will only sign certificates for legitimate organizations, then I can trust any organization who presents me with a certificate signed by ca.zaltek.

That's SSL authentication in a nutshell

Target Architecture

In the next section we will create 2 certificates and get them signed by a certificate authority. We are going to use the same certificate authority for both, but equally we could used different ones - as long as we trust them. We will then set up an apache ssl reverse proxy and emulate a remote SSL web servers.

Generate certificates

Will generate private keys and certificates for both the client and the server using openssl and tinyca:

mkdir c:/ssl
mkdir c:/ssl/client
mkdir c:/ssl/server
mkdir c:/ssl/cacerts
mkdir c:/ssl/apache

# Generate private keys and remove password protection
cd c:/ssl/client
openssl genrsa -des3 -out key_.pem 512 (use password)
openssl rsa -in key_.pem -out key.pem
rm key_.pem
# Generate CSR (Certificate Signing Request)
openssl req -out csr.pem -key key.pem -new (cn=client.coder36)

cd ../server
openssl genrsa -des3 -out key_.pem 512 (use password)
openssl rsa -in key_.pem -out key.pem
rm key_.pem
# Generate CSR (Certificate Signing Request)
openssl req -out csr.pem -key key.pem -new (cn=server.coder36)

The next step is to get the CSR's signed. Openssl can do this, however I prefer tinyca See here for a quick guide to using tinyca. Copy the signed certificates into their respective folders ssl/client/cert.pem and ssl/server/cert.pem. Copy the ca certificate into ssl/cacerts/ca.pem.

Certificates can be viewed using:
openssl x509 -in cert.pem -noout -text

Checking certificates and keys are all working:

The easiest way to confirm that the keys and certificates are playing nicely is to use openssl as a server.
cd c:/ssl/server
openssl s_server -accept 443 -www -key key.pem -cert cert.pem -CAfile ../cacerts/ca.pem -Verify 1

This will set the server listening on port 443 (the SSL port)

In another session, start the client:

openssl s_client -connect localhost:443 -verify 1 -CAfile ../cacerts/ca.pem -cert cert.pem -key key.pem
get /

If everything is as it should be, the client will perform the SSL handshake without any errors as per the above picture.

Okay, so this is the base line. We know the certificates work. The next step is to get the certificates working with apache.

Setting up apache

Download the latest version of apache (2.4) from here. Extract to c:/

Combine server certificate and key into single file
cd c:/ssl/server
cat cert.pem > certandkey.pem
cat key.pem >> certandkey.pem

Create apache config c:/ssl/apache/proxy.conf :

LoadModule authz_core_module modules/mod_authz_core.so
LoadModule logio_module modules/mod_logio.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule ssl_module modules/mod_ssl.so

LogFormat "%h %l %u %t \"%r\" %>s %b" common
CustomLog "c:/ssl/apache/access.log" common
ErrorLog "c:/ssl/apache/error.log"
LogLevel warn

Listen 80

<VirtualHost localhost:80>
SSLProxyEngine on
SSLProxyCACertificateFile "c:/ssl/cacerts/ca.pem"
SSLProxyVerify require
SSLProxyVerifyDepth 2
SSLProxyMachineCertificateFile "c:/ssl/server/certandkey.pem"
<Proxy *>
</Proxy>

<Location />
ProxyPass https://server.coder36:443/
ProxyPassReverse https://server.coder36:443/
</Location>
</VirtualHost>

Update hosts file:
127.0.0.1 server.coder36
127.0.0.1 client.coder36

Testing

Start up apache:
cd c:/apache24/bin
httpd -f c:/ssl/apache/proxy.conf

In another session, start the openssl server:

cd c:/ssl/server
openssl s_server -accept 443 -www -key key.pem -cert cert.pem -CAfile ../cacerts/ca.pem -Verify 1

Test 1

http https
Browser -------> Apache -------> Openssl Server

Fire up a browser and go to: http://server.coder36. If everything is working you should be presented with the openssl server diagnostic page.

Test 2
http https
Java App -------> Apache -------> Openssl Server

To test from a java app use the following code:

import java.io.*;
import java.net.*;

public class Client {
public static void main( String [] args ) {
try {
URL url = new URL( args[0] );
URLConnection con = url.openConnection();
InputStream is = con.getInputStream();
BufferedReader br = new BufferedReader( new InputStreamReader( is ));
String s = "";
while ( (s = br.readLine()) != null ) {
System.out.println( s );
}
is.close();
}
catch( Exception e ) {
e.printStackTrace();
}
}
}

To run::
java Client http://server.coder36

Resources

CA Certificate:
-----BEGIN CERTIFICATE-----
MIIEGzCCA4SgAwIBAgIJALf/i8hTdAG8MA0GCSqGSIb3DQEBBQUAMIGPMQswCQYD
VQQGEwJVSzEQMA4GA1UECBMHRW5nbGFuZDEcMBoGA1UEBxMTTmV3Y2FzdGxlLXVw
b24tdHluZTETMBEGA1UEChMKWmFsdGVrIEx0ZDESMBAGA1UEAxMJY2EuemFsdGVr
MScwJQYJKoZIhvcNAQkBFhhtYXJreW1pZGRsZXRvbkBnbWFpbC5jb20wHhcNMTIw
NDA1MDI0MzQ2WhcNMjIwNDAzMDI0MzQ2WjCBjzELMAkGA1UEBhMCVUsxEDAOBgNV
BAgTB0VuZ2xhbmQxHDAaBgNVBAcTE05ld2Nhc3RsZS11cG9uLXR5bmUxEzARBgNV
BAoTClphbHRlayBMdGQxEjAQBgNVBAMTCWNhLnphbHRlazEnMCUGCSqGSIb3DQEJ
ARYYbWFya3ltaWRkbGV0b25AZ21haWwuY29tMIGfMA0GCSqGSIb3DQEBAQUAA4GN
ADCBiQKBgQCv/X87xEwkmpoKMQE+KA5BHhCnF70OOcvn9sO/FONvNIX7DA7HUe6Y
WrrE8nKHLWt8XTDvbJKzbq2DyxOhPJZ759IbDBlmUpHQDfPexBL5Rg7AJR41TMAw
SQAqjGk03hPckptWxK1mEzib9X8XO2dkUe9KUuSqoYiht6ScxBXsNwIDAQABo4IB
ezCCAXcwHQYDVR0OBBYEFEpA7/Lo5wBFi7t725ViT6+8hp5SMIHEBgNVHSMEgbww
gbmAFEpA7/Lo5wBFi7t725ViT6+8hp5SoYGVpIGSMIGPMQswCQYDVQQGEwJVSzEQ
MA4GA1UECBMHRW5nbGFuZDEcMBoGA1UEBxMTTmV3Y2FzdGxlLXVwb24tdHluZTET
MBEGA1UEChMKWmFsdGVrIEx0ZDESMBAGA1UEAxMJY2EuemFsdGVrMScwJQYJKoZI
hvcNAQkBFhhtYXJreW1pZGRsZXRvbkBnbWFpbC5jb22CCQC3/4vIU3QBvDAPBgNV
HRMBAf8EBTADAQH/MBEGCWCGSAGG+EIBAQQEAwIBBjAJBgNVHRIEAjAAMCsGCWCG
SAGG+EIBDQQeFhxUaW55Q0EgR2VuZXJhdGVkIENlcnRpZmljYXRlMCMGA1UdEQQc
MBqBGG1hcmt5bWlkZGxldG9uQGdtYWlsLmNvbTAOBgNVHQ8BAf8EBAMCAQYwDQYJ
KoZIhvcNAQEFBQADgYEACVt3BMuusRAOW3G6w4T11FZFbJnKn5/GY5KFZQtn5hFV
AheWyWMrDPcHsxTW8EiGakjqbIjepOuHHFS21NkTWnwaLcQ2WtxwZyuEbee4kUWr
yRxjzsPWfShkAPE+ky6rvPmpORXXjq7ryEJdtdVdW9oobpHqA51j/K1r3wsmwpM=
-----END CERTIFICATE-----

Client Certificate
-----BEGIN CERTIFICATE-----
MIID5TCCA06gAwIBAgIBATANBgkqhkiG9w0BAQUFADCBjzELMAkGA1UEBhMCVUsx
EDAOBgNVBAgTB0VuZ2xhbmQxHDAaBgNVBAcTE05ld2Nhc3RsZS11cG9uLXR5bmUx
EzARBgNVBAoTClphbHRlayBMdGQxEjAQBgNVBAMTCWNhLnphbHRlazEnMCUGCSqG
SIb3DQEJARYYbWFya3ltaWRkbGV0b25AZ21haWwuY29tMB4XDTEyMDQwNTAyNTIy
NVoXDTE3MDQwNDAyNTIyNVowgZQxCzAJBgNVBAYTAlVLMRAwDgYDVQQIEwdFbmds
YW5kMRwwGgYDVQQHExNOZXdjYXN0bGUtdXBvbi10eW5lMRMwEQYDVQQKEwpaYWx0
ZWsgTHRkMRcwFQYDVQQDEw5jbGllbnQuY29kZXIzNjEnMCUGCSqGSIb3DQEJARYY
bWFya3ltaWRkbGV0b25AZ21haWwuY29tMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJB
AKmb4tqyOly9t934sK7MNytO1rmJiNi/VOC0DoNImzxArVV78LznE5oqCjVxcZ5T
8Z44HMt9a5SfshJ4YH2961cCAwEAAaOCAYwwggGIMAkGA1UdEwQCMAAwEQYJYIZI
AYb4QgEBBAQDAgSwMCsGCWCGSAGG+EIBDQQeFhxUaW55Q0EgR2VuZXJhdGVkIENl
cnRpZmljYXRlMB0GA1UdDgQWBBTpnreeFat6HsREismDmymwrnGgpzCBxAYDVR0j
BIG8MIG5gBRKQO/y6OcARYu7e9uVYk+vvIaeUqGBlaSBkjCBjzELMAkGA1UEBhMC
VUsxEDAOBgNVBAgTB0VuZ2xhbmQxHDAaBgNVBAcTE05ld2Nhc3RsZS11cG9uLXR5
bmUxEzARBgNVBAoTClphbHRlayBMdGQxEjAQBgNVBAMTCWNhLnphbHRlazEnMCUG
CSqGSIb3DQEJARYYbWFya3ltaWRkbGV0b25AZ21haWwuY29tggkAt/+LyFN0Abww
IwYDVR0SBBwwGoEYbWFya3ltaWRkbGV0b25AZ21haWwuY29tMCMGA1UdEQQcMBqB
GG1hcmt5bWlkZGxldG9uQGdtYWlsLmNvbTALBgNVHQ8EBAMCBaAwDQYJKoZIhvcN
AQEFBQADgYEAIgOaDKqrgdw3CEojzb6ZRWGI7x/W8pMy9VtSlYzpoDOeQl9GMTZ/
uyLKCA6Xm6rLP81G0aZ/u1KndL9Y5aTFQjPhd1ackNGymT/LuOK4pJlq0DJJuH4q
9JD9b2NwwjTtZ4UHx/W5Q5uq03kFIKIPwgqFnitPJ9xx18kyhDcZx9c=
-----END CERTIFICATE-----

Client Private Key
-----BEGIN RSA PRIVATE KEY-----
MIIBOwIBAAJBAKmb4tqyOly9t934sK7MNytO1rmJiNi/VOC0DoNImzxArVV78Lzn
E5oqCjVxcZ5T8Z44HMt9a5SfshJ4YH2961cCAwEAAQJACz4B820M4UjR/yyUyYFP
xtca9IFhSLyl658Dt0AAyiWDqb0oOiEN6KA4nNjxe2bFmii10y37V6b8FrknWhie
AQIhAN4N7ubIgwOKsE24o4c4zz36ocgNkh8x2hTkfdzY7vhhAiEAw4mBm60gK8b7
UDnEHQ7BddLVCRRxR1bZMdKtx9AvHrcCIQC2oQUE4JMIfa+FLL+qPmoMkq8uhyLP
70Nq/YYa9nyM4QIgA8PsGQFWc0FbqYnTOvQDya8Lpycbn1IO0KTL7+MUZTUCIQDa
AsruwTONhb1A96ljtFci82oER9tkwTdvKsBBvBvJjA==
-----END RSA PRIVATE KEY-----

Server Certificate
-----BEGIN CERTIFICATE-----
MIID2DCCA0GgAwIBAgIBAjANBgkqhkiG9w0BAQUFADCBjzELMAkGA1UEBhMCVUsx
EDAOBgNVBAgTB0VuZ2xhbmQxHDAaBgNVBAcTE05ld2Nhc3RsZS11cG9uLXR5bmUx
EzARBgNVBAoTClphbHRlayBMdGQxEjAQBgNVBAMTCWNhLnphbHRlazEnMCUGCSqG
SIb3DQEJARYYbWFya3ltaWRkbGV0b25AZ21haWwuY29tMB4XDTEyMDQwNTAyNTgy
NloXDTE3MDQwNDAyNTgyNlowgZQxCzAJBgNVBAYTAlVLMRAwDgYDVQQIEwdFbmds
YW5kMRwwGgYDVQQHExNOZXdjYXN0bGUtdXBvbi1UeW5lMRMwEQYDVQQKEwpaYWx0
ZWsgTHRkMRcwFQYDVQQDEw5zZXJ2ZXIuY29kZXIzNjEnMCUGCSqGSIb3DQEJARYY
bWFya3ltaWRkbGV0b25AZ21haWwuY29tMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJB
AN+SXjkPPUjpdglZ1VeoTGjmW+YgWzkrr3a0ZzLRiV1WKi/UzbtTv37PWWDBuRLB
KYxLyfbGLOiBwwurJXMtFtUCAwEAAaOCAX8wggF7MAkGA1UdEwQCMAAwEQYJYIZI
AYb4QgEBBAQDAgZAMCsGCWCGSAGG+EIBDQQeFhxUaW55Q0EgR2VuZXJhdGVkIENl
cnRpZmljYXRlMB0GA1UdDgQWBBTZgIKxgIeIxlvxIovV1fKyvTL8ejCBxAYDVR0j
BIG8MIG5gBRKQO/y6OcARYu7e9uVYk+vvIaeUqGBlaSBkjCBjzELMAkGA1UEBhMC
VUsxEDAOBgNVBAgTB0VuZ2xhbmQxHDAaBgNVBAcTE05ld2Nhc3RsZS11cG9uLXR5
bmUxEzARBgNVBAoTClphbHRlayBMdGQxEjAQBgNVBAMTCWNhLnphbHRlazEnMCUG
CSqGSIb3DQEJARYYbWFya3ltaWRkbGV0b25AZ21haWwuY29tggkAt/+LyFN0Abww
IwYDVR0SBBwwGoEYbWFya3ltaWRkbGV0b25AZ21haWwuY29tMCMGA1UdEQQcMBqB
GG1hcmt5bWlkZGxldG9uQGdtYWlsLmNvbTANBgkqhkiG9w0BAQUFAAOBgQAYjpjF
W4S40VJtqDJ5jciXlzrCs2VmIC3MG22iRKdUzrASH3kphpqFSsV1KXnaZvrRCxZu
+2ADrTFGAzVkS80Zqwhk7+Snq1MANEtt4qCKaBfaKMvaVKDH6JD3BF55M1Rh0fiM
Ey2Qq/X86qjJ4zRlfa8EBEDCLAeg+XoTcOXNBw==
-----END CERTIFICATE-----

Server Key
-----BEGIN RSA PRIVATE KEY-----
MIIBOgIBAAJBAN+SXjkPPUjpdglZ1VeoTGjmW+YgWzkrr3a0ZzLRiV1WKi/UzbtT
v37PWWDBuRLBKYxLyfbGLOiBwwurJXMtFtUCAwEAAQJAH+EeIGLE6V2o4CJ4Ily0
MI0Qc5sPhvJsva9xu7RXMwCvAzpN/7bijBXzanSuveKuVoNQyrv5rQgfP86/s5lg
YQIhAP0lhGFxXepBEZmGEChyQyI3a7i1HtOUXyYQo+/21IEtAiEA4heCcFe+KRiU
svC+TEMz+cQbEQIv+wmEwmc1WruD5UkCIBK17y70nihuXsqmKhOTPnsM9Z1VZc3l
rZK2sCdQayyRAiBGPReA30Fscv5/ONtlxdr31TyzZAKT8rkTzAx/S5IGQQIhAOi+
6gm1IHJagF7n3BJj4K5Sqx2ca4Bo/sn+5ogBuoOp
-----END RSA PRIVATE KEY-----

Certificate Trust chain

Wednesday, 26 September 2012

Spring batch with Hibernate Part 1

Hibernate is a great ORM for web applications (it's traditional usage pattern), but it can also be very good at batch. A word of warning, though, you need to be very careful as your HQL query can result in multiple SQL calls, which would be a disaster for batch.

This post describes how hibernate can be configured to operate in a batch mode to minimizes the number of SQL calls to the underlying database.

Enabling batch mode

Configure hibernate with the following properties (see improving hibernate performacne):

hibernate.jdbc.batch_size=1000
hibernate.default_batch_fetch_size=1000

eg.

<bean id="sessionFactory" class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
<property name="dataSource" ref="dataSource"/>
<property name="packagesToScan">
<list>
<value>coder36.sbent.sample.domain</value>
<value>coder36.sbent.sample.domain.stage</value>
</list>
</property>
<property name="hibernateProperties">
<props>
<prop key="hibernate.dialect">org.hibernate.dialect.MySQLDialect</prop>
<prop key="hibernate.jdbc.batch_size">1000</prop>
<prop key="hibernate.default_batch_fetch_size">1000</prop>
</props>
</property>
</bean

A good tip, is to also enable hibernate SQL logging, so you can see exactly what hibernate is doing behind the scenes (see here for the types of logging output) :

<prop key="hibernate.show_sql">true</prop>
<prop key="hibernate.format_sql">true</prop>
<prop key="hibernate.use_sql_comments">true</prop>

How does hibernate batch mode work?

The best way to describe it is via an example taken from sbent. Have the following entities:

@Entity
public class Transaction {
...

@ManyToOne(fetch=FetchType.LAZY)
Customer customer;

public Customer getCustomer() {
return customer;
}
}

and

@Entity
public class Customer {
...

@Column( nullable=false, length=60 )
private String name;

public String getName() {
return name;
}

}

To demonstrate what happens under the covers, load some data, then retrieve it with the following code:

List<Transaction> trans =
session.createQuery( "select t from Transaction t" ).list();
for ( Transaction t: trans ) {
System.out.println( "getting customer");
Customer c = t.getCustomer();
System.out.println( "name: " + c.getName() );
}

With hibernate SQL logging turmed on, the following is sent to stdout:

getting list of Transaction's
Hibernate:
/* select t from Transaction t */
select
transactio0_.id as id6_,
transactio0_.amount as amount6_,
transactio0_.bank_id as bank4_6_,
transactio0_.customer_id as customer5_6_,
transactio0_.version as version6_
from
Transaction transactio0_
getting customer
Hibernate:
/* load coder36.sbent.sample.domain.Customer */
select
customer0_.id as id1_0_,
customer0_.name as name1_0_,
customer0_.nino as nino1_0_,
customer0_.version as version1_0_
from
Customer customer0_
where
customer0_.id in ( ?, ?, ? )
name: Mark Middleton
getting customer
name: John Smith
getting customer
name: Fred Dibnah

What's happening here?

The Transaction entity is configured to lazily load its Customer. So when the customer is first accessed, (c.getName()), the lazy load is triggered. As batch mode is enabled, hibernate also triggers the lazy load of all Transaction.customer's for all Transaction entities in the cache - hence the customer0_.id in ( ?,?,? ) bit.

Summarize

Enable hibernate batch mode by setting hibernate.jdbc.batch_size and hibernate.default_batch_fetch_size
Use lazy loading - ensure hibernate entities are correctly configured with fetch=FetchType.LAZY
Enable SQL logging during unit testing, to prove hibernate is behaving as expected.

Tuesday, 25 September 2012

Spring Batch in the Enterprise

Large organizations by their nature are reluctant to move wholesale over to new technologies and this can result in a mishmash of new, legacy and bespoke components, all of which want a piece of the action. They also stick with what they know works. I'm always looking for software solutions which are a good fit, and sometimes I'm not so sure when it comes to spring batch and the enterprise.

Why Spring batch?

The great thing about spring batch, is that it provides the majority of what you need in terms of the framework. This allows you to invest your effort and time writing business code (this is the genius of spring). What it doesn't do so well, is to bridge onto legacy components, bespoke components and working patterns that prevail in the enterprise.

How can spring batch be made a better fit for the enterprise?

1) Work out batch patterns which fit with the enterprise. i.e. How does hibernate fit with batch?
2) Bridge onto enterprise components (i.e. the enterprise scheduler)
3) Work with enterprise data strategies (i.e. stream from database, rather than a file)
4) Monitoring tools (i.e. need to know if errors have occured)
5) Testing tools - testing to align with system testers. In the enterprise it's all about integration testing.

What is SBENT?

Spring Batch Extensions for the Enterprise (SBENT) is a thin software bridge built to span 'out of the box' spring batch and the corporate enterprise. It's a set of Readers, Writers, batch admin extensions, XML processing tasklets and design patterns.

SBENT is released under Apache License Version 2.0.

The sample batch project

The best way to describe SBENT is via a reference batch project. spent-sample makes use of the various

ItemWriters, ItemReaders and so on provided by SBENT.

The Batch scenario

1) Bank to push a return in the form of XML onto an AMQP Queue (RabbitMQ). The return contains all transactions its customers have made across a given period:

<bankName>Brookside Bank PLC</bankName>

</header>

<name>Mark Middleton</name>

</customer>

<name>John Smith</name>

</customer>

</return>

2) Pick up the XML submission, and save it to the database.

3) Parse the XML submission and save into staging tables

4) Process the staged data, saving it into the "domain model"

5) Generate "ticker" reporting customers overall transaction balance (a person can be a customer in many banks)

6) Publish ticker to AMQP Queue.