Apache SSL reverse proxy tutorial

In this tutorial, we will take a high level requirement, work out a design, then prototype a solution.

High level Requirement

• All communication  must be encrypted and trusted

Analysis

In terms of our java application which calls a remote web service, this means:

1. We need to authenticate the remote web service - they are who they say they are
2. The remote web service needs to authenticate us -  we are who we say we are
3. http calls to be encrypted

A good technology fit for this problem is  SSL with mutual authentication.  We will use apache as an SSL  reverse proxy (this will forward our plain http requests to the remote web service, applying SSL).  Will use certificate based authentication to prove the authenticity of the server and client.

What is a Certificate trust chain?

Certificate chains are all about trust.  In the diagram below the certificate authority ca.zaltek has signed the certificates for "Bob Ltd" and "Fred Ltd".  The certificate authority is saying "I trust Bob Ltd and Fred Ltd".

Now the leap - If I trust that certificate authority (ca.zaltek) will only sign certificates for legitimate organizations, then I can trust any organization who presents me with a certificate signed by ca.zaltek.  

That's SSL authentication in a nutshell

Target Architecture

   

In the next section we will create 2 certificates and get them signed by a certificate authority.  We are going to use the same certificate authority for both, but equally we could used different ones - as long as we trust them.  We will then set up an apache ssl reverse proxy and emulate a remote SSL web servers.

Generate certificates

Will generate private keys and certificates for both the client and the server using openssl and tinyca:

mkdir c:/ssl
mkdir c:/ssl/client
mkdir c:/ssl/server
mkdir c:/ssl/cacerts
mkdir c:/ssl/apache

# Generate private keys and remove password protection
cd c:/ssl/client
openssl genrsa -des3 -out key_.pem 512      (use password)
openssl rsa -in key_.pem -out key.pem      
rm key_.pem
# Generate CSR (Certificate Signing Request)
openssl req -out csr.pem -key key.pem -new  (cn=client.coder36)

cd ../server
openssl genrsa -des3 -out key_.pem 512      (use password)
openssl rsa -in key_.pem -out key.pem
rm key_.pem
# Generate CSR (Certificate Signing Request)
openssl req -out csr.pem -key key.pem -new  (cn=server.coder36)

The next step is to get the CSR's signed.  Openssl can do this, however I prefer tinyca  See here for a quick guide to using tinyca.  Copy the signed certificates into their respective folders ssl/client/cert.pem and ssl/server/cert.pem.  Copy the ca certificate into ssl/cacerts/ca.pem.

Certificates can be viewed using:
openssl x509 -in cert.pem -noout -text

Checking certificates and keys are all working:

The easiest way to confirm that the keys and certificates are playing nicely is to use openssl as a server.
cd c:/ssl/server
openssl s_server -accept 443 -www -key key.pem -cert cert.pem -CAfile ../cacerts/ca.pem -Verify 1

This will set the server listening on port 443 (the SSL port)

In another session, start the client:

openssl s_client -connect localhost:443 -verify 1 -CAfile ../cacerts/ca.pem -cert cert.pem -key key.pem
get /

If everything is as it should be, the client will perform the SSL handshake without any errors as per the above  picture.

Okay, so this is the base line.  We know the certificates work.  The next step is to get the certificates working with apache.

Setting up apache

Download the latest version of apache (2.4) from here.  Extract to c:/

Combine server certificate and key into single file
cd c:/ssl/server
cat cert.pem > certandkey.pem
cat key.pem >> certandkey.pem

Create apache config c:/ssl/apache/proxy.conf :

LoadModule authz_core_module modules/mod_authz_core.so
LoadModule logio_module modules/mod_logio.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule ssl_module modules/mod_ssl.so

LogFormat "%h %l %u %t \"%r\" %>s %b" common
CustomLog "c:/ssl/apache/access.log" common
ErrorLog "c:/ssl/apache/error.log"
LogLevel warn

Listen 80

<VirtualHost localhost:80>
   SSLProxyEngine on
   SSLProxyCACertificateFile "c:/ssl/cacerts/ca.pem"
   SSLProxyVerify require
   SSLProxyVerifyDepth  2
   SSLProxyMachineCertificateFile "c:/ssl/server/certandkey.pem"
   <Proxy *>
   </Proxy>

   <Location />
     ProxyPass          https://server.coder36:443/
     ProxyPassReverse   https://server.coder36:443/
   </Location>
</VirtualHost>


Update  hosts file:
127.0.0.1 server.coder36
127.0.0.1 client.coder36

Testing

Start up apache:
cd c:/apache24/bin
httpd -f c:/ssl/apache/proxy.conf

In another session, start the openssl server:

cd c:/ssl/server
openssl s_server -accept 443 -www -key key.pem -cert cert.pem -CAfile ../cacerts/ca.pem -Verify 1

Test 1

        http            https
Browser -------> Apache -------> Openssl Server

Fire up a browser and go to: http://server.coder36.  If  everything is working you should be presented with the openssl server diagnostic page.

Test 2
         http            https
Java App -------> Apache -------> Openssl Server


To test from a java app use the following code:

import java.io.*;
import java.net.*;

public class Client {
  public static void main( String [] args ) {
    try {
      URL url = new URL( args[0] );
      URLConnection con = url.openConnection();
      InputStream is = con.getInputStream();
      BufferedReader br = new  BufferedReader( new InputStreamReader( is ));
      String s = "";
      while ( (s = br.readLine()) != null ) {
     System.out.println( s );
      }
      is.close();
    }
    catch( Exception e ) {
      e.printStackTrace();
    }
  }
}

To run::
java Client http://server.coder36

Resources

CA Certificate:
-----BEGIN CERTIFICATE-----
MIIEGzCCA4SgAwIBAgIJALf/i8hTdAG8MA0GCSqGSIb3DQEBBQUAMIGPMQswCQYD
VQQGEwJVSzEQMA4GA1UECBMHRW5nbGFuZDEcMBoGA1UEBxMTTmV3Y2FzdGxlLXVw
b24tdHluZTETMBEGA1UEChMKWmFsdGVrIEx0ZDESMBAGA1UEAxMJY2EuemFsdGVr
MScwJQYJKoZIhvcNAQkBFhhtYXJreW1pZGRsZXRvbkBnbWFpbC5jb20wHhcNMTIw
NDA1MDI0MzQ2WhcNMjIwNDAzMDI0MzQ2WjCBjzELMAkGA1UEBhMCVUsxEDAOBgNV
BAgTB0VuZ2xhbmQxHDAaBgNVBAcTE05ld2Nhc3RsZS11cG9uLXR5bmUxEzARBgNV
BAoTClphbHRlayBMdGQxEjAQBgNVBAMTCWNhLnphbHRlazEnMCUGCSqGSIb3DQEJ
ARYYbWFya3ltaWRkbGV0b25AZ21haWwuY29tMIGfMA0GCSqGSIb3DQEBAQUAA4GN
ADCBiQKBgQCv/X87xEwkmpoKMQE+KA5BHhCnF70OOcvn9sO/FONvNIX7DA7HUe6Y
WrrE8nKHLWt8XTDvbJKzbq2DyxOhPJZ759IbDBlmUpHQDfPexBL5Rg7AJR41TMAw
SQAqjGk03hPckptWxK1mEzib9X8XO2dkUe9KUuSqoYiht6ScxBXsNwIDAQABo4IB
ezCCAXcwHQYDVR0OBBYEFEpA7/Lo5wBFi7t725ViT6+8hp5SMIHEBgNVHSMEgbww
gbmAFEpA7/Lo5wBFi7t725ViT6+8hp5SoYGVpIGSMIGPMQswCQYDVQQGEwJVSzEQ
MA4GA1UECBMHRW5nbGFuZDEcMBoGA1UEBxMTTmV3Y2FzdGxlLXVwb24tdHluZTET
MBEGA1UEChMKWmFsdGVrIEx0ZDESMBAGA1UEAxMJY2EuemFsdGVrMScwJQYJKoZI
hvcNAQkBFhhtYXJreW1pZGRsZXRvbkBnbWFpbC5jb22CCQC3/4vIU3QBvDAPBgNV
HRMBAf8EBTADAQH/MBEGCWCGSAGG+EIBAQQEAwIBBjAJBgNVHRIEAjAAMCsGCWCG
SAGG+EIBDQQeFhxUaW55Q0EgR2VuZXJhdGVkIENlcnRpZmljYXRlMCMGA1UdEQQc
MBqBGG1hcmt5bWlkZGxldG9uQGdtYWlsLmNvbTAOBgNVHQ8BAf8EBAMCAQYwDQYJ
KoZIhvcNAQEFBQADgYEACVt3BMuusRAOW3G6w4T11FZFbJnKn5/GY5KFZQtn5hFV
AheWyWMrDPcHsxTW8EiGakjqbIjepOuHHFS21NkTWnwaLcQ2WtxwZyuEbee4kUWr
yRxjzsPWfShkAPE+ky6rvPmpORXXjq7ryEJdtdVdW9oobpHqA51j/K1r3wsmwpM=
-----END CERTIFICATE-----

Client Certificate
-----BEGIN CERTIFICATE-----
MIID5TCCA06gAwIBAgIBATANBgkqhkiG9w0BAQUFADCBjzELMAkGA1UEBhMCVUsx
EDAOBgNVBAgTB0VuZ2xhbmQxHDAaBgNVBAcTE05ld2Nhc3RsZS11cG9uLXR5bmUx
EzARBgNVBAoTClphbHRlayBMdGQxEjAQBgNVBAMTCWNhLnphbHRlazEnMCUGCSqG
SIb3DQEJARYYbWFya3ltaWRkbGV0b25AZ21haWwuY29tMB4XDTEyMDQwNTAyNTIy
NVoXDTE3MDQwNDAyNTIyNVowgZQxCzAJBgNVBAYTAlVLMRAwDgYDVQQIEwdFbmds
YW5kMRwwGgYDVQQHExNOZXdjYXN0bGUtdXBvbi10eW5lMRMwEQYDVQQKEwpaYWx0
ZWsgTHRkMRcwFQYDVQQDEw5jbGllbnQuY29kZXIzNjEnMCUGCSqGSIb3DQEJARYY
bWFya3ltaWRkbGV0b25AZ21haWwuY29tMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJB
AKmb4tqyOly9t934sK7MNytO1rmJiNi/VOC0DoNImzxArVV78LznE5oqCjVxcZ5T
8Z44HMt9a5SfshJ4YH2961cCAwEAAaOCAYwwggGIMAkGA1UdEwQCMAAwEQYJYIZI
AYb4QgEBBAQDAgSwMCsGCWCGSAGG+EIBDQQeFhxUaW55Q0EgR2VuZXJhdGVkIENl
cnRpZmljYXRlMB0GA1UdDgQWBBTpnreeFat6HsREismDmymwrnGgpzCBxAYDVR0j
BIG8MIG5gBRKQO/y6OcARYu7e9uVYk+vvIaeUqGBlaSBkjCBjzELMAkGA1UEBhMC
VUsxEDAOBgNVBAgTB0VuZ2xhbmQxHDAaBgNVBAcTE05ld2Nhc3RsZS11cG9uLXR5
bmUxEzARBgNVBAoTClphbHRlayBMdGQxEjAQBgNVBAMTCWNhLnphbHRlazEnMCUG
CSqGSIb3DQEJARYYbWFya3ltaWRkbGV0b25AZ21haWwuY29tggkAt/+LyFN0Abww
IwYDVR0SBBwwGoEYbWFya3ltaWRkbGV0b25AZ21haWwuY29tMCMGA1UdEQQcMBqB
GG1hcmt5bWlkZGxldG9uQGdtYWlsLmNvbTALBgNVHQ8EBAMCBaAwDQYJKoZIhvcN
AQEFBQADgYEAIgOaDKqrgdw3CEojzb6ZRWGI7x/W8pMy9VtSlYzpoDOeQl9GMTZ/
uyLKCA6Xm6rLP81G0aZ/u1KndL9Y5aTFQjPhd1ackNGymT/LuOK4pJlq0DJJuH4q
9JD9b2NwwjTtZ4UHx/W5Q5uq03kFIKIPwgqFnitPJ9xx18kyhDcZx9c=
-----END CERTIFICATE-----


Client Private Key
-----BEGIN RSA PRIVATE KEY-----
MIIBOwIBAAJBAKmb4tqyOly9t934sK7MNytO1rmJiNi/VOC0DoNImzxArVV78Lzn
E5oqCjVxcZ5T8Z44HMt9a5SfshJ4YH2961cCAwEAAQJACz4B820M4UjR/yyUyYFP
xtca9IFhSLyl658Dt0AAyiWDqb0oOiEN6KA4nNjxe2bFmii10y37V6b8FrknWhie
AQIhAN4N7ubIgwOKsE24o4c4zz36ocgNkh8x2hTkfdzY7vhhAiEAw4mBm60gK8b7
UDnEHQ7BddLVCRRxR1bZMdKtx9AvHrcCIQC2oQUE4JMIfa+FLL+qPmoMkq8uhyLP
70Nq/YYa9nyM4QIgA8PsGQFWc0FbqYnTOvQDya8Lpycbn1IO0KTL7+MUZTUCIQDa
AsruwTONhb1A96ljtFci82oER9tkwTdvKsBBvBvJjA==
-----END RSA PRIVATE KEY-----


Server Certificate
-----BEGIN CERTIFICATE-----
MIID2DCCA0GgAwIBAgIBAjANBgkqhkiG9w0BAQUFADCBjzELMAkGA1UEBhMCVUsx
EDAOBgNVBAgTB0VuZ2xhbmQxHDAaBgNVBAcTE05ld2Nhc3RsZS11cG9uLXR5bmUx
EzARBgNVBAoTClphbHRlayBMdGQxEjAQBgNVBAMTCWNhLnphbHRlazEnMCUGCSqG
SIb3DQEJARYYbWFya3ltaWRkbGV0b25AZ21haWwuY29tMB4XDTEyMDQwNTAyNTgy
NloXDTE3MDQwNDAyNTgyNlowgZQxCzAJBgNVBAYTAlVLMRAwDgYDVQQIEwdFbmds
YW5kMRwwGgYDVQQHExNOZXdjYXN0bGUtdXBvbi1UeW5lMRMwEQYDVQQKEwpaYWx0
ZWsgTHRkMRcwFQYDVQQDEw5zZXJ2ZXIuY29kZXIzNjEnMCUGCSqGSIb3DQEJARYY
bWFya3ltaWRkbGV0b25AZ21haWwuY29tMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJB
AN+SXjkPPUjpdglZ1VeoTGjmW+YgWzkrr3a0ZzLRiV1WKi/UzbtTv37PWWDBuRLB
KYxLyfbGLOiBwwurJXMtFtUCAwEAAaOCAX8wggF7MAkGA1UdEwQCMAAwEQYJYIZI
AYb4QgEBBAQDAgZAMCsGCWCGSAGG+EIBDQQeFhxUaW55Q0EgR2VuZXJhdGVkIENl
cnRpZmljYXRlMB0GA1UdDgQWBBTZgIKxgIeIxlvxIovV1fKyvTL8ejCBxAYDVR0j
BIG8MIG5gBRKQO/y6OcARYu7e9uVYk+vvIaeUqGBlaSBkjCBjzELMAkGA1UEBhMC
VUsxEDAOBgNVBAgTB0VuZ2xhbmQxHDAaBgNVBAcTE05ld2Nhc3RsZS11cG9uLXR5
bmUxEzARBgNVBAoTClphbHRlayBMdGQxEjAQBgNVBAMTCWNhLnphbHRlazEnMCUG
CSqGSIb3DQEJARYYbWFya3ltaWRkbGV0b25AZ21haWwuY29tggkAt/+LyFN0Abww
IwYDVR0SBBwwGoEYbWFya3ltaWRkbGV0b25AZ21haWwuY29tMCMGA1UdEQQcMBqB
GG1hcmt5bWlkZGxldG9uQGdtYWlsLmNvbTANBgkqhkiG9w0BAQUFAAOBgQAYjpjF
W4S40VJtqDJ5jciXlzrCs2VmIC3MG22iRKdUzrASH3kphpqFSsV1KXnaZvrRCxZu
+2ADrTFGAzVkS80Zqwhk7+Snq1MANEtt4qCKaBfaKMvaVKDH6JD3BF55M1Rh0fiM
Ey2Qq/X86qjJ4zRlfa8EBEDCLAeg+XoTcOXNBw==
-----END CERTIFICATE-----


Server Key
-----BEGIN RSA PRIVATE KEY-----
MIIBOgIBAAJBAN+SXjkPPUjpdglZ1VeoTGjmW+YgWzkrr3a0ZzLRiV1WKi/UzbtT
v37PWWDBuRLBKYxLyfbGLOiBwwurJXMtFtUCAwEAAQJAH+EeIGLE6V2o4CJ4Ily0
MI0Qc5sPhvJsva9xu7RXMwCvAzpN/7bijBXzanSuveKuVoNQyrv5rQgfP86/s5lg
YQIhAP0lhGFxXepBEZmGEChyQyI3a7i1HtOUXyYQo+/21IEtAiEA4heCcFe+KRiU
svC+TEMz+cQbEQIv+wmEwmc1WruD5UkCIBK17y70nihuXsqmKhOTPnsM9Z1VZc3l
rZK2sCdQayyRAiBGPReA30Fscv5/ONtlxdr31TyzZAKT8rkTzAx/S5IGQQIhAOi+
6gm1IHJagF7n3BJj4K5Sqx2ca4Bo/sn+5ogBuoOp
-----END RSA PRIVATE KEY-----




Certificate Trust chain

Spring batch with Hibernate Part 1

Hibernate is a great ORM for web applications (it's traditional usage pattern), but it can also be very good at batch.  A word of warning, though, you need to be very careful as your HQL query can result in multiple SQL calls,  which would be a disaster for batch.

This post describes how hibernate can be configured to operate in a batch mode to minimizes the number of SQL calls to the underlying database.

Enabling batch mode

Configure hibernate with the following properties (see improving hibernate performacne):

hibernate.jdbc.batch_size=1000
hibernate.default_batch_fetch_size=1000

eg.

<bean id="sessionFactory" class="org.springframework.orm.hibernate4.LocalSessionFactoryBean">
  <property name="dataSource" ref="dataSource"/>
  <property name="packagesToScan">
    <list>
      <value>coder36.sbent.sample.domain</value>
      <value>coder36.sbent.sample.domain.stage</value>
    </list>
  </property>
  <property name="hibernateProperties">
    <props>
      <prop key="hibernate.dialect">org.hibernate.dialect.MySQLDialect</prop>
      <prop key="hibernate.jdbc.batch_size">1000</prop>
      <prop key="hibernate.default_batch_fetch_size">1000</prop>   
    </props>
  </property>
</bean

A good tip, is to also enable hibernate SQL logging, so you can see exactly what hibernate is doing behind the scenes (see here for the types of logging output) :

<prop key="hibernate.show_sql">true</prop>
<prop key="hibernate.format_sql">true</prop>
<prop key="hibernate.use_sql_comments">true</prop>

How does hibernate batch mode work?  

The best way to describe it is via an example taken from sbent.  Have the following entities:

@Entity
public class Transaction {
  ...

  @ManyToOne(fetch=FetchType.LAZY) 
  Customer customer;

  public Customer getCustomer() {
    return customer;
  }
}

and

@Entity
public class Customer {
...

@Column( nullable=false, length=60 )
private String name;

public String getName() {
return name;
}

}

To demonstrate what happens under the covers, load some data, then retrieve it with the following code:

List<Transaction> trans = 
    session.createQuery( "select t from Transaction t" ).list();
for ( Transaction t: trans ) {
  System.out.println( "getting customer");
  Customer c = t.getCustomer();
  System.out.println( "name: " + c.getName() );
}

With hibernate SQL logging turmed on, the following is sent to stdout:

getting list of Transaction's
Hibernate: 
    /* select t from Transaction t */ 
    select
        transactio0_.id as id6_,
        transactio0_.amount as amount6_,
        transactio0_.bank_id as bank4_6_,
        transactio0_.customer_id as customer5_6_,
        transactio0_.version as version6_ 
    from
        Transaction transactio0_
getting customer
Hibernate: 
    /* load coder36.sbent.sample.domain.Customer */ 
    select
        customer0_.id as id1_0_,
        customer0_.name as name1_0_,
        customer0_.nino as nino1_0_,
        customer0_.version as version1_0_ 
    from
        Customer customer0_ 
    where
        customer0_.id in ( ?, ?, ? )
name: Mark Middleton
getting customer
name: John Smith
getting customer
name: Fred Dibnah

What's happening here?

The Transaction entity is configured to lazily load its Customer.  So when the customer is first accessed, (c.getName()), the lazy load is triggered.  As batch mode is enabled, hibernate also triggers the lazy load of all Transaction.customer's for all Transaction entities in the cache - hence the customer0_.id in ( ?,?,? ) bit.

Summarize

• Enable hibernate batch mode by setting hibernate.jdbc.batch_size and  hibernate.default_batch_fetch_size
• Use lazy loading - ensure hibernate entities are correctly configured with  fetch=FetchType.LAZY
• Enable SQL logging during unit testing, to prove hibernate is behaving as expected. 

Spring Batch in the Enterprise

Large organizations by their nature are reluctant to move wholesale over to new technologies and this can result in a mishmash of new, legacy and bespoke components, all of which want a piece of the action.  They also stick with what they know works.  I'm always looking for software solutions which are a good fit, and sometimes I'm not so sure when it comes to spring batch and the enterprise.

Why Spring batch?

The great thing about spring batch, is that it provides the majority of what you need in  terms of the framework.  This allows you to invest your effort and time writing business code (this is the genius of spring).  What it doesn't do so well, is to bridge onto legacy components, bespoke components and working patterns that prevail in the enterprise.

How can  spring batch be made a better fit for the enterprise?

1) Work out batch patterns which fit with the enterprise. i.e. How does hibernate fit with batch?
2) Bridge onto enterprise components (i.e. the enterprise scheduler)
3) Work with enterprise data strategies (i.e. stream from database, rather than a file)
4) Monitoring tools (i.e. need to know if errors have occured)
5) Testing tools - testing to align with system testers.  In the enterprise it's all about integration testing.

What is SBENT?

Spring Batch Extensions for the Enterprise (SBENT) is a thin software bridge built to span 'out of the box' spring batch and the corporate enterprise. It's a set of Readers, Writers, batch admin extensions, XML processing tasklets and design patterns.

SBENT is released under Apache License Version 2.0.

The sample batch project

The best way to describe SBENT is via a reference batch project.  spent-sample makes use of the various

ItemWriters, ItemReaders and so on provided by SBENT.

The Batch scenario

 1) Bank to push a return in the form of XML onto an AMQP Queue (RabbitMQ).  The return contains all transactions its customers have made across a given period:

<return>

  <header>

    <bankName>Brookside Bank PLC</bankName>

    <periodEnd>2012-10-10</periodEnd>

  </header>

  <customer>

    <name>Mark Middleton</name>

    <nino>AB123001A</nino>

    <transaction><amount>45.11</amount></transaction>

  </customer>

  <customer>

    <name>John Smith</name>

    <nino>AB123002A</nino>

    <transaction><amount>44.44</amount></transaction>

  </customer>

</return>   

   

2) Pick up the XML submission, and save it to the database.

3) Parse the XML submission and save into staging tables

4) Process the staged data, saving it into the "domain model" 

5) Generate "ticker" reporting customers overall transaction balance (a person can be a customer in many banks)

6) Publish ticker to AMQP Queue.

Running the sample batch project

Install the following:

Maven

MySQL Server

MySQL Workbench - create a database called 'test'

RabbitMQ

RabbitMQ management plugin

Git

Instructions:

mkdir src

cd src

git clone git://github.com/coder36/sbent.git

cd sbent

mvn clean install

cd sbent-sample

mvn jetty:run    (wait for jetty to start up)

Open another prompt:

cd src/sbent/sbent-sample/bin

sample.bat  (this emulates what a scheduler would do and see)

You should see something along the lines of:

Monitoring links:

Spring batch admin front end - http://localhost:8080/sbent-sample

RabbitMQ Queues - http://localhost:55672/mgmt (guest/guest)

Source code:

https://github.com/coder36/sbent

Bytepusher for Java

In this post I discuss a java implementation of bytepusher.  Java is cross platform, so will work on windows, Linux and Macs.

I won't go through the Bytepusher specs as this was discussed in a previous post, however I will re-iterate the architecture and apply it to a java swing application.

Aim

Create a Bytepusher implementation in Java.

Tools

JDK 1.6
Gradle
Eclipse IDE
Git

Architecture

I will reuse the architecture from my c# solution, refined to take into account of how swing works:

BytePusherVM

public class BytePusherVM {

  private char[] mem = new char[0xFFFFFF];

  private BytePusherIODriver ioDriver;

  public BytePusherVM(BytePusherIODriver ioDriver) {

    this.ioDriver = ioDriver;

  }

  /**

   * Load ROM into memory

   * @param rom

   */

  public void load(InputStream rom) throws IOException {

    mem = new char[0xFFFFFF];

    int pc = 0;

    int i = 0;

    while ((i = rom.read()) != -1) {

      mem[pc++] = (char)i;

    }

  }

  /**

  * CPU loop, to be called every 60th of a second

  */

  public void run() {

    // run 65536 instructions

    short s = ioDriver.getKeyPress();

    mem[0] = (char) ((s & 0xFF00) >> 8);

    mem[1] = (char) (s & 0xFF);

    int i = 0x10000;

    int pc = getVal(2, 3);

    while (i-- != 0) {

      mem[getVal(pc + 3, 3)] = mem[getVal(pc, 3)];

      pc = getVal(pc + 6, 3);

    }

    ioDriver.renderAudioFrame(copy(getVal(6, 2) << 8, 256));

    ioDriver.renderDisplayFrame(copy(getVal(5, 1) << 16, 256 * 256));

  }

  private int getVal(int pc, int length) {

    int v = 0;

    for (int i = 0; i < length; i++) {

      v = (v << 8) + mem[pc++];

    }

    return v;

  }

  private char[] copy(int start, int length) {

    return Arrays.copyOfRange(mem, start, start + length);

  }

}

As java does not do unsigned byte, I've used a char which is effectively an unsigned byte.  In a deviation from the c# solution, I've decoupled the VM from the underlying file system by providing the load() method with an  InputStream.

Hardware Abstraction

public interface BytePusherIODriver {

  /**

   * Get the current pressed key (0-9 A-F)

   */

  short getKeyPress();

  /**

   * Render 256 bytes of audio 

  */

  void renderAudioFrame(char[] data);

  /**

   * Render 256*256 pixels.  

  */

  void renderDisplayFrame(char[] data);

}

Java Bytepusher Driver

public class BytePusherIODriverImpl extends KeyAdapter implements BytePusherIODriver {

  private SourceDataLine line;

  private int keyPress;

  private BufferedImage image;

  /**

   * Initializes the audio system

   */

  public BytePusherIODriverImpl() {

    try {

      AudioFormat f = new AudioFormat(15360, 8, 1, true, false );

      line = AudioSystem.getSourceDataLine(f);

      line.open();

      line.start();

    }

    catch( LineUnavailableException e ) {

      throw new RuntimeException( e );

    }

  }

  /**

   * Get the current pressed key (0-9 A-F

   */

  public short getKeyPress() {

    short k = 0;

    switch(  keyPress ) {

      case KeyEvent.VK_0: k+=1; break;

      case KeyEvent.VK_1: k+=2; break;

      case KeyEvent.VK_2: k+=4; break;

      case KeyEvent.VK_3: k+=8; break;

      case KeyEvent.VK_4: k+=16; break;

      case KeyEvent.VK_5: k+=32; break;

      case KeyEvent.VK_6: k+=64; break;

      case KeyEvent.VK_7: k += 128; break;

      case KeyEvent.VK_8: k += 256; break;

      case KeyEvent.VK_9: k += 512; break;

      case KeyEvent.VK_A: k += 1024; break;

      case KeyEvent.VK_B: k += 2048; break;

      case KeyEvent.VK_C: k += 4096; break;

      case KeyEvent.VK_D: k += 8192; break;

      case KeyEvent.VK_E: k += 16384; break;

      case KeyEvent.VK_F: k += 32768; break;

    }

    return k;

  }

  /**

   * Render 256 bytes of audio 

   */

  public void renderAudioFrame(char[] data) {

    // convert from char [] to byte []

    byte [] b = new byte[256];

    for ( int i=0; i < 256; i++ ) {

      b[i] = (byte) data[i];

    }

    // send buffer to audio device

    line.write( b, 0, 256);

  }

  /**

   * Render 256*256 pixels.  

   */

  public void renderDisplayFrame(char[] data) {

    image = new BufferedImage( 256, 256, BufferedImage.TYPE_INT_RGB );

    int z = 0;

    for ( int y=0; y < 256; y++ ) {

      for ( int x=0; x < 256; x++ ) 

      {

        int c = data[z++];

        if ( c < 216 ) {

          int blue = c % 6;

          int green = ((c - blue) / 6) % 6;

          int red = ((c - blue - (6 * green)) / 36) % 6;

          image.setRGB(x, y, ( red *0x33 << 16 ) + (green * 0x33 <<8) + (blue * 0x33 ) );

        }

      }

    }

  }    

  /**

   * Invoked when a key has been pressed.

   * See the class description for {@link KeyEvent} for a definition of

   * a key pressed event.

   */

  public void keyPressed(KeyEvent e) {

    keyPress = e.getKeyCode();

  }

  /**

    * Detect the key being released so that we can clear 

    * the key press. 

  */

  public void keyReleased(KeyEvent e) { 

    keyPress=0;

  }    

  /**

   * Get the image

   * @return the bufferedImage

   */

  public BufferedImage getDisplayImage() {

    return image;

  }

}

Audio System - Very similar to the XNA audio model where you submit audio buffers.  In the case of java you write to an AudioDataLine.  It also natively supports 8 bit audio samples which cuts down on a conversion algorithm.

Graphics rendering - Much simpler to use than the c# .net libraries.  No messing about with unsafe arrays etc.  Very simplistically, you create a BufferedImage then set each pixel.

Keyboard input - You need to register listeners for KeyEvents.  The issue here is that these keyevent's can occur outside of the 60th of a second run cycle.  The solution is to just store the keyevent, then respond to it within the getKeyPress() call.  In this way everything is synchronized and happy.

Swing Application

public class BytePusher extends JFrame {

  private BytePusherVM vm;

  private BytePusherIODriverImpl driver;

  private Canvas c;

  private FrameTask frameTask;

  /**

   * Entry point

   * @param args

   */

  public static void main( String [] args ) {

    BytePusher b = new BytePusher();

    b.setVisible(true);

  }

  /**

   * Constructor

   */

  public BytePusher() {

    setUpWindow();

    setUpVm();

  }

  /**

   * Create a JFrame with a single canvas within.  

   * Setup a key listener which will record the keypress.  

   * This will subsequently be handled by the FrameTask

   * which is setup to run every 60th of a second.

   */

  private void setUpWindow() {

    // create window

    setTitle( "Bytepusher for Java" );

    setLayout( new GridBagLayout() );

    setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);

    getContentPane().setPreferredSize(new Dimension(256*3, 256*3) );

    c = new Canvas();

    getContentPane().add(c);

    // canvas must be mon focusable otherwise key listeners 

    // don't work

 2  c.setFocusable(false); 

    c.setSize(new Dimension(256*3, 256*3) );

    pack();

    c.createBufferStrategy(2);

    // when window is resized, also resize canvas

 1  this.addComponentListener( new ComponentAdapter() {

      public void componentResized(ComponentEvent e) {

        c.setSize( getWidth()-15, getHeight()-38 );

      }

    });

  }

  /**

   * Load ROM into VM.  

   * @param rom

   */

  private void loadRom( String rom ) {

    try {

      FileInputStream fis = new FileInputStream( rom );

      vm.load( fis );

      fis.close();

    }

    catch( IOException e ) {

      throw new RuntimeException( e );

    }

  }

  private void setUpVm() {

    // set up bytepusher vm

    driver = new BytePusherIODriverImpl();

    vm = new BytePusherVM( driver );;

    // register key listened which will be used by the driver

    this.addKeyListener( driver );

    loadRom( "roms/audio.BytePusher" );

    // startup vm

3   frameTask = new FrameTask();

    new Timer().schedule(frameTask, 0, 1000/60);

  }

  /**

   * TimerTask which is setup to fire every 60th of a second

  */

  private class FrameTask extends TimerTask {

    /**

     * Runs the VM every 60th of a second and renders graphics

    */

    public void run() {

      vm.run();

      // render vm image to screen

      Graphics g = c.getBufferStrategy().getDrawGraphics();

      g.drawImage(driver.getDisplayImage(), 0, 0, c.getWidth(), c.getHeight(), null);

      // flip buffer 

      c.getBufferStrategy().show();

      g.dispose();

    }

  }    

}

1  Resizing window

The JFrame can be resized.  To detect this, we add a listener which triggers when the canvas sitting inside is resized. In order for the full bytpusher display to be visible, the canvas needs to be smaller than the JFrame. To be exact. the width needs to be 15 pixels smaller and the height 38 pixels smaller.  

2  KeyEvent

If you click on the canvas, then seemingly the JFrame stops responding to KeyEvent's.  To get around this, the canvas needs to be set to be non focusable.

3  Scheduled Task

This sets up a scheduled task which runs every 60th of a second.

Downloads

Binary: bytepusher4j.zip
Github: Source code
Git repository: git@github.com:coder36/bytepusher4j.git

Linux Install instructions:
         unzip bytepusher4j.zip   
    cd bytepusher
    bytepusher.sh

Windows Install instructions:
         Unzip bytepusher4j.zip   
         double click bytepusher.bat

The problem of checked exceptions in java

One thing that I really don't like about java is the use and abuse of checked exceptions.  They:

1) Pollute the code.  

With checked exceptions you either need to have try catch blocks all over the place or you need declare that the method throws a checked exception ie.

public void myMethod() {
  try {
     FileInputStream fis = new FileInputStream( "c:/file" );
  }
  catch( IOException e ) {
    //do something
  }
}
or

public void myMethod() throws IOException {
    FileInputStream fis = new FileInputStream( "c:/file" );
}

The problem with the second example is that in the calling class you then need to deal with the exception.

2) Lead to naive developers thinking they know best 

public void myMethod() throws CorporateException {
  try {
     FileInputStream fis = new FileInputStream( "c:/file" );
  }
  catch( IOException e ) {
    throw new CorporateException( "An error has occured: " + e.getMessage() );
  }
}


This is not particularly helpfull as the actual real reason of the failure (ie. the IOException) has been swallowed.

The number of times I've seen this, with the only thing to go on a message saying "CorporateException : message: java.lang.NullPointeException".  It's useless to me as I've got no line number to go on as the original NullPointerException has been swallowed.

3) Nested stack traces

public void myTopLevelMethod() throws CorporateException {
  try {
     myMethod1();
  myMethod2();
      FileInputStream fis = new FileInputStream( "c:/file" );
  }
  catch(  Exception  e ) {
     throw new CorporateException( "An error orccured in the top level method", e );
  }
}



This could result in a very long stack trace full of repeated exceptions!  The number of times I've seen this.

4) Transactions not rolling back

When using EJB's or Spring to manage transactions, the key is to ensure that a RuntimeException crosses the transaction boundary if the rollback is to be triggered.  If the naive developer declares that his methods throws a checked exception, then the transaction management will not work as expected.

------------------


What can we do?

1. Do not use checked exceptions.
2. Do not create a CorporateException class
3. When a method does trigger a checked exception convert it immediately into a RuntimeException
4. Policy that no methods are allowed to throw checked exceptions.
5. If you do need to add context do this in a dedicated single layer which all code will fall through ie. a Servlet in a web app. or via a hook in whatever framework you are using eg. spring batch @OnWriteError. This should worked out as part of the software architecture and not left to each and every developer.

Gradle Build automation

Gradle

Gradle is a next generation build automation tool.  "Gradle combines the power and flexibility of Ant with the dependency management and conventions of Maven into a more effective way to build. Powered by a Groovy DSL and packed with innovation, Gradle provides a declarative way to describe all kinds of builds through sensible defaults. Gradle is quickly becoming the build system of choice for many open source projects, leading edge enterprises and legacy automation challenges."

I've used gradle to automate the build of bytepusher4j. I haven't really used it before in a commercial setting (big companies prefer ant or maven), but I can really see how its simplicity is making it more popular.  The entire gradle config for bytepusher4j consists of:

apply plugin: 'java'
apply plugin: 'eclipse'


task zip(type: Zip) {
  from(jar.outputs.files) {
    into('bytepusher/lib')
  }
  from('roms') {
    into('bytepusher/roms')
  }
  from('src/bin') {
    into('bytepusher')
    fileMode(0755)
  }
}

This simple configuration allows a java project to be compiled into a jar file, and then packaged up into a zip file.  It also does dependency management in much the same way as maven (although I don'y need it for bytepusher4j).

Demo

This walkthroguh assumes that you already have  JDK6 ,  GIT and  Eclipse IDE installed.  It also assumes a windows OS, but could equally be applied to linux.

Setup Gradle:
1) Download  gradle and extract zip to c:/tools.  This will create a folder called c:/tools/gradle-1.0-nnn-n.

2) Add a new system environment variable GRADLE_HOME=c:/tools/gradle-1.0-nnn-n

3) Update the PATH environment variable adding %GRADLE_HOME%/bin; to the start

Download git source
4) mkdir c:/src
5) cd c:/src
6) git clone git@github.com:coder36/bytepusher4j.git
7) cd bytepusher4j

Build the code
8) gradle zip
9) gradle eclipse

10) Start Eclipse
11) File ->Import->Existing Projects into Workspace -> Browse
12) Open up C:/src/bytepusherj and select bytepusher4j
13) Finish
17) Select BytePusher.java then hit the green play button to start up the application.