When hosting a cluster of web application servers it’s common to have a reverse proxy (HAProxy, Nginx, F5, etc.) in between the cluster and the public internet to load balance traffic among app servers.  In order to perform deep packet inspection, SSL must be terminated at the load balancer (or earlier), but traffic between the load balancer and the app servers would be unencrypted.  Wouldn't early termination of SSL leave the app servers vulnerable to packet sniffing or ARP poisoning?

Should SSL be offloaded? If so, how can it be done without compromising the integrity of the data being served? My main concern is for a web application where message layer encryption isn't an option.

It seems to me the question is "do you trust your own datacenter".  In other words, it seems like you're trying to finely draw the line where the untrusted networks lie, and the trust begins.  

In my opinion, SSL/TLS trust should terminate at the SSL offloading device since the department that manages that device often also manages the networking and infrastructure.  There is a certain amount of contractual trust there.  There is no point of encrypting data at a downstream server since the same people who are supporting the network usually have access to this as well.  (with the possible exception in multi-tenant environments, or unique business requirements that require deeper segmentation).  

A second reason SSL should terminate at the load balancer is because it offers a centralized place to correct SSL attacks such as CRIME or BEAST.  If SSL is terminated at a variety of web servers, running on different OS's you're more likely to run into problems due to the additional  complexity .  Keep it simple, and you'll have fewer problems in the long run.

That being said


Yes, terminate at the load balancer and SSL offload there. Keep it simple.
The Citrix Netscaler load balancer (for example) can deny insecure access to a URL.  This policy logic, combined with the features of TLS should ensure your data remains confidential and tamper-free (given that I properly understand your requirement of integrity) 


Edit:

It's possible (and common) to 


Outsource the load balancer (Amazon, Microsoft, etc)
Use a 3rd party CDN (Akamai, Amazon, Microsoft, etc)
Or use a 3rd party proxy to prevent DoS attacks


... where traffic from that 3rd party would be sent to your servers over network links you don't manage.  Therefore may not trust those unencrypted links.  In that case you should re-encrypt the data, or at the very least have all of that data travel through a point-point VPN.

Microsoft does offer such a VPN product and allows for secure outsourcing of the perimeter.
