  This paper proposes a new obfuscation technique of a communication protocol
that is aimed at making the reverse engineering of the protocol more complex.
The obfuscation is based on the transformation of protocol message format
specification. The obfuscating transformations are applied to the Abstract
Syntax Tree (AST) representation of the messages and mainly concern the
ordering or aggregation of the AST nodes. The paper also presents the design of
a framework that implements the proposed obfuscation technique by automatically
generating, from the specification of the message format, a library performing
the corresponding transformations. Finally, our framework is applied to two
real application protocols (Modbus and HTTP) to illustrate the relevance and
efficiency of the proposed approach. Various metrics recorded from the
experiments show the significant increase of the complexity of the obfuscated
protocol binary compared to the non-obfuscated code. It is also shown that the
execution time and memory overheads remain acceptable for a practical
deployment of the approach in operation.
