Please note that republishing this article in full or in part is only allowed under the conditions described here.
This is part eight in a series which explains the evasions done by HTTP Evader. This part looks into the excessive and inconsistent robustness attempts done by the browser vendors and how this can be used to evade firewalls. As an example, in the following HTTP response the character "\000" inside the field name will be simply ignored by Chrome and Opera, while several firewalls will not understand it and pass it through:
HTTP/1.1 200 ok Transfer\000-Encoding: chunked malware with chunked encoding
The Robustness Principle effectively means, that a software should be strict (i.e. standard conforming) in what it sends but tolerant in what it accepts.
Unfortunately browsers are more tolerant against bad HTTP than most firewalls expect. Together with the common behavior of firewalls to pass everything which does not really looks bad this will lead to several evasions. Let's have a closer look against the abyss of robustness.
The character "\000" (i.e. byte 0x0) has a long tradition in causing security problems. The Chrome browser (and other browsers based on the same engine, like Opera) ignores this character in almost all places so this example is a HTTP response header were a chunked body encoding is expected by Chrome:
\000HTTP/1\000.1 200 ok Transfer\000-Encoding\000:\000chun\000ked\000
Ignorance if \000 inside field names and field values is unique for Chrome and Opera, but the other cases are also accepted by more browsers. Sure enough such kind of behavior is usually not expected by firewalls.
For HTTP 1.0 and HTTP 1.1 the status line of the response header starts with the string "HTTP/" (all upper case!) followed by the HTTP version. There are no other version than 1.0 and 1.1 allowed but browsers happily accept variations:
- http/1.1: the lower case version is accepted by all browsers. About 30% of the firewalls I've seen fail to handle this properly and simply pass the data.
- HTTP/0.9, HTTP/1.2, HTTP/2.0: all browsers except Safari handle these and treat versions > 1.0 similar to 1.1, i.e. chunked encoding is supported. Interestingly Firefox accepts HTTP/2.0 but not HTTP/2.1.
- HTTP/1.01: This is treated as 1.0 by Chrome and Opera, i.e. no support for chunked encoding. IE, Edge and Firefox instead treat is as >1.0 and support chunked encoding. Safari correctly considers the response invalid. The same inconsistent behavior can be found in the firewalls so about 15% can be bypassed in the rest reports I've seen.
- "HTTP/1.1foobar\r\n": Chrome, Opera and Firefox accept status lines which have junk after the HTTP version instead of space and status code.
While Attack of the White-Space covers different line endings I did not notice then how different the browsers behave regarding the end of the header. The end of the header according to the standard is an empty line, i.e. "\r\n\r\n" and all browsers except "\n\n" too. But, for IE and Edge "empty" has a slightly different meaning and they accept space and tabulator as part of the emptiness. This behavior difference is best used with compression, because in this case the compressed data must start with the first byte of the body.
Thus this HTTP response works for IE and Edge:
HTTP/1.1 200 ok\r\n Content-Encoding: gzip\r\n \t\r\n gzipped data
While all the other browsers will need an additional line after the pseudo-empty line containing the tabulator.
HTTP/1.1 200 ok\r\n Content-Encoding: gzip\r\n \t\r\n \r\n gzipped data
Of course firewalls behave differently and often can be bypassed this way. Another variation of the header end is "\n\r\r\n" instead of "\r\n\r\n" (first two bytes swapped) which is accepted by IE, Edge and Safari.
Experiments with spaces, non-ASCII characters or simply complete invalid lines (like no "key:value") in the header show that most browsers simply ignore such lines. Other broken lines get interpreted in a creative way, like in the case of a double colon between key ("Transfer-Encoding") and value ("chunked"):
HTTP/1.1 200 ok Transfer-Encoding::chunked data
With this HTTP response Firefox and Safari treat the data as chunked, while all the other browsers treat it as plain (non-chunked) data. Interpretation in the firewalls differ, so that about 25% of the firewalls in my test reports can be bypassed.
These are just a few examples where the protocol handling in firewalls is no match against the overly broad and inconsistent robustness of browsers. If you are interested to look into more detail at these browser quirks or want to see how the firewall at your site deals with these problems you can find out yourself at the the HTTP Evader test site. Or maybe you want to know more about HTTP Evader and read about other bypasses.
From the point of security the robustness principle is a nightmare, because it means that you have to protect a client where the software behaves in an unknown way. Apart from that nobody will really notice if the behavior of a web server conflicts with the standard because the browsers will handle it anyway. Thus you will find enough broken servers out there. And there is no good way for the browsers to make the behavior strict again because otherwise they might fail to work with existing servers.
With HTML it is widely known that browsers behave differently in edge cases or with invalid HTML and it is extensively documented how this can be abused to evade filters in the XSS Filter Evasion Cheat Sheet or the HTML5 Security Cheat Sheet. I also recommend to read The Tangled Web by Michal Zalewski which gives a deep look into the associated problems for securing web applications.
I'm not aware of any similar documentation for the borderline robustness at the HTTP level. But apart from the firewall evasion tests HTTP Evader also provides an extensive test suite to examine the robustness behavior of the web browsers.