How do you know when you have the right key when brute-forcing? Let's say that they test the right key. They then have to check that the decrypted text makes sense.  To do so, they can test whether there is a word in the dictionary. 

But then let's say that the message is encrypted with Enigma then with RSA encryption. They try to brute force the RSA encryption but how do they do to know that they got the right key, since the text itself is encrypted with Enigma and has no word contained in the dictionary ?

First, you do not break RSA through brute force. RSA is an asymmetric encryption algorithm, with a public/private key pair. The public key has a strong internal structure, and unravelling it yields access to the private key (basically, the main component of the public key is the modulus, which is a big composite integer, and the private key is equivalent to the knowledge of the prime factors of the modulus). This can be done a lot more efficiently than trying out possible private keys; in fact, trying prime factors one at a time is a method known as trial division, and is about the least efficient of all integer factorization methods.

The bottom-line is that the attacker perfectly knows when he has found the private key, and he does not need to see any encrypted text for that.



Now let's suppose that you are not talking about RSA encryption, but a symmetric encryption algorithm such as AES. In that case, your question makes more sense, because:


In a symmetric encryption algorithm, there is no public key to work with, so the attack must proceed from some intercepted ciphertexts.
In symmetric encryption algorithms (at least the "secure" ones), the best known attack method is, indeed, brute force. Brute force is the generic method that necessarily works (at least in theory -- in practice we use keys that are large enough to make it totally infeasible); an encryption algorithm is said to be secure if no better attack method is known.


Enigma uses as input and output sequences of latin letters -- the 26 letters from 'A' to 'Z'. Modern encryption systems work on bits and octets, i.e. "binary data". If you cascade AES with Enigma, then you need some conventional representations of letters into bits, e.g. ASCII. You would, for example, map each letter of the Enigma output into an 8-bit byte, to be used as input to AES. Since 26 is not a power of two, it is unavoidable that this mapping will have "holes": there necessarily are some sequences of bits that could be a valid AES input, but are not possible as an encoded Enigma output.

This thus yields a valid strategy for our attacker: for each possible AES key, decrypt the AES layer, and see if the result is a valid encoded Enigma output. If not, the AES key is wrong, and proceed with the next one. If you used ASCII encoding, then a 16-byte output from AES decryption is a valid Enigma output encoding only with probability 1 in 7803076729492126.75 (approximatively). This comes from the fact that there are 25616 possible sequences of 16 bytes, but only 2616 possible sequences of 16 uppercase letters. This is an awful lot of power of recognition of bad keys by the attacker. If the attacker has access to two encrypted blocks (32 bytes, corresponding to 32 characters of Enigma output), then he will be able to pinpoint the single AES key that "makes sense". In all of this, the attacker never has to even begin to consider how he will break through the Enigma.



Of course, the above is quite artificial. You may imagine cascading algorithms that live in the same world; e.g. two symmetric encryption algorithms that both work over bits.

Cascading algorithms is a method of construction of bigger encryption algorithms. It is not a very good one, though. For instance, suppose that you cascade AES with another block cipher such as Serpent; or maybe you use two AES instances with two keys chosen independently. The hope may be to thereby produce a synthetic block cipher that will have twice the key length. But it does not deliver all these promises: the double-encryption method falls prey to the Meet-in-the-Middle attack (not to be confused with the unrelated Man-in-the-Middle attack). This schema may be enlightening:



So, by cascading two algorithms, you double the operational cost, but you do not get your money worth in security increase. To really get extra security, you must go to triple-encryption. This is why Triple-DES is triple-DES, not double-DES: the latter would not be substantially stronger than simple-DES.



It still is much simpler not to use weak algorithms at all. If you cascade AES with anything else, then either the "anything else" is completely useless, because the AES is already strong enough to defeat brute force attacks; or you use the AES really poorly and that is what you should fix first.
