Everyone knows that ECB operation mode with a block cipher should be avoided because of clear and obvious weaknesses. But little attention is given to comparison of the other modes in the context of security, and people instead appear to simply prefer the oldest mode: CBC.

Are there security caveats with respect to other common operation modes? This is specifically in the context of stream encryption rather than special-purpose functions that might have very specific operational requirements (i.e. TrueCrypt et.al.). 

For example, the simplicity of OFB mode is appealing as it completely masks the nature of the underlying block cipher, turning into an easy-to-use stream cipher. But the fact that the cipher "output" is directly XORed with the plaintext to produce the ciphertext is vaguely unsettling, and smells like there's room for a chosen plaintext vulnerability.

Of the common operation modes, are there any that we should avoid for stream encryption, or situations in which we should avoid any given one of them, or reasons to prefer one over another? Besides ECB, of course.

Specifically, CBC, OFB, and CFB operation modes, as those are nearly universally supported. And possibly CTR 'cause everyone knows what it is.

OFB and CTR turn the block cipher into a stream cipher. This means that care must be taken when using them, like for RC4; reusing the same stream for encrypting two distinct messages is a deadly sin. OFB is more "delicate" in that matter. Since OFB consists in encrypting the same value repeatedly, it is, in practice, an exploration of a permutation cycle: the IV selects a point and OFB walks the cycles which contains this point. For a block cipher with block size n bits, the average size of a cycle should be around 2n/2, and if you encrypt more than that, then you begin to repeat a previous segment of the stream, and that's bad. This can be a big issue when n = 64 (e.g. you use 3DES). Moreover, this is an average: you can, out of (bad) luck, hit a smaller cycle; also, if you encrypt two messages with the same key but distinct IV, you could (there again if unlucky) hit the same cycle than previously (only at a different point).

The bad point of OFB is that it is hard to test for these occurrences (if the implementation includes the necessary code, it can test whether these unwanted situations occur, but this cannot be done in advance, only when part of the encryption has already been done). For CTR, things are easier: CTR is encryption of successive counter values; trouble begins when a counter value is reused. But a counter behaviour is easy to predict (it is, after all, a counter) hence it is much easier to ensure that successive messages encrypted with the same key use distinct ranges of counter values.

Also, when encrypting with CTR, the output begins to be distinguishable from pure random after about 2n/2 blocks, but that's rarely lethal. It is a worry and is sufficient to warrant use of block ciphers with big blocks (e.g. AES with 128-bit blocks instead of 3DES and its 64-bit blocks), but that's a more graceful degradation of security than what occurs with OFB.

To sum up, don't use OFB; use CTR instead. This does not make CTR easy to use safely, just easier. To avoid botching it, you should try to use one if the nifty authenticated encryption modes which do things properly and include integrity check, a necessary but often overlooked component. EAX and GCM are my preferred AE modes (EAX will be faster on small architectures with limited L1 cache, GCM will be faster on modern x86, especially those with the AES opcodes which were defined just for that).

To my knowledge, CFB does not suffer as greatly as OFB from the cycle length issues, but encrypting a long sequence of zeros with CFB is equivalent to OFB; therefore, it seems safer to prefer CTR over CFB as well.

Almost all block cipher modes of operation have trouble when reaching the 2n/2 barrier, hence it is wise to use 128-bit blocks anyway.

Note: CFB and OFB have an optional "feedback length". Usually, we use full-block feedback, because that's what ensures the maximum performance (production of n bits of ciphertext per invocation of the block cipher). Modes with smaller feedback have also been defined, e.g. CFB-8 which encrypts only one byte at a time (so it is 8 times slower than full-block CFB when using a 64-bit block cipher). Such modes are not as well supported by existing libraries; also, small feedback loops make the OFB issues worse. Therefore, I do not recommend using CFB or OFB will less than full block feedback.



As pointed out by @Rook: CBC mode, like ECB but unlike CFB, OFB and CTR, processes only full blocks, therefore needs padding. Padding can imply padding oracle attacks, which is bad (arguably, a padding oracle attack is possible only if no MAC is used, or is badly applied; the proper way being encrypt-then-MAC). For this reason, padding-less modes are preferable over CBC.

This leads us to a clear victory of CTR over other modes, CFB being second, then CBC and OFB in a tie, then ECB (this is a bit subjective, of course). But, really, use EAX or GCM.
