  We present a systematic algorithm testing for the existence of collective
self-organization in the behavior of agents in social systems, with a concrete
empirical implementation on the Dow Jones Industrial Average index (DJIA) over
the 20th century and on Hong Kong Hang Seng composite index (HSI) since 1969.
The algorithm combines ideas from critical phenomena, the impact of agents'
expectation, multi-scale analysis and the mathematical method of pattern
recognition of sparse data. Trained on the three major crashes in DJIA of the
century, our algorithm exhibits a remarkable ability for generalization and
detects in advance 8 other significant drops or changes of regimes. An
application to HSI gives promising results as well. The results are robust with
respect to the variations of the recognition algorithm. We quantify the
prediction procedure with error diagrams.
