I've heard it suggested that "solid tantalum" capacitors are dangerous and may cause fire, may fail short circuit and are fatally sensitive to even very short over voltage spikes.

Are tantalum capacitors reliable?

Are they safe for use in general circuits and new designs?

Summary:
"When used properly" tantalum capacitors are highly reliable.
They have the advantage of high capacitance per volume and good decoupling characteristics due to relatively low internal resistance and low inductance compared to traditional alternatives such as aluminum wet electrolytic capacitors.
The 'catch' is in the qualifier "when used properly".
Tantalum capacitors have a failure mode which can be triggered by voltage spikes only 'slightly more' than their rated value. When used in circuits that can provide substantial energy to the capacitor failure can lead to thermal run-away with flame and explosion of the capacitor and low resistance short-circuiting of the capacitor terminals.
To be "safe" the circuits they are used in need to be guaranteed to have been rigorously designed and the design assumptions need to be met. This 'does not always happen'.
Tantalum capacitors are 'safe enough' in the hands of genuine experts, or in undemanding circuits, and their advantages make them attractive. Alternatives such as "solid aluminum" capacitors have similar advantages and lack the catastrophic failure mode.
Many modern tantalum capacitors have built in protection mechanisms which implement fusing of various sorts, which is designed to disconnect the capacitor from its terminals when it fails and to limit PCB charring in most cases.
If 'when', 'limit' and 'most' are acceptable design criteria and/or you are a design expert and your factory always gets everything right and your application environment is always well understood, then tantalum capacitors may be a good choice for you.

Longer:
Solid Tantalum capacitors are potentially disasters waiting to happen.
Rigorous design and implementation that guarantees that their requirements are met can produce highly reliable designs.  If your real world situations are always guaranteed to not have out of spec exceptions then tantalum caps may work well for you, too.
Some modern tantalum capacitors have failure mitigation (as opposed to prevention) mechanisms built in. In a comment on another stack exchange question Spehro notes:

The data sheet for Kemet's Polymer-Tantalum caps says (in part) : "The KOCAP also exhibits a benign failure mode which eliminates the ignition failures that can occur in standard MnO2 tantalum types.".

Strangely, I can find nothing about the "ignition failure" feature in their other data sheets.
Solid Tantalum electrolytic capacitors have traditionally had a failure mode which makes their use questionable in high energy circuits that cannot be or have not been rigorously designed to eliminate any prospect of the applied voltage exceeding the rated voltage by more than a small percentage.
Tantalum caps are typically made by sintering tantalum granules together to form a continuous whole with an immense surface area per volume and then forming a thin dielectric layer over the outer surface by a chemical process. Here "thin" takes on a new meaning - the layer is thick enough to avoid breakdown at rated voltage  - and thin enough that it will be punched through by voltages not vastly in excess of rated voltage. For an eg 10 V rated cap, operation with say 15V spikes applied can be right up there with playing Russian Roulette. Unlike Al wet electrolytic caps which tend to self heal when the oxide layer is punctured, tantalum tends not to heal. Small amounts of energy may lead to localised damage and removal of the conduction path. Where the circuit providing energy to the cap is able to provide substantial energy the cap is able to offer a correspondingly low resistance short and a battle begins. This can lead to smell, smoke, flame, noise and explosion. I've seen all these happen sequentially in a single failure. First there was a puzzling bad smell for perhaps 30 seconds. Then a loud shrieking noise, then a jet of flame for perhaps 5 seconds with gratifying wooshing sound and then an impressive explosion. Not all failures are so sensorily satisfying.
Where the complete absence of overvoltage high energy spikes could not be guaranteed, which would be the case in many if not most power supply circuits, use of tantalum solid electrolytic caps would be a good source of service (or fire department) calls. Based on Spehro's reference, Kemet may have removed the more exciting aspects of such failures. They still warn against minimal overvoltages.
Some real world failures:

Wikipedia - tantalum capacitors

Most tantalum capacitors are polarized devices, with distinctly marked positive and negative terminals. When subjected to reversed polarity (even briefly), the capacitor depolarizes and the dielectric oxide layer breaks down, which can cause it to fail even when later operated with correct polarity. If the failure is a short circuit (the most common occurrence), and current is not limited to a safe value, catastrophic thermal runaway may occur (see below).

Kemet - application notes for tantalum capacitors

Read section 15., page 79 and walk away with hands in sight.

AVX - voltage derating rules for solid tantalum and niobium capacitors

For many years, whenever people have asked tantalum capacitor manufacturers for
general recommendations on using their product, the consensus was “a minimum
of 50% voltage derating should be applied”. This rule of thumb has since become
the most prevalent design guideline for tantalum technology. This paper revisits this
statement and explains, given an understanding of the application, why this is not
necessarily the case.

With the recent introduction of niobium and niobium oxide capacitor technologies,
the derating discussion has been extended to these capacitor families also.
Vishay - solid tantalum capacitor FAQ

. WHAT IS THE DIFFERENCE BETWEEN A FUSED (VISHAY SPRAGUE 893D) AND STANDARD,
NON-FUSED (VISHAY SPRAGUE 293D AND 593D) TANTALUM CAPACITOR?

A. The 893D series was designed to operate in high-current applications (> 10 A) and employs an “electronic” fusing mechanism. ... The 893D fuse will not “open” below 2 A because the I2R is below the energy required to activate the fuse. Between 2 and 3 A, the fuse will eventually activate, but some capacitor and circuit board
“charring” may occur. In summary, 893D capacitors are ideal for high-current circuits where capacitor “failure” can cause system failure.
Type 893D capacitors will prevent capacitor or circuit board “charring” and usually prevent any circuit interruption that can be associated with capacitor failure. A “shorted” capacitor across the power source can cause current and/or voltage transients that can trigger system shutdown. The 893D fuse activation time is sufficiently fast in most instances to eliminate excessive current drain or voltage swings.
Capacitor guide - tantalum capacitors

... The downside to using tantalum capacitors is their unfavorable failure mode which may lead to thermal runaway, fires and small explosions, but this can be prevented through the use of external failsafe devices such as current limiters or thermal fuses.

What a cap-astrophe

I was working at a manufacturer that was experiencing unexplained tantalum-capacitor failure. It wasn't that the capacitors were just failing, but the failure was catastrophic and was rendering PCBs (printed-circuit boards) unfixable. There seemed to be no explanation. We found no misapplication issues for this small, dedicated microcomputer PCB. Worse yet, the supplier blamed us.

I did some Internet research on tantalum-capacitor failures and found that the tantalum capacitors' pellets contain minor defects that must be cleared during manufacturing. In this process, the voltage is increased gradually through a resistor to the rated voltage plus a guard-band. The series resistor prevents uncontrolled thermal runaway from destroying the pellet. I also learned that soldering PCBs at high temperatures during manufacturing causes stresses that may cause microfractures inside the pellet. These microfractures may in turn lead to failure in low-impedance applications. The microfractures also reduce the device's voltage rating so that failure analysis will indicate classic overvoltage failure. ...

Related:
AVX - surge in solid tantalum capacitors
Failure modes and mechanisms in solid tantalum capacitors - Sprague / IEEE abstract only.  - OLD 1963.
AVX - FAILURE MODES OF TANTALUM CAPACITORS MADE BY DIFFERENT
TECHNOLOGIES - Age ? - about 2001?
Effect of Moisture on Characteristics of Surface Mount Solid Tantalum
Capacitors - NASA with AVX assistance - about 2002?
Hearst - How to spot counterfeit components
Sometimes it's easy :-) :


Added 1/2016:
Related:
Test for reverse polarity for standard wet-aluminium metal can capacitors.
Brief:
For correct polarity can potential is ~= ground.
For reverse polarity can potential is a significant percentage of applied voltage.
A very reliable test in my experience.
Longer:
For standard wet Al caps I long ago discovered a test for reverse insertion which I've not ever seen mentioned elsewhere but is probably well enough known. This works for caps which have the metal can accessible for testing - most have a convenient clear spot at top center due to the way the sleeve is added.
Power up circuit and measure voltages from ground to can of each cap. This is a very quick test with a volt-meter - -ve lead grounded and zip around cans.

Caps of correct polarity have can almost at ground.

Caps of reverse polarity have cans at some fraction of supply - maybe ~~~= 50%.


Works reliably in my experience.
You can usually check using can markings but this depends on intended orientation being known and clear. While that is usually consistent in a good design this is never certain.
