Dimos Dikoudis and Theodoros Papaloukas celebrate victory
Guard Vassilis Spanoulis starred with 22 points as the European champions overcame a 12-point deficit.
Carmelo Anthony scored 27 points and Dwayne Wade 19, but the joint captains were unable to save the favourites.
Greece, with no current NBA players on their roster, will next face Spain, who knocked out Olympic champions Argentina 75-74 in a thrilling second semi-final.
The final buzzer in the first semi-final sparked scenes of pandemonium as the Greeks danced in the centre circle as if they had won the gold medal.
"I thank my players," said Greek coach Panagiotis Yannakis. "They did something incredible. We had faith and we never lost concentration."
This historic victory has made Greeks all over the world proud
Greek Prime Minister Costas Karamanlis
US coach Mike Krzyzewski was gracious in defeat.
"Obviously the Greek team did an amazing job," he said. "It's not surprising. We knew that they had great heart and togetherness. They deserved it."
The shock win also caused huge celebrations to break out all over Greece while the country's political leadership rushed to offer praise and congratulations.
Prime Minister Costas Karamanlis cut short scheduled meetings to watch the end of the game.
"This historic victory has made Greeks all over the world proud," he said in a statement.
LeBron James and Carmelo Antony trudge off as the Greeks party
"It has proven that when Greece can do anything when it believes in itself."
Sports Minister George Orfanos told state NET television: "Nobody could believe this. The Greek team made the Americans kneel to them."
Offices and services came to a standstill in Athens as Greeks watched the game and then left their workplace to celebrate.
Workers at one downtown office threw hundreds of sheets of paper out of windows onto a celebrating crowd.
Others waved national blue-and-white flags and sounded car horns as they celebrated the country's place in the final.
Traffic information screens had flashed the final score, prompting some motorists to abandon their cars to join the crowds.
Greek fans are in joyous mood at the final buzzer
"I congratulate Greece for coming back from 12 points down and going on a 14-0 run," said Anthony.
"I take my hat off to them for playing so well and so hard down the stretch."
The USA have suffered a string of failures and will now be forced to qualify for the 2008 Olympics in Beijing.
They were crowned world champions in 1994 and were also Olympic champions in 2000, but they missed out on a medal at the last world championship in 2000 and could only win bronze at the Athens Games in 2004.
Jose Calderon's free throw with 19 seconds to play clinched victory for Spain.
Chicago Bulls' player Andres Nocioni had a chance to snatch a win for Argentina but his three-point attempt with two seconds left bounced off the rim.
However, their joy was tempered by the news that star player Pau Gasol could miss the final because of a foot injury.
The Memphis Grizzlies forward limped out late in the fourth quarter and Spain coach Pepu Hernandez said it was unlikely that he would play against Greece.
"We cannot be fully happy because it looks like Pau Gasol suffered a serious injury," he said.
"We will try to come out strong on Sunday to win the gold for Pau."
With a very tough defence and an attack of 101 points, the European Champions, Greece, beat the national team of the United States of America with a score 101-95, on the semi-final of the 2006 Fiba World Championship which took place in Saitama, Japan.

The Greek team had a very good day - especially Vassilis Spanoulis and Dimitris Diamantidis - while the zone defence organized by the Greek coach Panayiotis Giannakis blocked the NBA stars who, despite their good performance, will not take part in the final. The best player of the USA team was Denver Nuggets star, Carmello Anthony, who scored 25 points. The other semi-final of the 2006 Fiba World Championship is the game between Argentina and Spain. 

====Statistics (points and fouls)====

Greece: Papaloukas 8, Diamantidis 12(2), Τsartsaris 3(1), Hadjivrettas 2, Vasilopoulos, Schorchianitis 12, Papadopoulos 8, Kakiouzis 13(1), Fotsis 9(1), Dikoudis 8, Spanoulis 20(3).

United States: Paul 3, Wade 22(1), Arenas, Heinrich 12(4), Bowen, Anthony 25(3), Jameson, Battier 1, James 15, Johnson 3(1), Bosh 2, Miller, Howard 10.