Despite the defeat, Mr Abe has vowed to press on with reforms
Mr Abe told a news conference that he took the result "very seriously" and pledged a reshuffle of his Cabinet.
But he said that it was his duty to continue as premier and push forward his reform agenda.
Support for Mr Abe's government has plummeted in recent months following a series of gaffes and scandals.
"It was a severe result. We accept the people's judgement seriously and sincerely," Mr Abe said.
Voters gave a clear failing mark... The prime minister should face the results seriously and step down
Asahi newspaper
Tough road ahead for Abe
"However, it is my responsibility to continue with my mission in building a new nation and pushing for reform."
He said a Cabinet reshuffle would take place after a special session of parliament.
"Voters said we must reflect on our shortcomings and refresh the line-up," Mr Abe said. "I plan to reshuffle the Cabinet and top party posts at an appropriate time."
Despite Mr Abe's determination to stay in his job, the deputy leader of the LDP, Hidenao Nakagawa, has decided to resign over Sunday's polls.
'Failing mark'
Mr Abe achieved solid levels of popular support when he took office in September 2006.
FACTS AND FIGURES There are 242 seats in the upper house, half of which were up for grabs Pre-polls, the LDP-led coalition controlled 133 seats The coalition won 46 seats (LDP: 37, New Komeito: 9)* Opposition DPJ won 60 seats* LDP-led coalition now controls 105 seats* *Source: Kyodo news agency
Press calls for new polls
But many voters began to question his leadership skills following a series of ministerial scandals and a crisis over lost pension records that affected millions of payments.
In Sunday's polls, Mr Abe's Liberal Democratic Party-led ruling coalition fell far short of the 64 seats it needed to retain its majority in the upper house.
The opposition Democratic Party of Japan made huge gains, becoming the largest party in the upper house for the first time in history.
Final official results are expected later on Monday.
The ruling coalition still has a sizeable majority in the more powerful lower house, which chooses the prime minister.
HAVE YOUR SAY The Japanese people have at last demonstrated that when something is not right they are going to vote against the government
Yuji, Tokyo, Japan
Send us your comments
Several newspapers have called for Mr Abe to step down or call a snap election.
"Voters gave a clear failing mark," the Asahi newspaper said. "The prime minister should face the results seriously and step down."
Mr Abe has been damaged by this heavy defeat, says the BBC's Chris Hogg in Tokyo.
In fact, commentators say, the only reason he has not been forced to step down is that no-one else in his party wants to take over after such a drubbing at the polls, our correspondent adds.
Japanese House of Councillors election, 2007     - Liberal Democratic Party
     - New Komei Party
     - Democratic Party of Japan
     - Japanese Communist Party
     - Social Democratic Party
     - The People's New Party
     - New Party Nippon
     - Others

Democratic Party of Japan (DPJ) could claim victory in Sunday's House of Councillors elections. The House of Councillors is the upper house in the bicameral National Diet of Japan. Of the 242 seats, DPJ went from 82 to 109 seats, making it the largest party in the House of Councillors.

This gain came largely at the expense of the ruling Liberal Democratic Party (LDP), which went from 119 seats to 83. Also losing seats was LDP's coalition partner, New Komeito Party (NKP), which lost 4 councillor seats and now holds 20.
Shinzo Abe on November 18, 2006
Japan's Prime Minister Shinzo Abe said that while he takes the result of the election "very seriously," he does not intend to resign his post.

"We must take these results very seriously and reflecting on what we must reflect on," Abe said at a news conference. "I want to fulfill my responsibility to proceed with reform to build the nation and promote economic growth that the people can feel."

The election result will not force the government out of power, since the LDP holds 296 of the 480 seats in the House of Representatives. Nonetheless, Abe said that he would reshuffle the cabinet. "Voters said we must reflect on our shortcomings and refresh the line-up," Abe said. "I plan to reshuffle the Cabinet and top party posts at an appropriate time."

"The election expressed the frustrations of the people. He really needs to accept the results of the election and think about the future. This might include quitting," 38-year-old civil servant Akihiro Kodaira told ''Reuters''.