Mr Karamanlis has a new mandate, albeit with a smaller majority
Mr Karamanlis' centre-right New Democracy party won 152 seats in the 300-seat parliament, a loss of 13 seats since 2004.
But his main opponents, the socialist Pasok party, also lost ground, falling to its worst result since 1977.
The poll was overshadowed by forest fires that killed dozens in August.
On Monday Mr Karamanlis met President Karolos Papoulias, who asked him to form the new government.
"We are fully aware of our responsibility and we must go forward quickly and with determination to make the country progress," Mr Karamanlis said after the meeting.
Under the constitution, Mr Karamanlis has three days to form the administration, which he has said will be smaller and contain "new blood".
Electoral change
His job of reforming pensions and higher education, included in New Democracy's electoral manifesto, may now be harder given the slim majority in the new parliament - only four more seats than the combined opposition.
New Democracy polled four percentage points less than in 2004 and lost 13 seats, but analysts say the loss might have been magnified by a change in the electoral system, which is now less favourable to the party that tops the poll.
However, Mr Karamanlis is the first conservative prime minister to win re-election since 1977 and conversely, George Papandreou is the first Socialist leader to lose two consecutive elections in the past 30 years.
Mr Papandreou has made a speech thanking his supporters for showing their "love and trust" in him, adding: "Pasok fought a battle and lost it."
With Mr Papandreou under pressure to resign, former Culture Minister Evangelos Venizelos has already declared his interest in becoming party leader.
Fires pledge
Mr Karamanlis had been expected to triumph in the poll, after calling elections six months before the end of his term of office.
George Papandreou is facing a challenge for the Pasok leadership
But his support fell when many Greeks felt the government was slow to react to the forest fires, which killed 65 people.
However, the BBC's Malcolm Brabant in Athens says the prime minister rallied the nation with a steady, competent and compassionate display during the election campaign.
Mr Karamanlis has pledged to use his new four-year term to improve the economy, fight poverty, raise pensions and create a social state for the future.
He has also told the victims of the fires that the government's commitment to them would be fulfilled completely.
The Papandreou and Karamanlis families have dominated the Greek political scene for most of the past 50 years.
But both main parties suffered a fall in votes this year, with the three smaller parties from the far left and right gaining support.
George Papandreou faces leadership challenge. 
The PASOK party in Greece faces an internal conflict after the result of Sunday's elections in which the party lost from the conservative New Democracy. The 38,5% of the votes is the lowest percentage of PASOK since 1981 and many top party members accused the President of the party, George Papandreou. After the elections' result, former Minister of Culture and MP for the region of Thessaloniki, Evangelos Venizelos expressed his will to be candidate for the presidency of PASOK, however, it is not final whether Papandreou and Venizelos will be the only candidates. At the same time, PASOK’s National Committee meeting from 22-23 September will prove decisive, while the new president will be elected in the two or three following weeks.

As per an opinion poll, conducted by GPO company and commissioned by Television station Mega Channel, four out of ten civilians hold G. Papandreou responsible for PASOK’s defeat, while 73% of the voters considers Mr Evangelos Venizelos the most suitable leader for the party and 15% Mr George Papandreou. A former Prime Minister Kostas Simitis’ statement about PASOK’s poignant defeat has caused sensation. Mr Simitis appeared willing to guarantee the party’s unity and democratic procedures for the election of a new president, setting the party’s restructuring its top priority.

Other potential candidates, according to Greek media, are ex-Ministers Michalis Chrisohoidis, Kostas Skandalidis and Theodoros Pangalos as well as the former EU commissioner Anna Diamantopoulou.