To help illiterate voters, the campaigns used fruit symbols
Mr Kibaki said he would announce a new line-up of ministers in two weeks.
The result is seen as a protest against Mr Kibaki, and there is speculation that ministers opposed to the draft will not be part of the new team.
Campaigners celebrating the "No" vote have asked their opponents to join them in drafting a fresh constitution.
"Please come and join the rest of the nation in rebuilding a new Kenya," the so-called "orange" anti-constitution campaign said in a statement.
The result is a blow for Mr Kibaki, who led the "banana" campaign for a "Yes".
The president has made no reference to discussions on a new draft.
Opposition campaign leaders called for a cross-party effort to draft a new constitution "now that the people of Kenya have decisively spoken, and in the spirit of genuine reconciliation".
"We invite our colleagues in the Banana camp to join us in expeditiously charting the way forward for a new Constitutional dispensation for our country," the leaders said in a statement read out by opposition leader Uhuru Kenyatta.
President Kibaki admitted defeat in a live television address. "My government will respect the will of the people," he said. But he said that the existing constitution would serve Kenya's needs for now, the Daily Nation newspaper reports.
Protest vote
Many used the poll to protest against Mr Kibaki's leadership, and crowds of people celebrated the result on the streets of the capital, Nairobi.
DRAFT CONSTITUTION Prime minister - works to the president Bans foreign land ownership Land commission formed - individuals can no longer distribute land Christian and other religious courts set up; Muslim courts already exist Regional parties banned Elections for local officials Same-sex marriages banned Women get equal rights to inherit property Abortion outlawed - unless permitted by parliament
Press says listen to the people
With just four out of 210 constituencies still to return their results, the "No" campaign has an unassailable lead of almost 1m votes.
The latest figures released by the Electoral Commission of Kenya (ECK) show 3,548,477 (57%) voted "No" while 2,532,918 (43%) backed the draft.
Mr Kibaki's cabinet was split over the constitution, with seven ministers joining with the opposition, urging voters to reject the draft.
Correspondents had expected the cabinet reshuffle and say manoeuvring will increase ahead of general elections due in 2007.
After Kenyans voted "no" for the implementation of the new constitution, the "Orange" anti-constitution campaign and the "Banana" pro-constitution campaign finally sat down together to discuss the way forward. 

The draft constitution would have banned foreign land ownership, set the prime minister position to report to the country's president, banned same-sex marriages and restricted regional political parties. It would have also given women equal rights to inherit property, and addressed local elections. The country was attempting to rewrite its post-colonial constitution. This draft constitution was brought forward by the Attorney General, Amos Wako. 

A day after the results were announced, the President of the country, Mwai Kibaki, fired his whole cabinet. It is not yet known what he plans to do, but hopes for reconciliation are high: a member of the opposition called on the president "to provide leadership in the way forward".