Haji Yar Mohammad Karzai was killed during a night raid in the early hours of Thursday, south of the city of Kandahar.
The presidential palace said Mr Karzai was "extremely sad" at the news and once again called on Nato forces "to avoid killing civilians".
Afghan officials said troops had raided a compound in Karz village of Dand district which is the home of the Karzai clan.
During the raid at around 2am, Yar Mohammad, a cousin of the president's late father, had apparently tried to leave his home and had been mistakenly shot.
Ahmad Wali Karzai, the president's brother and Kandahar's main powerbroker, said: "It was a mistake. The forces conducted an operation, he was at his home, he came out and was shot."
"It was a mistake. What can you do about it?"
A Nato press statement had said the raid targeted a Taliban leader who smuggled weapons and bombs to insurgents in Kandahar.
It said an "armed individual", believed to be the target's father, had been shot when he emerged from a neighbouring building inside the compound carrying an AK-47.
When reports it was Yar Mohammad later emerged, the coalition said it was "aware of conflicting reports about the identities of those involved" and had begun an investigation.
Civilian deaths cause deep resentment of foreign forces among Afghans and are a constant source of tension between Mr Karzai and Nato.
He recently refused initially to accept an apology from Gen David Petraeus after a coalition helicopter mistakenly killed nine boys in an airstrike as they collected firewood in Kunar province.
Yar Mohammad, who was aged around 60, was the principal player in a vendetta which has split the Karzai family for 30 years.
He allegedly killed the father of Hashmat Karzai, a first cousin of the president, in Quetta in a row over a failed arranged marriage.
Two years ago Yar Mohammad's son, Waheed, was shot dead by unknown assassins who burst into his home and murdered him in front of his younger sister.
According to reports at the time, Waheed used his dying breath to claim Hashmat was behind the killing.
Yar Mohammad later complained that his son's death had never been adequately investigated.
Meanwhile a senior Afghan official said Kabul would take control of security in up to four provinces and three major cities this July in the first stage of a handover which will allow Nato troops to leave the country.
Hamid Karzai is expected to announce the first areas to be handed over in a speech to mark Afghan new year on March 21.
Hamid Karzai was said to be "extremely sad" over the shooting
A relative of the Afghan president Hamid Karzai has been mistakenly shot dead by NATO troops, according to officials. Yar Mohammad Khan, who was in his 60s was killed during a night raid in the . Khan, who also possibly used the surname of Karzai, was believed to have been a cousin of either the president or a cousin of the president's father.

The president's brother, Ahmad Wali Karzai, released a statement over the shooting. He said, "It was a mistake. The forces conducted an operation, he was at his home, he came out and was shot. It was a mistake. What can you do about it?" Several officials have apologised for the incident including United States president Barack Obama, US defence secretary , and commander of international troops in Afghanistan .

Khan's death come only days after President Karzai called the killing of civilians unacceptable. The president urged the United Nations to pressure NATO to take more care during operations when searching for insurgents. Karzai rejected a previous apology from General Petraeus over the killing of nine boys during an air strike.