Iranian and Iraqi officials said the building was an Iranian consulate and the detainees its employees.
The US military said it was still investigating, but that the building did not have diplomatic status.
The troops raided the building at about 0300 (0001GMT), taking away computers and papers, according to local media.
AFP news agency quoted Pentagon spokesman Bryan Whitman as saying he did not know the nationality of the six but said they were "suspected of being closely tied to activities targeting Iraq and coalition forces".
"I can confirm for you through our forces there that this is not a consulate or a government building," he said.
However, Tehran said the attack violated all international conventions. It has summoned ambassadors from Switzerland, representing US interests, and Iraq.
A spokesman for Iran's foreign ministry described the raid as an attempt to sabotage Tehran's relations with Iraq. One Iranian MP said it showed America's cruelty and meanness.
The raid comes amid high Iran-US tension.
In a major speech on Wednesday, President George W Bush said the US would take a tough stance towards Iran and Syria, whom he accused of destabilising Iraq.
The US also accuses Iran of seeking nuclear arms. Iran denies both charges.
Tehran counters that US military involvement in the Middle East endangers the whole region.
Pressure
A local TV station said Kurdish security forces had taken over the building after the Americans had left.
Irbil lies in Iraq's Kurdish-controlled north, about 350km (220 miles) from the capital Baghdad. Reports say the Iranian consulate there was set up last year under an agreement with the Kurdish regional government to facilitate cross-border visits.
Dozens of casualties resulted from a truck bombing in Samarra
One Iranian news agency with a correspondent in Irbil says five US helicopters were used to land troops on the roof of the Iranian consulate.
It reports that a number of vehicles cordoned off the streets around the building, while US soldiers warned the occupants in three different languages that they should surrender or be killed.
In December, US troops detained a number of Iranians in Iraq, including two with diplomatic immunity who were later released.
Thursday's raid came as US President George W Bush unveiled his new strategy in Iraq, which included increasing troop numbers and a commitment to stop Iranian support for "our enemies in Iraq".
BBC diplomatic correspondent Jonathan Marcus says the raid could signal a ratcheting-up of pressure on the Iranians, in line with the rhetorical thrust of his speech.
Meanwhile in the Iraqi capital, the five off-duty policemen were killed in an ambush in the western al-Khadra neighbourhood, hospital officials said
Security sources said another man was killed wounded in an attack on a money changer in downtown Baghdad.
In the restive Anbar province, the US military said that one of its troops was killed on Tuesday by a roadside bombing.
Other violence was reported in Mosul, where gunmen killed a professor driving home from work, and Samarra where a suicide truck bomber attacked the mayor's house, killing three people and wounding 33, including the mayor.
Erbil province in northern Iraq
The U.S. armed forces detained five Iranians working at a "liaison office" located in Erbil, Iraq before dawn on Friday. Sources said that the U.S. forces first landed their helicopters around the building, then broke through the office's gate, disarmed the guards, confiscated some documents and certain objects, arrested five suspected terrorists, and then left for an undisclosed location.

However, there was no raid on the Iranian Consulate General as earlier reported. Pentagon spokesman Bryan Whitman told Agence France-Presse that the building raided was "not a consulate or a government building", which appears to be confirmed by the Iraqi foreign minister, Hoshyar Zebari, who said the Iranians were working at an office that had government approval and was in the process of being approved as a consulate, but was not a Consulate nor the Iranian Consulate General.  The US is strategically insisting that they raided a "liaison office" instead of a Consulate to avoid being found in violation of Article 36 of the Vienna Convention on Consular Relations.

Russian Ministry of Foreign Affairs Mikhail Kamynin said that the raid was absolutely unacceptable and was a violation of the Vienna Convention on Consular Relations. The Kurdistan Regional Government also expressed their shock and disapproval of the raid.

On Thursday's hearing on Iraq, Senator Joseph Biden (D-Delware), chairman of the Senate Foreign Relations Committee, told Secretary of State Condoleezza Rice that the Bush Administration did not have the authority to send U.S. troops on cross-border raids. Biden said, "I believe the present authorization granted the president to use force in Iraq does not cover that, and he does need congressional authority to do that. I just want to set that marker."  After the meeting, Biden sent a follow-up letter to the White House asking for an explanation from the Bush Administration on the matter.

On Thursday morning, Iran's foreign ministry official sent a letter to Iraq's foreign ministry asking Iraq to stop the Bush Administration from interfering with Iraq-Iran relations, and has protested the raid on its liaison office. The official said, "We expect the Iraqi government to take immediate measures to set the aforesaid individuals free and to condemn the U.S. troopers for the measure. Following up on the case and releasing the arrestees is a responsibility of primarily the Iraqi government and then the local government and officials of the Iraqi Kurdistan."