DOHA, Qatar (CNN) -- Security guards outside the Taliban's new offices here said Thursday that they were empty and there was no one available to talk, but the building itself said plenty.

The Taliban's white flag, which had flown Wednesday over the building, was lowered behind the walls of the compound, and a sign that had read "Islamic Emirates of Afghanistan" -- the name used by the group during its rule in Afghanistan from 1996-2001 -- was nowhere to be seen.

The symbols -- interpreted as signs that the office was representing an alternative government -- had infuriated Afghan President Hamid Karzai.

He announced on Wednesday that Afghanistan was pulling out of peace talks with the Taliban and out of talks with the United States about a long-term foreign military presence in his country after the departure next year of Western military forces.

Though the peace process remains a priority, "the Afghan government will never allow for an Afghan peace process to be hijacked by the enemies of Afghanistan for reaching their nefarious designs that they have failed to achieve on the battlefield of war in Afghanistan," said Deputy Foreign Minister Ershad Ahmadi.

The Afghan government's plans to negotiate this week with a U.S. team in Kabul were suspended "so that we could signal our serious displeasure about the breach of the written assurances given to us by the U.S. government about the opening of the Taliban office in Doha," Ahmadi said in a statement.

"If the Taliban office in Doha is brought back into compliance with the written assurances given to us by the U.S. government, the Afghan government will review its decision about BSA negotiations with the U.S."

It was not clear Thursday whether the changes to the building in Doha would suffice to coax Karzai back to talks.

His concerns came after the Taliban claimed responsibility for the deaths of four U.S. troops in a rocket attack on Bagram air base, which led a member of the House Armed Services Committee to lambaste the Taliban.

"They cannot be trusted," said Rep. Duncan Hunter Jr., R-California. "And even if they give their word on something you can't know for sure that will filter down."

Karzai has a legitimate complaint, Hunter said. "We are trying to legitimize the guys who are blowing up Americans and Afghans in Afghanistan."

Still, a senior U.S. administration official told CNN on Wednesday that peace talks between the United States and the Taliban will likely be held "in the next few days."

The official, who spoke on condition of anonymity, did not say when such talks might take place.

Speaking Wednesday in Berlin, U.S. President Barack Obama said he wasn't surprised by Karzai's response.

"We had anticipated that at the outset there were going to be some areas of friction, to put it mildly, in getting this thing off the ground," Obama said.

But he said that he believes Karzai remains committed to political reconciliation, and that he needs to be.

"We don't expect that it will be easy," Obama told reporters. "But we do think ultimately we're going to need to see Afghans talking to Afghans about how they can move forward and end the cycle of violence so they can start actually building their country."

The Taliban opened the Doha office with a promise to renounce international terrorism and commit to peace negotiations, conditions the United States had set before it would support establishing the office as part of peace talks.

The Taliban were expected to raise the issue of a possible exchange of a captured U.S. soldier for Taliban prisoners being held at Guantanamo.

"This has been on the table since about Tuesday," said Col. Tim Marsano, a spokesman for the parents of Sgt. Bowe Bergdahl, 27, who was captured in 2009.

"Anytime that there are discussions and negotiations that may lead to the freeing of their son, that is encouraging news, especially after not much encouraging news over the past four years."

Karzai's decision on Wednesday to suspend the talks came a day after NATO-led troops transferred security responsibility to Afghan forces.

Reported by CNN's Reza Sayah in Doha, written in Atlanta by Tom Watkins with contributions from Khushbu Shah in Atlanta, Chris Lawrence and Jill Dougherty in Washington, Paul Vercammen in Los Angeles and Masoud Popalzai in Kabul.