PLP backbencher Randy Horton told the House that he had been "bombarded" with complaints from whites and blacks alike in recent days.
"They've not been complaining about the PLP so much as the leader of the PLP Government," he said.
He acknowledged the Premier had made many positive strides during his tenure, but said his decision to bring in four Guantanamo detainees without consultation was the last straw.
"On the last decision that he's made, he's overstepped his boundaries," Mr. Horton said. "I have no problem with the humanitarian aspect of a decision to bring these gentlemen into Bermuda. I may not agree with the decision because I don't.
"Particularly I don't agree that we have them here without having the consultation, without the benefit of the wisdom of so many people in this country in leadership positions."
He noted that the recent plaudits from US Senators and a phone call Dr. Brown received from President Obama "are absolutely expected", especially given that Bermuda has done the US a service.
"We've taken the US out of the hole these are four people they've detained," he said.
Mr. Horton was particularly vocal on the issue of naturalisation for the Uighurs, telling the House that his American wife had to wait ten years to become a citizen.
"I believe firmly they should not be given full citizenship while they're here," he said. "That is something indeed they should earn."
He said he knew this week's rallies would not have the impact they might have because the "optics" were wrong, noting all the white protesters present.
He went on to commend Janice Battersbee, one of the key organisers of the protests, calling her a "staunch PLP supporter" who has been active in party meetings.
Mr. Horton said he was "disturbed" by the Premier's dismissal of the rally, alluding to Dr. Brown's comments on Tuesday that he'd been to larger and longer demonstrations .
"He was on the other side protesting," he said of Dr. Brown's history of activism.
"Our honourable Premier has not been checked enough," he said in conclusion. "I don't think we've held him accountable enough."
Cabinet Minister Dale Butler told the House that the Premier should have consulted the UK before his controversial decision, but admitted Dr. Brown was not the type to constantly seek permission.
"It's not part of his modus operandi," he said.
Mr. Butler praised this week's rallies as "democracy in action", and noted that he saw "Bermudians" at the protests rather than white people.
"I'm not even surprised or upset or want to belabour the point that the group was mostly white. So what?"
He read a selection of letters he received this week aloud in order to demonstrate the variety of viewpoints on the no confidence motion.
Mr. Butler then told the House that he'd initially joined the PLP as a 16-year-old because of his perception that the party was committed to working people and to the law. He then launched into a condemnation of the Premier's actions, calling his lack of consultation with Cabinet "reprehensible".
"As a Minister I am very disappointed I was not consulted," he said. "I was absolutely stunned I think we all were when we were not informed."
He concluded his speech by asking for an apology from the Premier, and withdrawal of the motion should Dr. Brown comply.
"The Premier should apologise to the House, his Cabinet, and the people of Bermuda," he said. "Withdraw your motion; I would like to accept an apology."
UBP MP Pat Gordon-Pamplin took issue with Mr. Horton's comments from earlier in the evening on the "optics" of this week's demonstrations, when he noted that the two rallies were predominantly white.
"Isn't this Government, the PLP Government, the Government of all people of Bermuda?" she asked.
She called the Premier's actions "indefensible", asking: "Do you believe might should override right?"
Ms Gordon-Pamplin drew murmurs from the Government MPs across the floor when she mentioned the Finance Minister, E-Commerce Minister and Education Minister as "options" for leadership.
She went on to discuss reports of the four Uighurs' training in Afghanistan and Tora Bora, but said we know little about how intimately they were involved in any terrorist activity. She expressed concern that the US, who had detained the four men, had now washed their hands of them.
"If I had spent eight years of doing something wrong, I would want to put it right," she said. "We are all humanitarians within this honourable House and I believe in the country at large."
Ms Gordon-Pamplin was highly critical of the decision to give the Uighurs citizenship when there were expat workers who have put "time, energy and money" into Bermuda, but who will never be full citizens.
"Why can we not have something that helps our people that have been here, that helps them first?" she asked.
She closed by saying that the PLP Government has an obligation to every single Bermudian regardless of ethnicity.