Mr Bercow infuriated Brexiteers and government ministers by allowing a vote on whether the prime minister should be forced to reveal her Plan B within three days, should her Brexit deal be defeated next week.
The government subsequently lost the vote, raising the stakes further over MPs' looming decision on Theresa May's agreement, scheduled for next Tuesday.
The speaker was accused of "unprecedented" action ahead of the continuation of the debate on the deal on Wednesday.
Angry MPs highlighted how Mr Bercow has previously revealed he supported Remain at the 2016 EU referendum, as well as reports he displays a "B******* to Brexit" sticker on his car.
They questioned why Mr Bercow had allowed a vote on an amendment, tabled by Tory Remainer Dominic Grieve, which demanded a speedier process for Mrs May revealing her Plan B.
A senior Downing Street source said they were "surprised" Mr Bercow permitted the vote because of a belief the motion on Mrs May's Brexit deal was "unamendable".
Leader of the House of Commons Andrea Leadsom told the Speaker there were "concerns" about his action, as she challenged whether the Commons clerk Sir David Natzler agreed with his decision.
Mr Bercow did not provide confirmation of Sir David's agreement, but stated they "discussed" the matter.
He told MPs: "I'm trying to do the right thing and make the right judgements. That is what I have tried to do and what I will go on doing."
Challenging Mr Bercow on the decision in a further point of order, former Tory minister Crispin Blunt called on the Speaker to "reflect" on his position.
He said: "For many of us we will now have an unshakeable conviction that the referee of our affairs, not least because you gave your opinion and your vote on the issue of Brexit publicly, that we we will have an unshakeable conviction that the referee is no longer neutral.
"I just invite you to reflect on the conclusion that many of us will have inevitably have come to."
Mr Blunt also highlighted how Mr Bercow had promised to serve no longer than nine years when he took on his role in June 2009, as well as recent bullying allegations against the Speaker, which he denies.
Senior Labour MP Dame Margaret Beckett has previously said Mr Bercow should be kept in place despite the bullying claims because his role over Brexit "trumps bad behaviour".
Fellow Tory Brexiteer Adam Holloway said: "We've all noticed in recent months a sticker in your car making derogatory comments about Brexit. Have you driven that car with the sticker on?"
Mr Bercow said that was a "factual error" and told MPs the car belonged to his wife.
He said: "That sticker on the subject of Brexit happens to be affixed to or in the windscreen of my wife's car, and I'm sure he wouldn't suggest for one moment that a wife is somehow the property or chattel of her husband.
"She is entitled to her views, that sticker is not mine and that's the end of it."