Chris Grayling added that the millions who voted to leave the European Union would feel "cheated", and it could end the centuries of "moderate" politics that the UK has enjoyed since the Civil War.
The cabinet minister also urged his Conservative colleagues to back Theresa May's Brexit deal.
Just days before the critical Commons vote, Mr Grayling told the Daily Mail: "People have to think long and hard about how they are going to vote.
"This is too important for political game-playing and I urge Conservative MPs who back Brexit and others to back the deal.
"If not, we risk a break with the British tradition of moderate, mainstream politics that goes back to the Restoration in 1660.
"MPs need to remember that Britain, its people and its traditions are the mother of parliaments.
"We ignore that and the will of the people at our peril."
Mr Grayling said there would be a "different tone" in British politics if Brexit doesn't go ahead, and predicted a "less tolerant society" and a "more nationalistic nation".
He told the paper: "It will open the door to extremist populist political forces in this country of the kind we see in other countries in Europe.
"If MPs who represent seats that voted 70% to leave say 'sorry guys, we're still going to have freedom of movement', they will turn against the political mainstream."
Labour MP Luciana Berger, a leading supporter of the People's Vote campaign, said: "Chris Grayling is now cowering behind the small threat of far-right extremism as an excuse for refusing to give the British people a democratic final say on Brexit.
"These remarks are not only grossly irresponsible but also show just how desperate supporters of this proposed withdrawal agreement have become.
"We absolutely need to heal divisions in our society but we will not do so with a Brexit that makes us poorer and offers less control.
"And the answer to a small band of far-right thugs roaming the streets must never be to capitulate and restrict our democratic engagement - it must be more democracy."
It comes as the veteran Labour politician Roy Hattersley is set to declare his support for the People's Vote campaign in a speech in Sheffield on Saturday.
Lord Hattersley, who was a Labour minister in the Wilson and Callaghan governments, said the "vast majority" of members want the party to campaign for another referendum if hopes of an early general election disappear.
The former deputy Labour leader is expected to tell Jeremy Corbyn to "put out of his mind all the outdated nonsense about a socialist economy being impossible in Europe".
Speaking alongside Labour former cabinet minister Dame Margaret Beckett and Liberal Democrat leader Sir Vince Cable, Lord Hattersley will say there is "no conceivable deal which is remotely as beneficial to Great Britain as full membership of the European Union".
The 86-year-old has previously said leaving the European Union will be a "disaster".
Lord Hattersley will say: "Jeremy Corbyn promised the party, that policy would be made by the membership. Conference resolutions would, he said, be implemented.
"That means he must, difficult as it may be, put out of his mind all the outdated nonsense about a socialist economy being impossible in Europe.
"The vast majority of party members now expect that when the hope of an early general election is extinguished, Labour will campaign for a People's Vote."
He is also expected to warn that young people will "pay the price" from Brexit, while the elderly will be "protected" from the long-term "penalties".
Tory former minister Anna Soubry and Labour MP for Wakefield Mary Creagh will also speak at the event as part of the People's Vote campaign's national "Day of Action".
It comes amid reports that some cabinet ministers believe Mrs May has run out of time to get crucial exit legislation through parliament before 29 March - even if she wins the critical vote next week.
Two of the biggest donors to the Leave campaign said they believed Brexit would eventually be abandoned by the government and that the UK would stay in the EU.
Billionaire businessman Peter Hargreaves, who pumped more than £3m into the Brexit campaign, said: "I have totally given up. I am totally in despair, I don't think Brexit will happen at all."
Hedge fund manager Crispin Odey, who donated more than £870,000 to pro-Leave groups, said: "My view is that it ain't going to happen. I just can't see how it happens with that configuration of parliament."
They attributed a lack of direction from Brexiteers as one of the reasons for their pessimism.