Sir Mark Sedwill is leaving both posts in September - a dramatic move that confirms the prime minister is planning a major Whitehall shake-up.
The news means he will have served a relatively short term of less than two years.
David Frost - the UK's Brexit sherpa - will become the new national security adviser, combining it with his current job until negotiations with the EU on a trade deal are complete.
He will meet Michael Barnier on Monday morning for the first round of face-to-face Brexit talks in Brussels since the COVID-19 pandemic began.
Mr Frost said he was looking forward to the talks in the wake of the outbreak, adding: "These meetings will be smaller and focused on seeing whether we can begin to make genuine and rapid progress towards an agreement."
The UK has a deadline of 1 July if it wants to extend the transition period beyond the end of next year.
Sky News's deputy political editor Sam Coates said the appointment of Mr Frost "changes that job of national security adviser from a Whitehall civil service neutral one to a political one".
He added: "The question is the extent to which the Whitehall shake-up that will be rolled out over the next few days is actually about giving political figures in Number 10 more power and stopping things like the civil service and other checks and balances in the system from gumming up what they want to do, even when they might have to do and say difficult things to ministers."
Sir Mark was simultaneously in charge of 400,000 civil servants as cabinet secretary and the most senior official advising on security, defence, foreign policy and intelligence matters.
In an exchange of letters, Mr Johnson hailed his "outstanding service" and "calm and shrewd advice", adding Sir Mark had "done much to keep this country safe".
"After serving for decades with great distinction - and unflappable good humour - I believe you have earned the gratitude of the nation," the prime minister wrote.
Sir Mark responded that he had stepped in when his predecessor as cabinet secretary fell ill and was asked to stay on during the "acute phase" of coronavirus.
"As you are setting out this week, the government's focus is now shifting to domestic and global recovery and renewal," he continued.
"I wish you well and, of course, remain at your disposal in the years ahead. It has been a privilege to serve."
Lord Robert Kerslake, a former head of the civil service, told Sky News that the tone of the letters exchanged was contradictory to "days of hostile briefing", which he said had been "frankly unacceptable".
He added: "I think this sort of approach to making changes is not good."
Sir Mark has been given a life peerage to sit in the House of Lords, as is customary.
Speculation his job was under threat began to grow on Friday, when the Financial Times reported ministers were briefing he was expected to be gone by the end of 2020.
The Sunday Telegraph then claimed he could be ousted within a day.
Home Secretary Priti Patel did not dampen the rumours, failing to deny Sir Mark was facing the axe by 10 Downing Street.
Speaking to Sky News' Sophy Ridge on Sunday, she said: "Reform of the civil service is a matter that gets discussed in government and obviously a big role like that is subject to the prime minister…
"Any reforming government will be based around the type of delivery that our prime minister wants to drive for our great country and obviously have the right kind of support around him to deliver that."
Part of a speech by Cabinet Office minister Michael Gove released over the weekend also contained a curiously-timed warning.
"If this government is to reform so much, it must also reform itself," he said.
"The structures, ambitions and priorities of the government machine need to change if real reform is to be implemented and endure."
Helen Hayes, shadow minister for the Cabinet Office, said: "We pay tribute to the work Mark Sedwill has done. He has been a dedicated public servant and has run the civil service in difficult times.
"On the day it was revealed millions of jobs across the country could be under threat in the coming months, it is very concerning that Boris Johnson and Dominic Cummings are preoccupied with reshuffling Whitehall."
Dave Penman, head of the civil servants' FDA union, said Sir Mark was "one of the outstanding public servants of his generation" but attacked his treatment by Downing Street.
"Number 10 - or those around it - has sought to undermine Sir Mark and the leadership of the civil service, with a series of anonymous briefings against him over many months," he said.
"Not only is it a self-defeating and corrosive tactic, it's also a cowardly one, safe in the knowledge that those who are briefed against are unable to publicly respond."
Mr Johnson hailed his new national security adviser Mr Frost as an "experienced diplomat, policy thinker, and proven negotiator, with a strong belief in building Britain's place in the world".
Notably Downing Street said it was a political appointment and he will not be a civil servant.
The cabinet secretary is a role that dates back just over 100 years, and the few occupants who have held the title are said to be "at power's elbow".
But they are usually only obscurely known amongst the general public because they are a backroom figure and politically neutral.
Sky News defence correspondent Alistair Bunkall said: "Frost has no previous national security experience - Sedwill was ambassador to Afghanistan, a diplomat in Pakistan, Cairo and Iraq, UN weapons inspector and permanent secretary at the Home Office.
"Frost was ambassador to Denmark and chief negotiator of Task Force Europe.
"Frost's statement places importance of a security deal with the EU. This is important, but no mention of Islamic State, China, Russia, the far right, dissident Republicanism.
"Frost will at some point have to co-ordinate a COVID-delayed Integrated Security, Defence and Foreign Policy Review, not only has he said in his statement that he will 'remain chief negotiator for the EU talks and these will remain my top single priority', but these security reviews are notoriously testy and usually descend into tribal agendas.
"Hard for someone to oversee who, to the best of my knowledge, has little previous relationship with the agencies and military."
Sir Mark has directly served two prime ministers - Theresa May and then Boris Johnson - during a time when the civil service has faced upheaval over Brexit and coronavirus, with workers seconded between departments.
It has also faced criticism, and challenges - such as recently when the official civil service Twitter account posted minutes after the PM’s political aide Dominic Cummings' news conference: "Arrogant and offensive. Can you imagine having to work with these truth twisters?"
Whitehall is already braced for a shake-up in the autumn, following the news that the international development department is being abolished and merged with the Foreign Office.
The latter's permanent secretary was due to stay on to manage the transition - but was let go early by Mr Johnson.