The health secretary told Sky News he was aware of the cost for families holidaying abroad and that the testing measures should not be in place "for a second longer than is absolutely necessary".
Mr Javid said he had asked officials to remove the PCR rule "at the moment we can".
He also told Sky News' Trevor Phillips that he did not "like the idea" of vaccine passports and hoped to avoid introducing them - but later took a firmer line and said the proposal "will not be going ahead".
"We have got a huge number of defences; of course we still want to remain very cautious, and there are some things that - when it comes to travel for example - there are some rules that are going to have to remain in place," Mr Javid told Sky News.
"But the PCR test that is required upon your return to the UK from certain countries... I want to try and get rid of that as soon as I possibly can.
"I am not going to make that decision right now, but I have already asked officials that at the moment we can, let's get rid of these kind of intrusions, the costs that generates for families, particularly families just trying to go out and holiday.
"We shouldn't be keeping anything like that in place for a second longer than is absolutely necessary."
At present, passengers returning from green list countries, or amber list destinations if they are fully vaccinated, must take PCR tests on or before day two after they arrive back in England.
The government last month announced the cost of NHS coronavirus tests for international travel were being reduced from £88 to £68 each, with the sum for two rests reducing from £170 to £136.
The travel industry has long complained that the costs of tests are too high.
In August, Mr Javid announced a "rapid internal review" of prices charged by government-approved companies after claims holidaymakers were being exploited.
Asked whether Labour would support scrapping PCR tests for travel, shadow health secretary Jonathan Ashworth told Sky News his party would "have to see what the proposal is when it comes before parliament".
The debate comes as the Welsh government announced that returning international travellers would be offered a wider choice of test providers from 21 September.
It said it would make the change as "new standards and spot checks are being introduced, which will help to address long-standing concerns and issues about the market for PCR tests".
Meanwhile, on the use of vaccine passports, Mr Javid began his Sunday interviews by saying they would only be introduced if there was "no alternative".
Plans had been announced that would have required proof of two vaccine doses to gain entry to nightclubs and other large events in England - a proposal heavily criticised by businesses.
"I think if we went down the road of vaccine passports or vaccine certification, as it is sometimes called, that is a big decision for any government to make," he told Sky's Trevor Phillips.
"We have been looking at that, we have been open about that. Instinctively, I do not like the idea at all of people having to, let's say, present papers to do basic things.
"So if we do that, it has to be something that is looked at very carefully and something that we believe has to be done with no alternative."
The health secretary said rising vaccination rates needed considering when making "a final decision", but that "I hope we can avoid it".
He continued: "I am not here today to rule that out, we haven't made a final decision as a government.
"We have been looking at it, we have been very open about that, but as I have said, I think that if we did something like that it has to be supported by the evidence and it has to be something that is absolutely, absolutely necessary with no alternative.
"So as I say, I hope we can avoid it."
But in a U-turn later on Sunday morning, Mr Javid confirmed the idea had been scrapped.
He told the BBC's Andrew Marr Show: "What I can say is that we have looked at it properly and while we should keep it in reserve as a potential option, I am pleased to say that we will not be going ahead with plans for vaccine passports."
Labour's deputy leader Angela Rayner said the government's approach to COVID passports "has been shambolic from the start".
"There has never been any clarity from ministers about what vaccine passports were supposed to achieve, how they would work and what was expected from businesses and workers," she said in a statement.
"Days ago, the vaccine minister stood before Parliament to confirm the introduction of COVID passports and stress their importance, now they've been scrapped.
"This is the culmination of a summer of chaos from ministers and they urgently need to get a grip before winter."
Earlier this week, Culture Secretary Oliver Dowden said an extension to use of vaccine passports would be looked at if there was a "public health need".
He told Sky News the government wanted "as few restrictions for as short a period as possible", but that if the situation worsened ministers would consider vaccine certification for more venues.
Scotland's first minister, Nicola Sturgeon, has announced vaccine passports will be introduced for entry into venues such as nightclubs and football grounds from 1 October.
MSPs voted in favour of the measure, which will be introduced after all Scottish adults have had chance to receive both vaccines.
Ms Sturgeon told Sky's Trevor Phillips the measure was "part of a package of measures, it has a part to play".
"Anybody who thinks there is one single magic wand solution to this virus probably haven't learned a lot over the last 18 months, and secondly, any measure we take has upsides and it has downsides."
She added: "This is a very limited scheme, it is in fact similar to what is being proposed in England as well, and it has a part to play."
Many companies welcomed the news that the government has scrapped plans to introduce vaccine passports for large events.
Michael Kill, chief executive of the Night Time Industries Association (NTIA), said: "Following an intense political and public campaign by the NTIA, its members and wider industry supporters, we welcome the comments from the Health Secretary this morning regarding the government's decision to scrap the planned mandate of COVID passports from the end of September."
Sacha Lord, who is night time economy adviser for Greater Manchester, added: "I'm pleased to see that the government has listened to the events and nightlife industry and dropped the plans for vaccine passports. The plans were untenable and illogical and there were multiple factors which would have been discriminatory and legally questionable."
And the Music Venue Trust (MVT) also welcomed the announcement, describing vaccine passports as "problematic".