Just 16 per cent of Kiwis have Mr Luxon as their preferred leader in the latest TVNZ-Verian poll, which also shows his right-wing coalition would be turfed from office if an election were held last week.
"I get it. Not everyone will want me at their barbecue," Mr Luxon told Newstalk ZB on Monday.
In a press conference later in the day, he added "not everyone will want to have a beer with me", acknowledging his failure to win over New Zealanders.
The key question now is whether a majority of his National party caucus want him to remain as leader, as doubts crystallise ahead of the November 7 election.
TVNZ has reported an anonymous National MP stating the partyroom numbers were "probably there" to remove Mr Luxon, but the party wanted him to resign to avoid a messy confrontation.
Mr Luxon insists he hasn't considered resigning and retains the support of his caucus.
National has been trending downwards since it won 38 per cent of the vote to win office in 2023.
They now sit on 29.7 per cent - just below a psychologically important 30 per cent threshold - according to the TVNZ poll, well behind Labour on 36.
Should the result transpire at the November 7 election, it would make Mr Luxon's coalition the first one-term government in his party's history.
Other pollsters have the coalition of National, NZ First and ACT holding onto government, but with fewer National members, which is the primary cause of concern for Mr Luxon's MPs.
A report in the NZ Herald stated that earlier in April, concerns from the back bench about Mr Luxon's performance had reached such a level that party whip Stuart Smith felt obliged to raise the issue, requesting a meeting with the prime minister.
However, in what the Herald called a "rare and extraordinary" rebuke, Mr Luxon reportedly "ghosted" Mr Smith and didn't meet with him.
Mr Luxon disputes a meeting was sought.
It is not clear how firm or widespread dissatisfaction with Mr Luxon was then, or will be at a showdown caucus meeting on Tuesday.
Mr Luxon said he understood "there were five people that are moaning and frustrated ... that are possibly speaking to media about their frustrations".
This is a shift, with Mr Luxon arguing as recently as Friday that he had "full support" of his caucus.
In March, Mr Luxon shook up his cabinet and re-election team, removing moderate MP Chris Bishop from the key posting of campaign chair in favour of conservative MP Simeon Brown.
Mr Bishop has denied being part of a plot to remove Mr Luxon.
"Some people have been talking out of school ... that is unhelpful and untidy and indicates that the National Party is focused on ourselves rather than focused on the country," Mr Bishop told TVNZ.
Mr Luxon has vowed not to resign and contest the next election.
"We know we want to do better. We need to do better as a National Party. We know it's also been very volatile and challenging times," he said.
"What this country needs is strong economic management and a stable coalition, and that's what we're delivering."