The Pentagon announced his departure in a brief statement on Wednesday, saying he was leaving the administration "effective immediately".
It did not provide a reason or say whether it was his decision to go.
The sources, who spoke on condition of anonymity, said Phelan was dismissed in part because he was moving too slow to implement reforms to speed shipbuilding and because he had fallen out with key Pentagon leadership.
One source cited bad relationships with Hegseth, Hegseth's deputy, Steve Feinberg, as well as the navy's No. 2 civilian, Hung Cao, who the Pentagon said would take over as acting navy secretary.
The source also cited an ethics investigation into Phelan's office.
A billionaire seen as having close ties to President Donald Trump, Phelan is the first administration-picked service secretary to be fired since Trump came back into office in 2025.
His departure fits within a broader context of upheaval at all levels of leadership at the Pentagon under Hegseth's watch, including the firing in 2025 of the chairman of the Joint Chiefs of Staff, air force general CQ Brown, as well as the chief of naval operations and air force vice-chief of staff.
On April 2, Hegseth fired army chief of staff Randy George without citing a reason.
Two US officials said the decision was tied to tensions between Hegseth and Army Secretary Daniel Driscoll.
The latest departure comes during a tense ceasefire with Iran, as the US flows more naval assets into the Middle East.
The US military is relying on naval assets to carry out a blockade of Iran, which President Donald Trump is hoping will pressure Tehran to negotiate an end to the conflict on his terms.
