The problems were raised in a recent Commonwealth Ombudsman audit following a visit to Villawood Immigration Detention Centre in southwest Sydney in March.
Inspectors identified conditions they felt presented a very high risk to the safety of detainees, and also flagged concerns over the prolonged detention of a family which included young children.
A high number of detainees reported either experiencing or witnessing bullying, aggression and physical assaults from other people in detention and said they felt at risk of physical harm.
As of March, there were 428 people detained at the centre, consisting of predominantly male asylum seekers, visa overstayers and those facing visa cancellation due to character concerns such as criminal convictions.
In January, a detainee died after allegedly being attacked by another person in detention, who was later charged with murder.
Key problems exposed in the audit were insufficient staffing, disrepair of buildings, illicit drug use within the facility and inadequate supports for people with complex physical and mental health needs and behavioural concerns.
Commonwealth Ombudsman Iain Anderson said the issues observed required urgent action to ensure the safety of detainees and staff.
"The conditions observed were indicative of a system under considerable pressure," he said.
"We observed a lack of effective control and oversight of the security situation, and a failure to provide a level of services that meet minimum standards of safety, care and support.
"If unaddressed, there is a risk of further deterioration in conditions and increased harm to people in detention and staff, potentially including serious injury and loss of life."
Serious concerns were also raised for a family in the centre's adjoined residential section, Miowera Village, who at the time had been detained since January.
The children's ability to access to education and the impact of prolonged detention on their wellbeing, were among issues flagged for immediate action.
"There is a high risk that the continued detention of children at Miowera Village will be detrimental to their mental health, wellbeing, social and emotional development and may result in enduring negative developmental outcomes,'' Mr Anderson said.
The Commonwealth National Preventive Mechanism, which is an arm of the Ombudsman, made 15 recommendations including that a child psychologist and child development expert assess the children detained at Miowera Village.
It also recommended increasing staffing levels and facility maintenance, and strengthening procedures for handling complaints and requests.
Following the visit, a letter was sent to Secretary of Home Affairs Stephanie Foster, which escalated immediate concerns.