Tensions boiled over on Friday as residents of an on-site tent embassy were forcibly moved on from Brisbane's Victoria Park.
Five people were arrested, including a man tackled to the ground by officers, in what critics described as a surprise raid by police and council staff.
But it was a tamer affair on Sunday with no clashes reported when hundreds of people rallied at the sprawling park in the city's north.
Bulldozers are expected to begin clearing the way for construction of the $3.6 billion 2032 Olympics main arena as early as Monday.
Indigenous elders have declared the park a place of significance to many First Nations peoples and are calling for their concerns to be heard.
The state government has converted the park to freehold land, but the project faces several challenges under national Aboriginal heritage laws.
Environment Minister Murray Watt said he received 10 submissions seeking to "issue a declaration to protect a significant Aboriginal area that is under threat of injury or desecration".
But he confirmed two had already been declined, while the others remained under consideration.
"I encourage all parties to engage in this process and the proposed development in an open and transparent way," he said.
But Queensland Deputy Premier Jarrod Bleijie vowed his government was pushing on with its timeline despite the vociferous opposition, including from former Liberal National premier Campbell Newman.
"The reality is it's a construction site and no one's going to be able to be on the construction site, so they're going to have to be moved on," he told reporters.
However, the plan's opponents remain defiant.
Save Victoria Park rally organisers insist the stadium is not a done deal.
Hundreds gathered, including children, carrying placards with messages saying "parks over profits" and "hands off Victoria Park".
The activists described police actions as a "heavy-handed and disproportionate" response, adding it was a "shameful scene belonging to another era".
Many had been camping in the park for months in the tent embassy.
Authorities initially declared the site would be fenced off for five years to build the 2032 Games centrepiece.
However, protesters appeared blindsided on Friday when police and Brisbane City Council asked them to move before imposing a 2pm deadline.
The Crisafulli government and council have defended the decision to move against the protesters, saying their presence had become a safety issue with construction imminent.
Save Victoria Park is calling on the state government to respect a legal process under way and read an expert report warning against large-scale construction.
It has also appealed to the International Olympic Committee in Lausanne, Switzerland to enact the principles it espouses of inclusion, sustainability and peace.