The NT Council of Social Services (NTCOSS) has slammed the system as belittling, impenetrably complicated and in need of reform.
The NTCOSS will front a parliamentary committee today and call on the Federal Government to abolish the cashless debit card.
The debit cards were introduced by the Liberal-National Coalition in 2016 and set up in several communities including, Ceduna in South Australia and Cape York in Far North Queensland.
Under the scheme, up to 80 per cent of a person's welfare payment was added to a card and the funds could not be withdrawn for cash or spent on gambling or alcohol.
The Albanese Government's proposal would shift more than 17,000 welfare recipients away from the scheme.
There is evidence to suggest large numbers of Territorians don't have any form of social security, NTCOSS chief Deborah Di Natale said in a submission to the committee.
"Feedback suggests that many people become disengaged as they find the social security system repelling, belittling, impenetrably complicated, and damaging," she said.
"The Government's desire to co-design a workable voluntary income-management scheme with affected communities will not be hampered by the immediate repeal of compulsory income management."
However, despite pushback from many, a leading Indigenous think tank said models of income management had been successful in some remote communities.
The Cape York Institute urged the committee not to apply a blanket approach but rather allow communities to decide on a model for themselves.
The Government was consulting with communities about what they wanted, Social Services Minister Amanda Rishworth said.
"The cashless debit card was a broad-based income-management card that was foisted on communities and foisted on people that did not want it," Rishworth told Sky News on Tuesday.
"It has had negative impacts on many of those people, and the positive social outcomes that were promised never eventuated."