NYC AI Child Welfare Risks
- New York City's Administration for Children’s Services (ACS) is under fire for its use of an algorithmic tool to categorize families as "high risk." The AI tool, which...
- Developed internally by ACS's Office of Research Analytics, the algorithm scores families based on 279 variables, subjecting those deemed highest risk to increased scrutiny.
- The algorithm's variables are derived from cases in 2013 and 2014 where children suffered serious harm.
NYC’s AI child welfare algorithm faces critical scrutiny, as an algorithmic tool used by the Administration for Children’s Services (ACS) to identify “high-risk” families raises many red flags. The system, which employs 279 variables, is under fire for lack of clarity and potential bias against Black families, who are disproportionately affected by ACS investigations. Similar AI systems in other jurisdictions have faced similar challenges, including concerns about racial disparities and the automation of discriminatory practices.This article, brought to you by News Directory 3, dives into the algorithm’s mechanics and potential impacts. Discover what’s next as experts call for independant audits to ensure the prevention of social inequalities.
NYC child Services’ AI Algorithm Faces Scrutiny Over Bias Concerns
New York City’s Administration for Children’s Services (ACS) is under fire for its use of an algorithmic tool to categorize families as “high risk.” The AI tool, which considers factors such as neighborhood and a mother’s age, has sparked concerns about transparency, potential bias, and the risk of unwarranted family separation.
Developed internally by ACS’s Office of Research Analytics, the algorithm scores families based on 279 variables, subjecting those deemed highest risk to increased scrutiny. This can include home visits, calls to teachers and family, and consultations with outside experts.critics argue that the lack of transparency and accountability surrounding the system echoes past failures in child services.
The algorithm’s variables are derived from cases in 2013 and 2014 where children suffered serious harm. However, the specifics of the data analysis, auditing, and testing remain unclear. This raises questions about whether the inclusion of data from other years would alter the risk scoring.
Data shows Black families in New York City face ACS investigations at seven times the rate of white families. ACS staff have acknowledged the agency’s more punitive approach toward Black families. Critics fear the algorithm could automate and amplify existing discriminatory practices.
Families and even caseworkers often lack insight into why a case is flagged, making it difficult to challenge the system’s decisions. This lack of transparency and accountability has drawn criticism from advocates and legal experts.
Similar AI tools in child services have faced challenges related to systemic biases. In 2022, the Associated Press reported that an algorithm used in Allegheny County, Pennsylvania, flagged 32.5% of Black children for mandatory investigation, compared to 20.8% of white children. Social workers often disagreed with the algorithm’s risk scores.
The Allegheny system, like the one in New york City, operated with secrecy, denying families access to their algorithmic scores. A judge’s request to view a family’s score was resisted, with the county claiming it didn’t want to influence legal proceedings with algorithmic numbers.
Other jurisdictions have rejected or abandoned similar systems. New Zealand rejected the Allegheny tool due to concerns it would disproportionately affect Māori families. California scrapped a similar project in 2019, citing racial equity concerns.
What’s next
As scrutiny of AI in government intensifies, experts call for independent audits and greater transparency to ensure public trust and prevent the perpetuation of social inequalities.