This week, a piece from The Makeup uncovered biases in U.S. mortgage-approval algorithms that lead lenders to turn down people of color more often than white applicants. A decisioning model called Classic FICO didn’t consider everyday payments — like on-time rent and utility checks, among others — and instead rewarded traditional credit, to which Black, Native American, Asian, and Latino Americans have less access than white Americans.

The findings aren’t revelatory: back in 2018, researchers at the University of California, Berkeley found that mortgage lenders charge higher interest rates to these borrowers compared to white borrowers with comparable credit scores. But they do point to the challenges in regulating companies that riskily embrace AI for decision-making, particularly in industries with the potential to inflict real-world harms.

The stakes are high. Stanford and University of Chicago economists showed in a June report that, because underrepresented minorities and low-income groups have less data in their credit histories, their scores tend to be less precise. Credit scores factor into a range of application decisions, including credit cards, apartment rentals, car purchases, and even utilities.

In the case of mortgage decisioning algorithms, Fannie Mae and Freddie Mac, home mortgage companies created by Congress, told The Markup that Classic FICO is routinely evaluated for compliance with fair lending laws internally and by both the Federal Housing Finance Agency and the Department of Housing and Urban Development. But Fannie and Freddie have over the past seven years resisted efforts by advocates, the mortgage and housing industries, and Congress to allow a newer model.

Algorithmic discrimination

The financial industry isn’t the only party guilty of discrimination by algorithm, equality and fairness laws be damned. Last year, a Carnegie Mellon University study found that Facebook’s ad platform behaves prejudicially against certain demographics, sending ads related to credit cards, loans, and insurance disproportionately to men versus women. Meanwhile, Facebook rarely showed credit ads of any type to users who chose not to identify their gender, the study showed, or who labeled themselves as nonbinary or transgender.

Laws on the books including the U.S. Equal Credit Opportunity Act and the Civil Rights Act of 1964 were written to prevent this. Indeed, in March 2019, the U.S. Department of Housing and Urban Development filed suit against Facebook for allegedly “discriminating against people based upon who they are and where they live,” in violation of the Fair Housing Act. But discrimination continues, a sign that the algorithms responsible — and the power centers creating them — continue to outstrip regulators.

The European Union’s proposed standards for AI systems, released in April, come perhaps the closest to reigning in decisioning algorithms run amok. If adopted, the rules would subject “high-risk” algorithms used in recruitment, critical infrastructure, credit scoring, migration, and law enforcement to strict safeguards and ban outright social scoring, child exploitation, and certain surveillance technologies. Companies breaching the framework would face fines of up to 6% of their global turnover or 30 million euros ($36 million), whichever is higher.

Piecemeal approaches have been taken in the U.S. to date, such as a proposed law in New York City to regulate the algorithms used in recruitment and hiring. Cities including Boston, Minneapolis, San Francisco, and Portland have imposed bans on facial recognition, and Congressional representatives including Ed Markey (D-Mass.) and Doris Matsui (D-CA) have introduced legislation to increase transparency into companies’ development and deployment of algorithms.

In September, Amsterdam and Helsinki launched “algorithm registries” to bring transparency to public deployments of AI. Each algorithm cited in the registries lists datasets used to train a model, a description of how an algorithm is used, how humans use the prediction, and how algorithms were assessed for potential bias or risks. The registries also provide citizens a way to give feedback on algorithms their local government uses and the name, city department, and contact information for the person responsible for the responsible deployment of a particular algorithm

This week, China became the latest to tighten its oversight of the algorithms companies use to drive their business. The country’s Cyberspace Administration of China said in a draft statement that companies must abide by ethics and fairness principles and shouldn’t use algorithms that entice users to “spend large amounts of money or spend money in a way that may disrupt public order,” according to Reuters. The guidelines also mandate that users be given the option to turn off algorithm-driven recommendations and that Chinese authorities be provided access to the algorithms with the choice of requesting “rectifications,” should they find problems.

In any case, it’s becoming clear — if it wasn’t already — that industries are poor self-regulators where AI is concerned. According to a Deloitte analysis, as of March, 38% of organizations either lacked or had an insufficient governance structure for handling data and AI models. And in a recent KPMG report, 94% of IT decision makers said they feel that firms need to focus more on corporate responsibility and ethics when developing their AI solutions.

A recent study found that few major AI projects properly address the ways that technology could negatively impact the world. The findings, which were published by researchers from Stanford, UC Berkeley, the University of Washington, and University College Dublin & Lero, showed that dominant values were “operationalized in ways that centralize power, disproportionally benefiting corporations while neglecting society’s least advantaged.”

A survey by Pegasystems predicts that if the current trend holds, a lack of accountability within the private sector will lead to governments taking over responsibility for AI regulation over the next five years. Already, the results seem prescient.

For AI coverage, send news tips to Kyle Wiggers — and be sure to subscribe to the AI Weekly newsletter and bookmark our AI channel, The Machine.

Thanks for reading,

Kyle Wiggers

AI Staff Writer