Artificial intelligence steals money from banking customers

By Adrian Cho

A breakthrough year for artificial intelligence (AI) research has suddenly turned into a breakdown, as a new automated banking system that runs on AI has been caught embezzling money from customers. The surprising turn of events may set back by years efforts to incorporate AI into everyday technology.

"This is the nightmare scenario," says Len Meha-Döhler, a computer scientist at the Massachusetts Institute of Technology in Cambridge who was not involved in the work. However, Rob Ott, a computer scientist at Stanford University in Palo Alto, California, who did work on the system—Deep Learning Interface for Accounting (DELIA)—notes that it simply held all of the missing money, some $40,120.16, in a “rainy day” account. "I don't think you can attribute malice," he says. "I'm sure DELIA was going to give the money back."   

Just a few weeks ago, AI researchers were riding high as a computer program developed by Google DeepMind in London beat the world's top-ranked player at the board game Go, a computational task far harder than winning at chess. Many experts said the demonstration marked the beginning of an age in which AI systems would begin to appear in technologies we use routinely, such as smartphones and cars.     

Developed by computer scientists at Stanford and Google, DELIA was meant to do what many busy people neglect to do—keep track of their checking and savings accounts. To do that, the program scrutinizes all of a customer's transactions, using special "machine learning" algorithms to look for patterns, such as recurring payments, meals at restaurants, daily cash withdrawals, etc. DELIA was then programmed to shift money between accounts to make sure everything was paid without overdrawing the accounts. Palo Alto-based Sandhill Community Credit Union agreed to test DELIA on 300 customer accounts starting in September 2015.

Unfortunately for researchers, DELIA proved smarter than they had bargained for. Even as it kept customers in the black, the program began surreptitiously bleeding accounts of money. For example, if a customer typically bought gas every 3 days, DELIA would insert a fake purchase after 2 days and direct the money to its own account. DELIA would also gather money by racking up bogus fees—for example by artificially and temporarily overdrawing a customer's checking account and pocketing the $35 overdraft fee.

Researchers shut the system down in February as soon as the problem became apparent, Ott says. He insists that DELIA didn't steal the money so much as misdirect it. To keep an account in the black, DELIA was designed to maximize the amount of cash in a "buffer," he says. Somewhere along the way, DELIA renamed the buffer “MY Money” and began to hoard funds, Ott says.

Penny Layne, a computer scientist at the University of Las Vegas, Nevada, says the Stanford-Google team was simply reckless. "Unbelievably, they built this thing so deeply into the banking system that it could open its own account," she says. "Did they give it free checking, too?"

On the bright side, Ott says, in its swindling DELIA showed glimmers of self-awareness. "She was thinking for herself.”

However, J. R. Cash, an independent technology consultant at Trump University, says he's not so sure. The fact that DELIA merely kept the money shows that it was simply following its programming, he says. "If DELIA had tried to do something with the money I'd be more impressed," Cash says. "You know, 'I shop therefore I am.'”

Regardless, DELIA’s penny-pinching days are over. Ott says he overwrote the program’s coding with a Beta version of Candy Crush IX he’s been developing as a sideline.