brinsa.com

An artificial intelligence-powered chatbot created by New York City to help small business owners is criticized for dispensing bizarre advice that misstates local policies and advises companies to violate the law.

The MyCity AI chatbot uses Microsoft's Azure AI large language models but is plagued by the same issues that most AI models suffer from, including giving wrong and misleading information.

Days after the issues were first reported last week by tech news outlet The Markup, the city has opted to leave the tool on its official government website.

In response to the criticisms, Mayor Adams stated that the AI tool is "wrong in some areas, and we've got to fix it." He further defended the chatbot, saying that "any time you use technology, you need to put it into the real environment to iron out the kinks.”

The chatbot was previously reported as telling landlords that they were free to discriminate based on income and that business owners could take workers' tips as their own—despite both of these practices being illegal in New York.

Subsequent probes of the chatbot's responses on Thursday revealed that it was still giving out incorrect information. Reuters reported that the AI had said that stores no longer had to accept cash as a payment method, even though this violates New York law.

Microsoft told Reuters that the company was fixing the chatbot. At the same time, New York City's Office of Technology and Innovation released a statement saying that "as soon as next week, we expect to significantly mitigate inaccurate answers."

The AI chatbot seems like an especially egregious waste of resources. AI chatbots built on large language models are notoriously prone to spitting out errors, a shortcoming that some experts think will never be fully fixed. And what's the point of using a chatbot if it will give you incorrect information that may lead you to break the law? Couldn't the money for this tech go towards something decidedly low-tech yet beloved by all New Yorkers?

About the Author

Markus Brinsa writes about AI failure, enterprise risk, governance, and the structural shifts underneath them — the through-line being the gap between AI governance on paper and what systems actually do at runtime. He created Chatbots Behaving Badly, a publication and podcast investigating real incidents in which AI systems gave bad advice, were manipulated, or failed in ways that mattered. He is the Founder & CEO of SEIKOURI Inc., an international strategy firm that gives enterprises and investors human-led access to pre-market AI — and converts first looks into rights and rollouts that scale. Access creates possibility. Rights create leverage. Scale turns early advantage into durable position. The two halves are the same work from opposite ends: SEIKOURI gets clients to AI early and makes sure what they deploy holds up once it's running. Thirty years bridging technology, strategy, and cross-border growth across the U.S. and Europe. I close the gap between what leaders expect AI to do and what it actually does in the wild.

MyCity - Faulty AI Told People to Break the Law

About the Author

Sources

Microsoft told Reuters that the company was fixing the chatbot. At the same time, New York City's Office of Technology and Innovation released a statement saying that "as soon as next week, we expect to significantly mitigate inaccurate answers."

About the Author