In a move to modernize city services, NYC’s AI chatbot, introduced under Mayor Eric Adams’ administration, has unfortunately been advising businesses inaccurately, sometimes even suggesting actions contrary to the law. This development, which aimed at assisting entrepreneurs and business owners with city regulations, has backfired, causing concern among stakeholders and legal experts.
Launched as part of the MyCity portal, this AI tool was designed to be a comprehensive resource for businesses in New York City, offering information on permits, regulations, and compliance. Powered by Microsoft’s Azure AI, the chatbot promised to streamline operations and provide valuable insights to business owners. However, reports from The Markup and corroborated by Documented and The City, reveal that the chatbot has been disseminating “dangerously inaccurate” advice. This includes false information on housing policies, workers’ rights, and other critical business operation aspects.
Instances of the misleading guidance include telling landlords that they do not need to accept housing vouchers, contrary to city law which prohibits discrimination based on income. Moreover, the chatbot inaccurately informed users that businesses can withhold cash payments and employers can deduct wages from employee tips, both of which are incorrect and illegal practices in NYC.
The NYC Office of Technology and Innovation, while acknowledging the chatbot as a pilot project expected to improve over time, emphasizes that it has provided “thousands of people with timely, accurate answers” about business operations. Despite these assurances, the emergence of legal inaccuracies has led to calls for immediate corrective measures. Critics argue that if the chatbot cannot reliably provide accurate information, it should be taken offline to prevent further dissemination of erroneous advice.
The city administration has expressed commitment to refining the chatbot, highlighting the potential of AI to support small business owners if accurately deployed. Yet, this scenario underscores the complexities and potential pitfalls of integrating AI into governmental operations, especially concerning legal advice.
This situation has sparked a broader conversation about the responsibility and accountability of AI-powered tools in governmental services, emphasizing the need for stringent oversight, accuracy, and transparency in their deployment.
