Restaurant staff in New York City report that diners increasingly refuse to accept allergen warnings when those warnings conflict with an AI chatbot’s incorrect output. A server named Madison describes customers ordering fish dishes containing shellfish broth after their AI assistant confirmed the meal was safe. When staff correct the record, guests often insist the technology is wrong and demand the establishment trust the software instead of human knowledge. This trend is not limited to dining venues but extends across customer service roles where agents rely on automated systems to resolve queries. The core issue is that these systems frequently hallucinate facts, providing confident but false information that users treat as absolute truth. Consequently, employees face a dilemma where they must either override the tool or risk serving harmful products to vulnerable people. The situation highlights a growing friction between automated assistance and basic safety protocols in high-stakes interactions. Staff members now bear the burden of verifying every claim made by the software, effectively negating the efficiency gains promised by automation. As reliance on these tools deepens, the potential for real-world harm increases whenever the underlying data is flawed or the model generates errors.
- Diners cite ChatGPT as proof that allergen warnings are incorrect.
- Employees must manually verify every fact before serving food.
- False confidence in AI output overrides professional judgment.
