AI and chatbots sell on the promise of being the future of customer service. We present a project using IBM Watson in the financial sector that tried to live up to this promise. On paper it was hugely successful — nevertheless when looking back we were left with mixed feelings regarding the User Experience. What happened? What could we have done differently or improved on? In this talk we discuss our findings with regards to UX process, research, and machine learning.