Since searching the web didn't produce results from OSX 10.9 Mavericks I had to figure it out myself. There is no way to change this in settings, and the way it was done in OSX 10.8 Mountain Lion (right click the conversation area and choose "Show Smileys)" was not available.
The simple way to disable the auto-emoticons is to right click in the text input area, select Substitutions and click on "Emoji" (or "Text Replacement" in general if you do not like that).
The screenshot shows where to locate the option.
