How to test a chat bot?

How would you know a decent chat bot if you found one? I'm wondering what kinds of tricks you would use to figure out it's a bot, and to figure out how (un)sophisticated it was.

So, how would you probe a bot? What would be the dead giveaways, and what would give you pause / impress you if you saw the bot could handle it?