There are various degree of penalty, from losing your page rank completely to a certain keyword from you site being filtered..
If its a complete ban you keyword or even site name will not show up on search results.

How do you get your site banned then?

Not that I want to get one banned, but I'm curious....

normally google will just sandbox you (slow or halt ur rank climbing). but if someone reports you then a human will look into it and you can get properly blacklisted.

if this happens and you want to argue it, make sure you're tracks are clean, then report the problem to google and keep pestering them (politely of course) until they fix it. use formal language and sound professional and authoritative