Google penalties come in two forms: algorithmic penalties and manual action penalties.
You can easily tell if you’ve been hit by a manual action penalty as Google will inform you via your Webmaster Tool that a page on your site or your entire site has been penalized. Manual action penalties are often accompanied by a big decline in rankings and, possibly, complete deindexation of your site. Manual action penalties are very difficult to recover from.
Algorithmic penalties are harder to identify as you’re not directly informed by Google. Some clues to identify that you’ve been hit by an algorithmic penalty are:
You can use Google Analytics’s Fruition Google Penalty Checker, which creates a graph of all Google penalties and shows you the probability that you’ve been affected by one. Another tool is Barracuda, which also shows a graph highlighting all Google penalty updates, but does not give you the probability of your site being impacted by one.