Is it harmful to to do athletics without a sports bra?

Ive never actually thought about using one becase im quite small (b/c cup) but ive heard that you shouldnt even consider doing sports without wearing a sports bra... =S