Is it legal to put an exclamation point in a baby's (legal) name?
Not in California.
Naming Laws vary from country to country and, in the US, from state to state. While few places specifically ban particular names, county clerks have some discretion about refusing to certify names.
California requires that names be composed of the 26 letters of the alphabet; no diacritical marks, like accents or umlauts, are officially recognized on a California birth certificate. This is specifically inconvenient to Spanish names with accents, like José. The law only applies to the name on official records, and does not affect what people unofficially call their children. Kentucky, by contrast, has no naming restrictions on its books at all.
If you want to name your baby girl KABLAMMO!, there's no legal penalty for doing so. Just don't expect the public records clerk or the local school district to play along.