OK, I admit that this sounds unlikely, but I have two coins from quite different periods with this theme.

On the left, a republican denarius of Mn Fonteius C F, Roman Republic, from 85 BCE. The winged infant Genius (or possibly Cupid, depending where you look it up) is riding a goat. Above him are the caps of the Dioscuri (Romulus and Remus) and below is the thyrsus of Bacchus. All this is surrounded by a laurel wreath. The goat looks polite and well-groomed. This coin imitates (or possibly just resembles) a statue in the temple of Veiovis in Rome.

Next to it, an antoninanus of Valentinian II from 257-258 CE. Valerian II was the son of Gallienus and the grandson of Valerian I. The image is of the infant Jupiter, astride the goat Amalthea, who cared for him and fed him. He is seated confronting us, his head turned to our left; his arms are spread wide. The legend is IOVI CRESCENTI, "Jupiter is growing." Jupiter here represents the young Valerian, and the inference is that he, like the young Jupiter, will grow to become all-powerful. In fact he died young; possibly killed by Ingenuus, a rebellious general who was supposed to have been educating him.

Whether tidy and well-groomed, or wild and shaggy, on both coins this is the same goat whose hide was made into the aegis of Jupiter (Zeus) and Minerva (Athena). See also my pages on the aegis and the story of Medusa.