Thanks to the both of you!
I was also trying to rule out every option before conceding to myself that it could be a cannonball.
Yeah, catapults didn't really need to smooth the surface like that, that's true.
Someone brought a really big cannon to siege the small fort.