Gavin, it does indeed look like something once occupied this spot, but i'm not sure i could give any reliable answer to your question of exactly what it was. Very possibly a stone, but what type is a pretty open. I'm not sure i have ever seen a stone set into this type of hilt before so i have no precedent to draw from.
It does look like someone my have chipped the ivory a bit digging the object out if that chip is a sign of that then it happened a little ways back.