It looks like they have never been so many
I see that Robert Knox arrived in Ceylon in 1659, one year after Portuguese Joăo Ribeiro left the island (captured by the Dutch in 1658), where he has been since 1640. Ribeiro wrote his work (Historic Fatality of Ceylon) in 1680 and this was only published 1685. Although Knox wrote his work in 1681, Ribeiro’s experiences are earlier, especially in what touches local war episodes. However both descriptions of the Vedas don’t differ so much in the essential.
On the other hand, i am surprised in that Knox places the Vedas in the 'Land of Bintan' (Indonesia?), whereas Ribeiro places them in the Northern Ceylonese lands of Vanni, between Jaffna and Trincomalee, in the middle of two separating rivers, along 10 leagues of coast and 8 leagues inland, an area of very dense bushes; and they were so few that within these 10 leagues they wandered in those bushes and almost wouldn’t communicate ones with the others… despite a legend told by locals that would implicate in these people being in large number.
Ribeiro also stresses that they wouldn’t stay in permanent villages. Each family stayed in a place no more than six months, enough to plant seed and harvest the result; then they moved somewhere else.