A student asked me this question in class yesterday, and I wasn't sure (haven't looked at the Principia in a long time, but always found the arguments a bit hard to follow). I imagine he could do it from a rough estimate of the mass of the Earth, mass of the Moon, and distance to the Moon. With the Moon's period you could get a value for G. Is this how he did it? I know that the direct measurement wasn't done until later, by Cavendish.
Further, did he have any way of estimating the distance to the Sun? I couldn't think of one that was available at his time, but he was more clever than I. :)