I've noted this illusion many times. I also wear bifocals, and
I've always considered it possible that it is due to chromatic
aberration, in which case the virtual images of predominately
shorter wavelength objects and predominately longer wavelength
objects appear to form at different depths. I have not done
calculations, but I once figured out that the sign is correct.
I believe the effect also exists for naked eyes, though perhaps
it is weaker.