One can go very far conceptually with an action-at-a-distance theory in
which E and B are merely measures of forces on a hypothetical test charge.
This is logically how fields first enter our intro E/M study. Indeed the
naive re-ification of field lines often leads to serious calculational error
in many engineering problems.
OTOH, one conceptual difficulty with the action-at-a-distance model is
contending with a "here and now" force on a neuron in my retina due to an
accelerating charged particle which once existed on a distant star but has
long since been "annihilated". Newton's force concept suffers painfully
here!
Whatever course one takes, beyond a certain level of analysis any model of
reality will resist description in human terms
On the way to QED, Feynman and Wheeler tried to construct a consistent and
complete action-at-a-distance EM theory and finally abandoned the idea.