This is joint work with Primož Potočnik. In this paper we offer a classification of vertex-transitive graphs whose motion (smallest number of vertices moved by a non-trivial automorphism) is $4$ or a prime number $p$.
You can see the paper’s details here and the paper itself here.