Router moving

Is it normal for the router to move side to side while carving. I tightened the rollers so it still goes free up and down. But during carving it moves a little side to side. Do I tighten them more?

I tighten my wheels with the eccentric washer/nuts just so I can move a wheel slightly with a finger.