If it’s not losing position, you’re probably not pushing it too hard. This also depends on the path being cut. A slot is harder on the endmill than a pocket. As @HaldorLonningdal said, the speed setting on your router will matter too.
Can you give more details on the project? What speed are you using for the router? What type of bit are you using?