Yes you should worry. Or you will end up with a limit switch graveyard like this one.
The advise about the pencil is good, as well as keeping your hand on the off switch. Also, double check your connections before starting the machine each time you use it. Especially the little molex pin connectors for the limit switches, those can easily come loose.
