If you are comfortable with using a PC, that is a good start, and having manual machine experience already gives you a very good idea of what to expect, and even better, how cutting occurs, and what is reasonable for good cutting conditions.
You only need to be a mathematician to calculate arc center coordinates for circular cutting, and to deduce taper angles I am reasonably comfortable with math, including trigonometry, but I cannot stand doing toolpath calculations. So I got cadcam very shortly after I bought my first cnc (which was a lathe) and never looked back. There are plenty of inexpensive cadcam programs around to play with until such time as you decide to go serious and invest in professional cadcam packages.
Go for it. Learning and running CNC is a great hobby, if nothing else, and one that has potential for payback.
Keep in mind that the purchase of the major piece is only the beginning. There are many accessories that run the price of the hobby up quite fast.