Is there a penalty for taking money out for non-qualified purposes?

Yes. The earnings portion of funds withdrawn for non-qualified purposes are typically subject to federal income tax, as well as an additional 10% federal penalty tax. Taxable withdrawals, on the other hand, are subject to applicable federal income tax on earnings, if any, but are not subject to the 10% additional federal penalty tax on earnings.