Do I have to pay back my employer for paying my tax without withholding?

my former boss failed to withhold my tax, and he was paying my tax out of he's own pay check and didn't inform me all this time for the whole year.I was paid the whole wages without deductions, and all this time I thought he's paying for me, didn't have much information about this tax thing, since this was my first job(nanny), so now he wants me to pay him back, since he was paying my tax and didn't even discuss about the tax issue with me, and he tells me that he wasn't aware and didn't know that he has to with-hold tax from me.So all I want to know is do I really have to pay him, or is there other way this matter can be resolved?