customer refund

I run a small business and have customer who i have been doing work for for several years. about 6 mo ago, he wanted me to do a job and gave me deposit to buy parts. the job couldn't be done the way it was originally bid, so after several service calls I told cust since he knew I didn't work with tile I could not finish the job unless he brought in a tile contractor. he did not want to do this. some time later I was contacted by his son-in-law who threatened to have me arrested, etc so i made a trip to the customer's home with an invoice and a check for $4oo. (I took out $250 for work that was done). the customer refused the check and threatened to punch me in the face (he is phy ill) so i told his wife i should go after much apologizing from her. What is my legal responsibility to this customer? he is threatning me with all sorts of legal action and arrest, but i should be paid for work that was done to their satisfaction at the time. thanks you