Not sure how “your lawyer” can refuse your case. An attorney does not have to take on a case, just like a doctor cannot be forced to treat a patient. If you signed a retainer with the attorney then that is another story and there could be legal reasons why he/she no longer wants to handle the matter.
DISCLAIMER: David J. McCormick is licensed to practice law in the State of Wisconsin and this answer is being provided for informational purposes only because the laws of your jurisdiction may differ. This answer based on general legal principles and is not intended for the purpose of providing specific legal advice or opinions. Under no circumstances does this answer constitute the establishment of an attorney-client relationship.
There is no ethical violation by refusing to take on a case. He may feel the case is without merit in that there is a lack of liability or an inability to prove damages. Certainly he can refuse. Go shop around if you feel you have a case.