If the title to the car is solely in your name then he cannot legally take it; he may be able to go to court - depending on the facts - but I don't think he can just take it away from you if the title is in your name alone. If his name is on the title too, that would be a different story as you would both have a legal right to to possession.
I am a former federal and State prosecutor and now handle criminal defense and personal injury/civil rights cases. Feel free to check out my web site and contact me at (212) 385-8015 or via email at Eric@RothsteinLawNY.com. The above answer is for informational purposes only and not meant as legal advice.