First you need to know if the car is titled: "HIS AND HERS" or "HIS OR HERS" This will determine if you can only transfer title with ("AND"), or can transfer title without ("OR") his consent.
You do not have to refinance the car, and get his name off of the loan, but it is a good idea for him to insist on it, since you have possession of the car.
You are at risk that if the title is held HIS OR HERS, that he could possibly sell the car out from under you, without your consent, but he would have to pay the loan off at the time of the sale. If that happens, you will have lost the value of your trade in.
Consult with an attorney or the DMV if you are unsure about your title, and rights.
I am an attorney who is only licensed in the State of Florida. My answer is general legal advice based upon what I perceive your question to be, and should not be relied upon because every person's facts and circumstances are unique, and because specific laws vary from state to state. To completely evaluate a legal issue requires reviewing and evaluating all relevant facts, applicable laws and other information. My answer does not create an attorney-client relationship, and offered for informational purposes only.