I have read so many versions of this same question on Avvo that I posted a Legal Guide in the hopes of preventing others from doing this same thing. Here's the answer:
If you added him to title he is an owner of the property - but he only has half an interest and he cannot "take" the property from you. He is only entitled to the interest you gave him (if it wasn't spelled out in the deed, then his interest is presumed to be half).
One thing you did not say in your posting, is whether he, in fact, made the payments. If he did not, you might consider suing him to return the interest as there was no "consideration" given for the transfer of the interest. If he has made payments, that argument would be knocked down.
This answer does not constitute legal advice and does not and is not intended to create an attorney-client relationship. The law may vary depending on the state in which you reside. It is intended only to give some direction in which to seek assistance.
The deed determines ownership of the property. If he's on the deed, he owns part of the property, but can't take your part from you. If he's not on the deed, he owns nothing, and can't take the house from you.
The deed of trust is the document that allows the mortgage company to foreclose on the house if the mortage is not paid. Mortgage companies routinely include the co-borrower or spouse on the deed of trust, to make sure there's no problem with foreclosing on their interest in the property, if any.
I would want to know why he is on the deed of trust if he's not on the deed or the mortgage loan, and you aren't married.