If the man asked you to do a job for X, and you agreed by performing the job, that is a contract. It's an oral contract - rather than written. If you performed per the agreed upon terms, now he must also. Just because someone else quoted something different, does not change his obligations under the oral contract. If you need help enforcing this, don't hesitate to email.
Whether the customer must pay your invoice amount likely depends on whether you and the customer agreed on a set price before you did the job. If you didn't agree on a price, the customer would need to pay the fair value for the work based on a quasi contract theory called unjust enrichment. The customer received a benefit, kept the benefit and should have to pay the value of the benefit.
This is not intended as legal advice. It is only provided for educational purposes and cannot be relied upon as legal advice. Further, no attorney client relationship is or has been formed by answering this question.