Generally the earlier in time "intent to use" registration trumps subsequent actual use, but usually intent to use applications don't last for "a few years," because trademarks aren't meant to be locked up for that long without actually being put into use.
Have you filed extension requests every 6 months and reiterated your bona fide intentions for use? Did you get a Notice of Allowance from the USPTO more than 3 years ago? If so, it's too late to file your statement of use for this TM, and I suspect that you no longer have a pending TM application and that your competitor's claim is right, and there's nothing you can do to stop them.
TM applications are much more complex than they seem - hire a TM lawyer for help if you don't know where you are with this process.
Disclaimer: Please note that this answer does not constitute legal advice, and should not be relied on, since each state has different laws, each situation is fact specific, and it is impossible to evaluate a legal problem without a comprehensive consultation and review of all the facts and documents at issue. This answer does not create an attorney-client relationship.
Bear in mind that having a trademark registration is different from holding a trademark. Your "intent to use" application does not yet amount to a trademark registration. However, you may or may not have a trademark and, possibly, superior rights over another user of the the same or a substantially similar mark, regardless of whether or not you have completed the trademark registration process. Are you actually using the mark in commerce either within your state, or in interstate commerce?
If so, you may have earned common law trademark rights that enable you to enjoin another's later commencement of use of a confusingly similar mark in your channel of commerce. My colleague is correct that without registration, you do not have some of the arsenal of enforcement rights you would have against a later user if your registration process were complete.
You should also examine your trademark registration application. Have you renewed the intent to use application on a regular basis so that it is still active and you have not lost your initial filing date? Are you ready now -- actually using the mark in commerce -- so that you can file a use affidavit and complete the process. Don't forget, you are only allowed a limited number of renewals of an intent to use application.
Finally, you must always be careful before acusing another of trademark infringement that you are indeed the superior user with trademark rights. There is a risk that the user you've recently noticed actually had accumulated rights before you started your actual use of the mark. People who have assumed they held a valid trademark have found themselves on the defense side of an infringement action after accusing what turned out to be a prior user of trademark infringement. For this reason, a thorough search by a reliable search firm is also an advisable first step before commencing a trademark application or asserting a trademark right.
Q: ...they wrote back to say that we had no rights until registration. Is this true?
Q: It seems unfair that they can use a mark that is so similar!
R: "Life's not fair. Get Over it." Bill Gates.
Q: What happens when we get the registration since they will have been using the mark before we started to.
R: Your trademark rights will "relate back" to the date of your intent-to-use application. Which means that your trademark rights will be "senior" to theirs. And what flows from that fact is that you can sue to enjoin their continued use of the mark -- but ONLY if you and the other company are doing business in the same geographic area and the two marks are confusingly similar and the marks are used to brand related goods or services. Assuming all those facts, and that you can prove them, you would still not be entitled to any monetary damages because you have not suffered any.
Q: Is there anything we can do to stop them?
Hire a trademark attorney. For all the reasons my colleagues have mentioned, I seriously question whether you're even on the track that you think you're on. Good luck.