The feds don't waste their time on users. Your dealer can say whatever he wants, but if they do not have additional evidence, then they will not prosecute you. Usually by the time the feds arrest a person, they have pretty much completed their investigation of that conspiracy, and just get a few cooperators to help tell the story at trial & to tie up loose ends (or to go after bigger fish - not smaller ones.)
If what you say is correct, you have little worry. However, if the feds do try to talk to you, I strongly suggest that you hire a lawyer immediately. It is a crime to lie to a federal agent, and they can be very intimidating & make you forget things.
This answer is for educational purposes & not legal advice. The advice I would give is to hire an attorney and fully discuss the situation with the attorney.
Up front, draw a distinction between what you have in fact actually done or participated in versus what your dealer (and possibly others) have said you did or participated in. In the federal jurisdictions within which I practice, if the law enforcement officers who are pushing your prosecution believe the dealer and the latter is saying you were an active participant in his distribution process, then you are at risk for indictment and arrest. Merely being arrested or indicted is not based upon the same quantum of evidence as that required for conviction, proof beyond a reasonable doubt. Merely being a "user" if you have never carried, transferred or been present for a drug transaction should not bring you to the attention of federal authorities--but your dealer will only be getting "credit" for his cooperation when he names people who are in the distribution business--so he has an incentive to name you as one of those persons, whether or not you were actually involved.
People are frequently convicted in federal court on the mere statements of others that they bought or sold drugs from them, without the actual drugs that they are discussing ever being seized or observed by law enforcement officers. These "historical" prosecutions rely on the government having multiple cooperating defendants/informants who name the accused party as an active participant in the drug trade. Therefore, if you are approached by law enforcement officers who wish to discuss your prior drug habit or dealings with your dealer, you should and have the right to request to speak with a lawyer first and to have that lawyer present during any questioning or statements you may wish to make.