Hearsay is in the eye of the beholder. That is the presiding judge determines if the hearsay objection will be sustained or overruled. Another approach is to attack the "conversation" via a Motion in Limine. The ruling on the motion will provide your attorney with a preview of the judge's view in this regard. If this is a hypothetical question keep researching and appreciate the vagaries of evidence law application. If this is a real case scenario your experienced local federal criminal defense lawyer is in the best position to advocate for the accused.
This answer/response is based on the information provided in the question asked and requires a much more complete context than is available in this public forum. Please do NOT use this answer/response to say or do anything regarding your situation. BEFORE you say or do anything consult with an experienced Federal and/or state criminal defense attorney in your jurisdiction who will listen to you and your concerns.
There is really no way to give you a definite answer without reviewing all of the evidence. There may be some other evidence to link you to the conspiracy that has not been mentioned here. It is best to hire a lawyer that practices in the area of Federal law if you have been charged with conspiracy in federal court. Usually a conversation alone is not the basis for a federal "conspiracy" indictment. There may be statements given by others in the conspiracy, surveillance or drug seizures. All a lawyer on this site would be able to do is speculate (without having access to the prosecutor's file or more details), which seems like what you have been getting so far.
The information provided is not advice but a legal perspective and you should schedule a consultation with the lawyer of your choice.
If the government can prove to the court (not the jury) by a preponderance of the evidence that you were a part of the conspiracy, the conversation is admissible because it is excluded from the hearsay rule. Pursuant to Federal Rule of Evidence 801(d)(2)(E) states that a co-conspirator's statement is not hearsay if the statement is made during and in furtherance of the conspiracy. Unfortunately, district courts liberally apply this hearsay exclusion. Regardless, you seem to have a strong factual defense based on the scenario that you have described. - Joshua Sabert Lowther, Esq., National Federal Defense Group.
Whether or not a given statement is hearsay depends on the purpose for which it is offered into evidence. It might be hearsay for one purpose but not for another. Hearsay is not necessarily inadmissible. There are many exceptions to the hearsay rule. Furthermore, statements by co-conspirators may under some circumstances be offered into evidence against all members of the conspiracy, provided that the existence of the conspiracy is established by independent evidence. The point is that nobody can give you an answer to this question in the context of a simple Q&A forum like this. The issue is much too complicated and there are too many variables. All evidentiary questions are intensely case-specific, and this question is even more so than most. The attorneys are going to have to thrash this one out as they go along. The trial may be halfway finished before the judge gives a final ruling on which conversations will be admitted, which will not, and what restrictions will apply.