The prosecution must have sufficient, aissible evidence. Typically, that's done with testimony if the victim in a case like this. Because you have a right to confrontation (cross-examination), without the alleged victim, they may not have a case. Your attorney is in the best position to assess this.
The above answer is for general information only and is based on the information you posted. Every case is fact dependent, so to get a thorough analysis of your situation, you will need to consult face to face with an attorney licensed to practice in the jurisdiction where the incident took place. Do not conclusively rely on any information posted online when deciding what to do about your case.
Many victims of assaultive crimes are afraid to testify and some even fail to show up when subpoenaed. If the DA has a strong case based on other evidence he can proceed without the missing victim but it really weakens his position. If the DA cannot go forward without the victim he can have the victim arrested and jailed until they do testify (no longer than the length of the trial). This is called a body attachment where I practice. If they can't find the victim at all and the other evidence is weak many DAs will dismiss.
You need to tell your attorney about these statements being made by the accuser. An investigator needs to jump into action and develop impeachment evidence when the accuser shows up to testify.
If the accuser is telling everybody, perhaps he/she is willing to make statement to your attorney's investigator as well.
With some evidence on your side perhaps your attorney can get charges dismissed or at least be in better position to win at trial.
One pitfall is where a victim makes statements under trustworthy circumstances where the statements are not sought for prosecution purposes(like to a nurse) or as an "excited utterance". Courts can look past the hearsay challenge and admit those statements even if the victim isn't in court. Consult a lawyer on this.