How do she apply for a student visa/ social security number?

This may be problematic, and you & she should speak with an immigration lawyer.

First, she needs to renew her passport through the UK consulate. She will need a valid passport no matter what she does.

She should also speak with the officers who handle foreign student matters at the schools she is considering attending (the "DSOs"). They are often quite knowledgeable.

Some issues:

First, once out of status, it is normally impossibly to get back into valid status while in the U.S. - one needs to leave, get a new visa abroad and return. HOWEVER: once most people accrue 180 days of "unlawful presence" here, the act of leaving the U.S. subjects them to a three-year bar to coming back. Once here one year of unlawfule presence, leaving gets them a ten-year bar to coming back!

BUT: people don't start accruing unlawful presence until they are over 18. So, being 16, your cousin dodges that bullet.

Second, and a bigger problem in reality: Like the visitor visa, the F-1 student visa requires "nonimmigrant intent" - the intent to stay only temporarily for the purpose of the visa (the length of the educational program) and then return to their home abroad. Having already overstayed a visitor visa by three times its allotted length or more, and obviously having family here to stay with during that time, she will have a pretty tough time convincing the consulate of her nonimmigrant intent. There may be ways to convince them otherwise, but it will be a struggle.

A conversation with the DSOs will probably explain that financial aid is difficult to come by for non-U.s. students. She will not be able to get a social security number until she gets some form of authorization to work - very limited as a student and probably not available up front; again - speak with the DSOs about this.