You are going to have some challenges. The first is that you didn't "accept" an invite, you normally have to do the invitation and he has to accept - depending on the program - it shows clearly on the your program that both have happened. So, it appears you published the video to him and did send him the "pictures".
Now, here is the mixed news: If your ex's mom is trying to black mail you, you can go to the police - especially if you have proof - and they may help you get it.
As to the positng of the pictures, depening on how much time you want to spend on it, you may want to hire an attorney to send a cease and desist letter to his wife (and possibly him), as well as the websites they were posted on - make sure the person knows about I.P. addresses to include that in the letter (basically stating that the original of the computer where the pictures were posted from can be tracked).
Final 2 things - make sure you know how to use your messenger program so this doesn't happen again and there are literally millions of pictures of topless women on the internet-the only people that are probably evening noticing are you (how did you find out?) and your ex-s wife. They are literally a drop in an ocean (not even a bucket).
Good luck with the situation and the above is general information and not specific legal advice as I haven't interviewed you to know all the facts.