You could read 17 USC 512 and try to figure this out or you could do the smart thing and see an IP attorney or Internet attorney that handles DMCA matters. You will need to be prepared to file DMCA counternotices. That doesn't "get the complaint withdrawn", it just trumps it legally. That forces the alleged copyright owner to sue you and get a court order to take your material down. Most museums will not do that as they don't want to risk a determination that you and others like you are legal.
Bottom line: Time to lawyer up.
I am not your lawyer and you are not my client. Free advice here is without recourse and any reliance thereupon is at your sole risk. This is done without compensation as a free public service. I am licensed in IL, MO, TX and I am a Reg. Pat. Atty. so advice in any other jurisdiction is strictly general advice and should be confirmed with an attorney licensed in that jurisdiction.
Your option is to sue the person who made the DMCA complaint and establish that your view of the case law is correct. But I think it is clear that your view of the case law is not correct. Photographs of public domain works are protected by copyright if the photographs meet the standard of originality in the copyright law. It takes very little to make something original and satisfy the standard. Thus, selection of outstanding lighting alone can be sufficient to render a photograph original. The angle of the photograph can be sufficient to create some original. The way the underlying work is framed in the photograph can satisfy the originality requirement. The clarity and color tone of the photograph can establish the originality requirement. A well-crafted photograph that embodies a public domain work and faithfully translates the elements of that underlying work in a clear and compelling manner can be sufficiently original to justify copyright protection. Museums often hire very accomplished professional photographers to create these photographs and I can assure you that the vast majority of courts will recognize them as sufficiently original to qualify for copyright protection. You should not be using such photographs without a license, nor should you use them to create your fine art prints without a license. Fine art prints that copy public domain works, like photographs of the works, can be sufficiently original to justify copyright protection. If your business is built upon using other people's photographs to create your prints, then frankly, your business should be shut down. Make our own original prints of the public domain works from your own photographs---then you contribute something to society. Copying the photographs of others is not cool---and that seems to be your modus operandi.
I am not your lawyer and this is not intended to be legal advice on which you rely. My answer is merely intended to assist you in understanding some of the issues that you face so that you can make an intelligent choice when you hire legal counsel.
There are two ways you could go - you could either get a license (a time consuming and expensive endeavor) or fight the DMCA take down notice. Have any online retailers in fact received DMCA takedown notices for your prints? Are your prints "produced solely for documentary purposes"? It sounds more like for a commercial purpose but it may be both. You should check the DMCA take down notice policy of any site you plan to use. Feel free to contact me.
THIS SITE IS ATTORNEY ADVERTISING AND PRIOR RESULTS DO NOT GUARANTEE A SIMILIAR OUTCOME. My Avvo site contains general information only, is not legal advice and does not create an attorney-client relationship.