More facts are needed. What is meant by "modified duties" and why were your duties "modified." You are not protected under Family Med. Leave Act or Calif. Family Rights Act because your employer does not have enough employees. However, it is unlawful to discriminate against an employee because she is pregnant, took maternity leave, etc. If you feel that you are being retaliated against because of those reasons and not work performance, restructuring, etc., call an employment law attorney to discuss.
Are you living on site? This sounds highly suspicious. Have you made any complaints recently? I agree with the other attorney response, that if there is anything retaliatory in their decision making you may have a case.
You can always talk with an EEOC local office and make a complaint to be investigated.
This is my opinion and should not be construed as legal advise for your specific case as there are many more facts which you have not provided.