Without being willing to go through probate, you are not likely to get anything from your brother based on your facts. Due to your concerns about your mother's health, there is no action you can take that would leave her out of it. Since it sounds like a long term marriage, she is likely the sole heir of your father's unless there is a will. Your concern should be what happens with your mother's estate and your brother's influence. I would take very detailed notes of what your brother is doing and prepare yourself for a potential issue if a will of hers shows up favoring your brother after she is gone. Good luck.
Your mother, or you, can file a Petition for Probate seeking Letters of Administration to administer your father's estate. If there is a Will, this will force your brother to produce it. If he doesn't, you or your mother can be appointed to administer the assets, and the probate code will dictate who inherits your father's property in the probate. I would get the help of a probate attorney right away.
What is your moms competency? Does she need a conservator. You can't get from a to b without court given your brothers conduct. You need to consult a probate attorney with elder law/conservatorship experience.
The general advice above does not constitute an attorney-client relationship: you haven't hired me or my firm or given me confidential information by posting on this public forum, and my answer on this public forum does not constitute attorney-client advice. IRS Circular 230 Disclosure: In order to comply with requirements imposed by the Internal Revenue Service, we inform you that any U.S. tax advice contained in this communication (including any attachments) is not intended to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing, or recommending to another party any transaction or matter addressed herein. While I am licensed to practice in New York and California, I do not actively practice in New York. Regardless, nothing said should be deemed an opinion of law of any state. All readers need to do their own research or pay an attorney for a legal opinion if one is necessary or desired.
I'm certain that a qualified probate attorney could help you convince your brother to share the will (or admit that it doesn't exist). An attorney would probably need to ask you many questions before recommending the best course of action, however.
You can find good Contra Costa probate attorneys here on avvo.com, and you can also look for attorney certified specialists in probate law by going to http://members.calbar.ca.gov/fal/MemberSearch/QuickSearch#legallink . Most probate attorneys in your area will give free consultations, as will my firm.
Please call me for a free consultation at 925-362-4230. The information you obtain at this site is not, nor is it intended to be, legal advice. You should consult an attorney for individual advice regarding your own situation.