You cannot simply put your brother out, even though there is no rental agreement.
If you are afraid of him, if he is threatening you, or harming you, or harassing you, you can file for a Domestic Violence Restraining Order, and have him kicked out.
If you are not afraid of him or harassed by him, you will have to file an Unlawful Detainer action and have him evicted.
Please note that this answer does not constitute legal advice, and should not be relied on, as each situation is fact specific, and it is not possible to evaluate a legal problem without a comprehensive consultation and review of all the facts and court pleadings filed in the case. This answer does not create an attorney-client relationshipAsk a similar question