Store security are typically not in uniform. See http://data.opi.mt.gov/bills/mca/46/6/46-6-506.htm for the statute on detention of shoplifters. I think it is risky for the store for store security not to have some identification that clearly identifies them and establishes their authority, but in this case the store may be willing to take that risk. Based on your description the man may have been poorly trained.
DISCLAIMER: The forgoing comment is for general educational purposes only, and is not legal advice upon which the reader may rely as the commenter has no actual knowledge of the facts of the case, has not interviewed persons or examined evidence, and has not researched the applicable law. The comment is based only on the facts provided, which are extremely limited, and may or may not be true. Complete defenses may prevent the success of any claim. Competent legal advice should always be obtained before taking any legal action or filing suit. Readers employ any information provided herein at their own risk.