I think some more facts would be helpful. Was this an interest-free loan? In that case the IRS might be interested because it is losing the tax on the interest paid to you. Are you writing it off as a loss on an investment, and looking for a tax deduction? Other than those situations (and there may be more), I see no reason for the IRS to be interested. Furthermore, I think the burden of proof, at least initially, is on the IRS to produce evidence that is not an arm's length transaction. If the amount of money is significant, I would suggest you consult a tax attorney.
Under IRC Section 408(e)(2) your IRA has lost its tax exempt status because it has engaged in a prohibited transaction under Section 4975. You need some very good advice to minimize the damage.