The answer your question depends upon the entity that owned the property and whether there was a will in place when your father died. I would contact a New York estate attorney.
You should retain an attorney to address this issue. The above answer is for informational purposes only and does not constitute legal advice. I have provided this answer only based on the information presented in the question. The specific actions or answers to this issue require additional information. The above information and answering this question does not constitute an attorney-client privilege. I am not responsible for any issues, problems or losses from any reliance on this answer and information, and the consequences of such reliance.
You will wish to review the underlying deed to the property. If the deed was titled as "joint tenants with right of survivorship" between your father, grandfather and other family members, then your grandfather is telling you the truth. That is, when a property is titled in this fashion, the property simply passes to the survivor(s) on the deed when an individual passes away.
If, however, the property was titled as "tenants in common", then there should have been a probate administration in New York in order to transfer your father's interest. In that case, you may have had the right to inherit the property depending on the terms of your father's will.
Finally, it is possible that the property was owned by a family entity such as an Family Limited Partnership or LLC. In that case, you will need to review the underlying documentation to determine what happened to your father's interest upon his death.
This information is presented as a public service. It should not be construed to be formal legal advice nor considered to be the formation of a lawyer/client relationship.