Mortgages in communities that are more than 50% rental properties?

I live in a small (20 unit) townhome community. The current HOA covenants state that only 20% can be rented out. Home values in the community have plummeted and homeowners want to increase that 20%. The HOA is saying that they have in writing that if we go more than 50% rentals, that the only way a unit could sell is to a cash buyer. They are claiming that potential buyers would not qualify for FHA, Fannie Mae, Freddie Mac or Conventional loans any more. So if an owner occupied unit wants to sell, they'd have to find a buyer that doesn't need a loan to buy. Does anyone know if there is any truth to that?