More than four in 10 U.S. adults would not be allowed to enter their own country except for the fact that they are American.