It not illegal but you will likely be denied entry if they check. Here is the policy:

Denial of Visitor Access:
a. All collected information is considered “law enforcement” sensitive and is not disclosed to non law enforcement personnel. Some of the more common reasons individuals are denied installation access are:
1) Currently on parole and/or probation
2) Registered sex offender
3) History of gang affiliation or extensive criminal history
4) Identified on the Terrorist Watch list
5) Currently have an active warrant
6) Currently barred from any U.S. military installation
7) Any felony conviction for murder, assault, sex offenses, etc.

It’s not like they are doing background checks at the gates, but you would always be asked for identification and if your driver’s license denotes that you’re a sex offender, good luck getting in.