Update: I had to add a second connector-va to be used for external authentication.
The original Connector-va had Windows Authentication enabled with redirect enabled, so external users were hitting the gateway-va and being sent to the connector-va for authentication. As this wasn't available externally it timed out.
The second Connector-va is joined to the domain but Windows Auth is not enabled.
There is also valid internal IP ranges added to the original connector-va so that it will only attempt to authenticate from systems in those ranges. Everything else (external) is sent to the second connector-va.
/opt/vmware/c2/c2instance/logs/connector.log is your friend.
Thanks to VMware support for troubleshooting this one.