Nex!™ Routing - Site can't be reached
This article provides resolution instructions in case the routing layer of Nex!™ is not responding anymore and your browser displays that the "site can't be reached" (connection timeout).
1 - Context
If you are getting a connection timeout error when attempting to access an app on Nex!™ it is likely that something wrong is going on with all the routing racks. When this issue occurs you are likely to observe the following error in Chrome:
2 - Checking Nginx on the routing racks
The following steps assume that you have setup the Nex!™ command line client and that you have administrative access to the platform.
You can display the list of routing racks and access one of them using the following commands:
# Display all routing racks nex-cli racks --type routing # Access rack by internal IP nex-cli racks:ssh 10.1.1.1
The first thing to do is to check that Nginx is up. If it's not attempt to restart it.
# Check if any nginx process is running ps -ef | grep nginx # Perform a full restart of nginx service nginx restart # Check if nginx is up ps -ef | grep nginx
If Nginx has not come back up after restarting then you will need to start it in the foreground to understand the issue:
# Run the following command # This will start nginx in foreground mode. Any startup issue will appear here. /usr/local/openresty/nginx/sbin/nginx
Nginx not starting is usually related to default SSL certificates not being setup properly. If the above command yields an error related to SSL, Private Key, Public Key then go to Section 3.
3 - Fixing the default SSL certificates
If the default Nginx certificates are not properly setup - e.g. missing public or private key, keys not matching - Nginx will simply refuse to start. This problem typically indicates that the Nex!™ Orchestrator configuration is incorrect.
3.1 - Immediate resolution
The fastest way to resolve this issue is to manually re-setup the Nginx default SSL certificates. For that you will need access to:
- Your wildcard certificate public key: e.g. mydomain.crt
- Your wildcard certificate private key: e.g. mydomain.key
- You Certificate Authority (CA) chain: e.g. my_cert_provider_bundle.crt
On the routing rack copy the content of your certificate private key into the following file (e.g. using vim):
/etc/nginx/default-certificates/default.key
On your local computer concatenate your certificate public key and CA chain:
# Cat both files to obtain a chained certificate cat mydomain.crt my_cert_provider_bundle.crt
On the routing rack copy the content of the above into the following file (e.g. using vim):
/etc/nginx/default-certificates/default.crt
Once done start Nginx in the foreground to check that everything is fine:
# This will start nginx in foreground mode. Any startup issue will appear here. # Use Ctrl-C to exit the process /usr/local/openresty/nginx/sbin/nginx
Finally restart the Nginx service to bring it back up:
service nginx restart
After bringing Nginx back up and verifying that your web applications were accessible again proceed to section 3.2 to permanently resolve the issue.
3.2 - Permanent resolution
As we mentioned at the beginning of the section this kind of SSL issue is symptomatic of a configuration issue with the orchestrator itself. You should therefore review the SSL certificates configuration variables in Nex!™ to ensure they match your certificate keys.
a) Using Ansible
If you are using Maestrano's Ansible framework to deploy the Nex!™ orchestrator then you should review your ssl configuration in the *_secret.yml file. See this example of configuration file to understand what to look for:
https://github.com/maestrano/mno-deploy-nexmin/blob/master/ansible/vars/nexmin_prd_secrets.yml#L111
The following variables must be set properly:
- cert_key_cube_default: your wildcard certificate private key with newline characters ("\n"). The content of the variable must be enclosed with SINGLE quotes
- cert_chained_cube_default: the concatenation of your certificate and your Certificate Authority bundle. The variable must be a single line with newline characters ("\n") and must be enclosed with SINGLE quotes
Once your mno-deploy-* configuration package has been rebuilt (e.g. using Codeship) redeploy the orchestrator itself to update its configuration. You can do this through your favorite deployment tool (e.g. Rundeck) or by running the following commands on the Nex!™ orchestrator boxes directly:
# E.g on AWS bash <(curl -s http://169.254.169.254/latest/user-data) # E.g. on Azure bash /opt/maestrano/redeploy.sh
b) Using Rails configuration
If you have deployed the Nex!™ orchestrator manually or using any other deployment framework (e.g. Chef) then you will need to modify your deployment variables to ensure that the Nex!™ config/application.yml file is setup properly.
On one of the orchestrator boxes navigate to the Nex!™ configuration folder:
# Go to the Rails config folder cd /apps/nex/current/config # Edit the application.yml file vi application.yml
Ensure that the following configuration parameters are set properly:
ssl_cert_key_cube_default: your wildcard certificate private key with newline characters ("\n"). The content of the variable must be enclosed with DOUBLE quotes.
ssl_cert_chained_cube_default: the concatenation of your certificate and your Certificate Authority bundle. The variable must be a single line with newline characters ("\n") and must be enclosed with DOUBLE quotes
c) Final steps
Finally you need to instruct the Nex!™ orchestrator to reconfigure the routing racks. You can do so through the Nex!™ orchestrator console:
# Access one of the Nex!™ orchestrator boxes via SSH # Access the rails console cd /apps/nex/current bundle exec rails c <uat|production> # Reconfigure all routing racks RoutingRack.where(status:'running).each(&:sync_base_config)