...

...

1 - Context

If you are getting a connection timeout error when attempting to access an app on Nex!™ it is likely that something wrong is going on with all the routing racks. When this issue occurs you are likely to observe the following error in Chrome:

Image RemovedImage Added

2 - Checking Nginx on the routing racks

The following steps assume that you have installed setup the Nex!™ CLIcommand line client and that you have administrative access to the platform.

You can display the list of routing racks and access one of them using the following commands:

# Display all routing racks
nex-cli racks --type routing


# Access rack by internal IP
nex-cli racks:ssh 10.1.1.1

...

# Check if any nginx process is running
ps -ef | grep nginx


# Perform a full restart of nginx
service nginx restart


# Check if nginx is up
ps -ef | grep nginx

...

Nginx not starting is usually related to default SSL certificates not being setup properly. If the above command yields an error related to SSL, Private Key, Public Key then go to section Section 3.

Anchor
fixingsslcertificate fixingsslcertificate

3 - Fixing the default SSL certificates

If the default Nginx certificates are not properly setup - e.g. missing public or private key, keys not matching - Nginx will simply refuse to start. This problem typically indicates that the Nex!™ Orchestrator configuration is incorrect.

3.1 - Immediate resolution

The fastest way to resolve this issue is to manually re-setup the Nginx default SSL certificates. For that you will need access to:

...

After bringing Nginx back up and verifying that your web applications were accessible again proceed to section 3.2 to permanently resolve the issue.

3.2 - Permanent resolution

As we mentioned at the beginning of the section this kind of SSL issue is symptomatic of a configuration issue with the orchestrator itself. You should therefore review the SSL certificates configuration variables in Nex!™ to ensure they match your certificate keys.

a) Using Ansible

If you are using Maestrano's Ansible framework to deploy the Nex!™ orchestrator then you should review your ssl configuration in the *_secret.yml file. See this example of configuration file to understand what to look for:

https://github.com/maestrano/mno-deploy-nexmin/blob/master/ansible/vars/nexmin_prd_secrets.yml#L111

The following variables must be set properly:

• cert_key_cube_default: your wildcard certificate private key with newline characters ("\n"). The content of the variable must be enclosed with SINGLE quotes
• cert_chained_cube_default: the concatenation of your certificate and your Certificate Authority bundle. The variable must be a single line with newline characters ("\n") and must be enclosed with SINGLE quotes

Once your mno-deploy-* configuration package has been rebuilt (e.g. using Codeship) redeploy the orchestrator itself to update its configuration. You can do this through your favorite deployment tool (e.g. Rundeck) or by running the following commands on the Nex!™ orchestrator boxes directly:

# E.g on AWS
bash <(curl -s http://169.254.169.254/latest/user-data)

# E.g. on Azure
bash /opt/maestrano/redeploy.sh

b) Using Rails configuration 

If you have deployed the Nex!™ orchestrator manually or using any other deployment framework (e.g. Chef) then you will need to modify your deployment variables to ensure that the Nex!™ config/application.yml file is setup properly.

On one of the orchestrator boxes navigate to the Nex!™ configuration folder:

# Go to the Rails config folder
cd /apps/nex/current/config

# Edit the application.yml file
vi application.yml

Ensure that the following configuration parameters are set properly:

ssl_cert_key_cube_default: your wildcard certificate private key with newline characters ("\n"). The content of the variable must be enclosed with DOUBLE quotes.
ssl_cert_chained_cube_default: the concatenation of your certificate and your Certificate Authority bundle. The variable must be a single line with newline characters ("\n") and must be enclosed with DOUBLE quotes

c) Final steps

Finally you need to instruct the Nex!™ orchestrator to reconfigure the routing racks. You can do so through the Nex!™ orchestrator console:

# Access one of the Nex!™ orchestrator boxes via SSH

# Access the rails console
cd /apps/nex/current
bundle exec rails c <uat|production>

# Reconfigure all routing racks
RoutingRack.where(status:'running).each(&:sync_base_config)