Troubleshoot Express Deployments

Owned by Xaun Lopez (Unlicensed)
Last updated: May 26, 2017

Viewing the status of the Application:

nex-cli apps:info my-app

If the App Details section "STATUS" is "active", and the Source Control Management section "LAST COMMIT" is correct, if so, your deployment should be ok & running.


If the status is "restarting" and it has been some time, then something may have gone wrong and it's worth investigating the cubes.


Viewing the status of the cubes, and checking the logs

List the cubes for your app:

nex-cli cubes --app my-app --all


If one or more cubes have the "STATUS" => "running", then it appears the app is healthy and you should try trigger a redeployment (replay the github webhook).


Triggering a redeployment

nex-cli apps:restart my-app

The command above should trigger a redeployment and restart the cubes.

To view the events to see what is going on run:

nex-cli apps:events my-app

Note that the github webhook replay does not get logged in the event feed at the moment.


If one or more cubes have the "STATUS" => "performing:starting", most likely the cube has crashed. 

SSH'ing into the rack

Using the crashed cube's IP address, SSH into the rack (the IP is printed via the cubes command shown above):

nex-cli racks:ssh 10.0.1.234

Viewing the logs for the cube:

sudo su
docker logs CUBE-INSTANCE-ID

This should give you more information, typically there has been a 500 and identifying this log should give you your answer.


Logs don't look like your new code has been applied?

Whilst ssh'ed in the rack, list the docker containers to check the status:

sudo su
docker ps -a

If the status of the container with the same NAME as the crashed cube instance ID has exited hours or days ago, and you recently deployed, then the new code hasn't been applied properly. Trying "triggering a redeployment" via `nex-cli apps:restart my-app`.


Scaling down / up the application's cubes

As a form of "hard reset", we can scale down & then back up the application's cubes.

Scale applications down as many times as there are Cube 

nex-cli apps:down my-app

After the `nex-cli apps:info` "STATUS" => "stopped" for all Cube Instances, scale apps up as many times are there are Cube Instances:

nex-cli apps:up my-app

When all Cubes apps:info "STATUS" => "running", see if it's now working. If not, repeat these steps, especially viewing the crashed cube's logs as it may be different after it's restarted with the latest code.


Deleting the Nex! Locks & Restarting the Cubes

The lock files are used to control the state of actions currently undertaken on containers. If a containers crashes during provisioning the lock will stay there and the container will be forever considered as “starting” or “provisioning”.

SSH into the rack, and then delete the Nex! Locks:

rm -r /var/lock/nex/*