Investigating a Frontend downtime
- Francois Gourichon (Unlicensed)
Owned by Francois Gourichon (Unlicensed)
Problem
A frontend (myfrontend.maestrano.io in our example) is reported as down by the alerting system, or a user reports it
Step by step resolution and investigation
The assumption is that the Frontend infrastructure is deployed on Nex!
As a reminder, Nex! environment needs to be properly setup in order to run command line tools:
export NEX_API_KEY=your Api Key
export NEX_ENV=production
Resolution
The first step is to get the Frontend online in order to minimize user impact.
nex-cli will be used to determine app name and retrieving its status
nex-cli domains | grep myfrontend.maestrano.io
will output something similar to this:
| 5569ff29-0c86-4228-ba20-5e9a8a853c30 | myfrontend.maestrano.io | true | persevering-dolphin-5 |
so now we want to check its status
export app=persevering-dolphin-5
nex-cli apps:info $app
+----------------+--------+--------------------+------+---------+------------------+------+-------+------------+-----------------+------------------+
| App Details |
+----------------+--------+--------------------+------+---------+------------------+------+-------+------------+-----------------+------------------+
| NAME | STATUS | IMAGE | SSL | STORAGE | PREFERRED REGION | SIZE | NODES | OWNER | DESCRIPTION | TAGS |
+----------------+--------+--------------------+------+---------+------------------+------+-------+------------+-----------------+------------------+
| persevering-dolphin-5 | active | maestrano/web-ruby | true | false | ap-southeast-1 | 4 | 0/12 | u:tartempion | My cool frontend | development,ruby |
+----------------+--------+--------------------+------+---------+------------------+------+-------+------------+-----------------+------------------+
...
+--------------------------------------+------------------+------------+----------------+---------+----+------+---------+
| Cube Instances |
+--------------------------------------+------------------+------------+----------------+---------+----+------+---------+
| ID | STATUS | CONTAINER | REGION | STORAGE | IP | PORT | CLUSTER |
+--------------------------------------+------------------+------------+----------------+---------+----+------+---------+
| d2a2d9d9-ef8b-41ea-9e45-3cc911254a80 | stopped | terminated | ap-southeast-1 | false | - | - | - |
| 9b3d4de3-405d-4624-bced-80145bac28da | stopped | terminated | ap-southeast-1 | false | - | - | - |
+--------------------------------------+------------------+------------+----------------+---------+----+------+---------+
So we see here that no node is running (0/12 nodes runnings, and no cube started)
First thing we want to do is to restart the service , unless you can allow to keep it down longer, to avoid erasing clues on why it went down.
nex-cli apps:up $app
Now that we have a least one node up (we will bring a second one later on), let's try to understand what happened.
Root cause analysis
Now that the service is back we can have a look at the logs to see what went wrong
nex-cli apps:logs $app
Node: 5f621ba9-e282-4396-8dba-8e0f6abd761f
--------------------------------------------------
2017-01-20T00:45:25.623127175Z 00:45:25 web.1 | * Listening on tcp://0.0.0.0:3000
2017-01-20T00:45:25.623150529Z 00:45:25 web.1 | Use Ctrl-C to stop
2017-01-20T00:45:36.100704636Z 00:45:36 web.1 | Started GET "/" for 127.0.0.1 at 2017-01-20 00:45:36 +0000
2017-01-20T00:45:36.106560955Z 00:45:36 web.1 | Processing by StaticPagesController#home as */*
2017-01-20T00:45:36.121811713Z 00:45:36 web.1 | Rendered static_pages/home.html.erb within layouts/application (1.2ms)
2017-01-20T00:45:36.134399184Z 00:45:36 web.1 | Rendered layouts/_shim.html.erb (0.3ms)
2017-01-20T00:45:36.139367950Z 00:45:36 web.1 | Rendered layouts/_header.html.erb (1.0ms)
2017-01-20T00:45:36.143246075Z 00:45:36 web.1 | Rendered layouts/_footer.html.erb (0.4ms)
2017-01-20T00:47:06.398298961Z 00:47:06 web.1 | Rendered layouts/_shim.html.erb (0.0ms)
2017-01-20T00:47:06.398304191Z 00:47:06 web.1 | Rendered layouts/_header.html.erb (0.6ms)
2017-01-20T00:47:06.398309625Z 00:47:06 web.1 | Rendered layouts/_footer.html.erb (0.2ms)
2017-01-20T00:47:06.398314432Z 00:47:06 web.1 | Completed 200 OK in 8ms (Views: 6.1ms | ActiveRecord: 0.2ms)
In case there is nothing to see here, or we erased the useful logs by restarting the nodes, we will have to check in cubes logs directly.
For each cubes (listed in nex-cli apps:info command) we can run
nex-cli cubes:events --tail 200 d2a2d9d9-ef8b-41ea-9e45-3cc911254a80
2017-01-19 23:59:04 UTC | container | ERROR | I, [2017-01-19T05:19:45.167568 #509] INFO -- : Writing /app/public/assets/mno_enterprise/config-d760054ad0107c504a224466629f81cb8145bd7c8897bfea3e22d76c72dde01a.js
2017-01-19 23:59:04 UTC | container | ERROR | I, [2017-01-19T05:19:45.168649 #509] INFO -- : Writing /app/public/assets/mno_enterprise/config-d760054ad0107c504a224466629f81cb8145bd7c8897bfea3e22d76c72dde01a.js.gz
2017-01-19 23:59:04 UTC | container | ERROR | ng-process: /usr/local/bundle/bundler/gems/mno-enterprise-d0b9588ff879/frontend/app/assets/javascripts/mno_enterprise/application_lib.js
2017-01-19 23:59:04 UTC | container | ERROR | ng-ignore: /usr/local/bundle/gems/jquery-rails-4.2.2/vendor/assets/javascripts/jquery_ujs.js
2017-01-19 23:59:04 UTC | container | ERROR | ng-process: /usr/local/bundle/bundler/gems/mno-enterprise-d0b9588ff879/frontend/app/assets/javascripts/mno_enterprise/lib/sortable.js
2017-01-19 23:59:04 UTC | container | ERROR | I, [2017-01-19T05:19:48.130491 #509] INFO -- : Writing /app/public/assets/mno_enterprise/mail-2d6d5c1db70669c34d3438c4a36dd26220f3f08f7958f8b5fa10fab46fb1ec29.css.gz
2017-01-19 23:59:04 UTC | container | ERROR | rake aborted!
...
2017-01-19 23:59:06 UTC | container | ERROR | /usr/local/bundle/gems/sprockets-3.7.1/lib/sprockets/processor_utils.rb:75:in `call_processor'
2017-01-19 23:59:12 UTC | container | ERROR | /usr/local/bundle/gems/sprockets-rails-2.3.3/lib/sprockets/rails/task.rb:70:in `block (3 levels) in define'
2017-01-19 23:59:13 UTC | container | ERROR | /usr/local/bundle/gems/sprockets-3.7.1/lib/rake/sprocketstask.rb:147:in `with_logger'
2017-01-19 23:59:13 UTC | container | ERROR | /usr/local/bundle/gems/sprockets-rails-2.3.3/lib/sprockets/rails/task.rb:69:in `block (2 levels) in define'
2017-01-19 23:59:13 UTC | container | ERROR | /usr/local/bundle/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
2017-01-19 23:59:13 UTC | container | ERROR | /usr/local/bin/bundle:22:in `load'
2017-01-19 23:59:13 UTC | container | ERROR | /usr/local/bin/bundle:22:in `<main>'
2017-01-19 23:59:13 UTC | container | ERROR | Tasks: TOP => assets:precompile
2017-01-19 23:59:13 UTC | container | ERROR | (See full trace by running task with --trace)
2017-01-19 23:59:13 UTC | container | ERROR | Could not start container. Aborting. Please see the nex-start logs for more details.
Gotcha!
An untested code version was pushed, causing the container to fail starting.