Recover a Mongo ReplicaSet

ARTICLE NOT POLISHED

It may happen upon restarting a mongo addon that the replicaSet is lost - meaning that all hosts in the replicaSet are not valid anymore (because of IP/host change) therefore meaning that the cluster is stale.

If this happens you can follow the instructions below to recover the replica set:

First choose a container in your mongo addon - preferrably the last master if you can identify it
```
nex-cli cubes --addon name_of_mongo_addon
```
Stop all other mongo containers
```
nex-cli cubes:stop other_container_id
```
SSH to the rack hosting the container that is still alive
```
nex-cli racks:ssh 10.1.1.1
```
Connect to the container
```
sudo nex-attach my_container_id
```
Connect to Mongo using the Shell

mongo -u $MONGO_USER -p $MONGO_PASSWORD admin
Delete the local replication information
```
use local
db.dropDatabase()
```
Exit the MongoShell and the container
On the rack issue a restart of the container. Upon restart the mongo instance will initiate a new replicaSet (previous one being deleted)
```
# Restart
nex-restart --name my_container_id


# Monitor logs
docker logs -f my_container_id
```
Exit the rack and bring up the other nodes using the nex-cli
```
nex-cli cubes:start other_container_id
```