Recover an application from server snapshot
- Arnaud Lachaume (Unlicensed)
In the unlikely event of an emergency it may be required at some point that you perform an application recovery from snapshot. Nex!™ storage racks are designed to be snapshotted every six hours with a two weeks retention when the underlying platform allows it (e.g. AWS EBS snapshots). The steps described in this article are only applicable for applications using persistent storage (e.g. database addons). Web applications - e.g. ones using the maestrano/web-ruby image - are stateless and therefore not subject to backups.
1 - Choosing the backup to use
Login to the Nex!™ console and adapt the steps below based on your recovery needs
# Retrieve the container that you wish to recover
container = App.find_by(name: "someapp").cube_instances.first
# OR
container = Addon.find_by(name: "someaddon").cube_instances.first
# List the recent backups for the storage rack associated with this container
# /!\ Keep in mind that the "created_at" timestamp is displayed in UTC time /!\
puts container.storage_rack.rack_backups.where("created_at > ?",2.days.ago).map { |b| "#{b.id} | #{b.created_at}" }
# From the list displayed above, select the storage backup you want
backup = container.storage_rack.rack_backups.find(123456)
2 - Provision a storage server from this backup
The next step is to make this backup available for retrieval. In order to do that, we are going to create a StorageRack using the AWS AMI attached to the backup.
# Build a new StorageRack from the backup image and flag it as "non usable" by setting the
# available storage units (su) to a large negative value
backup_storage = StorageRack.new(vpc_region: backup.vpc_region, machine_image: backup.backup_id)
backup_storage.available_su = -1000
# Save the storage rack and provision it
backup_storage.save!
backup_storage.provision!
# After a couple of minutes, your backup StorageRack should be ready
backup_storage.reload.status
#=> "running"
backup_storage.ssh("echo 1")
#=> {:stdout=>"1\n", :stderr=>"", :exit_code=>0, :exit_signal=>nil}
# The backup StorageRack is now available to us to perform the recovery
3 - Perform the recovery
Now that the backup StorageRack is available, we're going to stop the application, change its storage rack to the backup one then start the application. This will have the effect to start the application from backup and perform the recovery.
Finally we will reassign the previous storage rack to the application and terminate the backup StorageRack.
# Capture the current application StorageRack active_storage = container.storage_rack # Stop the application container.stop! # Wait until the application is stopped container.reload.status #=> "stopped" # Change the application StorageRack to point to the backup StorageRack container.storage_rack = backup_storage container.save! # Start the application container.start! # Wait until the application is up container.reload.status #=> "running" # The application is now rolled back. Let's proceed to re-assigning the previous StorageRack # To do so we perform a CubeStorageTransfer from the backup StorageRack to the active StorageRack # This transfer will: # 1) re-assign the cube to the StorageRack it should point to # 2) copy the **backup storage** cube data to the active StorageRack transfer = CubeStorageTransfer.create!(cube_instance: container, src_storage_rack: container.storage_rack, dst_storage_rack: active_storage) transfer.provision! # Wait for the transfer to be done transfer.reload.status #=> "provisioned" # Reload the cube and ensure it's pointing to the right storage rack container.reload.storage_rack == active_storage #=> true # Finally delete the backup storage rack backup_storage.terminate! unless backup_storage.reload.cube_instances.any? # Wait until the backup_storage is terminated backup_storage.reload.status #=> "terminated"