Longhorn volume backups are based on its implementation of volume snapshots.

When a volume snapshot is requested, a new read-only layer is created from the live filesystem data. Each snapshot creates an additional read-only layer, similarly on how docker container images are built. Thus, deleting a file that is stored on a previous layer doesn’t decrease the volume size.

When taking a backup, Longhorn takes first a snapshot of the volume. Then, all the layers are flattened and backed-up. This is possible because the new live data (created after the backup request) lives outside the snapshot.

A longhorn volume backup is atomic and consistent at the filesystem level (although not necessarily at the application level, because it could have some important data in memory not yet written to disk). This is different on how Velero with “File System Backup” works. For more info on how to use FSB on Oracle Cloud, see this blog post.

Longhorn supports storing volume backups directly on NFS or S3-compatible storage.

This tutorials shows how to configure longhorn to use Oracle Cloud Object Storage as backup destination and how to create and restore a backup.

Prerequisistes

• an Oracle Cloud tenancy (free tier is fine)
• longhorn installed on a kubernetes cluster

Configure bucket destination

1. Create a Oracle Cloud access key, bucket and fetch your tenancy’s object storage namespace and region. See this post for further informations.

2. Create the secret oracle-cloud-object-storage-bucket-backup in the longhorn-system namespace:

apiVersion: v1
type: Opaque
kind: Secret
metadata:
  name: oracle-cloud-object-storage-bucket-backup
  namespace: longhorn-system
data:
  AWS_ACCESS_KEY_ID: [KEY_ID]
  AWS_SECRET_ACCESS_KEY: [KEY_SECRET]
  AWS_ENDPOINTS: https://[OCI_OBJ_STORAGE_NAMESPACE].compat.objectstorage.[OCI_REGION].oraclecloud.com

and fill the data fields. In my case, AWS_ENDPOINTS is https://zrwilxxxxxxx.compat.objectstorage.eu-zurich-1.oraclecloud.com.

3. Access the longhorn webui portal:

kubectl port-forward -n longhorn-system svc/longhorn-frontend 8080:80

Then, browse to http://localhost:8080, go to “Settings > General”, and set

• “Backup Target” to s3://[OCI_BUCKET_NAME]@[OCI_REGION]/longhorn
• “Backup Target Credential Secret” to oracle-cloud-object-storage-bucket-backup

In my case, the bucket is named bucket-lab-k3s-velero-demo, so the field value is s3://bucket-lab-k3s-velero-demo@eu-zurich-1/longhorn.

Manual backup and restore demo

Create a backup

From the longhorn webui portal:

• click on “Volume” in the top bar, then on the the volume name and then “Create Backup”,
• click on “Backup” in the top bar and verify the backup concludes successfully.

Restore the backup

• scale down to 0 the pod using the PVC that will be restored,
• delete the PVC and the PV: this step is required to restore the volume using the same name,
• click on “Backup” in the top bar, then on the backup name, then select “Operation = Restore”,
• check “Use Previous Name” and leave the other settings default,

• click on “Volume” in the top bar: a new volume appeared in the “Detached” state. Click on “Operation = Create PV/PVC”. Insert the name of the deleted PV and PVC,
• check if the PV and PVC appear using kubectl,
• scale back up the pod: the volume should be attached automatically.

Sources

• https://joao-rocha.com/posts/longhorn-backup-s3/
• https://longhorn.io/docs/1.7.2/concepts/
• https://longhorn.io/docs/1.7.2/snapshots-and-backups/scheduling-backups-and-snapshots/