1 .. This work is licensed under a Creative Commons Attribution 4.0
2 .. International License. http://creativecommons.org/licenses/by/4.0
3 .. Copyright 2017-2020 Aarna Networks, Inc.
6 .. Github web page to download the latest version of velero open source tool
7 .. _Velero official website: https://velero.io
8 .. Reference link with more details on Instructions for setting up Velero server
9 .. _Setup Velero Server: https://velero.io/docs/v1.5/contributions/minio/#set-up-server
11 Backup and Restore Solution
12 ###########################
15 This description is not covering the Backup & Restore procedure of the full
16 ONAP platform, but gives examples for the usage of Velero.
17 The ONAP community will focus on Disaster Recovery including B&R
18 in the coming releases.
21 Problem Statement and Requirement (User Story)
22 ==============================================
24 As an ONAP Operator- We require the ability to backup and restore ONAP state data, We want to have Disaster recovery solution for ONAP deployment done over K8.
26 Basic Use case would be
28 1) Add/Update/Modify the POD Data or DB Data.
29 2) Simulate a Disaster
30 3) Restore using Backup.
31 4) POD Data/DB entries should be recovered.
36 Narrowed down upon a tool which can be used for K8 Backup and Restoration for ONAP deployments named as Velero (formerly Heptio-ARK)
38 Velero is an Opensource tool to back up and restore your Kubernetes cluster resources and persistent volumes. Velero lets you:
40 * Take backups of your cluster and restore in case of loss.
41 * Copy cluster resources across cloud providers. NOTE: Cloud volume migrations are not yet supported.
42 * Replicate your production environment for development and testing environments.
46 * A server that runs on your cluster
47 * A command-line client that runs locally
51 .. figure:: Backup-And-Restore.png
59 - Access to a Kubernetes cluster, version 1.7 or later.
60 - A DNS server on the cluster
62 - Labels should be defined there.
66 Velero is an open source tool to safely backup and restore, perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes.
68 Go to `Velero official website`_ and Click on DOWNLOAD VELERO BUTTON. It will take you to the github page to download the latest version of velero. Scroll down to the
69 bottom of the page and choose the binary corresponding to the OS where you want to run the Velero on.
71 Install and configure Velero Server and Client
72 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
74 To configure the latest version (1.5.2) of velero on Linux machine, please follow the below steps.
76 Run the below command. It will download the velero to your machine
80 wget https://github.com/vmware-tanzu/velero/releases/download/v1.5.2/velero-v1.5.2-linux-amd64.tar.gz
82 Extract it using below command
86 tar -zxvf velero-v1.5.2-linux-amd64.tar.gz
88 Once extracted, goto the directory "velero-v1.5.2-linux-amd64"
92 cd velero-v1.5.2-linux-amd64
94 Inside this you will find a directory called examples and then minio, just go inside it
100 Inside this you will find a file called 00-minio-deployment.yaml. Open this file using any editor and look for Velero service yaml portion, there you need to change
101 the type from ClusterIP to NodePort
103 Once made the changes then run the below command to deploy velero server
107 kubectl apply -f 00-minio-deployment.yaml
109 This will configure the Velero with Minio. Minio is nothing but local storage with aws s3 capabilities. Velero support many cloud providers as well like Azure,AWS,GCP
110 etc. You can configure any of those.
112 Once the Velero deployment is done, then we need to install and configure the velero server component using velero cli.
114 Create a Velero-specific credentials file (credentials-velero) in your local directory
119 aws_access_key_id=minio
120 aws_secret_access_key=minio123
122 Start the server and the local storage service. In the Velero directory, run
128 --plugins velero/velero-plugin-for-aws:v1.0.0 \
130 --secret-file ./credentials-velero \
131 --use-volume-snapshots=false \
132 --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://minio.velero.svc:9000
135 Refer this for more details: `Setup Velero Server`_
139 As Labels need to be defined, because that is a unique identity which we need to have for any backup of our k8 containers,
141 So in OOM code, Where -ever we don't have labels, We need to define that whether its configmap or secret, for eg below:-
145 app: {{ include "common.name" . }}
146 chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
147 release: {{ .Release.Name }}
148 heritage: {{ .Release.Service }}
150 Running Velero Example (Backup and Restoration with Logs)
151 ---------------------------------------------------------
153 1) Install SO component
154 ~~~~~~~~~~~~~~~~~~~~~~~
158 This is an optional step, if you have already installed SO or all other components of ONAP then you can skip this step.
160 Below is the example of installing SO component using helm
164 helm install so -n bkup --namespace test3
168 LAST DEPLOYED: Fri Jul 20 06:59:09 2018
178 NAME READY STATUS RESTARTS AGE
180 bkup-so-db-744fccd888-w67zk 0/1 Init:0/1 0 0s
182 bkup-so-7668c746c-vngk8 0/2 Init:0/1 0 0s
188 bkup-so-db Opaque 1 0s
198 so-docker-file-configmap 1 0s
200 so-filebeat-configmap 1 0s
202 so-log-configmap 11 0s
205 ==> v1/PersistentVolume
207 NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
209 bkup-so-db 2Gi RWX Retain Bound test3/bkup-so-db 0s
212 ==> v1/PersistentVolumeClaim
214 NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
216 bkup-so-db Bound bkup-so-db 2Gi RWX 0s
221 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
223 so-db NodePort 10.43.63.96 <none> 3306:30252/TCP 0s
225 so NodePort 10.43.59.93 <none> 8080:30223/TCP,3904:30225/TCP,3905:30224/TCP,9990:30222/TCP,8787:30250/TCP 0s
228 ==> v1beta1/Deployment
230 NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
232 bkup-so-db 1 1 1 0 0s
239 Get the application URL by running these commands
243 export NODE_PORT=$(kubectl get --namespace test3 -o jsonpath="{.spec.ports[0].nodePort}" services so)
245 export NODE_IP=$(kubectl get nodes --namespace test3 -o jsonpath="{.items[0].status.addresses[0].address}")
247 echo http://$NODE_IP:$NODE_PORT
249 2) Checking status of pod
250 ~~~~~~~~~~~~~~~~~~~~~~~~~
254 kubectl get pods --all-namespaces | grep -i so
256 NAMESPACE NAME READY STATUS RESTARTS AGE
258 test3 bkup-so-7668c746c-vngk8 2/2 Running 0 8m
260 test3 bkup-so-db-744fccd888-w67zk 1/1 Running 0 8m
263 3) Creating backup of deployment
264 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
266 Here I am using selector label as release name
270 ./velero backup create so-backup --selector release=bkup
272 Backup request "so-backup" submitted successfully.
274 Run `velero backup describe so-backup` for more details.
277 4) Checking backup logs
278 ~~~~~~~~~~~~~~~~~~~~~~~
282 ./velero backup describe so-backup
311 Label selector: release=bkup
323 Backup Format Version: 1
325 Started: 2018-07-20 07:09:51 +0000 UTC
327 Completed: 2018-07-20 07:09:53 +0000 UTC
329 Expiration: 2018-08-19 07:09:51 +0000 UTC
331 Validation errors: <none>
333 Persistent Volumes: <none included>
335 5) Simulating a disaster
336 ~~~~~~~~~~~~~~~~~~~~~~~~
340 helm delete --purge bkup
342 release "bkup" deleted
344 6)Restoring the kubernetes resources using velero
345 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
349 ./velero restore create --from-backup so-backup
351 Restore request "so-backup-20180720071236" submitted successfully.
353 Run `velero restore describe so-backup-20180720071236` for more details.
356 7) Checking restoration logs
357 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
361 ./velero restore describe so-backup-20180720071236
363 Name: so-backup-20180720071236
385 Excluded: nodes, events, events.events.k8s.io, backups.ark.heptio.com, restores.ark.heptio.com
390 Namespace mappings: <none>
392 Label selector: <none>
399 Validation errors: <none>
409 As we are using Minio which is local storage with aws s3 capabilities. Thus our all the backup files are being stored in locally in Minio Pod. Let's see where the backup files are being genreted.
413 kubectl get pod -n velero
414 NAME READY STATUS RESTARTS AGE
415 minio-d9c56ff5-cg8zp 1/1 Running 0 4d5h
416 minio-setup-ph8pk 0/1 Completed 0 4d5h
417 velero-74cdf64d76-t8wfs 1/1 Running 0 4d5h
421 kubectl exec -it -n velero minio-d9c56ff5-cg8zp ls storage/velero/backups/
425 kubectl exec -it -n velero minio-d9c56ff5-cg8zp ls storage/velero/backups/so-backup
426 so-backup-csi-volumesnapshotcontents.json.gz
427 so-backup-csi-volumesnapshots.json.gz
429 so-backup-podvolumebackups.json.gz
430 so-backup-resource-list.json.gz
431 so-backup-volumesnapshots.json.gz
443 NAME BACKUP STATUS WARNINGS ERRORS CREATED SELECTOR
445 so-backup-20180720071236 so-backup Completed 0 0 2018-07-20 07:12:36 +0000 UTC <none>
448 10) Check the pod status
449 ~~~~~~~~~~~~~~~~~~~~~~~~
453 kubectl get pods --all-namespaces | grep -i so
455 NAMESPACE NAME READY STATUS RESTARTS AGE
457 test3 bkup-so-7668c746c-vngk8 2/2 Running 0 8m
459 test3 bkup-so-db-744fccd888-w67zk 1/1 Running 0 8m
463 Another Example with DB and PV Backup
464 -------------------------------------
466 APPC component backup and restoration
467 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
471 kubectl get pods --all-namespaces | grep -i appc
472 onap bk-appc-0 1/2 Running 0 1m
473 onap bk-appc-cdt-7cd6f6d674-5thwj 1/1 Running 0 1m
474 onap bk-appc-db-0 2/2 Running 0 1m
475 onap bk-appc-dgbuilder-59895d4d69-7rp9q 1/1 Running 0 1m
478 Creating dummy entry in db
479 ~~~~~~~~~~~~~~~~~~~~~~~~~~
483 kubectl exec -it -n default bk-appc-db-0 bash
484 Defaulting container name to appc-db.
485 Use 'kubectl describe pod/bk-appc-db-0 -n onap' to see all of the containers in this pod.
489 root@bk-appc-db-0:/# mysql -u root -p
491 Welcome to the MySQL monitor. Commands end with ; or \g.
492 Your MySQL connection id is 42
493 Server version: 5.7.23-log MySQL Community Server (GPL)
495 Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
497 Oracle is a registered trademark of Oracle Corporation and/or its
498 affiliates. Other names may be trademarks of their respective
501 Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
507 Reading table information for completion of table and column names
508 You can turn off this feature to get a quicker startup with -A
511 Current database: mysql
515 mysql> select * from servers;
519 +-------------+----------+------+-----+---------+-------+
520 | Field | Type | Null | Key | Default | Extra |
521 +-------------+----------+------+-----+---------+-------+
522 | Server_name | char(64) | NO | PRI | | |
523 | Host | char(64) | NO | | | |
524 | Db | char(64) | NO | | | |
525 | Username | char(64) | NO | | | |
526 | Password | char(64) | NO | | | |
527 | Port | int(4) | NO | | 0 | |
528 | Socket | char(64) | NO | | | |
529 | Wrapper | char(64) | NO | | | |
530 | Owner | char(64) | NO | | | |
531 +-------------+----------+------+-----+---------+-------+
532 9 rows in set (0.00 sec)
534 mysql> insert into servers values ("test","ab","sql","user","pwd",1234,"test","wrp","vaib");
535 Query OK, 1 row affected (0.03 sec)
540 mysql> select * from servers;
541 +-------------+------+-----+----------+----------+------+--------+---------+-------+
542 | Server_name | Host | Db | Username | Password | Port | Socket | Wrapper | Owner |
543 +-------------+------+-----+----------+----------+------+--------+---------+-------+
544 | abc | ab | sql | user | pwd | 1234 | test | wrp | vaib |
545 +-------------+------+-----+----------+----------+------+--------+---------+-------+
546 1 row in set (0.00 sec)
554 root@bk-appc-db-0:/# exit
555 command terminated with exit code 127
556 kubectl get pods --all-namespaces | grep -i appc
557 onap bk-appc-0 1/2 Running 0 5m
558 onap bk-appc-cdt-7cd6f6d674-5thwj 1/1 Running 0 5m
559 onap bk-appc-db-0 2/2 Running 0 5m
560 onap bk-appc-dgbuilder-59895d4d69-7rp9q 1/1 Running 0 5m
563 Creating dummy file in APPC PV
564 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
567 kubectl exec -it -n onap bk-appc-0 bash
568 Defaulting container name to appc.
569 Use 'kubectl describe pod/bk-appc-0 -n onap' to see all of the containers in this pod.
573 root@bk-appc-0:/# cd /opt/opendaylight/current/daexim/
574 root@bk-appc-0:/opt/opendaylight/current/daexim# ls
575 root@bk-appc-0:/opt/opendaylight/current/daexim# ls
576 root@bk-appc-0:/opt/opendaylight/current/daexim#
577 root@bk-appc-0:/opt/opendaylight/current/daexim#
578 root@bk-appc-0:/opt/opendaylight/current/daexim# touch abc.txt
579 root@bk-appc-0:/opt/opendaylight/current/daexim# ls
581 root@bk-appc-0:/opt/opendaylight/current/daexim# exit
583 root@rancher:~/oom/kubernetes# kubectl get pods --all-namespaces | grep -i appc
584 onap bk-appc-0 1/2 Running 0 6m
585 onap bk-appc-cdt-7cd6f6d674-5thwj 1/1 Running 0 6m
586 onap bk-appc-db-0 2/2 Running 0 6m
587 onap bk-appc-dgbuilder-59895d4d69-7rp9q 1/1 Running 0 6m
590 Creating backup using velero
591 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
595 ./velero backup create appc-bkup1 --selector release=bk
596 Backup request "appc-bkup1" submitted successfully.
597 Run `velero backup describe appc-bkup1` for more details.
599 ./velero backup describe appc-bkup1
616 Label selector: release=bk
624 Backup Format Version: 1
626 Started: 2018-08-27 05:07:45 +0000 UTC
627 Completed: 2018-08-27 05:07:47 +0000 UTC
629 Expiration: 2018-09-26 05:07:44 +0000 UTC
631 Validation errors: <none>
633 Persistent Volumes: <none included>
636 Simulating disaster by deleting APPC
637 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
641 helm delete --purge bk
645 Restoration using velero
646 ~~~~~~~~~~~~~~~~~~~~~~~~
650 ./velero restore create --from-backup appc-bkup1
651 Restore request "appc-bkup1-20180827052651" submitted successfully.
652 Run `velero restore describe appc-bkup1-20180827052651` for more details.
657 Check the Restoration details immediately after restoration. Restoration process is in InProgress Phase. Please check the Phase.
661 ./velero restore describe appc-bkup1-20180827052651
662 Name: appc-bkup1-20180827052651
675 Excluded: nodes, events, events.events.k8s.io, backups.ark.heptio.com, restores.ark.heptio.com
678 Namespace mappings: <none>
680 Label selector: <none>
686 Validation errors: <none>
690 ./velero restore describe appc-bkup1-20180827052651
691 Name: appc-bkup1-20180827052651
704 Excluded: nodes, events, events.events.k8s.io, backups.ark.heptio.com, restores.ark.heptio.com
707 Namespace mappings: <none>
709 Label selector: <none>
715 Validation errors: <none>
718 Warnings: <error getting warnings: Get "http://minio.velero.svc:9000/velero/restores/dev-appc-1-20201108164330/restore-dev-appc-1-20201108164330-results.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20201108%2Fminio%2Fs3%2Faws4_request&X-Amz-Date=20201108T183923Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=847bdbb0a76718220c40767c4837aa999a4da9ff1344e9b42d3c93f7009e6898": dial tcp: lookup minio.velero.svc on 127.0.0.53:53: no such host>
720 Errors: <error getting errors: Get "http://minio.velero.svc:9000/velero/restores/dev-appc-1-20201108164330/restore-dev-appc-1-20201108164330-results.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20201108%2Fminio%2Fs3%2Faws4_request&X-Amz-Date=20201108T183923Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=847bdbb0a76718220c40767c4837aa999a4da9ff1344e9b42d3c93f7009e6898": dial tcp: lookup minio.velero.svc on 127.0.0.53:53: no such host>
722 This process might take some time to complete. When you check the Restoration details again after some time then the phase will show as Completed as shown below.
726 ./velero restore describe appc-bkup1-20180827052651
727 Name: appc-bkup1-20180827052651
740 Excluded: nodes, events, events.events.k8s.io, backups.ark.heptio.com, restores.ark.heptio.com
743 Namespace mappings: <none>
745 Label selector: <none>
751 Validation errors: <none>
753 Warnings: <error getting warnings: Get "http://minio.velero.svc:9000/velero/restores/dev-appc-1-20201108164330/restore-dev-appc-1-20201108164330-results.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20201108%2Fminio%2Fs3%2Faws4_request&X-Amz-Date=20201108T183923Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=847bdbb0a76718220c40767c4837aa999a4da9ff1344e9b42d3c93f7009e6898": dial tcp: lookup minio.velero.svc on 127.0.0.53:53: no such host>
755 Errors: <error getting errors: Get "http://minio.velero.svc:9000/velero/restores/dev-appc-1-20201108164330/restore-dev-appc-1-20201108164330-results.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20201108%2Fminio%2Fs3%2Faws4_request&X-Amz-Date=20201108T183923Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=847bdbb0a76718220c40767c4837aa999a4da9ff1344e9b42d3c93f7009e6898": dial tcp: lookup minio.velero.svc on 127.0.0.53:53: no such host>
764 NAME BACKUP STATUS WARNINGS ERRORS CREATED SELECTOR
765 appc-bkup-20180827045955 appc-bkup Completed 2 0 2018-08-27 04:59:52 +0000 UTC <none>
766 appc-bkup1-20180827052651 appc-bkup1 Completed 5 0 2018-08-27 05:26:48 +0000 UTC <none>
767 vid-bkp-20180824053001 vid-bkp Completed 149 2 2018-08-24 05:29:59 +0000 UTC <none>
769 Completed status means the Restoration is done successfully.
771 Restoration successful
772 ~~~~~~~~~~~~~~~~~~~~~~
776 kubectl get pods --all-namespaces | grep -i appc
777 onap bk-appc-0 1/2 Running 0 26m
778 onap bk-appc-cdt-7cd6f6d674-5thwj 1/1 Running 0 26m
779 onap bk-appc-db-0 2/2 Running 0 26m
780 onap bk-appc-dgbuilder-59895d4d69-7rp9q 1/1 Running 0 26m
781 kubectl exec -it -n onap bk-appc-db-0 bash
782 Defaulting container name to appc-db.
783 Use 'kubectl describe pod/bk-appc-db-0 -n onap' to see all of the containers in this pod.
789 Restoration of db successful
790 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
794 root@bk-appc-db-0:/# mysql -u root
795 ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
796 root@bk-appc-db-0:/# mysql -u root -p
798 Welcome to the MySQL monitor. Commands end with ; or \g.
799 Your MySQL connection id is 335
800 Server version: 5.7.23-log MySQL Community Server (GPL)
802 Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
804 Oracle is a registered trademark of Oracle Corporation and/or its
805 affiliates. Other names may be trademarks of their respective
808 Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
811 Reading table information for completion of table and column names
812 You can turn off this feature to get a quicker startup with -A
815 Current database: mysql
817 mysql> select * from servers;
818 +-------------+------+-----+----------+----------+------+--------+---------+-------+
819 | Server_name | Host | Db | Username | Password | Port | Socket | Wrapper | Owner |
820 +-------------+------+-----+----------+----------+------+--------+---------+-------+
821 | abc | ab | sql | user | pwd | 1234 | test | wrp | vaib |
822 +-------------+------+-----+----------+----------+------+--------+---------+-------+
823 1 row in set (0.00 sec)
827 root@bk-appc-db-0:/# exit
831 Restoration of PV successful
832 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
836 kubectl get pods --all-namespaces | grep -i appc
837 onap bk-appc-0 1/2 Running 0 27m
838 onap bk-appc-cdt-7cd6f6d674-5thwj 1/1 Running 0 27m
839 onap bk-appc-db-0 2/2 Running 0 27m
840 onap bk-appc-dgbuilder-59895d4d69-7rp9q 1/1 Running 0 27m
841 kubectl exec -it -n onap bk-appc-0 bash
842 Defaulting container name to appc.
843 Use 'kubectl describe pod/bk-appc-0 -n onap' to see all of the containers in this pod.
848 root@bk-appc-0:/# cd /opt/opendaylight/current/daexim/
849 root@bk-appc-0:/opt/opendaylight/current/daexim# ls
851 root@bk-appc-0:/opt/opendaylight/current/daexim#
852 root@bk-appc-0:/opt/opendaylight/current/daexim#
853 root@bk-appc-0:/opt/opendaylight/current/daexim# exit
863 Using Schedules and Restore-Only Mode
865 If you periodically back up your cluster resources, you are able to return to a previous state in case of some unexpected mishap, such as a service outage.
870 Using Backups and Restores
872 Velero can help you port your resources from one cluster to another, as long as you point each Velero Config to the same cloud object storage.
875 https://github.com/vmware-tanzu/velero