Giter Club home page Giter Club logo

safekeeping's Introduction

logo

safekeeping

Overview

Safekeeping is an online backup tool for VMware vSphere. There are two way to use Safekeeping

  1. by command line using safekeeping-cmd
  2. by Soap WebService using safekeeping-cxf daemon

Common features between cmd and cxf:

  1. Generation base repository.
  2. Support for Virtual Machine, Improved Virtual Disk, vApp and K8s (experimental).
  3. vCenter Tag support and management.
  4. Archive Compression.
  5. Archive Encryption
  6. Multi repository target support (AWS S3,Azure, File System, and more to come).
  7. Restore to a New Virtual Machine new vApp or a new Improved Virtual Disk.
  8. Full Recovery of a Virtual Machine and VAPP with identity (UUID and other metadata).
  9. Support VM and VAPP vApp property(OVF).
  10. Support for encrypted VM and IVD.
  11. Support for Storage Profile.
  12. Support for VM implementing Microsoft VBS.
  13. VMware Cloud on AWS and FedRamp support.
  14. Kubernetes backup support (work in progress).
  15. Full and incremental backup
  16. Data dedup on the fly
  17. Virtual full backup - forever incremental (create a full backup generation directly on the repository)

safekeeping-cmd Features

  1. Command-line Interface, Interactive mode, and batch mode.
  2. Single user
  3. single thread
  4. Support scripts
  5. Extended command support for any entity on vCenter (more or less like powercli)
  6. Works on Windows and Linux

safekeeping-cxf Features

  1. Soap Web Service
  2. Multi concurrent users with different logins
  3. authentication based on vSphere PSC or IVDM
  4. multiThreads operations
  5. multi targets supports on the same sessions (ex backup to 2 different S3 buckets)
  6. Rich SOAP Api
  7. Works on Windows and Linux

Try it out

Prerequisites

CentOS 7.x with the following packages:

  • open-vm-tools
  • wget
  • java-1.8.0-openjdk
  • gcc
  • zip
  • gcc-c++
  • make
  • unzip
sudo yum install open-vm-tools wget java-1.8.0-openjdk  gcc zip gcc-c++ make unzip

Note: Other distributions should work fine

Windows (any) with the following packages:

  • powershell
  • java-1.8.0-openjdk
  • Visual Studio 2017,2019 or 2022

Build

  • Extract the Safekeeping source tar file or sincronize from this GIT deposit
  • Download the Virtual Disk Development Kit and copy on the files on safekeeping/jdisklib/"OS-type"/vddk

Build the VDDK Wrapper

on Windows

cd safekeeping\jdisklib\windows 

Copy any VDDK library you want to use inside the safekeeping\jdisklib\windows\vddk directory

To build the VDDK wrapper use

  • with VisualStudio 2017 build-2017.cmd
  • with VisualStudio 2019 build-2019.cmd
  • with VisualStudio 2022 build-2022.cmd

on Linux

cd ./safekeeping/jdisklib/linux

Copy any VDDK library you want to use inside the ./safekeeping/jdisklib/linux/vddk directory

To build the VDDK wrapper use: ./buildVddkLibraries.sh

Build Java code

Important JAVA supported versions are 8.0.345,11.0.16 or greater. Java 12 and doesn't work.

Import the project on Eclipse

cd safekeeping
prepare.cmd <java_version>  
gradlew eclipse 

on Windows

cd safekeeping
prepare.cmd <java_version>  
gradlew installDist 

on Linux

cd safekeeping
./prepare.sh <java_version>  
./gradlew installDist 

<java_version> 8 jdk1.8 9 jdk1.9 10 jdk10 11 jdk11

Gradle main tasks

Application tasks

  • run - Runs this project as a JVM application

Build tasks

  • build - Assembles and tests this project.
  • clean - Deletes the build directory.

safekeeping-cmd and safekeeping-cxf only tasks

  • deb - Safekeeping Installation. - Create a deb file (requires Debian/Ubuntu)
  • dmg - Safekeeping Installation. - (requires MacOS )
  • msi - Safekeeping Installation. - (requires Windows and https://wixtoolset.org/ )
  • rpm - Safekeeping Installation. - (requires ReadHat/Centos )
  • installDist - Safekeeping Installation. - Create the installation directory

Run

safekeeping-cxf

Run the webservice in interactive mode using port 8080 for HTTP and 8043 for HTTPS

safekeeping-cxf -port 8080 -secure 8043 -interactive

safekeeping-cmd

To run in interactive mode

safekeeping
or 
/opt/vmware/safekeeping/safekeeping

To execute a command

safekeeping <command> <options>

For example to run a backup

safekeeping backup vm:myTestVM

Help

If you are looking for help, the following configure command will display a list of help options:

safekeeping --help

Documentation

Check the wiki for safekeeping-cmd

Blogs Articles

Cormac Hogan Blog

Directory Contents

Directories

  • safekeeping-cmd
    • Safekeeping Command Line version
  • safekeeping-cxf
    • Safekeeping SOAP Web Service daemon version
  • safekeeping-core
    • Core backup tool library written with Java
  • jdisklib
    • Native interface for VDDK written with C++ using VDDK library.
  • jvix
    • Java wrapper for jdisklib native library
  • safekeeping-common
    • Java common library used by the project
  • safekeeping-core-ext
    • Library used by the safekeeping-cmd to deal with: IVD, snapshots, etc.
  • safekeeping-external
    • Wrapper of the internal class. Used by safekeeping-cfx for reflection
  • jopt-simple
    • Modified jopt-simple parsing command line library nssm
    • Updated NSSM - the Non-Sucking Service Manager (support Visual Studio 2017 and 2019) PowerShell
    • Safekeeping SAPI cmd-let
  • lib
    • Support files
  • jar
    • Java jar files required by Safekeeping
  • doc
    • Documentation.
  • sample_scripts
    • Sample scripts to be used with Safekeeping
  • nfs-client
    • NFS library DEPRECATED

Files

  • LICENSE.txt
    • BSD-2 License file.
  • open_source_licenses.txt
    • Open source License file.
  • README.md
    • This file.
  • VERSION
    • Safekeeping version number
  • build.gradle
    • Gradle master build script
  • settings.gradle
    • Gradle settings file

Contributing

The safekeeping project team welcomes contributions from the community. Before you start working with safekeeping, please read our Developer Certificate of Origin. All contributions to this repository must be signed as described on that page. Your signature certifies that you wrote the patch or have the right to pass it on as an open-source patch. For more detailed information, refer to CONTRIBUTING.md.

License

BSD-2 License

safekeeping's People

Contributors

adrienchabot avatar mdaneri avatar svrc avatar vmwsrpbot avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

safekeeping's Issues

Default plugin selection ignored

WHen running the safekeeping application the prompts for S3, FileStorage, and NFSStorage all show a default of false. This implies that pressing enter should be the same as typing 'false' and pressing enter. When I just press enter, I'm then prompted for the settings of each plugin. Those prompts should be skipped if the default is being applied. Here is the output where I just pressed enter at each of the 3 plugin prompt defaults for false:

AMAZONS3
Activate the Plugin [false]
S3 Region [us-west-2]
S3 Backet Name/folder []
S3 Access Key []
S3 Secret Key <hidden>
	Repeat password:     

FILESTORAGE
Activate the Plugin [false]
File Archive directory [/opt/vmware/safekeeping/archive]

NFSSTORAGE
Activate the Plugin [false]
NFS Absolute Path (server:/Export) [localhost:/vol]
User UID [0]
User GID [0]
Max number of retry [5]
Use port <1024 [false]
Use portmap port <1024 [false]
NFS Timeout [20]

Add support for custom vCenter port

A number of our customers have vCenter configured on a non standard port, like 8443, for various reasons. The port for safekeeping appears to be here:

./jvix/src/com/vmware/jvix/jDiskLib.java:102

Please add support for configuring the vCenter port.

Crashes when attempting a backup

When trying to perform a backup Safekeeping crashes. OS is Centos 7. Safekeeping Version 1.0.1 - VDDK Version 6.7.3. Openjdk version "1.8.0_252". Tried different VDDK versions, different transport modes, specifying/not specifying the folder.

7/24/20 17:46:29.718 [INFO, 1, Command, showInfo] BackupInfo: [isNoVmdk false][isDryRun false][compression true][transportMode default][mode auto]
07/24/20 17:46:29.726 [INFO, 30, com.vmware.vmbk.soap.VimConnection, keepAlive] keep alive 2f1778fd-72a2-4e02-9e87-69faad7d4187 : Current Time 2020-07-24T21:46:29.725108Z
07/24/20 17:46:29.726 [INFO, 30, com.vmware.vmbk.soap.KeepAlive, keepAlive] keep alive f032287c-2bfc-46f5-b2d7-bd2f0bd256ea : Name PBM  Version 2.0
07/24/20 17:46:29.919 [WARNING, 1, com.vmware.vmbk.soap.VslmConnection, getIdList] com.vmware.vslm.RuntimeFaultFaultMsg: java.lang.NullPointerException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.sun.xml.internal.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:135)
	at com.sun.xml.internal.ws.client.sei.StubHandler.readResponse(StubHandler.java:238)
	at com.sun.xml.internal.ws.db.DatabindingImpl.deserializeResponse(DatabindingImpl.java:189)
	at com.sun.xml.internal.ws.db.DatabindingImpl.deserializeResponse(DatabindingImpl.java:276)
	at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:104)
	at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:77)
	at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:147)
	at com.sun.proxy.$Proxy49.vslmListVStorageObjectForSpec(Unknown Source)
	at com.vmware.vmbk.soap.VslmConnection.getIdList(VslmConnection.java:438)
	at com.vmware.vmbk.soap.VslmConnection.retrieveVStorageObjectAssociations(VslmConnection.java:933)
	at com.vmware.vmbk.soap.VslmConnection.retrieveVStorageObjectAssociations(VslmConnection.java:895)
	at com.vmware.vmbk.soap.ConnectionManager.retrieveVStorageObjectAssociations(ConnectionManager.java:817)
	at com.vmware.vmbk.command.CommandWithOptions.retrieveVStorageObjectAssociations(CommandWithOptions.java:680)
	at com.vmware.vmbk.command.CommandWithOptions.getVmTarget(CommandWithOptions.java:587)
	at com.vmware.vmbk.command.CommandWithOptions.getFcoTarget(CommandWithOptions.java:173)
	at com.vmware.vmbk.command.BackupCommand.action(BackupCommand.java:103)
	at com.vmware.vmbkCmd.command.BackupCommandInteractive.action(BackupCommandInteractive.java:88)
	at com.vmware.vmbkCmd.VmbkMain.dispatch(VmbkMain.java:260)
	at com.vmware.vmbkCmd.VmbkMain.run(VmbkMain.java:436)
	at com.vmware.vmbkCmd.VmbkMain.main(VmbkMain.java:102)

07/24/20 17:46:29.945 [WARNING, 1, com.vmware.vmbk.soap.VslmConnection, retrieveVStorageObjectAssociationsById] com.vmware.vslm.RuntimeFaultFaultMsg: InvalidArgument: ids is an empty array
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.sun.xml.internal.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:135)
	at com.sun.xml.internal.ws.client.sei.StubHandler.readResponse(StubHandler.java:238)
	at com.sun.xml.internal.ws.db.DatabindingImpl.deserializeResponse(DatabindingImpl.java:189)
	at com.sun.xml.internal.ws.db.DatabindingImpl.deserializeResponse(DatabindingImpl.java:276)
	at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:104)
	at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:77)
	at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:147)
	at com.sun.proxy.$Proxy49.vslmRetrieveVStorageObjectAssociations(Unknown Source)
	at com.vmware.vmbk.soap.VslmConnection.retrieveVStorageObjectAssociationsById(VslmConnection.java:976)
	at com.vmware.vmbk.soap.VslmConnection.retrieveVStorageObjectAssociations(VslmConnection.java:934)
	at com.vmware.vmbk.soap.VslmConnection.retrieveVStorageObjectAssociations(VslmConnection.java:895)
	at com.vmware.vmbk.soap.ConnectionManager.retrieveVStorageObjectAssociations(ConnectionManager.java:817)
	at com.vmware.vmbk.command.CommandWithOptions.retrieveVStorageObjectAssociations(CommandWithOptions.java:680)
	at com.vmware.vmbk.command.CommandWithOptions.getVmTarget(CommandWithOptions.java:587)
	at com.vmware.vmbk.command.CommandWithOptions.getFcoTarget(CommandWithOptions.java:173)
	at com.vmware.vmbk.command.BackupCommand.action(BackupCommand.java:103)
	at com.vmware.vmbkCmd.command.BackupCommandInteractive.action(BackupCommandInteractive.java:88)
	at com.vmware.vmbkCmd.VmbkMain.dispatch(VmbkMain.java:260)
	at com.vmware.vmbkCmd.VmbkMain.run(VmbkMain.java:436)
	at com.vmware.vmbkCmd.VmbkMain.main(VmbkMain.java:102)

07/24/20 17:46:29.945 [WARNING, 1, Command, action] java.lang.NullPointerException
	at java.util.LinkedList.addAll(LinkedList.java:408)
	at java.util.LinkedList.addAll(LinkedList.java:387)
	at com.vmware.vmbk.soap.ConnectionManager.retrieveVStorageObjectAssociations(ConnectionManager.java:817)
	at com.vmware.vmbk.command.CommandWithOptions.retrieveVStorageObjectAssociations(CommandWithOptions.java:680)
	at com.vmware.vmbk.command.CommandWithOptions.getVmTarget(CommandWithOptions.java:587)
	at com.vmware.vmbk.command.CommandWithOptions.getFcoTarget(CommandWithOptions.java:173)
	at com.vmware.vmbk.command.BackupCommand.action(BackupCommand.java:103)
	at com.vmware.vmbkCmd.command.BackupCommandInteractive.action(BackupCommandInteractive.java:88)
	at com.vmware.vmbkCmd.VmbkMain.dispatch(VmbkMain.java:260)
	at com.vmware.vmbkCmd.VmbkMain.run(VmbkMain.java:436)
	at com.vmware.vmbkCmd.VmbkMain.main(VmbkMain.java:102)

Question: How to enable First Class Disk Service

I'm working on getting the VMWare CSI driver installed on kubernetes. The biggest issue I'm running into right now is that the CSI pods say that the FCD service is not available. I'm not enough of a VMWare guru to know how to turn it on. Does FCD require vSAN? I started using this tool to take kube out of the picture and just try to talk to the vCenter API about FCDs directly. So far when I run safekeeping with ivd -list it just hangs and times out. I suspect because the FCD service is not there.

Installation requires admin privileges

Running: ant install results in /opt/vmware/safekeeping/jre/bin which doesn't have anything in it. The wrapper script to run safekeeping has:

export VMBK_HOME=/opt/vmware/safekeeping
export JAVA_HOME=$VMBK_HOME/jre
export VMBK_JAR=$VMBK_HOME/jar
export VMBK_LIB=$VMBK_HOME/lib/lib64

Which results in:

 /bin/safekeeping 
/bin/safekeeping: line 51: /opt/vmware/safekeeping/jre/bin/java: No such file or directory

Perhaps JAVA_HOME should not be pointing to the safekeeping directory structure and instead use the system value?

Default memory requirements are large

The default memory requirements to run safekeeping is 10GB, used right away, and set as the maximum allowed memory usage. I ran the application and the most memory that it used was 4GBs. The memory setting mean that on a small machine with only 8GBs of memory this application cannot be used:

$JAVA_HOME/bin/java $DEBUG -ea -server -Xms10G -Xmx10G -server -XX:NewSize=2G  $LIBRARY_PATH_OPTIONS -jar $VMBK_JAR/vmbkcmd.jar $@

vmbk configure does not apply changes until safekeeping is restarted

I entered the hostname of the vcenter server incorrectly during the configuration of safekeeping. After trying to run ivd -list and seeing the error I ran the configure command and changed the VMware Platform Service Controller FQDN or IP to the correct hostname. I then ran ivd -list again but got the same error message about the incorrect hostname. After closing safekeeping and starting it again the correct hostname is being used.

FCD disk remove operation fails

vmbk# ivd -list

Connecting...
Connected to Platform Service Controller: https://fcdvc101vapp.fmr.com/sts/STSService
LookupService reports 1 VimService Instance
Connected to VimService uuid: ddf67b17-fb54-446d-94ee-1c7ebff6adba url: https://fcdvc101vapp.fmr.com:443/sdk
VMware vCenter Server 6.7.0 build-15129973 - Api Version: 6.7.3
Connected to Storage Profile Service Ver.2.0 url: https://fcdvc101vapp.fmr.com/pbm
Connected to VMware Virtual Storage Lifecycle Manager Service 1.0.0 url: https://fcdvc101vapp.fmr.com/vslm/sdk
Connected to Vapi Service url: https://fcdvc101vapp.fmr.com/api
entity uuid name size cbt snapshot attached datastore path
ivd 06197122-4f0a-4922-b9ab-f6e8f2d17fa7 ivddisk2 1024MB false no yes datastore1 fcd/7932c62e061e4c44834d96a316964587.vmdk
ivd e7879fcd-2cd9-424e-ab20-6b5f21f9a1e7 ivddisk1 2048MB false no yes datastore1 fcd/1dbe0fa82dd2487688c4fddcfc8f4591.vmdk
vmbk#
vmbk#
vmbk# ivd -remove ivd:e7879fcd-2cd9-424e-ab20-6b5f21f9a1e7
Destroy ivd: ivd e7879fcd-2cd9-424e-ab20-6b5f21f9a1e7 ivddisk1
Confirm the operation (y/n)? y
Destroy Improved Virtual Disks ivddisk1 fails
vmbk#
vmbk#

I have created 2 new FCD disks. One observation is, it shows attached by default. Actually the disks are not attached to any VM.

Some *.sh files and lib errors

Describe the bug

hello, we tried to install the safekeeping files, but we could not install because the steps and file paths are different from the readme.md file and we had problems running some *.sh files. Do you have a cleaner explanation or installation guide for installation?
Thank you in advance for helping

Reproduction steps

1. install safekeeping 
2. unzip safekeeping
3. install vddk files
4. copy vddk /safekeepingGit/native-libraries/jdisk/linux/vddk/
5. start ./buildVddkLibraries.sh

Expected behavior

Ekran Resmi 2022-08-12 15 37 43

Additional context

again
hello, we tried to install the safekeeping files, but we could not install because the steps and file paths are different from the readme.md file and we had problems running some *.sh files. Do you have a cleaner explanation or installation guide for installation?
Thank you in advance for helping

archive -check or restore always throws No Valid target specified.

Describe the bug

Hi, I have a few good backups on S3 but when I try to restore or check any of the archive, I always get

-> archive -check vm-1215
No Valid target specified.

I can see the archives though

vm	       vm-1196	502ad6af-9569-0a82-f7c5-9908be36c4d5	1	15 Mar 2022 19:26:38 GMT	2	15 Aug 2022 16:28:12 GMT 	webserver

This also happens with restore no matter what other options I use with that. The logs are not throwing any useful information. Am I missing something here or this has something to do with the backups made with safekeeping-1.0.1 and now trying to restore using safekeeping-2.0.1 and the latest head. Though also tried with safekeeping-1.0.1 gives the same message.

Appreciate any support in this regard.

Thanks.

Reproduction steps

Vanilla install with only S3 configured.

Expected behavior

Should be able to restore or check the archive

Additional context

No response

CXF : unable to login on soap service

Describe the bug

Hi,

I'm trying to connect using soap service provided by safekeeping-cxf and experiencing issues.
The getVersion() soap operation is working fine.
When I login() with authServer/base64/user/password/port I get an Unsupported authentication type:class com.vmwar e.safekeeping.external.type.options.ConnectOptions.
It seems that "options" element (of login() operation) is linked to "ConnectOptions" causing the object to never beeing recognized as instanceOf pscConnectOptions ... and so failing with previous error.

Is it a bug or am i missing something ? :)

Reproduction steps

1. sended this request to /sdk entrypoint

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:cxf="http://cxf.safekeeping.vmware.com/">
   <soapenv:Header/>
   <soapenv:Body>
      <cxf:login>
         <options>
            <authServer>11.22.33.44</authServer>
            <base64>false</base64>
            <password>password</password>
            <port>902</port>
            <user>[email protected]</user>
         </options>
      </cxf:login>
   </soapenv:Body>
</soapenv:Envelope>


### Expected behavior

Got 500 error code with  `Unsupported authentication type:class com.vmwar
e.safekeeping.external.type.options.ConnectOptions` error

### Additional context

Debian 11 / OpenJDK 11 / Safekeeping master

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.