A lightweight high-level TSM API/library (called ltsmapi) supporting operations
- archiving
- retrieving
- deleting
- and querying.
In addition a simple console client (called ltsmc) is provided which demonstrates the use of ltsmapi.
Before using LTSM a working access to a TSM server is required. One can install for testing purposes a fully working TSM server (e.g. inside a KVM) for a period of 30 days before the license expires. A complete installation and setup guide is provided at TSM Server Installation Guide.
To compile ltsmc first clone the git repository
git clone https://github.com/tstibor/ltsm
Download and install the TSM API client provided at 7.1.4.0-TIV-TSMBAC-LinuxX86_DEB.tar.
Make sure to install tivsm-api64.amd64.deb
where the header files dapitype.h, dsmapips.h, ...
and the low-level library libApiTSM64.so
are provided.
The building of LTSMAPI/LTSMC is based on simple Makefile
make
Gives result when the header files and library are provided at the proper place:
gcc -m64 -DLINUX_CLIENT -std=c99 -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -O2 -DVERSION=\"0.0.1-19-g3f7bbbe\" -c src/qarray.c -I ibmtsm -o obj/src/qarray.o
gcc -m64 -DLINUX_CLIENT -std=c99 -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -O2 -DVERSION=\"0.0.1-19-g3f7bbbe\" -c src/tsmapi.c -I ibmtsm -o obj/src/tsmapi.o
gcc -m64 -DLINUX_CLIENT -std=c99 -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -O2 -DVERSION=\"0.0.1-19-g3f7bbbe\" -c src/ltsmc.c -I ibmtsm -o obj/src/ltsmc.o
gcc -m64 -DLINUX_CLIENT -std=c99 -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -O2 -DVERSION=\"0.0.1-19-g3f7bbbe\" -o bin/ltsmc obj/src/qarray.o obj/src/tsmapi.o obj/src/ltsmc.o -lApiTSM64 -lm
gcc -m64 -DLINUX_CLIENT -std=c99 -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -O2 -DVERSION=\"0.0.1-19-g3f7bbbe\" -c src/test/test_dsstruct64_off_t.c -I src -I ibmtsm -I cutest -o obj/src/test/test_dsstruct64_off_t.o
gcc -m64 -DLINUX_CLIENT -std=c99 -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -O2 -DVERSION=\"0.0.1-19-g3f7bbbe\" -c src/test/test_qarray.c -I src -I ibmtsm -I cutest -o obj/src/test/test_qarray.o
gcc -m64 -DLINUX_CLIENT -std=c99 -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -O2 -DVERSION=\"0.0.1-19-g3f7bbbe\" -c src/test/ltsmc_testsuite.c -I src -I ibmtsm -I cutest -o obj/src/test/ltsmc_testsuite.o
gcc -m64 -DLINUX_CLIENT -std=c99 -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -O2 -DVERSION=\"0.0.1-19-g3f7bbbe\" -c src/test/CuTest.c -I src -I ibmtsm -I cutest -o obj/src/test/CuTest.o
gcc -m64 -DLINUX_CLIENT -std=c99 -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -O2 -DVERSION=\"0.0.1-19-g3f7bbbe\" -o bin/ltsmc_testsuite obj/src/test/test_dsstruct64_off_t.o obj/src/test/test_qarray.o obj/src/test/ltsmc_testsuite.o obj/src/test/CuTest.o obj/src/qarray.o obj/src/tsmapi.o -lApiTSM64 -lm
To compile with DEBUG and VERBOSE flags for verbose output details set ENV
variables DEBUG=1 VERBOSE=1
:
DEBUG=1 VERBOSE=1 make
Gives result:
gcc -m64 -DLINUX_CLIENT -std=c99 -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -g -DDEBUG -DVERBOSE -DVERSION=\"0.0.1-19-g3f7bbbe\" -c src/qarray.c -I ibmtsm -o obj/src/qarray.o
gcc -m64 -DLINUX_CLIENT -std=c99 -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -g -DDEBUG -DVERBOSE -DVERSION=\"0.0.1-19-g3f7bbbe\" -c src/tsmapi.c -I ibmtsm -o obj/src/tsmapi.o
gcc -m64 -DLINUX_CLIENT -std=c99 -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -g -DDEBUG -DVERBOSE -DVERSION=\"0.0.1-19-g3f7bbbe\" -c src/ltsmc.c -I ibmtsm -o obj/src/ltsmc.o
gcc -m64 -DLINUX_CLIENT -std=c99 -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -g -DDEBUG -DVERBOSE -DVERSION=\"0.0.1-19-g3f7bbbe\" -o bin/ltsmc obj/src/qarray.o obj/src/tsmapi.o obj/src/ltsmc.o -lApiTSM64 -lm
gcc -m64 -DLINUX_CLIENT -std=c99 -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -g -DDEBUG -DVERBOSE -DVERSION=\"0.0.1-19-g3f7bbbe\" -c src/test/test_dsstruct64_off_t.c -I src -I ibmtsm -I cutest -o obj/src/test/test_dsstruct64_off_t.o
gcc -m64 -DLINUX_CLIENT -std=c99 -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -g -DDEBUG -DVERBOSE -DVERSION=\"0.0.1-19-g3f7bbbe\" -c src/test/test_qarray.c -I src -I ibmtsm -I cutest -o obj/src/test/test_qarray.o
gcc -m64 -DLINUX_CLIENT -std=c99 -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -g -DDEBUG -DVERBOSE -DVERSION=\"0.0.1-19-g3f7bbbe\" -c src/test/ltsmc_testsuite.c -I src -I ibmtsm -I cutest -o obj/src/test/ltsmc_testsuite.o
gcc -m64 -DLINUX_CLIENT -std=c99 -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -g -DDEBUG -DVERBOSE -DVERSION=\"0.0.1-19-g3f7bbbe\" -c src/test/CuTest.c -I src -I ibmtsm -I cutest -o obj/src/test/CuTest.o
gcc -m64 -DLINUX_CLIENT -std=c99 -Wall -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -g -DDEBUG -DVERBOSE -DVERSION=\"0.0.1-19-g3f7bbbe\" -o bin/ltsmc_testsuite obj/src/test/test_dsstruct64_off_t.o obj/src/test/test_qarray.o obj/src/test/ltsmc_testsuite.o obj/src/test/CuTest.o obj/src/qarray.o obj/src/tsmapi.o -lApiTSM64 -lm
After successful compiling you will find two binaries: bin/ltsmc_testsuite
and bin/ltsmc
. The former is a simple test suite, the latter the console client:
Syntax: ./bin/ltsmc
-a, --archive
-r, --retrieve
-q, --query
-d, --delete
-f, --fsname <STRING>
-h, --hl <STRING>
-l, --ll <STRING>
-c, --description <STRING>
-n, --node <STRING>
-u, --username <STRING>
-p, --password <STRING>
-s, --servername <STRING>
Version: 0.0.5-1-gb460ddf,© Thomas Stibor <[email protected]>
Make sure to setup proper TSM servername, nodename and IP address entries in file dsmopt/dsm.sys
or in the default file
/opt/tivoli/tsm/client/api/bin64/dsm.sys
. For TSM Server Installation Guide the following entries are working:
SErvername lxdv81-kvm-tsm-server
Nodename lxdv81
TCPServeraddress 192.168.254.101
SErvername polaris-kvm-tsm-server
Nodename polaris
TCPServeraddress 192.168.254.102
In addition make sure to set variable DSMI_CONFIG
pointing to the file dsm.sys
, that is: export DSMI_CONFIG=`pwd`/dsmopt/dsm.sys
or export DSMI_CONFIG=/opt/tivoli/tsm/client/api/bin64/dsm.sys
.
See also the sanity script fore more details.
For demonstrating of how to archive files and directories we first create an archive directories and place there some data.
mkdir -p /tmp/archives && wget --directory-prefix /tmp/archives https://www.kernel.org/pub/linux/kernel/Historic/linux-0.01.tar.gz && tar xvfz /tmp/archives/linux-0.01.tar.gz -C /tmp/archives && rm -rf /tmp/archives/linux-0.01.tar.gz
First we archive the single file /tmp/archives/linux/Makefile
bin/ltsmc --archive --fsname '/' -c 'Historic Linux Kernel Makefile' --node lxdv81 --password lxdv81 --servername=lxdv81-kvm-tsm-server /tmp/archives/linux/Makefile
For archiving the complete /tmp/archives/linux
directory (with all sub-directories) perform following command:
bin/ltsmc --archive --fsname '/' -c 'Complete Historic Linux Kernel' --node lxdv81 --password lxdv81 --servername=lxdv81-kvm-tsm-server /tmp/archives/linux
For querying all data stored in directory /tmp/archives/linux
bin/ltsmc --query --fsname '/' --node lxdv81 --password lxdv81 --servername=lxdv81-kvm-tsm-server --hl '/tmp/archives/linux/*' --ll '/*'
Querying the single file /tmp/archives/linux/Makefile
bin/ltsmc --query --fsname '/' --node lxdv81 --password lxdv81 --servername=lxdv81-kvm-tsm-server --hl '/tmp/archives/linux' --ll '/Makefile'
We first delete all data in /tmp/archives/linux
and the restore the data to that directory:
rm -rf /tmp/archives/linux && bin/ltsmc --retrieve --fsname '/' --node lxdv81 --password lxdv81 --servername=lxdv81-kvm-tsm-server --hl '/tmp/archives/linux/*' --ll '/*'
To delete data on the TSM perform the following command:
bin/ltsmc --delete --fsname '/' --node lxdv81 --password lxdv81 --servername=lxdv81-kvm-tsm-server --hl '/tmp/archives/linux/*' --ll '/*'
A subsequent delete or query command shows that there is no data left on the TSM side.
bin/ltsmc --delete --fsname '/' --node lxdv81 --password lxdv81 --servername=lxdv81-kvm-tsm-server --hl '/tmp/archives/linux/*' --ll '/*'
Klick on the screenshot to see the full screencast
- Integrate LTSMAPI into the Lustre HSM framework.
- Implement multiple objects per transaction, currently a single object is processed per transaction.
- Extend LTSMAPI with new compression stream algorithm such as LZ4 (perform a comparison to TSM built-in compression).
Large directories can contain empty files and naively setting the estimated file size as follows:
objAttrArea.sizeEstimate.hi = 0;
objAttrArea.sizeEstimate.lo = 0;
leads to error dsmSendData: ANS0344E (RC2107) Cannot Send data with a zero byte sizeEstimate.
Whenever you are using the low-level library libApiTSM64.so
make sure to catch this issue. A simple approach is:
dsStruct64_t hi_lo_size = to_dsStruct64_t(st_buf.st_size);
objAttrArea.sizeEstimate.hi = hi_lo_size.hi;
objAttrArea.sizeEstimate.lo = (hi_lo_size.hi == 0 && hi_lo_size.lo == 0) ? 1 : hi_lo_size.lo;
Before you can retrieve archived data, calling function dsmBeginQuery(...)
is required. Subsequently filling e.g. an array
of query replies is achieved with (multiple) function calls of dsmGetNextQObj(...)
. However, when retrieving data
make sure to call function dsmBeginGetData
with parameter dsmGetList
with not more than DSM_MAX_GET_OBJ
objects
values, otherwise your retrieving process will brake. A simple solution is to partition the query replies in chunks
of DSM_MAX_GET_OBJ
and call dsmBeginGetData
on these max chunk sizes.
If you experience the error
[ERROR] (src/tsmapi.c) dsmInitEx: handle: 0 ANS1035S (RC406) Options file '*' could not be found, or it cannot be read.
then the environment variable DSMI_CONFIG
is either not set or is pointing to an improper option file. When you checkout the repository, e.g. make sure to set DSMI_CONFIG
as follows
export DSMI_CONFIG=`pwd`/dsmopt/dsm.sys
If you see the error
[ERROR] (src/tsmapi.c) dsmInitEx: handle: 0 ANS1217E (RC409) Server name not found in System Options File
then as the message suggests the specified servername e.g. --servername=<lxdv81-kvm-tsm-non-server>
does not exists in the option file dsm.sys
.
Suppose you want to query and list all archived data, however you don't know exactly the file paths. The following simple command query all archived data for the registered node
bin/ltsmc --query --fsname '/' --node lxdv81 --password lxdv81 --servername=lxdv81-kvm-tsm-server --hl '*' --ll '/*'
A thorough description and code examples of IBM's low-level TSM API/library can be found in the open document Using the Application Programming Interface, 2007.
LTSM itself is licensed under the GPL-2 license. The code however, depends on IBM's low-level TSM API/library which are distributed under different licenses. See IBM Tivoli Storage Manager Server.