Intro to Virtual Tape Libraries (VTLs)

Start Here

Get in touch with a
TriCore Solutions specialist

Blog | Nov 22, 2017

Intro to Virtual Tape Libraries (VTLs)

VTL is more flexible, easy to manage, reliable, has lower latency, and enhanced performance that allows the disk integration of newer data encryption techniques. A virtual tape library emulates the tape drive and gives a feel of a physical tape. VTL combines the advantages of both real physical tapes and disks and delivers high speed backup with restore management in case of any loss.

Introduction:

A virtual tape library (VTL) is a data storage virtualization technology used typically for backup and recovery purposes. A VTL presents a storage component (usually hard disk storage) as tape libraries or tape drives for use with existing backup software.

Virtualizing the disk storage as tape allows integration of VTLs with existing backup software and existing backup and recovery processes and policies. The benefits of such virtualization include storage consolidation and faster data restore processes. For most mainframe data centers, the storage capacity varies, however protecting its business and mission critical data is always vital.

The shift to VTL also eliminates streaming problems that often impair efficiency in tape drives as disk technology does not rely on streaming and can write effectively regardless of data transfer speeds.

By backing up data to disks instead of tapes, VTL often increases performance of both backup and recovery operations. Restore processes are found to be faster than backup regardless of implementations.

The Advantages of Virtual Tape Libraries

The durability, low cost and high density of tape make it an ideal media for handling extremely large amounts of information. However, tapes also come with one significant drawback: SPEED.

 

To work effectively, automated tape libraries need a significant boost in speed to take them to the next level. This is the reason that the VTLs were created.

Instead of reading/writing directly to tape, VTLs, keep a large disk-based buffer of Virtual tapes. This expedites the read and write process. Often, this buffer represents as many as 256 or 512 tapes.
This approach provides two significant benefits:


Performance:

Virtual copies of tapes can be instantly accessed in the buffer without being loaded into the reader. Now you can access 20 different virtual tapes in the same time as it would take to find, load and read just 1 or 2 tapes.


Capacity:

With simple automated tape libraries, storage devices were often under-used. To save a file, you’d pick up a tape, write to it, and then put it back. Now, you can load up the entire virtual tape before putting it back. Or better yet, find a way to optimize file storage across many virtual devices. For example, several smaller virtual tapes can be copied to one physical tape to save space.

 

MHVTL is a free virtual tape library, as this is not production ready, it can be used to explore VTL and for non-prod environment. There are proprietary VTL available in the market. An organization can opt for any backup software with VTL. 

MHVTL to explain the concept step-by-step. 

If I am using two virtual box centos machine. The first represents the VTL and the second is the backup machine. 

One Centos machine which is running VTL

  1. Verify that selinux is disabled, if not then disable and reboot.
  2.  Install repositories from web 

[root@server2 ~]# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

[root@server2 ~]# rpm -Uvh http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm                                                                                                                   Retrieving http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm

Preparing...                ########################################### [100%]

   1:elrepo-release         ########################################### [100%] 

  1. c) Now after configureing new repo, install MHVTL software with below command:

[root@server2 ~]# yum install kmod-mhvtl mhvtl-utils  

       REBOOT the machine after install 

As configuring VTL from the command line is difficult, let’s install it through web interface to make it easy.

          1) mkdir /guivtl

          2) Download web interface console from:

                   [root@server2 ~]# cd /guivtl/

                   [root@server2 guivtl]# weget https://github.com/markh794/mhvtl-gui/tarball/master

          3) [root@server2 guivtl]# tar zxvf markh794-mhvtl-gui-1.4.4-41-gcfcd117.tar.gz

          4)  [root@server2 guivtl]# cd markh794-mhvtl-gui-cfcd117/

5)  [root@server2 markh794-mhvtl-gui-cfcd117]# yum install httpd php

6)  [root@server2 markh794-mhvtl-gui-cfcd117]# chkconfig httpd on

7) [root@server2 markh794-mhvtl-gui-cfcd117]# mkdir /var/www/html/mhvtl

      [root@server2 markh794-mhvtl-gui-cfcd117]# cp -r * /var/www/html/mhvtl

8) now web server user has to be unables to run all root user commaand

                   [root@server2 ~]# echo "apache ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

9)  under sudoers file comment out “Defaults    requiretty”

                   [root@server2 ~]# cat  /etc/sudoers | grep -i "Defaults    requiretty"

                   #Defaults    requiretty

10) install below packages:

                   [root@server2 ~]# yum install lsscsi mtx mt-st git sysstat iscsi-initiator-utils*

11) configure virtual alias in web server

/var/www/html/mhvtl

[root@server2 ~]# vi /etc/httpd/conf/httpd.conf

At end of file add below lines

Alias /mhvtl "/var/www/html/mhvtl"

<Directory "/var/www/html/mhvtl">

  Options None

  AllowOverride None

  Order allow,deny

  Allow from all

</Directory>

12) restart web server

[root@server2 ~]# /etc/init.d/httpd restart 

DONE. Now we have to test if web-ui is accessible from browser.

Enter url: http://192.168.171.142/mhvtl/, we should be able to below screen.

mhvtl.png

The default login password is “MHVTL”. If tgt is offline then we have to insdtall stgt by clicking on Install button. 

We must install iscsi target

[root@server2 ~]# yum list | grep -i scsi-target-util

scsi-target-utils.x86_64                   1.0.24-18.el6                 base

[root@server2 ~]# yum install scsi-target-utils.x86_64 

After installing, go to webui à console and start “SCCI target framework (tgt)”

mhvtl1.png

Once removed click on “Start MHVTL”

Return to SETUP section: 

Click on “Add” and then press “Standard” button (this will initiate wizard select “Next” to start):

mhvtl2.png

Once details are complete, click “Submit” button, “Finish” twice and then “Return”. From there go to “Console” and start the MHVTL engine by clicking on the “Start” button (green). 

You will notice that we have 2 VTLs now, if you want to erase the old one you can proceed. 

Let’s start configuring the robot STK L700 as an iscsi target to make it ready to be connected by host initiator using iscsi.

Go to the “iSCSI (tgt)” section. (as we have already installed tgt framework, we will get below screen).

For easy setup, click on “Quick Start” then “Start” and finally on “Return”: 

to configure that iSCSI target robot is configured, we have to press on “Target” 

Now we are done with the robot configuration. The robot tape drives are

configured in /opt/mhvtl directory: 

STK L700 robot and its tape drives are configured in /etc/mhvtl/device.conf. The MHVTL general configuration is stored in “/etc/mhvtl/mhvtl.conf”

[root@server2 ~]# cat /etc/mhvtl/device.conf 

VERSION: 5 

# VPD page format:

# <page #> <Length> <x> <x+1>... <x+n>

# NAA format is an 8 hex byte value seperated by ':'

# Note: NAA is part of inquiry VPD 0x83

#

# Each 'record' is separated by one (or more) blank lines.

# Each 'record' starts at column 1

# Serial num max len is 10.

# Compression: factor X enabled 0|1

#     Where X is zlib compression factor        1 = Fastest compression

#                                               9 = Best compression

#     enabled 0 == off, 1 == on

#

# fifo: /var/tmp/mhvtl

# If enabled, data must be read from fifo, otherwise daemon will block

# trying to write.

# e.g. cat /var/tmp/mhvtl (in another terminal) 

Library: 50 CHANNEL: 1 TARGET: 00 LUN: 00

 Vendor identification: STK

 Product identification: L700

 Product revision level: 0016

 Unit serial number: 90000050

 NAA: 50:11:22:33:ab:1:00:00

 Home directory: /opt/mhvtl

 Backoff: 400

.

.

.

.

Drive: 54 CHANNEL: 1 TARGET: 00 LUN: 4

 Library ID: 50 Slot: 4

 Vendor identification: STK

 Product identification: T10000A

 Product revision level: 1.27

 Unit serial number: 90000054

 NAA: 50:11:22:33:ab:1:00:4

 Compression: factor 1 enabled 1

 Compression type: lzo

 Backoff: 400 

Drive: 55 CHANNEL: 1 TARGET: 00 LUN: 5

 Library ID: 50 Slot: 5

 Vendor identification: STK

 Product identification: T10000A

 Product revision level: 1.27

 Unit serial number: 90000055

 NAA: 50:11:22:33:ab:1:00:5

 Compression: factor 1 enabled 1

 Compression type: lzo

 Backoff: 400 

Checking tgtd daemon is running: 

[root@server2 ~]# /etc/init.d/tgtd status

tgtd (pid 2080 2078) is running...

[root@server2 ~]# 

NOW CONNECTING Centos 6.x host to configured MHVTL 

first make sure “iscsi-initiator-utils” installed

[root@server3 ~]# yum install iscsi-initiator-utils

[root@server3 ~]# rpm -qa | grep -i iscsi-initiator-utils

iscsi-initiator-utils-6.2.0.873-22.el6_8.x86_64 

Now execute below command

[root@server3 ~]# iscsiadm -m discovery -t st -p 192.168.171.142

192.168.171.142:3260,1 iqn.1994-05.com.redhat:dbb9aea5f9f:mhvtl:stgt:1

[root@server3 ~]#iscsiadm -m node -T iqn.1994-05.com.redhat:dbb9aea5f9f:mhvtl:stgt:1 -p 192.168.171.142 -l 

[root@server3 ~]# dmesg

……………….

……………….

st: Version 20130315rh, fixed bufsize 32768, s/g segs 256

st 33:0:0:1: Attached scsi tape st0

st 33:0:0:1: st0: try direct i/o: yes (alignment 1 B)

osst :I: Tape driver with OnStream support version 0.99.4

osst :I: $Id: osst.c,v 1.73 2005/01/01 21:13:34 wriede Exp $

scsi 33:0:0:2: Sequential-Access STK      T10000A          1.27 PQ: 0 ANSI: 5

st 33:0:0:2: Attached scsi tape st1

st 33:0:0:2: st1: try direct i/o: yes (alignment 1 B)

st 33:0:0:2: Attached scsi generic sg4 type 1

scsi 33:0:0:3: Sequential-Access STK      T10000A          1.27 PQ: 0 ANSI: 5

st 33:0:0:3: Attached scsi tape st2

st 33:0:0:3: st2: try direct i/o: yes (alignment 1 B)

st 33:0:0:3: Attached scsi generic sg5 type 1

scsi 33:0:0:4: Sequential-Access STK      T10000A          1.27 PQ: 0 ANSI: 5

st 33:0:0:4: Attached scsi tape st3

st 33:0:0:4: st3: try direct i/o: yes (alignment 1 B)

st 33:0:0:4: Attached scsi generic sg6 type 1

scsi 33:0:0:5: Sequential-Access STK      T10000A          1.27 PQ: 0 ANSI: 5

st 33:0:0:5: Attached scsi tape st4

st 33:0:0:5: st4: try direct i/o: yes (alignment 1 B)

st 33:0:0:5: Attached scsi generic sg7 type 1

scsi 33:0:0:6: Medium Changer    STK      L700             0016 PQ: 0 ANSI: 5

scsi 33:0:0:6: Attached scsi generic sg8 type 8

SCSI Media Changer driver v0.25

ch0: type #1 (mt): 0x2c0+1 [medium transport]

ch0: type #2 (st): 0x400+25 [storage]

ch0: type #3 (ie): 0x300+5 [import/export]

ch0: type #4 (dt): 0x1+5 [data transfer] 

[root@server3 ~]# /etc/init.d/iscsi status

iSCSI Transport Class version 2.0-870

version 6.2.0-873.22.el6

Target: iqn.1994-05.com.redhat:dbb9aea5f9f:mhvtl:stgt:1 (non-flash)

        Current Portal: 192.168.171.142:3260,1

        Persistent Portal: 192.168.171.142:3260,1

                **********

                Interface:

                **********

                Iface Name: default

                Iface Transport: tcp

                Iface Initiatorname: iqn.1994-05.com.redhat:9738eb103a40

                Iface IPaddress: 192.168.171.145

                Iface HWaddress: <empty>

                Iface Netdev: <empty>

                SID: 1

                iSCSI Connection State: LOGGED IN

                iSCSI Session State: LOGGED_IN

                Internal iscsid Session State: NO CHANGE

                *********

                Timeouts:

                *********

                Recovery Timeout: 120

                Target Reset Timeout: 30

                LUN Reset Timeout: 30

                Abort Timeout: 15

                *****

                CHAP:

                *****

                username: <empty>

                password: ********

                username_in: <empty>

                password_in: ********

                ************************

                Negotiated iSCSI params:

                ************************

                HeaderDigest: None

                DataDigest: None

                MaxRecvDataSegmentLength: 262144

                MaxXmitDataSegmentLength: 8192

                FirstBurstLength: 65536

                MaxBurstLength: 262144

                ImmediateData: Yes

                InitialR2T: Yes

                MaxOutstandingR2T: 1

                ************************

                Attached SCSI devices:

                ************************

                Host Number: 33 State: running

                scsi33 Channel 00 Id 0 Lun: 0

                scsi33 Channel 00 Id 0 Lun: 1

                scsi33 Channel 00 Id 0 Lun: 2

                scsi33 Channel 00 Id 0 Lun: 3

                scsi33 Channel 00 Id 0 Lun: 4

                scsi33 Channel 00 Id 0 Lun: 5

                scsi33 Channel 00 Id 0 Lun: 6 

This is the basic setup and we are ready to use any backup software with it. 

We will work with very basic Linux utility “mt” and “mtx”.

Make sure you have installed required packages

[root@server3 ~]# rpm -qa | grep -i ^mt

mtr-0.75-5.el6.x86_64

mt-st-1.1-5.el6.x86_64

mtools-4.0.12-1.el6.x86_64

mtdev-1.1.2-5.el6.x86_64

mtx-1.3.12-5.el6.x86_64

Tape device names on Linux

/dev/rmt/0 or /dev/rmt/1 or /dev/rmt/[0-127] : Regular tape device name on Unix. The tape is rewound.

  • /dev/rmt/0n : This is know as no rewind i.e. after using tape, leaves the tape in current status for next command.
  • /dev/rmt/0b : Use magtape interface i.e. BSD behavior. More-readable by a variety of OS’s such as AIX, Windows, Linux, FreeBSD, and more.
  • /dev/rmt/0l : Set density to low.
  • /dev/rmt/0m : Set density to medium.
  • /dev/rmt/0u : Set density to high.
  • /dev/rmt/0c : Set density to compressed.
  • /dev/st[0-9] : Linux specific SCSI tape device name.

Tape device name examples

  • The /dev/rmt/1cnindicate that I’m using unity 1, compressed density and no rewind.
  • The /dev/rmt/0hbindicate that I’m using unity 0, high density and BSD behavior.
  • The auto rewind SCSI tape device name on Linux : /dev/st0
  • The non-rewind SCSI tape device name on Linux : /dev/nst0

listing installed SCSI tape devices

lsscsi

lsscsi -g

[root@server3 ~]# rpm -qa | grep -i ^lsscsi

lsscsi-0.23-2.el6.x86_64[root@server3 ~]# lsscsi -g

[0:0:0:0]    disk    VMware,  VMware Virtual S 1.0   /dev/sda   /dev/sg0

[4:0:0:0]    cd/dvd  NECVMWar VMware SATA CD01 1.00  /dev/sr0   /dev/sg1

[33:0:0:0]   storage IET      Controller       0001  -         /dev/sg2

[33:0:0:1]   tape    STK      T10000A          1.27  /dev/st0   /dev/sg3

[33:0:0:2]   tape    STK      T10000A          1.27  /dev/st1   /dev/sg4

[33:0:0:3]   tape    STK      T10000A          1.27  /dev/st2   /dev/sg5

[33:0:0:4]   tape    STK      T10000A          1.27  /dev/st3   /dev/sg6

[33:0:0:5]   tape    STK      T10000A          1.27  /dev/st4   /dev/sg7

[33:0:0:6]   mediumx STK      L700             0016  /dev/sch0  /dev/sg8 

now displaying status of tape/drive 

[root@server3 ~]# mtx -f /dev/sg8 status

  Storage Changer /dev/sg8:5 Drives, 30 Slots ( 5 Import/Export )

Data Transfer Element 0:Empty

Data Transfer Element 1:Empty

Data Transfer Element 2:Empty

Data Transfer Element 3:Empty

Data Transfer Element 4:Empty

      Storage Element 1:Full :VolumeTag=S00001TA

      Storage Element 2:Full :VolumeTag=S00002TA

      Storage Element 3:Full :VolumeTag=S00003TA

      Storage Element 4:Full :VolumeTag=S00004TA

      Storage Element 5:Full :VolumeTag=S00005TA

      Storage Element 6:Full :VolumeTag=S00006TA

      Storage Element 7:Full :VolumeTag=S00007TA

      Storage Element 8:Full :VolumeTag=S00008TA

      Storage Element 9:Full :VolumeTag=S00009TA

      Storage Element 10:Full :VolumeTag=S00010TA

      Storage Element 11:Full :VolumeTag=S00011TA

      Storage Element 12:Full :VolumeTag=S00012TA

      Storage Element 13:Full :VolumeTag=S00013TA

      Storage Element 14:Full :VolumeTag=S00014TA

      Storage Element 15:Full :VolumeTag=S00015TA

      Storage Element 16:Full :VolumeTag=S00016TA

      Storage Element 17:Full :VolumeTag=S00017TA

      Storage Element 18:Full :VolumeTag=S00018TA

      Storage Element 19:Full :VolumeTag=S00019TA

      Storage Element 20:Full :VolumeTag=CLN020TA

      Storage Element 21:Empty

      Storage Element 22:Empty

      Storage Element 23:Empty

      Storage Element 24:Empty

      Storage Element 25:Empty

      Storage Element 26 IMPORT/EXPORT:Empty

      Storage Element 27 IMPORT/EXPORT:Empty

      Storage Element 28 IMPORT/EXPORT:Empty

      Storage Element 29 IMPORT/EXPORT:Empty

      Storage Element 30 IMPORT/EXPORT:Empty 

Above drives are empty. 

Checking status of tape device 

[root@server3 ~]# mt -f /dev/st0 status

SCSI 2 tape drive:

File number=-1, block number=-1, partition=0.

Tape block size 0 bytes. Density code 0x0 (default).

Soft error count since last status=0

General status bits on (50000):

 DR_OPEN IM_REP_EN 

WE have to load tape on “Data Transfer Element 0” to make it usable

for this we have go back to MHVTL portal and load tape on slot 0 

Click on Operator → Mount
mhvtl3.pngSelect Element Slot and Drive

mhvtl4.png

NOW back to server3 i.e. backup server and check status again 

[root@server3 ~]# mtx -f /dev/sg8 status

  Storage Changer /dev/sg8:5 Drives, 30 Slots ( 5 Import/Export )

Data Transfer Element 0:Full (Storage Element 1 Loaded):VolumeTag = S00001TA

Data Transfer Element 1:Empty

Data Transfer Element 2:Empty

Data Transfer Element 3:Empty

Data Transfer Element 4:Empty

      Storage Element 1:Empty

      Storage Element 2:Full :VolumeTag=S00002TA

      Storage Element 3:Full :VolumeTag=S00003TA

      Storage Element 4:Full :VolumeTag=S00004TA

      Storage Element 5:Full :VolumeTag=S00005TA

      Storage Element 6:Full :VolumeTag=S00006TA

      Storage Element 7:Full :VolumeTag=S00007TA

      Storage Element 8:Full :VolumeTag=S00008TA

      Storage Element 9:Full :VolumeTag=S00009TA

      Storage Element 10:Full :VolumeTag=S00010TA

      Storage Element 11:Full :VolumeTag=S00011TA

      Storage Element 12:Full :VolumeTag=S00012TA

      Storage Element 13:Full :VolumeTag=S00013TA

      Storage Element 14:Full :VolumeTag=S00014TA

      Storage Element 15:Full :VolumeTag=S00015TA

      Storage Element 16:Full :VolumeTag=S00016TA

      Storage Element 17:Full :VolumeTag=S00017TA

      Storage Element 18:Full :VolumeTag=S00018TA

      Storage Element 19:Full :VolumeTag=S00019TA

      Storage Element 20:Full :VolumeTag=CLN020TA

To backup directory (tar format)

[root@server3 ~]# tar cvf /dev/st0 /etc/

tar: Removing leading `/' from member names

/etc/

/etc/hosts.deny

/etc/audit/

/etc/audit/audit.rules

/etc/audit/auditd.conf

/etc/lsb-release

/etc/abrt/

/etc/abrt/plugins/

/etc/abrt/plugins/CCpp.conf

/etc/abrt/plugins/python.conf

/etc/abrt/gpg_keys

/etc/abrt/abrt.conf

…..

…..

….. 

List or check tape contents (tar format)

[root@server3 ~]# tar tvf /dev/st0 

To restore directory (tar format)

[root@server3 ~]# tar xvf /dev/st0 -C /tmp &>/dev/null

[root@server3 ~]# ls -ld /tmp/etc/

drwxr-xr-x 114 root root 12288 Mar 22 02:25 /tmp/etc/

[root@server3 ~]# date

Wed Mar 22 02:48:10 IST 2017 

There are many more options to use like backing-up partition or using it with free backup software like Bacula. 

Conclusion:

Managing physical tape backup requires both physical effort, time and involves costs in terms of setting up and sending tapes offsite. This option is not feasible for small and medium size business for taking data backup.

Likewise, retrieving data from tape backup is inconvenient and searching for correct tape is also a tedious process. 

VTL is the simplest replacement for tapes and is a strong medium that bears the problems of the tape. VTL is more flexible, easy to manage, reliable, has lower latency, and enhanced performance that allows the disk integration of newer data encryption techniques. A virtual tape library emulates the tape drive and gives a feel of a physical tape. VTL combines the advantages of both real physical tapes and disks and delivers high speed backup with restore management in case of any loss.


Some of the benefits can be listed as:

  • Performance
  • Capacity
  • Non-disruptive, low-effort migration to new backup approa
  • Keep trusted backup software in place
  • Reduced datacenter infrastructure footprint
  • Streamlined backup processes, and reduced manual effort
  • Data durability
  • Low-cost data archiving for long-term retention, with predictable retrieval times

I hope you find the above details useful. In case you have any questions on the topic please feel free to reach out to me by clicking on the button below. You can also leave a comment in the field given below. For any questions on the topic click below: 

Ask Abhishek