Tape Libraries and LTFS

Caching tapes in memory for quick access

LTFS 2 Technology: Segment 5


You talked early on about how LTFS works. And you talked about the single drive implementation with the driver and such. Many of our listeners may have tape libraries. If I have a tape library in my infrastructure, how is sort of the concept of a robot supported, acknowledged or used in LTFS?

Interesting question. So this gets to the heart of why IBM was interested in developing LTFS in the first place from a business perspective. As a researcher and software architect for me building LTFS was useful and interesting as a technological challenge. But as a commercial company IBM wanted to know why they would invest in such an effort. The reason that IBM invested in this was that there’s a view that by solving the problems that we’ve have working with tape and introducing an open format that hopefully would be adopted by the industry and we’re seeing some widespread adoption by tape vendors which is very uplifting.

The idea was that a rising tide lifts all boats. If we make tape easier to use we make tape more appealing to users and applicable to more different scenarios of use. Then potential market for data tape would expand and IBM would benefit by getting more product sold alongside their competitors in the LTO tape space. But really by making the market bigger IBM wins, HP wins, Quantum wins because we have more people to sell to and tape just is better overall. So that was the business decision. Based on their business decision we developed the single drive implementation and released that as an open source implementation.

And then we used the core of that single drive implementation to implement a second version of LTFS called LTFS Library Edition. With LTFS Library Edition you can install, this is a proprietary offering from IBM. You can buy an IBM tape library whether it would be a 1U or 2U unit or even a room-sized full rack system and you can purchase the LTFS LE software from IBM. You install the LTF LE software and rather than mounting a tape drive as you do with the single drive edition you mount your tape library. And what this means is that below the mount point in your file browser for the LTFS LE software you see a folder for each cartridge that exists in your tape library.

And the name of the folder is either the bar code number that is pasted to the tape cartridge or optionally you can provide a human readable name when you format the cartridge for LTFS. So you could format a cartridge with the name “Football 2013” or “News October 2012” and those names would show up as the folder names below your mount point. And then to access files stored on those cartridges you simply change directory into the folder that represents the cartridge you want to access. When you change directory into that folder you could list the file stored on that cartridge and access them, copy them off do whatever you like with them.

Inside the LTFS LE software, what is happening is the software scanned your tape library when the library is mounted to determine which cartridges exist in the library. And then when you try to access a folder that represents a particular cartridge the LTFS LE software drives the robotics in the library to pick up the tape, move it to a tape drive that’s available in the tape library and then accesses the index stored in that tape and then shows you the list of files stored in that cartridge. And that list of files, that index information, is cached in memory for the length of the mount session for the library.

And so over time, all of the index of what stored on your tapes gets cached in memory so that you have very quick access. Once the index is cached in memory the primary time when LTFS LE moves the cartridge into a tape drive is when you access the file essence stored on a particular cartridge. And so that’s how tape libraries are supported in the LTFS LE software. I’m not familiar with library implementations by other vendors but I’m certain that they are interested in the space and working towards their own solutions. And in addition to the LTFS LE implementation IBM has been investigating what they refer to as LTFS Enterprise edition which takes a larger step beyond what I just described to provide what is effectively a hierarchical storage system based on top of LTFS. So that rather than having physical cartridge names as folder names, you just see a flat name space where you can put files into this name space and the LTFS Enterprise Edition software manages moving the files from that name space to and from the cartridges behind the scenes. So that gives a very abstract logical view of your file space rather than a cartridge-centric view. The LTFS Enterprise Edition software is something that I was involved in the architecture, designed very early on and the development of that software has continued by other members of IBM staff after I left the company.

The Speakers:

Michael Richmond
Jay Livens