LTFS as an Underlying Storage System

Developing a new way to access data with underlying LTFS storage

LTFS 3 Future: Segment 3


Jay Livens:
Michael, thanks. I appreciate your perspectives on the LTFS spec. I think it's clear that it's very standardized, that everyone understands it, that there might be some additional extensions that could be layered on top. But also that, the very nature of it, as a low-level, clearly-defined format, that that leaves a lot of opportunity for innovation by other vendors, or third parties, or even software manufacturers. One of the things that got my attention recently is that, a tape library company, Spectra Logic, released a technology where they appear to be using LTFS as an underlying file system, but they create a new way to access the data, using an object-oriented approach, kind of like Amazon S3. And in fact, they have their own version of the S3 commands called DS3, which is really optimized for longer-term data storage and such. I'm curious about your perspectives on a technology like that, and combining it with LTFS.

M. Richmond:
That technology, and that announcement by Spectra Logic, is precisely the type of system that I'm talking about where, software developers can build innovative solutions, but still be utilizing the LTFS format as their underlying data storage. I'm still learning about the Spectra Logic product. I've looked at some of their press releases, and from what I understand about their product, they're basically doing the following. They have this rich software layer that exposes an object-based interface to client/server software, and an object-based interface is somewhat like a simplified file system interface. A file system has nested folders and an object store amidst that, and a file system allows you to modify files in place, whereas an object-based interface generally allows you to push some data and get some data, but you can't modify it in place. What I understand that Spectra Logic has done is, they've implemented the software that allows you to put and get chunks of data, as objects, into this system.

And I suspect that, they are then taking the object that's being put into their system, and they're probably storing the object as an individual file on the underlying LTFS volume, and possibly, recording some metadata either beside that file, as an XML file, or a JSON file, to describe the metadata for that object. Or, potentially, they might just be tagging that file with the metadata as extended attributes in the LTFS file system. This is a great example of how developers try to solve a particular market need, can build on top of the common format. But this is an also an example of where LTFS has multiple benefits for developers like Spectra Logic.

LTFS allows their development team to use data tape as a storage medium, without having to go through the pain of working with tape at a low level. They can read and write tape the way that they can read and write any other file system. And for end users, this potentially reduces vendor lock-in to some extent. With any new software system, there's also the concern of, "Well, I'm giving you my data to store in the system, and that's great, so long as the software system continues to be functional, and continues to be developed and improved over time. But, to some extent, my data is locked away within the system.”

The fact that they've used LTFS as the underlying format, potentially means that, in a complete disaster where the customer is unable to continue using the software system for any reason, they potentially can go straight to the LTFS volumes, and retrieve the data directly. This is assuming that a single object is being stored as a single file, but this is probably the most natural way of building an object store on top of LTFS. In this way, LTFS is really, very closely analogous to the FAT file system format developed by Microsoft, way back at the beginning of Microsoft DOS.

The FAT file system format, on the media, has had a couple of major revisions over the years, but those revisions have been fairly large jumps from FAT16 to FAT32, for example. And those were, primarily, expanding the functionality that was already built into the format, but the format largely has been, largely unchanged. If you look at FAT in the marketplace, the FAT file system is used on thumb drives, it's used on hard drives, it's used in various systems internally, just because it's a well-defined format that, since it's well-defined, people know how to work with it.

And hopefully, it's certainly our goal with LTFS, was to introduce a similar common language and common data format for data tape, and other sequential media, that could be adopted by the industry, so that we can stop investing our energy in trying to work out how to read and write data to tape and instead, focus our energy on the interesting things that are going to be solving new problems. In that way, the LTFS format frees up some resources, so that we can do new and better things in the future.

Jay Livens:
Yes, I agree. And the funny thing, I have to say, that got me smiling as I thought about that. I know that FAT stands for File Allocation Table, but all I could think of when you were talking about that is, that is a name that only an engineer would love, calling something FAT.

M. Richmond:
Yes. The unintended consequences of acronyms keeps on coming back to bite us, sometimes.

The Speakers:

Michael Richmond
Jay Livens