Keeping an open-source format
LTFS 2 Technology: Segment 6
I think that you make a great point. I think the interesting thing in my viewpoint about LTFS is that it is indeed an open standard. While IBM has obviously gone to great lengths to develop different ways to do it – whether it is a future enterprise edition or the library edition – the openness of the standard certainly means that other vendors I would imagine may be doing other similar implementations. And in fact, some may be doing it today – I think some even are – to try to get to the vision perhaps of either cartridge-centric view of storage or even as the enterprise edition view as a single storage volume that could span multiple tapes transparently to the end user.
Correct me if I'm wrong but I think that's one of the things that's interesting about LTFS, is that because it's open, there's certainly other people who can take a similar strategy. But in the end I imagine, regardless of the strategy you're taking these different ways, the key hallmark of LTFS being a self-describing open format, always needs to be maintained. Isn't that a requirement, you think?
Yes, yes. It certainly is. No one at all is served by introducing confusion in the eyes of users between one version of LTFS is compatible with another version of LTFS. The format specification exists precisely to avoid that confusion. Strictly speaking, something is only in LTFS format if it complies with the specification.
In addition to the specification, though, the single drive software was implemented as open source software so that people can look at the code and work out really what's going on, and if they choose implement equivalent code internally within their own system.
But internally, within the single drive implementation, architecturally the software is implemented to allow reuse of that software by other vendors. This is going to get into some fairly deep programmer details. But I want to get the message out there. The way that I designed the software implementation is that the code that specifically deals with working with individual tape drives is highly compartmentalized behind an API that is specifically designed to work with tape.
That makes it easy to switch out any one implementation for IBM's tape drives, for example, and switch in the implementation of this low level code by a vendor such as HP. Specifically, there's also a tape backend implementation provided as part of the open source software, which actually uses files stored on local hard drives to simulate a tape drive for the purposes of the file system. This was convenient during development, and it's convenient for people who just want to evaluate LTFS and work out its functionality without necessarily having hardware available.
In addition to this low-level tape backend API, the core logic of the file system, all the operations that deal with driving the tape backend, manipulating data structures in memory and exposing operations such as openFile, readFile, writeFile, all of that code is actually implemented as a library that sits behind another well-defined API. This well-defined API at the top level is what then plugs into the fuse libraries to perform integration with the operating system that is hosting the software.
This core top level library is important for the purposes of adoption of LTFS, because it is completely possible for a vendor writing a backup software or writing a video-editing piece of software, to take this top level library and choose to integrate this top level library into their existing source code. So one way that you could work with LTFS is you can mount the cartridge using the LTFS single drive edition, and just read and write files to and from it from your software.
But if you don't want to expose the fact that there's a file system there, or if you don't want to support all of the operations afforded by a file system – so, for example, if you just wanted to create an LTFS volume, but you didn't want to be able to list files or modify files – then you could pick up the LTFS single edition code base, treat it as a library and just invoke the areas of the code base that are responsible for creating the format on the cartridge and writing files to the cartridge, and thereby save yourself a whole lot of development time, because you're reusing the code that IBM's already written and tested out.
Now, there's certainly licensing concerns there if this reuse is being done by a commercial entity. But we specifically chose the LGPL license as the open source license for the LTFS SD software, to allow other developers to pick up this code base and reuse it in their own applications.
Interesting. I think that's cool, how it allows LTFS to be easily integrated in other environments. Michael Richmon: Our goal here was really lowering the barrier of entry. Because when we were building all of this, we had no idea how people would respond to the technology. We were hopeful, and we really love the technology ourselves. But we were trying to smooth the way so that we didn't release it and then have the public say, "Yeah, so what?"
I think that the effort that we put into moving on multiple fronts – on the format front, on the open source front, and on just getting out there and beating the drum about LTFS – has been really quite successful. It is this open source software that has allowed other vendors who are outside the LTO space strictly, such as Oracle, to adopt the software and adopt the format for their own tape products.