An IOSlave Tutorial

Note

This document is a work in progress. Please feel free to add constructive comments, make additions and edit accordingly.

Introduction

The ADFS IOSlave presents the contents of ADFS floppy disk images to the user by inspecting the ADFS filesystem stored within each image and using the KIOSlave API to return this information to client applications. Although the underlying Python module was written with a command line interface in mind, it provides an interface which can be used by an IOSlave without too much of an overhead.

This document outlines the source code and structure of the ADFS IOSlave and aims to be a useful guide to those wishing to write IOSlaves, either in Python or C++.

Annotated code

It is convenient to examine the source code as it is written in the kio_adfs.py file. However, we can at least group some of the methods in order to provide an overview of the IOSlave and separate out the details of the initialisation.

Initialisation

Various classes and namespaces are required in order to communicate with the KIOSlave framework. These are drawn from the qt, kio and kdecore modules:

from qt import QByteArray
from kio import KIO
from kdecore import KURL

The os and time modules provide functions which are relevant to the operation of the IOSlave; the sys and traceback modules are useful for debugging the IOSlave:

import os, sys, traceback, time

The ADFSlib module is imported. This provides the functionality required to read disk images:

import ADFSlib

We omit the debugging code to keep this tutorial fairly brief. This can be examined in the distributed source code.

The slave class

We define a class which will be used to create instances of this IOSlave. The class must be derived from the KIO.SlaveBase class so that it can communicate with clients via the DCOP mechanism. Various operations are supported if the appropriate method (represented as virtual functions in C++) are reimplemented in our subclass.

Note that the name of the class is also declared in the details.py file so that the library which launches the Python interpreter knows which class to instantiate.

class SlaveClass(KIO.SlaveBase):

    """SlaveClass(KIO.SlaveBase)
    
    See kdelibs/kio/kio/slavebase.h for virtual functions to override.
    """

An initialisation method, or constructor, is written which calls the base class and initialises some useful attributes, or instance variables. Note that the name of the IOSlave is passed to the base class's __init__ method:

    def __init__(self, pool, app):
    
        # We must call the initialisation method of the base class.
        
        KIO.SlaveBase.__init__(self, "adfs", pool, app)
        
        # Initialise various instance variables.
        
        self.host = ""
        self.disc_image = None
        self.adfsdisc = None

We create a method to parse any URLs passed to the IOSlave and return a path into a disk image. This initially extracts the path from the KURL object passed as an argument and converts it to a unicode string:

    def parse_url(self, url):
    
        file_path = unicode(url.path())

The presence of a colon character is determined. If one is present then it will simply be discarded along with any preceding text; the remaining text is assumed to be a path to a local file.

        at = file_path.find(u":")
        
        if at != -1:
        
            file_path = file_path[at+1:]

Since we are implementing a read-only IOSlave, we can implement a simple caching system for operations within a single disk image. If we have cached a URL for a disk image then we check whether the URL passed refers to an item beneath it. This implies that the cached URL is a substring of the one given.

If the disk image has been cached then return the path within the image:

        if self.disc_image and file_path.find(self.disc_image) == 0:
        
            # Return the path within the image.
            return file_path[len(self.disc_image):]

An uncached URL must be examined element by element, as far as possible, comparing the path with the local filesystem. Since a valid path will contain at least one slash character then we can immediately discard any paths which do not contain one, returning None to the caller:

        elements = file_path.split(u"/")
        
        if len(elements) < 2:
        
            return None

Starting from the root directory, we apply each new path element to the constructed path, testing for the presence of the objects it refers to. If no object can be found at the path given then None is returned to the caller to indicate that the URL was invalid. If a file is found then it is assumed that an ADFS disk image has been located; a check could be performed to verify this. Finally, if all the path elements are added to the root directory, and the object referred to is itself a directory, then the URL is treated as invalid; it should have referred to a file.

        path_elements, elements = elements[:1], elements[1:]
        
        while elements != []:
        
            path_elements.append(elements.pop(0))
            
            path = u"/".join(path_elements)
            
            if os.path.isfile(path):
            
                break
            
            elif elements == [] and os.path.isdir(path):
            
                return None
            
            elif not os.path.exists(path):
            
                return None

At this point, it is assumed that a suitable image has been found at the constructed path. The characters following this path correspond to the path within the image file. We record the path to the image and construct the path within the image:

        self.disc_image = path
        
        image_path = u"/".join(elements)

If not already open, it is necessary to open the image file, returning None if the file cannot be found. (Its presence was detected earlier but it is better to catch any exceptions.)

        try:
        
            adf = open(self.disc_image, "rb")
        
        except IOError:
        
            return None

We attempt to open the disk image using a class from the support module. This will read and catalogue the files within the image, storing them in an internal structure. However, if a problem is found with the image, then an exception will be raised. We tidy up and return None to signal failure in such a case, but otherwise return the path within the image:

        try:
        
            self.adfsdisc = ADFSlib.ADFSdisc(adf)
        
        except ADFSlib.ADFS_exception:
        
            adf.close()
            return None
        
        return image_path

The get file operation

Various fundamental operations are required if the IOSlave is going to perform a useful function. The first of these is provided by the get method which reads files in the disk image and sends their contents to the client. The first thing this method does is check the URL supplied using the previously defined parse_url method, reporting an error if the URL is unusable:

    def get(self, url):
    
        path = self.parse_url(url)
        
        if path is None:
        
            self.error(KIO.ERR_DOES_NOT_EXIST, url.path())
            return

Having established that the disk image referred to is valid, we now have a path which is supposed to refer to a file within the image. It is now necessary to attempt to find this file. This is achieved by the use of the as yet undeclared find_file_within_image method which will return None if a suitable file cannot be found:

        adfs_object = self.find_file_within_image(path)
        
        if not adfs_object:
        
            self.error(KIO.ERR_DOES_NOT_EXIST, path)
            return

Since, at this point, an object of some sort was located within the image, we need to check whether it is a directory and return an error if so.

The details of the object returned by the above method is in the form of a tuple which contains the name, the file data and some other metadata.

If the second element in the tuple is a list then the object found is a directory:

        if type(adfs_object[1]) == type([]):
        
            self.error(KIO.ERR_IS_DIRECTORY, path)
            return

For files, the second element of the tuple contains a string. In this method, we are only interested in the file data. Using the base class's data method, which we can access through the current instance, we send a QByteArray to the client:

        self.data(QByteArray(adfs_object[1]))

The end of the data string is indicated by an empty QByteArray before we indicate completion of the operation by calling the base class's finished method:

        self.data(QByteArray())
        
        self.finished()

The stat operation

The stat method returns information about files and directories within the disk image. It is very important that this method works properly as, otherwise, the IOSlave will not work as expected and may appear to be behaving in an unpredictable manner. For example, clients such as Konqueror often use the stat method to find out information about objects before calling get, so failure to read a file may actually be the result of a misbehaving stat operation.

As for the get method, the stat method initially verifies that the URL supplied is referring to a valid disk image, and that there is a path within the image to use. Unlike the get method, it will redirect the client to the path contained within the URL if the parse_url fails to handle it. This allows the user to use URL autocompletion on ordinary filesystems while searching for images to read.

    def stat(self, url):
    
        path = self.parse_url(url)
        
        if path is None:
        
            # Try redirecting to the protocol contained in the path.
            redir_url = KURL(url.path())
            self.redirection(redir_url)
            self.finished()
            #self.error(KIO.ERR_DOES_NOT_EXIST, url.path())
            return

As before, non-existant objects within the image cause errors to be reported:

        adfs_object = self.find_file_within_image(path)
        
        if not adfs_object:
        
            self.error(KIO.ERR_DOES_NOT_EXIST, path)
            return

In the tuple containing the object's details the second item may be in the form of a list. This would indicate that a directory has been found which we must deal with appropriately. However, for ordinary files we simply generate a suitable description of the file to return to the client:

        if type(adfs_object[1]) != type([]):
        
            entry = self.build_entry(adfs_object)

If the object was not a file then we must ensure that the path given contains a reference to a directory. If, on the other hand, the path is either empty or does not end in a manner expected for a directory then it is useful to redirect the client to an appropriate URL:

        elif path != u"" and path[-1] != u"/":
        
            # Directory referenced, but URL does not end in a slash.
            
            url.setPath(unicode(url.path()) + u"/")
            self.redirection(url)
            self.finished()
            return

If the URL referred to a directory then a description can be returned to the client in a suitable form:

        else:
        
            entry = self.build_entry(adfs_object)

After a description of the object found has been constructed, it only remains for us to return the description (or entry in the filesystem) to the client by submitting it to the KIOSlave framework. This is performed by the following operation to the statEntry method, and is followed by a finished call to indicate that there are no more entries to process:

        if entry != []:
        
            self.statEntry(entry)
            
            self.finished()
        
        else:
        
            self.error(KIO.ERR_DOES_NOT_EXIST, path)

The mimetype operation

In many virtual filesystems, the mimetype operation would require a certain amount of work to determine MIME types of files, or sufficient planning to ensure that data is returned in a format in line with a predetermined MIME type. Since its use is optional, we do not define a method to determine the MIME types of any files within our virtual filesystem. The client will have to inspect the contents of such files in order to determine their MIME types.

The listDir operation

The contents of a directory on our virtual filesystem are returned by the listDir method. This works like the stat method, but returns information on multiple objects within a directory.

As for the previous methods the validity of the URL is checked. If no suitable directory found within the disk image, the path component of the original URL is extracted and the client redirected to this location. Note that the url argument is a kdecore.KURL object.

{{{ def listDir(self, url):

• path = self.parse_url(url) if path is None:
  • redir_url = KURL(url.path()) self.redirection(redir_url) self.finished() return

}}}

Having established that the path refers to a valid disk image, we try to find the specified object within the image, returning an error if nothing suitable is found:

        adfs_object = self.find_file_within_image(path)
        
        if not adfs_object:
        
            self.error(KIO.ERR_DOES_NOT_EXIST, path)
            return

If the path does not end in a slash then redirect the client to a URL which does. This ensures that either a directory will be retrieved or an error will be returned to the client:

        elif path != u"" and path[-1] != u"/":
        
            url.setPath(unicode(url.path()) + u"/")
            self.redirection(url)
            self.finished()
            return

If a file is referenced then an error is returned to the client because we can only list the contents of a directory:

        elif type(adfs_object[1]) != type([]):
        
            self.error(KIO.ERR_IS_FILE, path)
            return

A list of files is kept in the second item of the object returned from the support module. For each of these files, we must construct an entry which is understandable to the KIOSlave infrastructure in a manner similar to that used in the method for the stat operation.

        # Obtain a list of files.
        files = adfs_object[1]
        
        # Return the objects in the list to the application.
        
        for this_file in files:
        
            entry = self.build_entry(this_file)
            
            if entry != []:
            
                self.listEntry(entry, 0)
                
                # For old style disk images, return a .inf file, too.
                if self.adfsdisc.disc_type.find("adE") == -1:
                
                    this_inf = self.find_file_within_image(
                        path + "/" + this_file[0] + ".inf"
                        )
                    
                    if this_inf is not None:
                    
                        entry = self.build_entry(this_inf)
                        
                        if entry != []:
                        
                            self.listEntry(entry, 0)
        
        # We have finished listing the contents of a directory.
        self.listEntry([], 1)
        
        self.finished()

The dispatch loop

Although not entirely necessary, we implement a dispatchLoop method which simply calls the corresponding method of the base class:

    def dispatchLoop(self):
    
        KIO.SlaveBase.dispatchLoop(self)

Utility methods

We define some methods which, although necessary for this IOSlave to work, are not standard virtual methods to be reimplemented. However, they do contain code which might be usefully reused in other IOSlaves.

Building file system entries

We create a method to assist in building filesystem entries to return to the client via the KIOSlave infrastructure. For this example, some basic details of each file or directory in the disk image is derived from information contained within and stored within standard KIO.UDSAtom instances.

    def build_entry(self, obj):
    
        entry = []

We check the type of object passed to the method in order to determine the nature of the information returned. For files we do not provide a predetermined MIME type, leaving the client to determine this from data retrieved from the disk image.

        if type(obj[1]) != type([]):
        
            # [name, data, load, exec, length]
            name = self.encode_name_from_object(obj)
            length = obj[4]

Files stored in old disk images require accompanying .inf files to describe their original attributes. The following code provides details of these files which are not actually present in the disk image, but are generated automatically by this IOSlave:

            if self.adfsdisc.disc_type.find("adE") == -1 and \\
                obj[0][-4:] == ".inf":
            
                # For .inf files, use a MIME type of text/plain.
                mimetype = "text/plain"
            
            else:
            
                # Let the client discover the MIME type by reading
                # the file.
                mimetype = None
        
        else:
        
            name = self.encode_name_from_object(obj)
            length = 0
            mimetype = "inode/directory"

Having determined the MIME type we now declare all the relevant attributes of the object and return these to the caller:

        atom = KIO.UDSAtom()
        atom.m_uds = KIO.UDS_NAME
        atom.m_str = name
        
        entry.append(atom)
        
        atom = KIO.UDSAtom()
        atom.m_uds = KIO.UDS_SIZE
        atom.m_long = length
        
        entry.append(atom)
        
        atom = KIO.UDSAtom()
        atom.m_uds = KIO.UDS_MODIFICATION_TIME
        # Number of seconds since the epoch.
        atom.m_long = int(time.time())
        
        entry.append(atom)
        
        atom = KIO.UDSAtom()
        atom.m_uds = KIO.UDS_ACCESS
        # The usual octal permission information (rw-r--r-- in this case).
        atom.m_long = 0644
        
        entry.append(atom)
        
        # If the stat method is implemented then entries _must_ include
        # the UDE_FILE_TYPE atom or the whole system may not work at all.
        
        atom = KIO.UDSAtom()
        atom.m_uds = KIO.UDS_FILE_TYPE
        
        if mimetype != "inode/directory":
        
            atom.m_long = os.path.stat.S_IFREG
        
        else:
        
            atom.m_long = os.path.stat.S_IFDIR
        
        entry.append(atom)
        
        if mimetype:
        
            atom = KIO.UDSAtom()
            atom.m_uds = KIO.UDS_MIME_TYPE
            atom.m_str = mimetype
            
            entry.append(atom)
        
        return entry

Encoding filenames

The following two internal methods deal with the translation of paths and filenames within the disk image to and from canonical URL style paths. These are only of interest to those familiar with ADFS style paths.

    def encode_name_from_object(self, obj):
    
        name = obj[0]
        
        # If the name contains a slash then replace it with a dot.
        new_name = u".".join(name.split(u"/"))
        
        if self.adfsdisc.disc_type.find("adE") == 0:
        
            if type(obj[1]) != type([]) and u"." not in new_name:
            
                # Construct a suffix from the object's load address/filetype.
                suffix = u"%03x" % ((obj[2] >> 8) & 0xfff)
                new_name = new_name + "." + suffix
        
        return unicode(KURL.encode_string_no_slash(new_name))
    
    def decode_name(self, name):
    
        return unicode(KURL.decode_string(name))

Locating objects within a disk image

A key element in the construction of an IOSlave is the method used to map between the URLs given by client applications and the conventions of the virtual filesystems represented by the IOSlave. In this instance, the disk image contains a working snapshot of an ADFS filesystem which must be navigated in order to extract objects referenced by the client.

Since the ADFSlib support module provides objects to contain the directory structure contained within the disk image, only a minimal amount of work is required to locate objects, and this mainly involves a recursive examination of a tree structure. However, there are a few special cases which are worth mentioning.

In this method, the path argument contains the path component of the URL supplied by the client in the standard form used in URLs.

    def find_file_within_image(self, path, objs = None):

A convention we have adopted is the use of a default value of None for the final argument of this method. Omission of this argument indicates that we are starting a search from the root directory of the disk image. As we descend into the directory structure, recursive calls to this method will supply suitable values for this argument but, for now, a reasonable value needs to be substituted for None; this is a structure containing the entire filesystem:

        if objs is None:
        
            objs = self.adfsdisc.files

If an empty path was supplied then it is assumed that an object corresponding to the root directory was being referred to. In such a case a slash character is given as the name of the object, and the list of objects supplied is given as the contents of the root directory:

        if path == u"":
        
            # Special case for root directory.
            return [u"/", objs, 0, 0, 0]

For non-trivial paths, we split the path string into elements corresponding to the names of files and directories expected as we descend into the filesystem's hierarchy of objects, then we remove any empty elements:

        elements = path.split(u"/")
        
        elements = filter(lambda x: x != u"", elements)

For each object found in the current directory, we examine each object and compare its name to the next path element expected.

        for this_obj in objs:
        
            if type(this_obj[1]) != type([]):
            
                # A file is found. 

If a file is found in the filesystem, we translate its name so that we can compare it with the next path element expected:

                obj_name = self.encode_name_from_object(this_obj)
                
                if obj_name == elements[0]:
                
                    # A match between names.

For files, we perform the simple test that the current path element is the final one in the list, and return the corresponding object if this is the case. If this is not the case then the URL is likely to be invalid:

                    if len(elements) == 1:
                    
                        # This is the last path element; we have found the
                        # required file.
                        return this_obj
                    
                    else:
                    
                        # There are more elements to satisfy but we can
                        # descend no further.
                        return None

If a direct match between names is not possible then, for old-style disk images, it is possible that the path is referring to a .inf file; we check for this possibility, applying the same check on the remaining path elements as before:

                elif self.adfsdisc.disc_type.find("adE") == -1 and \\
                     elements[0] == obj_name + u".inf":

If successfully matched, a .inf file is created and returned, otherwise a None value is returned to indicate failure:

                    if len(elements) == 1:
                    
                        file_data = "%s\t%X\t%X\t%X\n" % \\
                            tuple(this_obj[:1] + this_obj[2:])
                        
                        new_obj = \\
                        (
                            this_obj[0] + ".inf", file_data,
                            0, 0, len(file_data)
                        )
                        
                        return new_obj
                    
                    else:
                    
                        # There are more elements to satisfy but we can
                        # descend no further.
                        return None
            
            else:
            
                # A directory is found.

As for files, the names of directories found in the filesystem are translated for comparison with the next path element expected:

                obj_name = self.encode_name_from_object(this_obj)
                
                if obj_name == elements[0]:
                
                    # A match between names.

Unlike files, directories can occur at any point in the descent into the filesystem. Therefore, we either return the object corresponding to the last path element or descend into the directory found:

                    if len(elements) == 1:
                    
                        # This is the last path element; we have found the
                        # required file.
                        return this_obj
                    
                    else:
                    
                        # More path elements need to be satisfied; descend
                        # further.
                        return self.find_file_within_image(
                            u"/".join(elements[1:]), this_obj[1]
                            )

At this point, no matching objects were found, therefore we return None to indicate failure:

        return None