EntityFS programmer's guide

Revision History
Revision 1.1.12009-03-16

Table of Contents

1. Introduction
Please help!
License and Copyright
Support
2. Getting started
Requirements
Downloading EntityFS
Installing EntityFS
Running unit tests (optional)
Running performance tests (optional)
3. What is a file, anyway?
Lockable
ReadLockable
WriteLockable
ReadWriteLockable
Named
Deletable
ReadableFile
NamedReadableFile
WritableFile
RandomlyAccessibleFile
ReadWritableFile
EntityHolder
EntityListable
EntityRepository
4. File systems
Creating a file system
Tuning file system parameters
Closing a file system
5. Entities
Entity locations
Entity
Directory
Directory views
EFile
6. Design principles
Exceptions and error handling
Logging
7. Locking and multi-threaded access
Locks
Locking strategies
Lock commands
Custom locks
Interrupting threads
Troubleshooting
8. Utility classes
Entity utility classes
Recursive iterators
Iterator utilities
Entity filters
Property utilities
Adapters
Input and output stream implementations
Other utility classes
9. Entity events
Polling observable
10. Capabilities
11. Metadata support
12. Entity attributes
Unix attributes
NTFS attributes
Custom attributes
13. Access controls
The access granter access controller
The Unix entity mode access controller
14. Zip and Jar files
Zip file systems
Building Zip files
Building Jar files
Reading from Zip and Jar files
A. Java IO to EntityFS and back again
From File to entity
From entity to File
Bibliography

List of Figures

5.1. Simplified inheritance chart for entity objects

List of Tables

3.1. Important ReadableFile implementations
3.2. ReadableFile adapters
3.3. Important NamedReadableFile implementations
3.4. NamedReadableFile adapters
3.5. Important WritableFile implementations
3.6. WritableFile adapters
3.7. Important RandomlyAccessibleFile implementations
3.8. RandomlyAccessibleFile adapters
3.9. ReadWritableFile adapters
3.10. Important EntityHolder implementations
3.11. Important EntityListable implementations
3.12. Important EntityRepository implementations
4.1. File system implementations
7.1. Lock types
7.2. Lock acquiring strategies
7.3. Lock commands
8.1. Entity utility classes
8.2. Recursive iterators
8.3. Iterator utilities
8.4. Entity filters
8.5. Adapter implementations
8.6. InputStream implementations
8.7. OutputStream implementations
8.8. RandomAccess implementations
8.9. Other utility classes
9.1. EntityEvent types
10.1. Entity capabilities
10.2. File system capabilities
10.3. Capability providers
11.1. Metadata capability providers
A.1. File adapters

List of Examples

4.1. Creating a file system-backed file system
4.2. Creating a Jar file-backed file system on a file in another file system
5.1. Moving an entity in a locking file system
5.2. Moving an entity using utility methods
5.3. Creating a file in the root directory
5.4. Creating a file in the root directory using Directories
5.5. Getting an entity in a subdirectory
5.6. Listing entities in a directory
5.7. Listing entities in a directory using Directories
5.8. A directory view
5.9. Using a Directory directly in a for loop
5.10. Using a DirectoryView directly in a for loop
5.11. Using a depth-first iterator
5.12. Views settings are inherited
5.13. Creating and opening a file
5.14. Creating and writing to a text file
5.15. Using a lock-aware stream
7.1. Using try-finally for locks
7.2. Releasing several locks in finally
7.3. Using a LockCommandExecutor
8.1. Creating a new file
8.2. Holding a write lock for several utility method calls
8.3. Simple IteratorDeleter implementation
8.4. An iterator for iterating recursively over XML files
8.5. Working with properties
8.6. Synchronizing directory hierarchies
9.1. Registering an observer on a directory
9.2. Adding a file to a directory
9.3. Modifying a file
9.4. Moving an entity
9.5. Using a polling directory monitorer to watch for incoming XML files
10.1. Using Unix entity attributes to get an entity's creation time
10.2. Using Unix entity attributes utility class to get an entity's creation time
10.3. Creating a Ram file system with gzip compression of file data
11.1. Creating a metadata enabled file system
11.2. Creating a file system with metadata support for directories
11.3. Setting a metadata property on an entity
11.4. Reading data from a named metadata file
12.1. Using the custom entity attributes capability
13.1. A custom access granter
13.2. The Unix entity mode access controller on a non-locking file system
13.3. The Unix entity mode access controller on a locking file system
14.1. Building a Zip file
14.2. Building a Zip file from files in two directory hierarchies
14.3. Building a Jar file
14.4. Reading entities from a Zip file
14.5. Unzipping a Zip file
A.1. Creating a custom file system on the File
A.2. Creating a file system on the File using a FileSystems method
A.3. Getting an existing Entity using a FSCFileResolvableUtil method
A.4. Adapting a File to the ReadableFile interface
A.5. Getting a File object referencing an entity

EntityFS is a Java library for working with file system entities – files and directories – in an object-oriented fashion. For an introduction to EntityFS, see the Architecture overview.

This book is the Programmer's guide. It is written for programmers who want to use EntityFS in their applications. It gives an overview of EntityFS concepts and features, with examples and pointers to the API documentation, which serves as the reference documentation.

Get EntityFS from http://www.entityfs.org.

There are two different distributions of EntityFS – the binary distribution and the source distribution. The binary distribution contains everything necessary for using EntityFS. The source distribution contains everything that the binary archive does as well as unit test classes, test data and the complete EntityFS source code in an Eclipse workspace.

The unit test suite runs all EntityFS unit tests for many capability permutations for two different file systems. The total number of permutations for this release is 14534, so all unit tests take a long time to run – approximately four to five hours on a dual core CPU.

To run unit tests, you have to have the build tool Schmant installed, the EntityFS source distribution and some six gigabytes free space on the hard disk partition where the platform's temporary directory (java.io.tmpdir) is.

Open a command window (terminal, cmd) and change directory to the distribution's build directory. Set the JAVA_HOME environment variable to point to a Java JDK 6 installation. (Just a JRE won't do.)

On Unix, run:

$ # For instance
$ export JAVA_HOME=/opt/java6
$ schmant.sh -cp lib/entityfs_build.jar run_unit_tests.js

On Windows, run:

> rem For instance. Note the absence of quotes in JAVA_HOME
> set JAVA_HOME=c:\Program Files\Java\jdk1.6.0_24
> schmant -cp lib\entityfs_build.jar run_unit_tests.js

To use one or several other JVM:s than the system's default for running unit tests, set the javaCmds property to a path separator (: on Unix, ; on Windows) separated list of all Java installations to use.

$ # For instance
$ export JAVA_HOME=/opt/java6
$ schmant.sh -cp lib/entityfs_build.jar \
  -p javaCmds=/opt/java6/bin/java:/opt/jrockit6/bin/java run_unit_tests.js

Or, on Windows:

> rem For instance. Note the absence of quotes
> set JAVA_HOME=c:\Program Files\Java\jdk1.6.0_24
> rem Note the quotes around javaHomes
> schmant -cp lib\entityfs_build.jar \
  -p "javaCmds=c:\Program\Java\jdk1.6.0_24\bin\javaw.exe;c:\Program\Java\jdk1.5.0_22\bin\javaw.exe" \
  run_unit_tests.js

Don't be scared by all "errors" about disappeared files or files that will not be overwritten. They are from the tests that test that those conditions are handled correctly.

On Windows, anti-virus software may interfere with the tests. Run the tests with the anti-virus software disabled.

Well, what is a file, anyway? The answer is, of course, it depends. It depends on what you intend it to do. Sometimes a file is just a container of binary data that a client can read from, and sometimes it is a container of data with a unique identity that can be manipulated and moved around in the context of a file system.

EntityFS supports several different ways of viewing files and directories, all depending on what the client intends to do with them. This chapter describes the simple, low-level file and directory interfaces, Chapter 4, File systems describes what a file system is in EntityFS and Chapter 5, Entities describes what it means when a file or a directory is an entity.

Clients are encouraged to take the simplest possible view on the objects that they use. If a client just wants to read data from a file, it should use a ReadableFile instead of the full file entity (EFile). This makes mocking and testing much easier.

The sections below contain a walkthrough through the simple low-level interfaces for files and directories. Graphical inheritance charts are in Inheritance chart for EFile, Inheritance chart for Directory and Inheritance chart for all entity objcets.

Implemented by: Directory, EFile

Lockable is a marker interface that identifies the implementing object as being capable of locking for exclusive access by a thread or a process somehow.

Entity locking is described in Chapter 7, Locking and multi-threaded access.

Implements: Lockable.

Implemented by: Directory, EFile

ReadLockable gives the implementing object the capability of being locked for exclusive read access by a thread, several reading threads or a process. It is up to the implementing object to decide whether to actually support locking and, if so, the locking semantics to use.

Entity locking is described in Chapter 7, Locking and multi-threaded access.

Implements: Lockable.

Implemented by: Directory, EFile

WriteLockable gives the implementing object the capability of being locked for exclusive write access by a thread or a process. It is up to the implementing object to decide whether to actually support locking and, if so, the locking semantics to use.

Entity locking is described in Chapter 7, Locking and multi-threaded access.

Implements: Lockable, ReadLockable.

Implemented by: Directory, EFile

Named is implemented by objects that has a name that can be read, such as a file name.

Implements: Lockable, WriteLockable.

Implemented by: Directory, EFile

Deletable is implemented by objects that can be deleted.

The FileSystem interface represents a file system. It contains a directory hierarchy containing EFile and Directory entities, and it has a set of policy and strategy objects. It always holds a reference to the root Directory.

A FileSystem is created from a FileSystemBuilder object[1].


All setter methods of a FileSystemBuilder return the builder object. This makes it possibly to chain setter method invocations together as is shown in many of the examples below.

File systems have an optional name that, if set, should be unique within the current system. [2] The name can be used by applications to identify different file system instances. It is not used by EntityFS at all.

Example 4.1. Creating a file system-backed file system

// Create a file system with default settings except for a custom name
FileSystem fs1 = new FSRWFileSystemBuilder().
  setName("My first file system").
  setRoot(new File("/tmp/fs1")).
  create();

// Create a file system with no event handling and relaxed validity controls
// for entities. (See FSRWFileSystemBuilder API docs.)
FileSystem fs2 = new FSRWFileSystemBuilder().
  setName("My second file system").
  setRoot(new File("/tmp/fs2")).
  disableEntityValidityControls().
  // Enable entity locking
  enableLocking().
  create();

Example 4.2. Creating a Jar file-backed file system on a file in another file system

// fs is another file system
EFile jarFile = Directories.getFile(fs.getRootDirectory(), "myapp.jar");

FileSystem jfs = new JarFileSystemBuilder().
  setName("My Jar file system").
  setJarFile(jarFile).
  create();

By default, file systems are created with a minimal set of features. This can be tuned by configuring the file system builder differently. Also, the default validity control settings for the file system may be more defensive than what is required by the client. By setting less safe values, the file system's performance may increase somewhat.

The following is a list of common file system builder properties:

accessController

A file system can use an AccessController to limit access to entities. It can be used by applications to implement their access control policies. See Chapter 13, Access controls. Access controls are disabled by default.

bufferSize

The size of temporary in-memory buffers used by the file system. Different file system implementations use this for different things. The default value is 8192 bytes.

entityValidityControlStrategy

The strategy used for verifying that entities are still present in the backend whenever they are used. The default strategy for this is file system-specific. If it is known when the file system is created that no other processes will ever modify its contents, this can be disabled by calling disableEntityValidityControls on the file system builder.

events

By calling enableEvents, a client can make the entire FileSystem as well as individual entities Observable for EntityEvent:s. See Chapter 9, Entity events.

lockAcquiringStrategy

The strategy for acquiring file system EntityLock:s. See Chapter 7, Locking and multi-threaded access. Entity locking is disabled by default. It can be enabled by calling enableLocking on the file system builder.

lockCommandExecutor

Utility class for locking entity locks in accordance with the file system's locking strategy. Entity locking is disabled by default. It can be enabled by calling enableLocking on the file system builder. See Chapter 7, Locking and multi-threaded access.

lockAdapterFactory

A factory used for creating locks. Entity locking can be enabled by calling enableLocking on the file system builder. See Chapter 7, Locking and multi-threaded access.

Different file system implementations may also have their own, implementation-specific tunable parameters. For details, see the API documentation for each file system implementation.



[1] The builder design pattern is discussed in for example GoF.

[2] The definition of the word system is application-specific; it might be a single JVM or a cluster of application servers.

A file system contains Entity:s such as files and directories. Entity objects are entities in the object-oriented sense of the word[3]. This means that, for instance for EFile (the same holds for Directory), as far as the client application is concerned, is the file itself. At any given time, in one FileSystem, there can only be one EFile object referencing that file. If the file is moved or renamed, it is still represented by the same EFile object. Compare this with how Java's File works.

An entity object always lives within the context of one and the same FileSystem. It cannot be moved to another file system.

The methods of file entities are defined by the EFile interface and the interfaces that it inherits. Directory methods are defined by the Directory interface and its inherited interfaces. The super interfaces of EFile and Directory each capture some aspect of files or directories. Just like when working with the Java Collections API, clients are recommended to use the simplest possible interface that meets their needs. For instance, a client just wanting to read from a file can use an implementation of the ReadableFile interface instead of the full EFile. This makes the client code easier to test since it is easier to replace a ReadableFile with a mock implementation. See Chapter 3, What is a file, anyway? and Figure 5.1, “Simplified inheritance chart for entity objects”.

Some entity types (Directory:s) are ViewCapable. For a Directory, this means that a client can create a view of the directory that uses one or several entity Filter:s to hide some of its child entities. EntityView:s such as the DirectoryView are not entities like the Entity objects; there can be several different active DirectoryView:s for a Directory entity, for instance.


CapabilityProvider:s may give entities or file systems extra capabilities, such as metadata support or support for entity attributes. See Chapter 10, Capabilities.

Note

Many of the examples in this and the following chapters do entity locking as described in Chapter 7, Locking and multi-threaded access.

An entity's AbsoluteLocation, its absolute path, is its unique location in its file system. An entity also has a different RelativeLocation relative to every directory in its file system.

The Entity interface is what identifies an object as an entity, with all the properties discussed above. It defines all common properties and methods for entity objects. It is augmented by static utility methods implemented in the Entities and EntityViews classes.

For the complete interface inheritance chart, see the Inheritance chart for all entity objcets.

Example 5.1. Moving an entity in a locking file system

// d1 and d2 are directories, e1 is an entity of any type that currently is in
// the directory d1.
Collection<EntityLock> locks = new ArrayList<EntityLock>();
try
{
  locks.add(d1.lockForWriting());
  
  // The write lock on d1 is also a read lock. Get the entity to move.
  EntityView e1 = d1.getEntityOrNull("e1");
  locks.add(e1.lockForWriting());
  locks.add(d2.lockForWriting());
  
  // Move the entity
  e1.rename(d2, "e1");
}
finally
{
  // Use the utility class EntityLocks to unlock all locks
  EntityLocks.unlockAll(locks);
}
// The entity is still represented by the variable e1, even though it has
// moved.


Concurrency note:

In the second example, no locks are held on d1 between e1 is retrieved and moved. Another thread may move or even delete e1 between the calls to getEntity and move. If this is a real risk (this depends on the application), lock d1 for writing before calling getEntity and release the lock after calling move. Locks are reentrant, so the locking performed by the utility classes will not be disturbed by that. See Chapter 7, Locking and multi-threaded access.

A Directory is a container for other entities, child entities. Basic directory operations are defined in the EntityHolder and EntityListable interfaces.

For the Directory inheritance chart, see the Inheritance chart for Directory.

The utility class Directories implements utility methods for working with directories.

Example 5.3. Creating a file in the root directory

// fs is a locking file system. Get the root directory
Directory root = fs.getRootDirectory();

EFile myFile;

// Lock the root directory for writing
EntityLock wl = root.lockForWriting();
try
{
  // Create an entity of the type ETFile.TYPE called myFile
  myFile = (EFile) root.newEntity(ETFile.TYPE, "myFile", null);
}
finally
{
  // Unlock the directory
  wl.unlock();
}

Example 5.4. Creating a file in the root directory using Directories

// fs is a file system. Get the root directory
Directory root = fs.getRootDirectory();

// Create a file called "myFile".
EFile myFile = Directories.newFile(root, "myFile");

Example 5.5. Getting an entity in a subdirectory

// d is a directory
// Create a directory under d with an empty file in it. This invocation
// returns a reference to the file, but we ignore it for the sake
// of the example
Directories.newFile(org.entityfs.util.Directories.newDirectory(d, "d"), "f");

// Directories.getDirectory returns a DirectoryView. Since the argument d
// is a Directory, the result can safely be cast to a Directory.
Directory dd = (Directory) Directories.getDirectory(d, "d");

// (We could have used the entity reference returned by newFile above.)
EFile f = Directories.getFile(d, new RelativeLocation("d/f"));

Example 5.6. Listing entities in a directory

Set<EntityView> s;
// d is a directory in a locking file system. Lock it for reading.
EntityLock rl = d.lockForReading();
try
{
  s = d.listEntities();
}
finally
{
  // Unlock the directory.
  rl.unlock();
}


A DirectoryView is a directory viewed through one or several entity Filter:s. The filter hides all child entities that do not match the filter criteria. Read more about entity filters in the section called “Entity filters”.

Example 5.8. A directory view

// d is a directory
EFile f1 = Directories.newFile(d, "f1");
DirectoryView d1 = Directories.newDirectory(d, "d1");

// Create a view that only shows files
DirectoryView dv = d.newView(EFileFilter.FILTER);
Set<EntityView> s = Directories.listEntities(dv);

The returned set will only contain f1.

The Directory interface inherits DirectoryView, so all directories can be considered to be views of themselves with no filters.

All EntityListable objects, including directories and directory views, are Iterable<EntityView>. They have an iterator() method that returns an Iterator<EntityView> for iterating over all visible child entities. This means that directories can be used directly in a for loop like in the examples below.

Example 5.9. Using a Directory directly in a for loop

// d is a directory
EntityLock rl = d.lockForReading();
try
{
  for (EntityView ev : d)
  {
    // process child entities...
  }
}
finally
{
  rl.unlock();
}

Example 5.10. Using a DirectoryView directly in a for loop

// d is a directory (or a view)
// Create a view that only shows XML files.
DirectoryView dv = d.newView(new EFileNameExtensionFilter("xml"));

// Even though this is called on the view, it locks the viewed directory
Lock rl = dv.lockForReading();
try
{
  for (EntityView ev : dv)
  {
    EFile xmlFile = (EFile) ev;
    // process the XML files...
  }
}
finally
{
  rl.unlock();
}

There are utility classes that implement other types of entity iterators, for instance recursive iterators over entity hierarchies. Read more about them in the section called “Recursive iterators” and the section called “Iterator utilities”.

Example 5.11. Using a depth-first iterator

// d is a directory
Directory dd = (Directory) Directories.newDirectory(d, "d");
Directory ddd = (Directory) Directories.newDirectory(dd, "d");
EFile f = Directories.newFile(ddd, "f");

Iterator<EntityView> itr = Directories.getDepthFirstIterator(d);

The iterator will return the entities f, ddd, dd and d, in that order.


View settings are inherited. If a view capable child entity is retrieved from a directory view, the returned object is a view of that entity with the same filters (filter object instances) as the directory view.

Example 5.12. Views settings are inherited

// d is a directory
EFile f1txt = Directories.newFile(d, "f1.txt");
EFile f1xml = Directories.newFile(d, "f1.xml");
Directory d1 = (Directory) Directories.newDirectory(d, "d1");
EFile d1f1txt = Directories.newFile(d1, "d1f1.txt");
EFile d1f1xml = Directories.newFile(d1, "d1f1.xml");

// Create a view that only shows directories and XML files
DirectoryView dv = d.newView(
  DirectoryFilter.FILTER.
    or(new EFileNameExtensionFilter("xml")));

Iterator<EntityView> itr = Directories.getDepthLastIterator(dv);

itr will return the entities / (view of), f1.xml, d1 (view of) and d1f1.xml (d1 may be returned before f1.xml).


Tip

In the the inherited view example above, a FilteringIterator may be used if you don't want the iterator to return any directories. Create the iterator like this instead:

Iterator itr = new FilteringIterator(
  Directories.getDepthLastIterator(dv),
  DirectoryFilter.FILTER.not());

File entities are containers for byte data. Basic file operations are defined in the ReadableFile, WritableFile and RandomlyAccessibleFile interfaces. They are inherited by the file entity interface, EFile.

For the EFile inheritance chart, see the Inheritance chart for EFile.

Example 5.13. Creating and opening a file

// d is a directory
EFile f = Directories.newFile(d, "f");

// Lock the file for writing
EntityLock wl = f.lockForWriting();
try
{
  OutputStream os = f.openForWrite();
  try
  {
    try
    {
      ... write to os
    }
    finally
    {
      os.close();
    }
  }
  catch (IOException e)
  {
    ... error handling
  }
}
finally
{
  wl.unlock();
}

The class Files contains utility methods for working with file entities.

Example 5.14. Creating and writing to a text file

// d is a directory
EFile f = Directories.newFile(d, "f");
Files.writeText(f, "Ceterum censeo Carthaginem esse delendam");

Sometimes an open input or output stream on a file has to be passed as an argument to another method. If using a locking file system, to make sure that the lock on the file is released when the stream is closed, use the lock-aware LockAwareInputStream or a lock-aware stream from Files.openForRead like in the example below.

Example 5.15. Using a lock-aware stream

// f is a file
myStreamUsingMethod(Files.openForRead(f));
// The openForRead method locks the file for reading. The lock is released when 
// the stream is closed. (And that is a responsibility of the stream-using 
// class.)

Methods that only are interested in files as sources or sinks of data, not of their entity qualities, are encouraged to use the ReadableFile and WritableFile interfaces, just like most of the methods in Files do. They are easier to mock for testing than the EFile interface.



[3] See, for instance, Domain Driven Design.

All EntityFS exceptions inherit FileSystemException, either directly or via some subclass. FileSystemException inherits RuntimeException which makes all EntityFS exceptions unchecked (see the Wikipedia article on checked exceptions).

In general, EntityFS methods do not try to handle any exceptions that they encounter; it is up to the client application to implement a strategy for that. Some classes with longer-running methods, such as the IteratorCopier, support an optional ErrorHandlingStrategy.

Every FileSystem uses a LogAdapter instance for logging. It is an adapter to the client application's logging system.

The following adapters are implemented:

StdOutLogAdapter

Logs to stdout and stderr.

Jdk14LogAdapter

Logs to the Logger with a configurable name. The default name is org.entityfs.log.

CommonsLoggingLogAdapter

Logs to the org.apache.commons.logging.Log with a configurable name. The default name is org.entityfs.log.

Adapters to other logging systems can be created by implementing the LogAdapter interface.

The file system's current log adapter is always referenced by its LogAdapterHolder instance. The same holder object is used throughout the file system's lifetime and it is shared among all of the file system's objects that need to log.

By keeping the log configuration in an object instance, rather than in static class configuration like Commons Logging and JDK 1.4 logging do, the configuration of a LogAdapter instance is only visible to the objects that have a reference to it. This has the advantage of automatically limiting the scope of configuration changes without having to bother with different class loaders, thread locals and such. This is important in environments such as application servers where several different applications may share the same JVM and hence, often, the same class loaders. The obvious drawback is that an instance of the logger object has to be passed around to everyone that needs it. In EntityFS, this is not a big problem since most objects have a reference to their parent file system object anyway, and can get a LogAdapter from there.

EntityFS entities are designed to be used concurrently by several threads. To avoid simultaneous, conflicting operations by different threads, entities can be locked for exclusive access by the thread holding the lock.

Locking is supported by read/write file system, but by default locking is disabled. It can be enabled by calling enableLocking on the FileSystemBuilder.

Locks are either read/write locks (the same semantics as Java's ReadWriteLock) or single locks (the same semantics as Java's synchronized keyword). Locks can also be fair (first come, first served for threads wanting to lock the entity, but with a performance penalty), or non-fair. All lock types are reentrant, so the same lock can safely be locked several times by the thread that owns it.

Every file system uses one lock type and one lock acquiring strategy for all of its entities. The type of locks is determined by the file system's LockAdapterFactory.


How a file system's locks are acquired is determined by its LockAcquiringStrategy:


Entity locks, regardless of their type, implement the EntityLock interface, which is an extension of Java's Lock interface. An EntityLock is aware of the Lockable object that it protects.

The EntityLocks utility class contains static utility methods for working with entity locks.

Entity locks should always be released in a finally clause of a try-finally block. In this way, the locks are always release even in the case of an error happening in the try block.

Example 7.1. Using try-finally for locks

// d is a directory
EntityLock rl = d.lockForReading();
try
{
  d.listEntities();
}
finally
{
  // This will always be called.
  rl.unlock();
}

Example 7.2. Releasing several locks in finally

Collection<EntityLock> locks = new ArrayList<EntityLock>();
try
{
  // d1 and d2 are directories
  locks.add(d1.lockForReading());
  locks.add(d2.lockForWriting());
  
  // do stuff...
}
finally
{
  EntityLocks.unlockAll(locks);
}

To avoid deadlocks, every application must have a strategy for in which order entity locks are locked. (This is another strategy than the LockAcquiringStrategy discussed above.) For a discussion on locking strategies, see for instance the book Java concurrency in practice. Entities do not assume any particular locking strategy. Utility classes, however, use a strategy where:

  1. Parent entities are locked before child entities.

  2. Entities in the same directory are locked in hash code order.

LockCommand:s are command objects for locking entities. Several commands may be run by calling the LockCommandExecutor's execute method. The executor makes sure that locks are acquired in a sequence that follows a the lock acquiring strategy that it implements.


A locking FileSystem has its own LockCommandExecutor instance that clients may use to execute LockCommand:s in an order that is compliant with that file system's locking strategy. The default implementation, LockCommandExecutorImpl, implements the utility classes' locking strategy that were described above.

The example below shows how a LockCommandExecutor can be used to lock all locks necessary for moving and then writing to an entity when using an AccessController that require locked entities. (See Chapter 13, Access controls.)

Example 7.3. Using a LockCommandExecutor

// Move the file f1 from directory d1 to directory d2 and append the text
// "moved" to it.
FileSystem fs = f1.getFileSystem();
AccessController ac = fs.getAccessController();
Collection<LockCommand> cmds = 
  new ArrayList<LockCommand>();

// Add the lock commands that are needed for moving the entity.
// Keep this lock command for future reference
LockCommand f1lc = new LockForWritingCommand(f1);
cmds.add(f1lc);
cmds.add(new LockForWritingCommand(d1));
cmds.add(new LockForWritingCommand(d2));

// Add the lock commands that are needed for performing access controls.
// Keep these lock commands for future reference
Collection<? extends LockCommand> f1acmds =
  ac.getLockCommandsForAccessControl(f1);
cmds.addAll(f1acmds);
cmds.addAll(ac.getLockCommandsForAccessControl(d1));
cmds.addAll(ac.getLockCommandsForAccessControl(d2));

// Lock all locks
LockCollection lc = fs.getLockCommandExecutor().execute(cmds);
try
{
  // Move the entity
  f1.rename(d2, f1.getName());
  
  // Keep all locks required for appending text to the file in a new lock 
  // collection.
  LockCollection lc2 = new LockCollection();
  try
  {
    // The write lock on f1
    lc2.addLock(f1lc, lc.removeLock(f1lc));
    // The locks required for access controls on f1
    lc2.addLocks(lc.removeLocks(f1acmds));
    
    // Unlock all other locks
    lc.unlockAll();
    
    Writer w = new OutputStreamWriter(
      f1.openForAppend(),
      Charset.forName("utf8"));
    try
    {
      w.write("moved");
    }
    finally
    {
      w.close();
    }
  }
  finally
  {
    lc2.unlockAll();
  }
}
finally
{
  // It is safe to call unlockAll several times an a LockCollection
  lc.unlockAll();
}

An application can create new locks for an entity in addition to the standard read and write locks by calling its createLock method. This can be used as an alternative to using synchronized or any other locking method and it makes it possible for the application to use the same LockAcquiringStrategy for its own locks as for EntityFS locks.

When creating new locks for an entity, give careful thought about how the new locks relate to the standard read and write locks. For instance, an application may require that a thread must hold at least a read lock on the entity before it lets it lock the custom lock. Without a good strategy, deadlocks may occur.

Entity utility classes contain static methods for working with entities. The entity interfaces are designed to be as small as possible, so they contain just the bare essentials for implementing an entity. They are complemented by utility classes which implement methods to make it easier for client code to work with the entities. Compare this with the role of Java's Collections class.

Example 8.1. Creating a new file

Compare the Directory entity method for creating a new file:

// d is a directory
EFile f;
EntityLock wl = d.lockForWriting();
try
{
  f = (EFile) d.newEntity(ETFile.TYPE, "f", null);
}
finally
{
  wl.unlock();
}

With the Directories utility method:

EFile f = Directories.newFile(d, "f");

In the example above, the utility class takes care of locking all required locks before creating the file. All utility classes use the locking strategy that were described in Chapter 7, Locking and multi-threaded access, which should be fine with most applications.

Since locks are reentrant, that the utility classes do their own locking does not prevent client code from doing its own locking too. For instance, client code might want to hold a write lock on an entity that is modified through several utility method invocations to be sure that no other thread accesses the entity in between calls.

An entity utility class is designed to work with one specific entity interface. There are also capability utility classes with designed for working with capability objects, see Chapter 10, Capabilities. The following utility classes are implemented for entity interfaces:


Example 8.2. Holding a write lock for several utility method calls

// d is a directory
EntityLock wl = d.lockForWriting();
try
{
  EFile f1 = Directories.newFile(d, "f1");
  // Since we hold a write lock on d here, no other thread can get access to
  // f1 since that would involve reading d.
  
  // writeDataToFile is a method that writes data to the file...
  writeDataToFile(f1);
  
  EFile f2 = Directories.newFile(d, "f2");
}
finally
{
  wl.unlock();
}

In addition to the iterators that iterate over a directory's child entities, there are two types of iterators that iterate recursively over the entities in an entire directory hierarchy – the DepthFirstIterator and the DepthLastIterator[4]. Both iterators can be instantiated directly or created by using Directories utility methods.

There are implementations of depth-first and breadth-first iterators, apart from the default implementations DepthFirstIteratorImpl and DepthLastIteratorImpl, that implement a strategy for locking the entities that they return. All implementations are listed in the table below.


The example below shows a simple alternative implementation of the IteratorDeleter class – an utility object that deletes entities from a directory tree.

Example 8.3. Simple IteratorDeleter implementation

// d is a directory (or view)
Iterator<EntityView> itr = Directories.getDepthFirstIterator(d);
while(itr.hasNext())
{
  EntityView ev = itr.next();
  DirectoryView parent;

  // Get a temporary read lock on ev. We need it to get ev's parent.
  EntityLock rl = ev.lockForReading();
  try
  {
    // getParent requires a read lock (a write lock would also do)
    parent = ev.getParent();
  }
  finally
  {
    rl.unlock();
  }

  // parent == null means the root directory. We cannot delete that.
  if (parent != null)
  {  
    // Lock the parent before ev.
    EntityLock pwl = parent.lockForWriting();
    try
    {
      EntityLock wl = ev.lockForWriting();
      try
      {
        // Is the parent still the parent of the entity to delete?
        if (ev.getParent() == parent)
        {
          ev.delete();
        }
        else
        {
          ev.getFileSystem().getLogAdapter().
            logWarning("Cannot delete " + ev + ". It has been moved!");
        }
      }
      finally
      {
        wl.unlock();
      }
    }
    finally
    {
      pwl.unlock();
    }
  }
}

EntityFS comes with a generic filter implementation defined by the Filter and ConvenientFilter interfaces. In EntityFS, the filters are used for filtering entities to, for instance, create entity views or select which entities an iterator should return. (Other projects such as Schmant and At4J use the filter library for other purposes.)

There are several implementations of entity filters, as shown in the table below. Different filters can be combined using logical operators by using the methods defined in the ConvenientFilter interface. All entity filters also implement the EntityFilter interface to make them easier to find in the API documentation.

Table 8.4. Entity filters

Entity filter classDescription
DirectoryContainsFilterMatches directories that contain at least x entities that match a supplied entity filter, where x is a configurable number.
DirectoryEmptyFilterMatches empty directories.
DirectoryFilterMatches directories.
EFileFilterMatches files.
EFileNameExtensionFilterMatches files with specific file name extensions.
EntityIdentityFilterMatches a specific entity object.
EntityLatestModificationTimeFilterMatches entities that were last modified before a specific point in time. It can be negated to match entities that were modified after a specific point in time.
EntityLocationGlobFilterMatches entities whose locations relative to a base directory match a Glob pattern, for instance */test/*.xml.
EntityLocationRegexpFilterMatches entities whose locations relative to a base directory match a Java regexp , for instance .*/test/.*\.xml.
EntityNameFilterMatches entities with a specific name.
EntityNameGlobFilterMatches entities with names that match a Glob pattern, for instance test_[ab]*.xml.
EntityNamePrefixFilterMatches entities with names that starts with a specific prefix.
EntityNameRegexpFilterMatches entities with names that match a Java regular expression , for instance test_[ab].*.xml.
EntityRecentModificationFilterMatches entities that have been modified recently. It can be negated to match entities that have not been recently modified.
EntityTypeFilterMatches entities of a specific EntityType (files, directories, etc.).
FalseEntityFilterDoes not match any entities at all.
FGrepFilterMatches text files that contain a certain text string.
GrepFilterMatches text files that contain text that match a regular expression Pattern.
LockedEntityFilterMatches entities that are locked by the current thread.
ParentFilterA filter that applies another filter to an entity's parent directory.
ReadLockedEntityFilterMatches entities that are read locked by the current thread.
SuperParentAndFilterThis filter matches if all of an entity's parent directories (up to the file system root directory) matches a filter.
SuperParentOrFilterThis filter matches if any of an entity's parent directories (up to the file system root directory) matches a filter.
SymbolicLinkFilterMatches symbolic link entities.
TrueEntityFilterMatches all entities.
WriteLockedEntityFilterMatches entities that are write locked by the current thread.


All Filter implementations in EntityFS also implement the ConvenientFilter interface. It adds the methods and, or, xor and not that can be used instead of the AndFilter, OrFilter, XorFilter and NotFilter to combine different filters. This gives a more compact and readable syntax.

The EntityFilters class contains static utility methods for working with entity filters.

EntityFS defines its own Properties interface. It extends Map<String,String> and defines methods for accessing properties of different data types. Static property utilities are implemented in the PropertiesUtil class.

Example 8.5. Working with properties

// f is a property file
Properties p = PropertiesUtil.loadFromFile(f);

int i = p.getIntValue("myIntProperty");

// The object value is stored as the object serialized to a 
// Base64-encoded String.
Object o = p.getObjectValue("myObjectProperty");

// If no default value is given, all getter methods throw
// a NoSuchPropertyException if a requested property is missing.

// Define a default value to use if the property is missing.
long l = p.getLongValue("myOptionalLongProperty", 17L);

EntityFS comes with several useful implementations of Java's InputStream and OutputStream. It also has some utility implementations of the RandomAccess interface.






[4] DepthLastIterator should really have been named BreadthFirstIterator

Entities as well as the entire file system, may be configured to be Observable for events. Clients that want to receive notification of entity updates, register a listener of the type EntityObserver on the entity or on the file system object. The listener will then receive all EntityEvent:s that involve the observed entity, or for a file system listener, all events in that file system.

Every event has a sender which is the entity that generated the event.

Example 9.1. Registering an observer on a directory

// d is a directory. Register a simple observer on it.
d.addObserver(
  new EntityObserver() 
  {
    public void update(EntityEvent<?> ee)
    {
      if (ee instanceof ChildEntityAddedEvent)
      {
        System.out.println("Child entity " + 
          ((ChildEntityAddedEvent) ee).getChildEntity() +
          " added to directory " + ee.getSender());
      }
      else
      {
        System.out.println("Event: " + ee);
      }
    }
  });

Directories.newFile(d, "f1");

The observer will print out that the file f1 was added.



The EntityObserver's update method is called in the thread that generated the event. This means that, if the file system is locking, the calling thread is holding a write lock on the event's sender when update is called. It also means that update should return quickly. If there is a lot of work to do in response to an event, use an Executor to schedule the work for execution in another thread.

The following examples show which events that are generated when the file system is modified in different ways.




For directories that can be modified by someone that does not use its file system's methods, for instance by another process, a DirectoryMonitorer can be used to detect updates. Generally, it is not recommended to let entities be modified without using their entity objects, but in some cases, such as when monitoring a directory for incoming files, it can be useful. It requires a good EntityValidityControlStrategy for the file system, though.

Example 9.5. Using a polling directory monitorer to watch for incoming XML files

// d is the directory to monitor.
// Create a view that only shows XML files.
DirectoryView dv = d.newView(new EFileNameExtensionFilter("xml"));

// Create a timer that will tell the monitorer when to scan the directory
// for updates. Any old Observable should work; the monitorer will
// scan its directory every time the Observable generates an event.
// Scan every minute.
TimerTicker tt = new TimerTicker(60000);

// Create a polling monitorer on the directory view.
PollingDirectoryMonitorer mon = 
  new PollingDirectoryMonitorer(dv, tt);
  
// Register an observer on the monitorer
mon.addObserver(new EntityObserver() 
  {
    public void update(EntityEvent ee)
    {
      if (ee instanceof ChildEntitiesAddedEvent)
      {
        System.out.println("New child entities: " + ee);
      }
    }
  });

// Start the monitorer
mon.start();

// Start the timer ticker.
tt.start();

// This will generate an event sometime in the next minute
Directories.newFile(d, "foo.xml");

// This will not
Directories.newFile(d, "bar.txt");

// Neither will this
Directories.newDirectory(d, "baz.xml");

// But this will
new File(
  ECFileResolvableUtil.getFileObject(d) + 
    File.separator +
    "doh.xml").createNewFile();

Standard file system and entity functionality is extended by capabilities. The capabilities that a file system and its entities have is decided when the file system is created by adding CapabilityProvider:s to its builder. All file system implementations also have built-in capabilities, see Table 4.1, “File system implementations”.

When mixing some capabilities, the order in which the capability providers are added to the file system may be important. Restrictions are documented in each capability provider's API documentation.

Capabilities are used by first retrieving a capability object from a file system or from an entity. The capability is identified by a capability-specific FileSystemCapabilityType or EntityCapabilityType constant.

Below is an example of how the ECUnixAttributes entity capability can be used to get the creation time for an entity. Entity attributes are explained more in detail in Chapter 12, Entity attributes.

Example 10.1. Using Unix entity attributes to get an entity's creation time

// Use the Unix entity attributes capability to find out when the EFile f was
// created

// First get the capability object for the file
ECUnixAttributes eua =
  f.getCapability(ECTUnixAttributes.TYPE);

long timestamp = eua.getAttributes().getCreationTime();

Most capabilities have their own utility classes that can be used to make the client code shorter and more readable. The example below accomplishes just the same as the previous example, but with less code.


The name of an entity capability object shows for which entity types it is used. If the name starts with FC, the capability is used for EFile:s. If it starts with DC, it is used for Directory:s. An entity capability name starting with EC is used for all entity types.

Table 10.1. Entity capabilities

CapabilityUtility classDescription
DCSymbolicLinknoneSymbolicLink entity support. This is still experimental.
ECFileResolvableECFileResolvableUtilFor entities that are backed by a File object, this capability gives access to that File.
ECJarEntryECJarEntryUtilFor entities that have a JarEntry object, this capability gives access to that object. This is true for all entities in a Jar file system. See Chapter 14, Zip and Jar files.
ECMetadataECMetadataUtilMetadata support for entities. This capability give an entity one metadata container. See Chapter 11, Metadata support.
ECNamedMetadataECNamedMetadataUtilMetadata support for entities. This capability give an entity any number of metadata containers. See Chapter 11, Metadata support.
ECNamedEntityAttributesnoneThis capability is supported by entities that have several, named EntityAttributes objects. This is currently not implemented by any capability provider.
ECNtfsAttributesECNtfsAttributesUtilNtfsAttributes support for entities. See Chapter 12, Entity attributes.
ECUnixAttributesECUnixAttributesUtilUnixAttributes support for entities. See Chapter 12, Entity attributes.
ECUriResolvableECUriResolvableUtilFor entities whose locations can be represented by a URI, this capability gives access to that URI.
ECZipEntryECZipEntryUtilFor entities that have a ZipEntry object, this capability gives access to that object. This is true for all entities in a Zip or a Jar file system. See Chapter 14, Zip and Jar files.
FCAppendablenoneThis capability marks that a file can be appended to. It does not have any methods.
FCFileBackedFCFileBackedUtilFor files that are backed by a File object, this capability has methods for manipulating that File.
FCRandomAccessnoneThis capability marks that a file can be opened for random access. It does not have any methods.

Table 10.2. File system capabilities

CapabilityUtility classDescription
FSCAppendableFilesnoneThis capability is supported by file systems whose files can be appended to. It does not have any methods.
FSCCompressionnoneThis capability is supported by file systems that compress file data. It does not have any methods.
FSCDirectoryMetadatanoneThis capability is supported by file systems that support metadata for its directories. It does not have any methods.
FSCFileResolvableFSCFileResolvableUtilCapability supported by file systems whose entities can be resolved using File:s.
FSCJarFileBackedFSCJarFileBackedUtilCapability supported by file systems whose entities are backed by JarFile:s.
FSCMetadatanoneThis capability is supported by file systems that support metadata for its entities. It does not have any methods.
FSCNamedEntityAttributesnoneThis capability is supported by file systems that support named entity attributes objects for its entities. It does not have any methods.
FSCNtfsAttributesnoneThis capability is supported by file systems that support NTFS attributes for its entities. It does not have any methods.
FSCPersistentnoneThis capability is supported by file systems that store entity data in some kind of persistent storage, such as in an ordinary file system.
FSCRandomAccessFilesnoneThis capability is supported by file systems with support for random access to its files.
FSCSymbolicLinknoneThis capability is supported by file systems that support symbolic link entities.
FSCUnixAttributesnoneThis capability is supported by file systems that support Unix attributes for its entities. It does not have any methods.
FSCUriResolvableFSCUriResolvableUtilCapability supported by file systems whose entities can be resolved using URI:s.
FSCZipFileBackedFSCZipFileBackedUtilCapability supported by file systems whose entities are backed by ZipFile:s.

Table 10.3. Capability providers



Metadata support for entities is defined by the ECMetadata and the ECNamedMetadata capability interfaces. ECMetadata uses a single metadata container for each entity, while ECNamedMetadata supports any number of named metadata containers per entity. Each named metadata container is identified by a name that is unique for the entity that it is associated with. A metadata container is represented as a ReadWritableFile.

There are two file system capabilities that indicate that a file system supports metadata; FSCMetadata and FSCDirectoryMetadata. FSCMetadata indicate metadata support for all entity types, while FSCDirectoryMetadata indicate metadata support for directory entities only. All file systems that support FSCMetadata also support FSCDirectoryMetadata.

Two capability providers provide metadata support:


If the file system that the metadata capability is added to is FSCPersistent, the metadata will also be persistent.

When copying entities with metadata to a file system that does not support metadata, the metatada capability provider's MetadataCopyStrategy determines what happens. If it is set to IGNORE, which is the default, metadata will not be copied. If it is set to COPY, metadata is copied and put in one or several separate files in the target file system.

ECMetadataUtil and ECNamedMetadataUtil contain static utility methods for working with metadata capabilities.

Example 11.1. Creating a metadata enabled file system

FileSystem fs = new FSRWFileSystemBuilder().
  setRoot(new File("/tmp/fs1")).
  addCapabilityProvider(
    new FileBasedMetadataCapabilityProvider().
      // Put metadata in separate files if copying entities to a metadata
      // unaware file system.
      setCopyStrategy(MetadataCopyStrategy.COPY)).
  create();

Example 11.2. Creating a file system with metadata support for directories

FileSystem fs = new FSRWFileSystemBuilder().
  setRoot(new File("/tmp/fs1")).
  addCapabilityProvider(
    new FileBasedDirectoryMetadataCapabilityProvider().
      // Don't copy metadata if copying directories with metadata to a file
      // system that does not support metadata.
      setCopyStrategy(MetadataCopyStrategy.IGNORE)).
  create();

Example 11.3. Setting a metadata property on an entity

// a is an entity (file or directory)
EntityLock wl = a.lockForWriting();
try
{
  Properties p = new FileBackedProperties(
    ECMetadataUtil.getCapability(a));
  p.putStringValue(
    "lastSeen", 
    new SimpleDateFormat("yyyyMMdd").format(new Date()));
}
finally
{
  wl.unlock();
}

Example 11.4. Reading data from a named metadata file

// a is an entity (file or directory)
EntityLock rl = a.lockForReading();
try
{
  // The metadata file is protected by it's parent entity's read lock
  InputStream is = ECNamedMetadataUtil.getMetadataFile(a, "icon64x64.bmp").
    openForRead();
  try
  {
    BufferedImage img = ImageIO.read(is);
  }
  finally
  {
    is.close();
  }
}
finally
{
  rl.unlock();
}

An EntityAttributes object captures a collection of attributes for an Entity, such as the information on who owns it. It can be used for representing metadata in a structured way.

The entity attributes of an entity are properties of its parent directory. The entity attributes for all of a directory's child entities are stored in one of the directory's metadata files, so all entity attribute capability implementations require that the file system they are used for support the FSCDirectoryMetadata capability.

The UnixAttributes contains metadata information about entities that can be found in a Unix file system. The attributes are the user and group id:s of the entity's owner, the UnixEntityMode of the entity and the entity's creation time.

UnixAttributes support is provided by the MetadataUnixAttributesCapabilityProvider. It uses a B+ Tree HeliDB database stored in a metadata file for a directory to store all the entity attributes for that directory's child entities. This has the consequence that the root directory itself does not support entity attributes.

The NtfsAttributes contains a subset of the metadata information about entities that can be found in an NTFS file system. The attributes are the user and group id:s of the entity's owner, the UnixEntityMode of the entity, the entity's creation time and the set of NtFileAttributes.

NtfsAttributes support is provided by the MetadataNtfsAttributesCapabilityProvider. It uses a B+ Tree HeliDB database stored in a metadata file for a directory to store all the entity attributes for that directory's child entities. This has the consequence that the root directory itself does not support entity attributes.

The MetadataCustomAttributesCapabilityProvider can be used to have custom EntityAttributes for entities. The client must provide factory objects for default attributes for different entity types and an object that is used to parse the entity attributes metadata file, a HeliDB Serializer.

For entity attributes that have records of a fixed size in the attributes metadata file, there are abstract stub implementations of the required classes. For entity attributes that don't have fixed record size, some more effort is required. See the API documentation for details.

The rather long example below shows how a simple EntityAttributes object and all other classes required to use it together with the MetadataCustomAttributesCapabilityProvider are implemented.

Example 12.1. Using the custom entity attributes capability

// Our custom entity attributes
// Implement the attributes object to be immutable.
public class IndexAndBackupAttributes implements EntityAttributes
{
  private final Date m_indexDate;
  private final Date m_backupDate;
  
  public IndexAndBackupAttributes(Date indexDate, Date backupDate)
  {
    // Null checks
    indexDate.getClass();
    backupDate.getClass();
    m_indexDate = indexDate;
    m_backupDate = backupDate;
  }
  
  public Date getIndexDate()
  {
    return m_indexDate;
  }
  
  public Date getBackupDate()
  {
    return m_backupDate;
  }
  
  // EntityAttributes inherits Cloneable
  public Object clone()
  {
    return new IndexAndBackupAttributes(m_indexDate, m_backupDate);
  }
  
  // EntityAttributes recommends that we implement equals
  @Override
  public boolean equals(Object o)
  {
    if (o instanceof IndexAndBackupAttributes)
    {
      IndexAndBackupAttributes a2 = (IndexAndBackupAttributes) o;
      return a2.m_indexDate.equals(m_indexDate) && a2.m_backupDate.equals(m_backupDate);
    }
    else
    {
      return false;
    }
  }
  
  // We implemented equals, so now we have to implement hashCode too...
  @Override
  public int hashCode()
  {
    // Implement as suggested in Joshua Bloch's Effective Java
    int res = ((int) m_indexDate.getTime() ^ (int) (m_indexDate.getTime() >>> 32));
    return 31 * res + 
      (int) (m_backupDate.getTime() ^ (int) (m_backupDate.getTime() >>> 32));
  }
}

// This is the Serializer that is used to serialize
// attributes to and interpret attributes from the file where they are stored.
// One instance is used by all objects.
public class IndexAndBackupSerializer
  implements Serializer<IndexAndBackupAttributes>
{
  // The size of a serialized object in bytes.
  // A long is 8 bytes long
  private static final int SERIALIZED_SIZE = 8 + 8;
  
  // Get the size in bytes of the object when serialized.
  public int getSerializedSize()
  {
    return SERIALIZED_SIZE;
  }
  
  public boolean isNullValuesPermitted()
  {
    // No.
    return false;
  }
  
  // This is used by both serialization methods
  private void serializeInternal(IndexAndBackupAttributes o, byte[] barr, int offset)
    // We are allowed to throw this
    throws ArrayIndexOutOfBoundsException
  {
    // Serialize the dates to long:s. Use the LongSerializer.
    int pos = offset;
    LongSerializer.INSTANCE.serialize(o.getIndexDate().getTime(), barr, pos);
    pos += LongSerializer.DATA_SIZE;
    LongSerializer.INSTANCE.serialize(o.getBackupDate().getTime(), barr, pos);
  }
 
  public byte[] serialize(IndexAndBackupAttributes o)
  {
    byte[] res = new byte[SERIALIZED_SIZE];
    serializeInternal(o, res, 0);
    return res;
  }
 
  public int serialize(IndexAndBackupAttributes o, byte[] barr, int offset)
  {
    serializeInternal(o, barr, offset);
    return SERIALIZED_SIZE;
  } 
  
  private void validateSize(int size)
  {
    if (size != SERIALIZED_SIZE)
    {
      throw new SerializationException(
        "Invalid size of data " + size + 
        ". Must be " + SERIALIZED_SIZE + " bytes");
    }
  }
  
  // This is used by both interpret methods.
  private IndexAndBackupAttributes interpretInternal(byte[] barr, int offset)
  {
    // Use the LongSerializer to read the dates
    // that are serialized as longs
    int pos = offset;
    long indexDate = 
      LongSerializer.INSTANCE.interpret(barr, pos, LongSerializer.DATA_SIZE);
    pos += LongSerializer.DATA_SIZE;
    long backupDate =
      LongSerializer.INSTANCE.interpret(barr, pos, LongSerializer.DATA_SIZE);
    return new IndexAndBackupAttributes(
      new Date(indexDate),
      new Date(backupDate));
  }
  
  public IndexAndBackupAttributes interpret(byte[] barr) 
    throws SerializationException
  {
    validateSize(barr.length);
    return interpretInternal(barr, 0);
  }
  
  public IndexAndBackupAttributes interpret(byte[] barr, int offset, int length)
    throws SerializationException
  {
    validateSize(length);
    return interpretInternal(barr, offset);
  }
  
  public IndexAndBackupAttributes read(InputStream is, int size)
  {
    validateSize(size);
    byte[] barr = new byte[SERIALIZED_SIZE];
    try
    {
      int noRead = is.read(barr);
      if (noRead != SERIALIZED_SIZE)
      {
        throw new NotEnoughDataException(SERIALIZED_SIZE, noRead);
      }
      return interpret(barr);
    }
    catch (IOException e)
    {
      throw new WrappedIOException(e);
    }
  }
  
  public IndexAndBackupAttributes read(RandomAccess ra, int size)
  {
    validateSize(size);
    byte[] barr = new byte[SERIALIZED_SIZE];
    int noRead = ra.read(barr);
    if (noRead != SERIALIZED_SIZE)
    {
      throw new NotEnoughDataException(SERIALIZED_SIZE, noRead);
    }
    return interpret(barr);
  }
}

// This ObjectFactory is used to create default values of the attributes
// for different entity types.
public class IndexAndBackupAttributesFactory 
  implements ObjectFactory<IndexAndBackupAttributes>
{
  public IndexAndBackupAttributes create()
  {
    // Initialize the object with the current time.
    Date d = new Date();
    return new IndexAndBackupAttributes(d, d);
  }
}

// This object creates a file system whose entities have 
// IndexAndBackupAttributes, creates a couple of entities and set their
// attribute values.
public class MyIndexAndBackupAttributesUsingClass
{
  public FileSystem createAndInitFileSystem(File root)
  {
    // Default attributes factories for the different entity types.
    IndexAndBackupAttributesFactory af = new IndexAndBackupAttributesFactory();
    Map<EntityType, ObjectFactory<IndexAndBackupAttributes>> daf =
      new HashMap <EntityType, ObjectFactory<IndexAndBackupAttributes>>();
    daf.put(ETFile.TYPE, af);
    daf.put(ETDirectory.TYPE, af);
    
    FileSystem res =
      new FSRWFileSystemBuilder().
        setRoot(root).
        // Add the necessary directory metadata capability provider first.
        addCapabilityProvider(
          new FileBasedDirectoryMetadataCapabilityProvider()).
        // Add the custom entity attributes capability provider
        addCapabilityProvider(
          new MetadataCustomAttributesCapabilityProvider().
            // Don't store entities' attributes lazily (this is the default, so
            // this invocation is unnecessary)
            setStoreAttributesLazily(false).
            setDefaultAttributesFactories(daf).
            // Set the types of attributes that will be handled by the 
            // capability
            setAttributesType(IndexAndBackupAttributes.class).
            // This factory creates ChildEntityAttributeManager
            // objects for all directories.
            setChildEntityAttributeManagerFactory(
              // This can be used for capabilities that store their attributes
              // in a HeliDB database.
              new DatabaseChildEntityAttributeManagerFactory<IndexAndBackupAttributes>(
                new IndexAndBackupSerializer(), daf, 8192))).create();
    Directory fsroot = res.getRootDirectory();
	
    EFile f = Directories.newFile(fsroot, "f");
    // This has not been backed up since the 28th of January 1982. Sorry.
    Calendar cal = Calendar.getInstance();
    cal.clear();
    cal.set(Calendar.YEAR, 1982);
    cal.set(Calendar.MONTH, 0);
    cal.set(Calendar.DAY_OF_MONTH, 28);
    IndexAndBackupAttributes iab = 
      new IndexAndBackupAttributes(new Date(), cal.getTime());
    ECEntityAttributesUtil.setAttributes(f, iab);
	
    return res;                
  }
}

File systems have support for using an AccessController to control access to its entities. Access controllers are tools for helping file system clients in setting access policies for their file systems. They are not full-blown security solutions in themselves, though, since they can always be circumvented by malicious code running in the same JVM.

Most access controllers use Subject objects to identify who is trying to access an entity. A Subject is assigned to a running thread by calling any of the Subject's doAs methods.

The simplest AccessController implementation is the AccessGranterAccessController. It uses the callback method hasAccess of an AccessGranter to decide if a client has access to an entity.

The example below shows how a custom AccessGranter can be used with custom Subject objects to limit access to entities.

Example 13.1. A custom access granter

// The access granter
public class MyAccessGranter implements AccessGranter
{
  public boolean hasAccess(Subject subj, Permission p)
  {
    // Alice may only write in directories on thursdays
    if (p instanceof WritePermission)
    {
      WritePermission wp = (WritePermission) p;

      // Is the entity we are controlling access to a directory entity?
      if (wp.getEntity().getType() == ETDirectory.TYPE)
      {
        String subjName = ((MySubject) subj).getName();
        if (subjName.equals("Alice"))
        {
          return Calendar.getInstance().get(Calendar.DAY_OF_WEEK) == 
            Calendar.THURSDAY;
        }
      }
    }
    
    // All other access is permitted
    return true;
  }
}

// A custom Subject implementation that has a name.
public class MySubject extends SimpleAccessControllerSubject
{
  private final String m_name;
  
  public MySubject(AccessController ac, String name)
  {
    super(ac);
    m_name = name;
  }
  
  public String getName()
  {
    return m_name;
  }
}

// A PrivilegedAction that creates a new temporary file in a directory
public class MyWriteToDirectoryAction implements PrivilegedAction<EFile>
{
  private final Directory m_directory;
  
  public MyWriteToDirectoryAction(Directory d)
  {
    m_directory = d;
  }
  
  public EFile run()
  {
    return Directories.newTempFile(m_directory, "privileged", ".tmp");
  }
}

// A class that uses the privileged action to try to create a temporary file.
public class MyDirectoryWriter
{
  public EFile createTempFile(MySubject subj, Directory dir)
  {
    try
    {
      return subj.doAs(new MyWriteToDirectoryAction(dir));
    }
    catch (AccessDeniedException e)
    {
      dir.getFileSystem().getLogAdapter().logWarning("Not today");
      throw e;
    }
  }
}

The UnixEntityModeAccessController uses an UidGidSubject, the owner user and group id:s, and the UnixEntityMode of an entity to decide if the subject has access to it. The UidGidSubject is a subject that has a user id, UID, a primary group id, GID, and zero or more secondary GID:s. The UnixEntityMode specifies what kind of access the entity's owner, members of the same group as the entity belongs to and all other subjects have to the entity.

The UnixEntityModeAccessController requires that the file system supports any of the ECNtfsAttributes or ECUnixAttributes capabilities. They are supplied by the MetadataNtfsAttributesCapabilityProvider or the MetadataUnixAttributesCapabilityProvider.

The following example shows how a UnixEntityModeAccessController can be used in a non-locking file system.

Example 13.2. The Unix entity mode access controller on a non-locking file system

// A PrivilegedAction that creates a new text file in a directory
public class NewTextFileAction implements PrivilegedAction<EFile>
{
  private final Directory m_directory;
  
  public NewTextFileAction(Directory d)
  {
    m_directory = d;
  }
  
  public EFile run()
  {
    EFile res = (EFile) m_directory.newEntity(ETFile.TYPE, "t.txt", null);
    Files.writeText(res, "Quidquid latine dictum sit, altum sonatur");
    return res;
  }
}

// A class that makes sure that the file is written
public class MyFileWriter
{
  public EFile createFile(Directory d)
  {
    // Create a subject with UID = 1, primary GID = 2 and one secondary GID = 3
    UidGidSubject subj = new SimpleAccessControllerUidGidSubject(
      d.getFileSystem().getAccessController(), 1, 2, 
      Arrays.asList(new Integer[] { Integer.valueOf(3) }));
      
    // Get the entity attributes for the directory. Make no assumption about 
    // what exact kind of entity attributes that the directory has, other than
    // that it implements UidGidConfigurable and UnixEntityModeConfigurable
    EntityAttributes ea = ECEntityAttributesUtil.getAttributes(d);

    // Set the directory's owner to be the new subject
    ((UidGidConfigurable) ea).setUid(1);
    
    // Set the directory's mode to permit write and execution by its
    // owner. 0755 is the mode's octal representation. It will allow writes only
    // by the directory's owner. See UnixEntityMode for more documentation.
    ((UnixEntityModeConfigurable) ea).setUnixEntityMode(0755);
    
    // Save the directory's attributes
    ECEntityAttributesUtil.setAttributes(d, ea);
    
    // Create the text file. It will automatically be owned by the creating
    // Subject
    return subj.doAs(new NewTextFileAction(d));
  }
}

Since the entity attributes of an entity is a property of its parent directory, the parent directory has to be locked for reading every time an access control is performed on the entity, in other words, every time any of the access controlling entity methods, which are most of them, are called. A consequence of this is that the locking strategy of the utility classes does not work with this access controller since the utility classes do not acquire locks on an entity's parent directory before calling an entity's methods. File system clients have to do their own entity locking, as is shown in the example below.

The following example does the same as the example above on a file system that is locking. The file system's LockCommandExecutor (see Chapter 7, Locking and multi-threaded access) is used for locking all locks in an order that is compliant with the file system's locking strategy.

Example 13.3. The Unix entity mode access controller on a locking file system

// A PrivilegedAction that creates a new text file in a directory
public class NewTextFileAction implements PrivilegedAction<EFile>
{
  private final Directory m_directory;
  
  public NewTextFileAction(Directory d)
  {
    m_directory = d;
  }
  
  public EFile run()
  {
    FileSystem fs = m_directory.getFileSystem();

    // Create the file.
    
    // Use the file system's LockCommandExecutor to
    // execute the lock commands according to the file system's locking strategy.
    Collection<LockCommand> lcmds = new ArrayList<LockCommand>();
	
    // We need lock commands for access controls on the directory
    lcmds.addAll(fs.getAccessController().getLockCommandsForAccessControl(m_directory));
    
    // ...and lock commands for locking the directory for writing
    lcmds.add(new LockForWritingCommand(m_directory));
    
    EFile res;
    LockCollection lc = fs.getLockCommandExecutor().execute(lcmds);
    try
    {
      res = (EFile) m_directory.newEntity(ETFile.TYPE, "t.txt", null);
    }
    finally
    {
      lc.unlockAll();
    }
    
    // Write to the file
    lcmds = new ArrayList<LockCommand>();
    
    // Lock commands for access controls on the file
    lcmds.addAll(fs.getAccessController().getLockCommandsForAccessControl(res));
    
    // ... and for locking the file for writing
    lcmds.add(new LockForWritingCommand(res));

    lc = fs.getLockCommandExecutor().execute(lcmds);
    try
    {
      Files.writeText(res, "Vescere bracis meis!");
    }
    finally
    {
      lc.unlockAll();
    }
    return res;
  }
}

// A class that makes sure that the file is written
public class MyFileWriter
{
  public EFile createFile(Directory d)
  {
    // Create a subject with UID = 1, primary GID = 2 and one secondary GID = 3
    UidGidSubject subj = new SimpleAccessControllerUidGidSubject(
      d.getFileSystem().getAccessController(), 1, 2, 
      Arrays.asList(new Integer[] { Integer.valueOf(3) }));
      
    // Get the entity attributes for the directory. Make no assumption about 
    // what exact kind of entity attributes that the directory has, other than
    // that it implements UidGidConfigurable and UnixEntityModeConfigurable
    EntityAttributes ea = ECEntityAttributesUtil.getAttributes(d);

    // Set the directory's owner to be the new subject
    ((UidGidConfigurable) ea).setUid(1);
    
    // Set the directory's mode to permit write and execution by its owner.
    // 0755 is the mode's octal representation. It will allow writes only by the
    // directory's owner. See UnixEntityMode for more documentation.
    ((UnixEntityModeConfigurable) ea).setUnixEntityMode(0755);
    
    // Save the directory's attributes
    ECEntityAttributesUtil.setAttributes(d, ea);
    
    // Create the text file. It will automatically be owned by the creating
    // Subject
    return subj.doAs(new NewTextFileAction(d));
  }
}

EntityFS has support for using Zip and Jar files as backends for file systems, and also support for creating new Zip and Jar files.

There are three different Zip file systems

Zip file system backed by At4J's Zip implementation

This file system, built by the At4jZipFileSystemBuilder, supports more kinds of Zip files than the file system backed by Java's Zip implementation. It requires the At4J library on the classpath.

The file system supports random access (FCRandomAccess) to files that are stored uncompressed in the Zip file. This makes it possible to build a new Zip file system on an uncompressed file in another Zip file system.

Zip file system backed by Java's Zip implementation

This Zip file system, built by the ZipFileSystemBuilder, supports the kinds of Zip files that Java's Zip implementation supports. It must be built on a File.

Jar file system backed by Java's JAR implementation

This file system, built by the JarFileSystemBuilder extends the Zip file system with support for JAR metadata.

A ZipCreator object can be used to create a new Zip file. It builds the file using ZipEntryInfo objects that are returned from an Iterator. A ZipEntryInfo object contains information that the ZipCreator uses for creating a ZipEntry for a file or a directory in the Zip file.

Note

EntityFS ZipCreator uses Java's ZipOutputStream to create Zip files. It is somewhat limited in how the created file can be configured. The At4J project has a Zip builder that can create more types of Zip files.

The EntityIteratorToZipEntryInfoIteratorAdapter can be used to adapt any kind of EntityView iterator to a ZipEntryInfo Iterator.

Below is a simple example of how a Zip file can be created.

Example 14.1. Building a Zip file

// Build a Zip file containing all files and directories under the directory d1.
// Put the new file zip.zip in the directory d2.
new ZipCreator(
  new EntityIteratorToZipEntryInfoIteratorAdapter(d1, 
    Directories.getDepthLastIterator(d1)),
  d2, "zip.zip").create();

// The ZipCreator will acquire read locks on the files and directories
// added to the Zip file when it needs them.

The next example is a slightly more advanced example that builds a Zip file from files in two different directory hierarchies.


A Jar file can be built just like a Zip file. The JarCreator works just like the ZipCreator, with the added setManifest method for setting a Manifest.

Example 14.3. Building a Jar file

// Build a Jar file containing all files and directories under the directory d1.
// Use a custom manifest. Put the new file jar.jar in the directory d2.
new JarCreator(
  new EntityIteratorToJarEntryInfoIteratorAdapter(d1, 
    Directories.getDepthLastIterator(d1)),
  d2, "jar.jar").
  setManifest(
    new Manifest(
      new ByteArrayInputStream(
        // Note the trailing \n after the last row...
        "Manifest-Version: 1.0\nCreated-By: 1.0 me\n".getBytes()))).create();

To read the contents from a Zip or Jar file, create a Zip or Jar file system on the file and read entities from the file system.

Example 14.4. Reading entities from a Zip file

// Read the file d1/text2.txt from the Zip file texts.zip in the directory d
FileSystem fs = new ZipFileSystemBuilder().
  setZipFile(Directories.getFile(d, "texts.zip")).
  create();
try
{
  info("Contents of text2.txt: " +
    Files.readTextFile(
      Directories.getFile(
        fs.getRootDirectory(),
        new RelativeLocation("d1/text2.txt"))));
}
finally
{
  // Don't forget to close the Zip file system when you're done with it.
  fs.close();
}
  

If you want to just unzip a Zip or Jar archive, use any of ZipFiles's unzip methods.


An application often has to deal with both EntityFS entities and plain, old Java IO File:s[5]. This appendix shows how you can move between the two worlds with the least effort.

Depending on what the File should be used for, there are several methods available that can represent it as an EntityFS entity.

1. Create a file-based FileSystem on the File

This method gives most flexibility in configuring how the desired entities should behave, but it is perhaps also the method requiring the most effort to use.

Example A.1. Creating a custom file system on the File

// f is a file File called f

// Create a read/write file system on the file's parent directory.
FileSystem fs = new FSRWFileSystemBuilder().
  setRoot(f.getParentFile()).
  create();
  
// This is the file, represented as a file entity
EFile ef = Directories.getFile(fs.getRootDirectory(), "f");

2. Use FileSystems to create a FileSystem on the File

The utility class FileSystems has the getEntityForDirectory and getEntityForFile shortcut methods for creating file systems on File objects.

Example A.2. Creating a file system on the File using a FileSystems method

// f is a file File called f

// The returned EFile entity is in a file system created using the default
// settings of FSRWFileSystemBuilder
EFile ef = FileSystems.getEntityForFile(f, false);

3. Use FSCFileResolvableUtil to get an entity in an existing FileSystem

If the File references a location in an existing FileSystem, the FSCFileResolvableUtil methods getEntityForFile or getEntityLocationForFile can be used to retrieve the referenced Entity.

Example A.3. Getting an existing Entity using a FSCFileResolvableUtil method

// f is a file File that references the file entity /d/f in the file
// system fs.

EFile ef = (EFile) FSCFileResolvableUtil.getEntityForFile(fs, f);

4. Use an adapter object to adapt the File to the desired interface

A File can be adapted different interfaces using an adapter object.


Example A.4. Adapting a File to the ReadableFile interface

// f is a File containing text

// Use the Files utility class to read the contents of the text file
// to a String
String s = Files.readTextFile(new FileReadableFile(f));

Conversions from entity objects to File objects can be useful when working with API:s that use File arguments. File objects can be created for an entities in a file system that is FSCFileResolvable using the FSCFileResolvableUtil method getFileForEntity.




[5] POJIOFS?