Large Files

Interfaces Supporting Large Files

There are two ways to support large files:

Data Types

These are the data types that are 64 bits in the alternate compile environment:

fsblkcnt_t (new data type) fsfilcnt_t (new data type) fpos_t off_t rlim_t

These data types remain 32 bits in the default environment, although the data types also have 64-bit equivalents in the 32-bit environment. The names end in "64" or "64_t", such as fsblkcnt_t and fsblkcnt64_t.

Data Structures

Below are the data structures that use the above data types.

dinode off64_t di_size flock off_t l_start off_t l_len ic_common off64_t ic_size inode off64_t i_size rlimit rlim_t rlim_cur rlim_t rlim_max stat off_t st_size fsblkcnt_t st_blocks statvfs fsblkcnt_t f_blocks fsblkcnt_t f_bfree fsblkcnt_t f_bavail fsblkcnt_t f_files fsblkcnt_t f_ffree fsblkcnt_t f_favail

In the alternate compile environment, the structures contain the 64-bit fields listed. In the default environment, these structures do not change, but there are parallel structures, such as flock64 for flock.

Subroutine Interfaces

Below are the subroutine interfaces that are different in the alternate compile environment, usually because they use one of the types or structures above. The name of the corresponding routine usable in the default environment is also listed:

Existing Interface New Interface ================== ============= creat() creat64() fgetpos() fgetpos64() fopen() fopen64() freopen() freopen64() fsetpos() fsetpos64() fstat() fstat64() fstatvfs() fstatvfs64() fstatvfsdev() fstatvfsdev64() ftruncate() ftruncate64() ftw() ftw64() getrlimit() getrlimit64() lockf() lockf64() lseek() lseek64() lstat() lstat64() mmap() mmap64() nftw() nftw64() nftw2() nftw2_64() open() open64() prealloc() prealloc64() setrlimit() setrlimit64() stat() stat64() statvfs() statvfs64() statvfsdev() statvfsdev64() tmpfile() tmpfile64() truncate() truncate64()

New Interfaces

New interfaces have been introduced for fseek() and ftell() because the POSIX definitions do not support file sizes larger than 2GB on machines that use 32-bit int values.

New 32-bit interfaces New 64-bit interfaces ===================== ===================== ftello() ftello64() fseeko() fseeko64()

Commands

The following commands have also been enhanced to provide command-line support for large files:

• Accounting and quota commands

• UFS and JFS file system administration commands

awk bdiff cat chgrp chmod chown cksum cmp comm compress cp csh csplit cut dd df du expand fbackup frecover file find fold getaccess grep/egrep/fgrep head hyphen join ksh ln ls mv od/xd paste prealloc rm/rmdir sed sh sort split strings tail tee touch uncompress unexpand uniq uudecode uuencode wc zcat

Complete details of the data types, structures, interfaces, and commands can be found in the Large Files White Paper.

Behavior of Existing Applications

The current small file (less than 2GB) environment has not changed. The implementation has only introduced new interfaces that enable applications to manipulate large files. Therefore, as long as an application has no need to create, open, or stat large files, no code modification or recompilation is required. Also, because the existing environment is not changing, existing code that is recompiled on a system that supports large files will still behave as it did before the recompilation.

When an existing application that does not utilize the new interfaces to access large files encounters a large file, an error will be returned. These applications can expect both the open() and stat() calls to fail with EOVERFLOW. This protection mechanism prevents applications unaware of large files from accessing a large file because the application may not be able to correctly handle large files.

There is also a file system-based protection mechanism to protect unaware applications from encountering large files. A file system can be designated as one of the following:

The mechanism is administered through the file system administration commands:

      Command                       Description
     =========        =================================================
 
     newfs(1M)        Creates file systems and now accepts a flag to
                      designate the file system either "largefiles" or
                      "nolargefiles" (the default is "nolargefiles").
 
     mount(1M)        Also accepts the "largefiles" or "nolargefiles"
                      options.
 
     fsadm(1M)        Modifies this attribute of the file system after it
                      has been initially designated.

Full details can be found in both the white paper and associated manpages.

Programming for Large Files

Large files are supported in two programming/compile environments. In the first, existing data types keep their old size and large files are accessed by explicitly calling the new 64-bit routines and using the 64-bit data structures and types. These new interfaces enable the full access of files up to the supported limit. Current 32-bit interfaces keep their old sizes and continue to work on files less than 2GB. The new and old interfaces can be mixed in the same program. Specific compilation flags are required to make these 64-bit interfaces available:

       Compilation Flag                        Provides
      ===================          ================================
      _LARGEFILE_SOURCE            fseeko() and ftello() interfaces
 
      _LARGEFILE64_SOURCE          all *64() interfaces

In the second environment, existing standard data types and routines use 64-bit fields, allowing programs using standard POSIX and XPG interfaces to support large files. Again, compilation flags are used to determine the interfaces sizes:

        Compilation Flag                        Provides
      ====================         ================================
      _FILE_OFFSET_BITS=32         32-bit environment (the default)
 
      _FILE_OFFSET_BITS=64         64-bit environment; causes 32-bit
                                   routines and data structures to
                                   to be 64 bits

Complete details of the programming/compile environments can be found in the white paper.

Using Large Files

As stated above, if your application does not need to operate on file data beyond 2GB, you do not have to make changes to the application. The built-in protection mechanisms will help prevent the unaware application from accessing a large file. However, if the application needs to support the large file interfaces, there are a few alternatives to consider:

Complete information can be found in the white paper.

Regardless of which interfaces are used (either the standard POSIX interfaces or *64() interfaces), all exhibit the same behavior on files less than 2GB. For example, the *64() interfaces can be used on small files just as they are used on large files. A small file that was created/modified by a large file application can be modified/accessed by a small file application, and vice versa. There are no incompatibilities between the two types of interfaces when operating on files less than 2GB.

EOVERFLOW Error

With large files, it is possible for a developer to mix and match 32- and 64-bit interfaces and data types. For this reason, many of the routines can return a new error, EOVERFLOW, indicating that the operation or returned data is too large for the environment or passed data structure. This can occur when a 64-bit call is made using a 32-bit data structure, or when an unconverted application tries to open a large file. The details of the error associated with each interface can be found in the interfaces' manpage.

Commands Support

The commands listed in the above "Commands" subsection support operations on large files. In addition, commands that display or accept as arguments byte, word, and line counts also handle values larger than 2GB. For example, wc(1) displays byte, word, and line counts that can be larger than 2GB; find(1) accepts size arguments that are larger than 2GB, and compress(1) operates on files larger than 2GB. An exception to this general rule is tail(1), which is (and has always been) limited to a 20K buffer for display. Therefore, tail(1) only displays up to 20K bytes of data, regardless of the line number argument that is passed. See the manpage for complete details.

Shells Support

All three shells (sh, ksh, and csh) support redirection of data to/from large files.

Backing Up Large Files

The dd, fbackup/frecover, and vxdump/vxrestore commands support large file archival and retrieval. The fbackup and vxdump data formats have been changed to accommodate the larger file sizes. The archives created by these versions of the commands are readable only by the corresponding frecover and vxdump, and not by their earlier versions. Archives created by older versions of fbackup and vxdump are readable by the new versions of frecover and vxdump.

Standards-conforming backup commands such as tar, cpio, ftio, and pax do not support archiving files greater than 2GB. These commands have limitations associated with standards and interoperability that prevents their support of large files. The manpages contain complete details.

The dump/restore commands do not support files greater than 2GB.

Performance

There can be some performance degradation due to the use of 64-bit data types, which, on 32-bit architectures, require more instructions to manipulate than 32-bit types.

Compatibility

The solutions for large files introduces new functionality. All prior functionality is preserved. The implementation is fully binary compatible with prior releases of HP-UX.