Frequently Asked Questions

From Ext4
Revision as of 23:03, 7 May 2008 by Adilger (Talk | contribs)

Jump to: navigation, search
This page is a stub

You can help Ext2/3/4 Wiki by expanding it.

Contents

Getting Started

Where do I get the latest version of e2fsprogs?

The latest version of e2fsprogs can be found at Soureforge or at kernel.org.

How do I build e2fsprogs?

The INSTALL file in the top of the source tree gives more detailed information, but e2fsprogs uses a standard configure script, so the standard "./configure; make" will build the e2fsprogs binaries. Note that if you wish to build the ELF shared libraries, you need to add the "--enable-elf-shlibs" option to the configure invocation.

How do I mount a fresh new storage device as ext4?

For example, if the new device has been detected as /dev/sdb1 (check "dmesg" output):

Using the newly installed e2fsprogs (using the OS default mke2fs most likely will fail - just compare the message returne d with the following messages):

cd misc/
./mke2fs -E test_fs /dev/sdb1

mke2fs 1.40.8 (13-Mar-2008)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
126976 inodes, 506044 blocks
25302 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67633152
62 block groups
8192 blocks per group, 8192 fragments per group
2048 inodes per group
Superblock backups stored on blocks: 
        8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409

Writing inode tables: done                            
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 22 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

cd ../debugfs/
./debugfs -w /dev/sdb1
debugfs 1.40.5 (27-Jan-2008)
debugfs: set_super_value s_flags 4
debugfs: quit

cd ../misc/
./tune2fs -j /dev/sdb1
./blkid /dev/sdb1
/dev/sdb1: UUID="3aa983ef-0dcd-474d-a9d3-2660ff6ef3e7" TYPE="ext4dev"
mkdir /mnt/test
mount -t ext4dev /dev/sdb1 /mnt/test

To verify the mounted filesystem:

df

Why did I get two different output for blkid?

When executing:

blkid /dev/sdb2
/dev/sdb2: UUID="5b8706af-208c-4e4c-a3bf-16f1c0864c82" TYPE="ext2" 

But when the latest e2fsprogs' version of blkid is executed:

/download/e2fsprogs/e2fsprogs/misc>./blkid /dev/sdb2
/dev/sdb2: UUID="5b8706af-208c-4e4c-a3bf-16f1c0864c82" SEC_TYPE="ext2" TYPE="ext4dev" 

So this explained why you get different output.

Why do I get "EXT4-fs: sdb1: not marked OK to use with test code." in my dmesg?

This error arises from not executing the "mke2fs -E test_fs /dev/sdb1", which will label the partition as test_fs. This signature is currently checked for in the ext4 development source codes.

Searching for this symbol in e2fsprogs package:

/download/e2fsprogs/e2fsprogs/misc>grep test_fs *.c
mke2fs.c:               } else if (!strcmp(token, "test_fs")) {
mke2fs.c:                       "\ttest_fs\n"),
tune2fs.c:              if (!strcmp(token, "test_fs")) {
tune2fs.c:              } else if (!strcmp(token, "^test_fs")) {
tune2fs.c:                      "\ttest_fs\n"
tune2fs.c:                      "\t^test_fs\n"));

To add test_fs signature to formatted partition, execute: "tune2fs -E test_fs /dev/sdb1".

How do I turn off the mballoc feature?

By default "mount" will enable mballoc feature when mounting. To turn it off:

mount -t ext4dev -o data=writeback,delalloc,extents,nomballoc /dev/sdb1 /mnt/test

How do I turn off the extent feature?

By default "mount" will enable extent feature when mounting. To turn it off:

mount -t ext4dev -o data=writeback,delalloc,noextents,mballoc /dev/sdb1 /mnt/test

CAUTION: Once extent feature has been turned on, it is not possible to mount it as ext3 anymore.

Why do I get "EXT4-fs: Unrecognized mount option "delalloc" or missing value"?

Please download the ext4 version from the git patchset, instead of using the kernel source.

What is mballoc feature? After I mounted a partition as mballoc, can I remount it as nomballoc?

Yes, it is possible, just dismount the partition first, and execute "mount" with the nomballoc feature.

Understanding how it works

What are the new features in Ext4 (vs Ext2/3)?

Check here: http://ext4.wiki.kernel.org/index.php/New_ext4_features.

How do I test the features in Ext4?

What external tools are available for testing Ext4 FS?

One commonly known simple tool is fsfuzzer: http://www.digitaldwarf.be/products/mangle.c, and its content is reproduced here:

/*
  trivial binary file fuzzer by Ilja van Sprundel.
  It's usage is very simple, it takes a filename and headersize
  as input. it will then change approximatly between 0 and 10% of 
  the header with random bytes (biased towards the highest bit set)
 
  obviously you need a bash script or something as a wrapper !

  so far this broke: - libmagic (used file)
                     - preview (osX pdf viewer)
		     - xpdf (hang, not a crash ...)
		     - mach-o loading (osX 10.3.7, seems to be fixed later)
		     - qnx elf loader (panics almost instantly, yikes !)
		     - FreeBSD elf loading 
		     - openoffice
		     - amp
		     - osX image loading (.dmg) 
		     - libbfd (used objdump)
		     - libtiff (used tiff2pdf)
		     - xine (division by 0, took 20 minutes of fuzzing)
		     - OpenBSD elf loading (3.7 on a sparc)
		     - unixware 713 elf loading
		     - DragonFlyBSD elf loading
		     - solaris 10 elf loading 
		     - cistron-radiusd
		     - linux ext2fs (2.4.29) image loading (division by 0)
		     - linux reiserfs (2.4.29) image loading (instant panic !!!)
		     - linux jfs (2.4.29) image loading (long (uninteruptable) loop, 2 oopses)
		     - linux xfs (2.4.29) image loading (instant panic)
		     - windows macromedia flash .swf loading (obviously the windows version of mangle needs a few tweaks to work ...)
		     - Quicktime player 7.0.1 for MacOS X
		     - totem
		     - gnumeric
                     - vlc
                     - mplayer
                     - python bytecode interpreter
                     - realplayer 10.0.6.776 (GOLD)
                     - dvips
 */
#include <stdio.h>
#include <sys/types.h>
#include <sys/mman.h>
#include <fcntl.h>

#define DEFAULT_HEADER_SIZE 1024
#define DEFAULT_NAME "test2"

int getseed(void) {
	int fd = open("/dev/urandom", O_RDONLY);
	int r;
	if (fd < 0) {
		perror("open");
		exit(0);
	}
	read(fd, &r, sizeof(r));
	close(fd);
	return(r);
}

int main(int argc, char **argv) {
	
	int fd;
	char *p, *name;
	unsigned char c;
	unsigned int count, i, off, hsize;

	if (argc < 2) {
		hsize = DEFAULT_HEADER_SIZE;
		name = DEFAULT_NAME;
	} else if (argc < 3) {
		hsize = DEFAULT_HEADER_SIZE;
		name = argv[1];
	} else {
		hsize = atoi(argv[2]);
		name = argv[1];
	}
	fd = open(name, O_RDWR);
	if (fd < 0) {
		perror("open");
		exit(0);
	}
	p = mmap(0, hsize, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
	if ((int) p == -1) {
		perror("mmap");
		close(fd);
		exit(0);
	}
	srand(getseed());
	count = (unsigned) rand() % (hsize / 10);
	for (i = 0; i < count; i++) {
		off = rand() % hsize;
		c = rand() % 256;
		/* we want the highest bit set more often, in case of signedness issues */
		if ( (rand() % 2) && c < 128) c |= 0x80;
		p[off] = c;
	}
	close(fd);
	munmap(p, hsize);
}

Its fuller content is packaged here (which shell scripts to run the compiled programs, but customization for ext4 will be needed): [1]

How do I benchmark the performance of Ext4 as against other FS? What are the tools available?

There exists a wide variety of tools and comparison, for more information on the different performance testing tools available: [2]

Anothere reference here: http://en.wikipedia.org/wiki/Comparison_of_file_systems.

Can I undelete files in Ext4?

No, in the same way that the ext3 journal requirements to be consistent after a crash prevent undelete of ext3 files, it isn't possible to undelete ext4 files.

Can I mount existing Ext3 as Ext4? And vice versa? Similarly from Ext2 to Ext4 and its reverse?

What is the information provided by /proc/fs/jbd2/<partition>/history?

Executing "cat /proc/fs/jbd2/<partition>/history" gives:

R/C  tid   wait  run   lock  flush log   hndls  block inlog ctime write drop  close
R    2     0     107692 0     0     0     1      1     2    

The purpose of this history is to provide a capture of the statistical properties on the performance of the Ext4 filesystem. It can be observed that the entries are added after a file has been copied to the file system - not immediately, but delayed by a short while, due to the asynchronous mechanism of jbd2 logging(*FIXED ME*).

What is the information provided by /proc/fs/jbd2/<partition>/info?

Executing "cat /proc/fs/jbd2/<partition>/info" gives:

56 transaction, each upto 2048 blocks average:

 0ms waiting for transaction
 57671ms running transaction
 0ms transaction was being locked
 28ms flushing data (in ordered mode)
 14ms logging transaction
 2383 handles per transaction
 6 blocks per transaction
 7 logged blocks per transaction

How to online resize the Ext4 filesystem?

Online resizing of ext4 works in a similar manner as ext3, using either resize2fs or ext2resize, but there is currently a limit (around 4TB or so) to the maximum filesystem size. Implementing online resize with the META_BG feature would allow this limit to be exceeded.

What is the difference between extents mapping and traditional indirect block mapping?

To quote from the paper: http://ext2.sourceforge.net/2005-ols/2005-ext3-paper.pdf:

Currently, the ext2/ext3 filesystem, like other traditional UNIX filesystems, uses a direct, indi-
rect, double indirect, and triple indirect blocks to map file offsets to on-disk blocks. This
scheme, sometimes simply called an indirect block mapping scheme, is not efficient for large
files, especially large file deletion. In order to address this problem, many modern filesystems
(including XFS and JFS on Linux) use some form of extent maps instead of the traditional
indirect block mapping scheme.

Since most filesystems try to allocate blocks in a contiguous fashion, extent maps are a more efficient 
way to represent the mapping between logical and physical blocks for large files. An extent is a single 
descriptor for a range of contiguous blocks, instead of using, say hundreds of entries to describe 
each block individually.

What is delayed allocation? What are its advantages in Ext4?

Delayed allocation worked by deferring the allocation of new blocks in the filesystem to disk blocks until writeback time. This helps in three ways:

1. Reduced fragmentation.
2. Reduced CPU cycles spent in get_block() calls.
3. It may avoid the need for disk updates for metadata creation, which in turn reduces impact on fragmentation.

What is multiblock allocation (mballoc)?

mballoc is a mechanism to allow many blocks to be allocated to a file in a single operation, in order to dramatically reduce the amount of CPU usage searching for many free blocks in the filesystem. Also, because many file blocks are allocated at the same time, a much better decision can be made to find a chunk of free space where all of the blocks will fit.

The mballoc code is active when using the O_DIRECT flag for writes, or if the delayed allocation (delalloc) feature is being used. This allows the file to have many dirty blocks submitted for writes at the same time, unlike the existing kernel mechanism of submitting each block to the filesystem separately for allocation.

What is this bitmap allocator?

Can you say something about the history of Ext4?

Check here: http://en.wikipedia.org/wiki/Ext4.

When was Ext4 first annouced to the LKML?

Check here: http://kerneltrap.org/node/6776

What are the key differences between ext3 and ext4?

The main new features in ext4 are below, and are described more fully in New_ext4_features:

  • extent-mapped files for more efficient storage of file metadata (EXTENTS)
  • multi-block and delayed allocation for faster/better file allocations
  • support for larger filesystems (up to 2^48 blocks, currently 2^60 bytes) (64_BIT)
  • optimized storage of filesystem metadata like bitmaps and inode table (FLEX_BG)
  • less overhead for e2fsck, on-disk checksum of group descriptors (GDT_CSUM)
  • removed 32000 subdirectory limit (DIR_NLINKS)
  • nanosecond inode timestamps (EXTRA_ISIZE)

What are the key differences between jbd and jbd2?

The code between jbd and jbd2 is nearly identical, but jbd2 adds a few new features in a compatible way:

  • support for 64-bit filesystems (64_BIT)
  • checksumming of journal transactions (CHECKSUM)
  • asynchronous transaction commit block write (ASYNC_COMMIT)

What are the key on-disk format differences between ext3 and ext4?

/*
 * This is the extent on-disk structure.
 * It's used at the bottom of the tree.
 */
struct ext4_extent {
        __le32  ee_block;       /* first logical block extent covers */
        __le16  ee_len;         /* number of blocks covered by extent */
        __le16  ee_start_hi;    /* high 16 bits of physical block */
        __le32  ee_start_lo;    /* low 32 bits of physical block */
};

/*
 * This is index on-disk structure.
 * It's used at all the levels except the bottom.
 */
struct ext4_extent_idx {
        __le32  ei_block;       /* index covers logical blocks from 'block' */
        __le32  ei_leaf_lo;     /* pointer to the physical block of the next *
                                 * level. leaf or next index could be there */
        __le16  ei_leaf_hi;     /* high 16 bits of physical block */
        __u16   ei_unused;
};

What are the key on-disk format differences between jbd and jbd2?

struct commit_header {

      __be32          h_magic;
      __be32          h_blocktype;
      __be32          h_sequence;
      unsigned char   h_chksum_type;
      unsigned char   h_chksum_size;
      unsigned char   h_padding[2];
      __be32          h_chksum[JFS_CHECKSUM_BYTES];

};

Personal tools