1163 lines
31 KiB
Text
1163 lines
31 KiB
Text
.HTML "Fossil, an Archival File Server
|
|
... .FP times
|
|
... .fp 1 R R.nomath
|
|
... .fp 5 CW LucidaSansCW83
|
|
.TL
|
|
Fossil, an Archival File Server
|
|
.AU
|
|
Sean Quinlan
|
|
Jim McKie
|
|
Russ Cox
|
|
jmk,rsc@plan9.bell-labs.com
|
|
.AB
|
|
This paper describes the internals and
|
|
operation of Fossil, an archival file server built for Plan 9.
|
|
Fossil has not yet replaced the current Plan 9 file server
|
|
and
|
|
.CW kfs ,
|
|
but that is our eventual intent.
|
|
Both fossil and this documentation are
|
|
works in progress. Comments on either are most welcome.
|
|
.AE
|
|
.de HP
|
|
.LP
|
|
..
|
|
.NH 1
|
|
Introduction
|
|
.HP
|
|
Fossil is an archival file server built for Plan 9.
|
|
In a typical configuration, it maintains a traditional file system
|
|
in a local disk partition and periodically archives snapshots of the file system
|
|
to a Venti server. These archives are made available through
|
|
a file system interface.
|
|
Fossil can also be run without a Venti server, in which case the
|
|
snapshots (if any) occupy local disk space.
|
|
.PP
|
|
The bulk of this paper explains the underlying data structures:
|
|
Venti trees, the Venti archival file system format, and finally Fossil's
|
|
file system format.
|
|
The end of the paper discusses the architecture of the Fossil server.
|
|
.PP
|
|
The presentation of the data structures is very detailed, perhaps
|
|
too detailed for most readers.
|
|
The intent is to record all the details necessary to make structural
|
|
changes to the file system format.
|
|
Feel free to jump ahead when boredom strikes.
|
|
.NH 1
|
|
Venti trees and directory hierarchies
|
|
.HP
|
|
Venti [3] is an archival block storage server.
|
|
Once a block is stored, it can be retrieved by presenting the 20-byte
|
|
SHA1 hash of its contents, called a
|
|
.I score .
|
|
Blocks on Venti have a maximum length of about 56 kilobytes,
|
|
though in practice smaller blocks are used.
|
|
To store a byte stream of arbitrary length, Venti uses a hash tree.
|
|
Conceptually, the data stream is broken into fixed-size (say,
|
|
.I dsize -byte)
|
|
chunks, which are stored on the Venti server.
|
|
The resulting scores are concatenated into a new pointer stream, which is
|
|
broken into fixed size (say,
|
|
.I psize -byte)
|
|
chunks, which are stored on the Venti server.
|
|
.I Psize "" (
|
|
is different from
|
|
.I dsize
|
|
so that we can ensure that each pointer block holds an
|
|
integral number of pointers.)
|
|
This yields a new pointer stream, and so on, until there is a single block
|
|
and finally a single score describing the entire tree.
|
|
The resulting structure looks like:
|
|
.PS
|
|
.ps 8
|
|
.vs 10
|
|
boxht=0.1
|
|
boxwid=0.1
|
|
|
|
B0: box invis wid 1 "\f(CWVtDataType\fP"
|
|
move right 0.1
|
|
L0a: box wid 0.2
|
|
move right 0.1
|
|
L0b: box wid 0.2
|
|
move right 0.1
|
|
L0c: box invis wid 0.2 "..."
|
|
move right 0.1
|
|
|
|
L0d: box wid 0.2
|
|
move right 0.1
|
|
L0e: box wid 0.2
|
|
move right 0.2
|
|
L0f: box invis wid 0.2 "..."
|
|
move right 0.2
|
|
|
|
L0g: box wid 0.2
|
|
move right 0.1
|
|
L0h: box wid 0.2
|
|
move right 0.1
|
|
L0i: box invis wid 0.2 "..."
|
|
move right 0.1
|
|
|
|
L0j: box wid 0.2
|
|
move right 0.1
|
|
L0k: box wid 0.2
|
|
move right 0.1
|
|
L0l: box invis wid 0.2 "..."
|
|
move right 0.1
|
|
L0m: box wid 0.2
|
|
|
|
define boxppddd {
|
|
line from 0.2<$1.nw,$1.ne> to 0.2<$1.sw,$1.se>
|
|
line from 0.4<$1.nw,$1.ne> to 0.4<$1.sw,$1.se>
|
|
X: box invis at 0.1<$1.nw,$1.ne>
|
|
Y: box invis at 0.1<$1.sw,$1.se>
|
|
line -> from 0.5<X,Y> to $2.nw
|
|
X: box invis at 0.3<$1.nw,$1.ne>
|
|
Y: box invis at 0.3<$1.sw,$1.se>
|
|
line -> from 0.5<X,Y> to $3.nw
|
|
"..." at 0.7<$1.w,$1.e>
|
|
}
|
|
|
|
define boxppdddp {
|
|
line from 0.2<$1.nw,$1.ne> to 0.2<$1.sw,$1.se>
|
|
line from 0.4<$1.nw,$1.ne> to 0.4<$1.sw,$1.se>
|
|
line from 0.8<$1.nw,$1.ne> to 0.8<$1.sw,$1.se>
|
|
X: box invis at 0.1<$1.nw,$1.ne>
|
|
Y: box invis at 0.1<$1.sw,$1.se>
|
|
line -> from 0.5<X,Y> to $2.nw
|
|
X: box invis at 0.3<$1.nw,$1.ne>
|
|
Y: box invis at 0.3<$1.sw,$1.se>
|
|
line -> from 0.5<X,Y> to $3.nw
|
|
"..." at 0.6<$1.w,$1.e>
|
|
X: box invis at 0.9<$1.nw,$1.ne>
|
|
Y: box invis at 0.9<$1.sw,$1.se>
|
|
line -> from 0.5<X,Y> to $4.nw
|
|
}
|
|
|
|
define boxpdddp {
|
|
line from 0.2<$1.nw,$1.ne> to 0.2<$1.sw,$1.se>
|
|
line from 0.8<$1.nw,$1.ne> to 0.8<$1.sw,$1.se>
|
|
X: box invis at 0.1<$1.nw,$1.ne>
|
|
Y: box invis at 0.1<$1.sw,$1.se>
|
|
line -> from 0.5<X,Y> to $2.nw
|
|
"..." at 0.5<$1.w,$1.e>
|
|
X: box invis at 0.9<$1.nw,$1.ne>
|
|
Y: box invis at 0.9<$1.sw,$1.se>
|
|
line -> from 0.5<X,Y> to $3.nw
|
|
}
|
|
|
|
bhd=0.4
|
|
L1abc: box wid 0.5 at 0.5<L0a, L0b>+(0,bhd)
|
|
boxppddd(L1abc, L0a, L0b)
|
|
L1def: box wid 0.5 at 0.5<L0d, L0e>+(0,bhd)
|
|
boxppddd(L1def, L0d, L0e)
|
|
L1ghi: box wid 0.5 at 0.5<L0g, L0h>+(0,bhd)
|
|
boxppddd(L1ghi, L0g, L0h)
|
|
L1jklm: box wid 0.5 at 0.5<L0j, L0k>+(0,bhd)
|
|
boxppdddp(L1jklm, L0j, L0k, L0m)
|
|
B1: box invis wid 1 "\f(CWVtPointerType0\fP" at B0+(0,bhd)
|
|
|
|
L2abcdef: box wid 0.5 at 0.5<L1abc,L1def>+(0,bhd)
|
|
boxppddd(L2abcdef, L1abc, L1def)
|
|
L2ghijklm: box wid 0.5 at 0.5<L1ghi,L1jklm>+(0,bhd)
|
|
boxpdddp(L2ghijklm, L1ghi, L1jklm)
|
|
B2: box invis wid 1 "\f(CWVtPointerType1\fP" at B1+(0,bhd)
|
|
|
|
L3atom: box wid 0.5 at 0.5<L2abcdef, L2ghijklm>+(0,bhd)
|
|
boxpdddp(L3atom, L2abcdef, L2ghijklm)
|
|
B3: box invis wid 1 "\f(CWVtPointerType2\fP" at B2+(0,bhd)
|
|
.PE
|
|
.LP
|
|
The leaves are the original data stream. Those blocks have type
|
|
.CW VtDataType .
|
|
The first pointer stream has type
|
|
.CW VtPointerType0 ,
|
|
the next has type
|
|
.CW VtPointerType1 ,
|
|
and so on.
|
|
The figure ends with a single block of type
|
|
.CW VtPointerType2 ,
|
|
but in general trees can have height up to
|
|
.CW VtPointerType6 .
|
|
For a
|
|
.I dsize
|
|
of 8192 bytes
|
|
and
|
|
.I psize
|
|
of 8180 bytes (409 pointers),
|
|
this gives a maximum stream size of approximately 10 zettabytes
|
|
(2\s-2\u73\d\s+2 or 10\s-2\u22\d\s+2 bytes).
|
|
.PP
|
|
Data blocks are truncated to remove trailing runs of zeros before
|
|
storage to Venti; they are zero-filled back to
|
|
.I dsize
|
|
bytes after retrieval from Venti.
|
|
Similarly, trailing runs of pointers to zero-length blocks are
|
|
removed from and added back to pointer blocks.
|
|
These simple rules happen to make it particularly efficient to store
|
|
large runs of zeros, as might occur in a data stream with ``holes:''
|
|
the zero-length block itself can be interpreted as a tree of any
|
|
depth encoding an all-zero data stream.
|
|
.PP
|
|
Reconstructing the data stream requires the score and type of the
|
|
topmost block in the tree, the data chunk size, the pointer chunk size,
|
|
and the data stream size.
|
|
(From the data stream size and the chunk sizes we could derive the
|
|
depth of the tree and thus the type of the topmost block, but it is convenient
|
|
to allow trees that are deeper than necessary.)
|
|
This information is kept in a 40-byte structure called a
|
|
.CW VtEntry :
|
|
.P1
|
|
VtEntry:
|
|
.ta +\w' 'u +\w' 'u
|
|
gen[4] \fRgeneration number\fP
|
|
psize[2] \fRsize of pointer blocks\fP
|
|
dsize[2] \fRsize of data blocks\fP
|
|
flags[1]
|
|
zero[5]
|
|
size[6] \fRlength of file\fP
|
|
score[20] \fRscore of root block in tree\fP
|
|
.P2
|
|
(In this notation,
|
|
.CW name[sz]
|
|
indicates a
|
|
.CW sz -byte
|
|
field called
|
|
.CW name .
|
|
Integers are stored in big-endian order.
|
|
.CW Size
|
|
really is a 48-bit field.)
|
|
.CW Flags
|
|
is made up of the following bit fields.
|
|
.P1
|
|
.ta +\w' 'u +\w' 'u
|
|
0x01 VtEntryActive \fRentry is allocated\fP
|
|
0x02 VtEntryDir \fRentry describes a Venti directory (q.v.)\fP
|
|
0x1C VtEntryDepthMask \fRmask for tree depth\fP
|
|
0x20 VtEntryLocal \fRreserved (q.v.)\fP
|
|
.P2
|
|
.LP
|
|
The depth of the described tree is stored in the 3 bits indicated:
|
|
a tree with a topmost node of type
|
|
.CW VtPointerType3
|
|
has depth 4.
|
|
.PP
|
|
With
|
|
.CW VtEntry
|
|
we can build more complicated data structures,
|
|
ones with multiple or nested data streams.
|
|
A data stream consisting of
|
|
.CW VtEntry
|
|
structures is called a Venti directory.
|
|
It is identical in structure to the Venti data stream
|
|
we described earlier except that the bottom-level type is
|
|
.CW VtDirType ,
|
|
and
|
|
the
|
|
.CW VtEntry
|
|
describing a Venti directory has the
|
|
.CW VtEntryDir
|
|
flag bit set.
|
|
The
|
|
.I dsize
|
|
for a Venti directory
|
|
is a multiple of 40 so that each data chunk holds
|
|
an integer number of
|
|
.CW VtEntry
|
|
structures.
|
|
By analogy with Venti directories,
|
|
we call the original data stream a
|
|
Venti file.
|
|
Note that Venti files are assumed
|
|
.I not
|
|
to contain pointers to other Venti blocks.
|
|
The only pointers to Venti blocks occur in
|
|
.CW VtEntry
|
|
structures in
|
|
Venti directories
|
|
(and in the internal hash tree structure of the
|
|
individual files and directories).
|
|
Note also that these directories are nothing more than pointer lists.
|
|
In particular, there are no names or metadata like in a file system.
|
|
.PP
|
|
To make it easier to pass hierarchies between applications,
|
|
the root of a hierarchy is described in a 300-byte structure
|
|
called a
|
|
.CW VtRoot :
|
|
.P1
|
|
VtRoot:
|
|
.ta +\w' 'u +\w' 'u
|
|
version[2] \f(CW2\fP
|
|
name[128] \fRname of structure (just a comment)\fP
|
|
type[128] \fRstring describing structure (\f(CWvac\fR)\f(CW
|
|
score[20] \fRpointer to \f(CWVtDirType\fP block\f(CW
|
|
blockSize[2] \fRmaximum block size in structure\fP
|
|
prev[20] \fRprevious \f(CWVtRoot\fP in chain, if any\fP
|
|
.P2
|
|
.LP
|
|
This structure is stored to Venti and its score is passed
|
|
between applications, typically in the form
|
|
``\fItype\f(CW:\fIrootscore\fR,''
|
|
where
|
|
.I type
|
|
is the type field from the
|
|
.CW VtRoot
|
|
structure, and
|
|
.I rootscore
|
|
is the score of the
|
|
.CW VtRoot
|
|
block.
|
|
.CW VtRoot
|
|
structures can be chained together using the
|
|
.I prev
|
|
field to encode an archival history
|
|
of the data structure.
|
|
.PP
|
|
For example, a small Venti hierarchy might look like:
|
|
.PS
|
|
.ps 8
|
|
.vs 10
|
|
boxwid=0.1
|
|
boxht=0.1
|
|
f=0.9
|
|
mb=0.16
|
|
|
|
VtRoot: [
|
|
right
|
|
B1: box
|
|
move right 0.1
|
|
"\f(CWVtRoot\fP" ljust
|
|
]
|
|
|
|
Root: [
|
|
right
|
|
B1: box fill f
|
|
B2: box fill f
|
|
B3: box fill f
|
|
move right 0.1
|
|
] with .nw at VtRoot.sw+(0.2,-.1)
|
|
Level1: [
|
|
RootMeta: [
|
|
box wid mb
|
|
]
|
|
MetaSource: [
|
|
right
|
|
B1: box wid 5*mb
|
|
] with .nw at RootMeta.sw+(0,-.1)
|
|
|
|
Source: [
|
|
right
|
|
B1: box fill f
|
|
B2: box fill f
|
|
B3: box fill f
|
|
B4: box fill f
|
|
B5: box fill f
|
|
B6: box fill f
|
|
B7: box fill f
|
|
B8: box fill f
|
|
] with .nw at MetaSource.sw+(0,-.1)
|
|
SB1: box invis at Source.B1
|
|
SB2: box invis at Source.B2
|
|
SB3: box invis at Source.B3
|
|
] with .nw at Root.sw+(0.4,-.1)
|
|
Level2: [
|
|
MetaSource: [
|
|
right
|
|
B1: box wid 5*mb
|
|
]
|
|
Source: [
|
|
right
|
|
B1: box fill f
|
|
B2: box fill f
|
|
B3: box fill f
|
|
B4: box fill f
|
|
B5: box fill f
|
|
B6: box fill f
|
|
B7: box fill f
|
|
B8: box fill f
|
|
] with .nw at MetaSource.sw+(0,-.1)
|
|
File: box wid 0.8 with .nw at Source.sw+(0,-.1)
|
|
] with .nw at Level1.sw+(0.6,-.1)
|
|
|
|
line -> from VtRoot.B1 down boxwid/2+0.1+boxwid/2 then to Root.w
|
|
line -> from Root.B3 down boxwid/2+0.1+boxwid/2 then to Level1.RootMeta.w
|
|
line -> from Root.B2 down boxwid/2+0.1+boxwid+0.1+boxwid/2 then to Level1.MetaSource.w
|
|
line -> from Root.B1 down boxwid/2+0.1+boxwid+0.1+boxwid+0.1+boxwid/2 then to Level1.Source.w
|
|
|
|
line -> from Level1.SB3 down boxwid/2+0.1+boxwid/2 then to Level2.MetaSource.w
|
|
line -> from Level1.SB2 down boxwid/2+0.1+boxwid+0.1+boxwid/2 then to Level2.Source.w
|
|
line -> from Level1.SB1 down boxwid/2+0.1+boxwid+0.1+boxwid+0.1+boxwid/2 then to Level2.File.w
|
|
|
|
[
|
|
KEY: box wid 1.5 invis "Key"
|
|
line from KEY.sw to KEY.se
|
|
k = -.1
|
|
kk=0.5
|
|
A: [
|
|
box wid 4*boxwid
|
|
"Venti file" ljust with .w at last box .w+(kk,0)
|
|
] with .nw at KEY.sw+(0,2*k)
|
|
B: [
|
|
box fill f
|
|
"Venti entry (\f(CWVtEntry\fP)" ljust with .w at last box .w+(kk,0)
|
|
] with .nw at A.sw+(0,k)
|
|
C: [
|
|
right
|
|
CC: box fill f
|
|
box fill f
|
|
box fill f
|
|
box fill f
|
|
"Venti directory" ljust with .w at CC.w+(kk,0)
|
|
] with .nw at B.sw+(0,k)
|
|
D: [
|
|
line -> right 3*boxwid
|
|
"Venti pointer (score)" ljust with .w at last line .w+(kk, 0)
|
|
] with .nw at C.sw+(0,k)
|
|
] with .nw at VtRoot.nw+(3,0)
|
|
.PE
|
|
.LP
|
|
Venti files are shown as white boxes, while directories are shown
|
|
as shaded boxes. Each shaded square represents a
|
|
.CW VtEntry .
|
|
Arrows represent pointers from
|
|
.CW VtEntry
|
|
structures to other
|
|
Venti files or directories.
|
|
.PP
|
|
The hierarchical structure provided by Venti files and directories
|
|
can be used as the base for more complicated data structures.
|
|
Because this structure captures all the information
|
|
about pointers to other blocks, tools written to traverse
|
|
Venti hierarchies can traverse the more complicated
|
|
data structures as well.
|
|
For example,
|
|
.I venti/copy
|
|
(see
|
|
.I venti (1))
|
|
copies a Venti hierarchy from one Venti server to another,
|
|
given the root
|
|
.CW VtEntry .
|
|
Because the traditional file system described in later sections is
|
|
layered on a Venti hierarchy,
|
|
.I venti/copy
|
|
can copy it without fully understanding its structure.
|
|
.NH 1
|
|
Vac file system format
|
|
.HP
|
|
The Venti archive format
|
|
.I vac
|
|
builds a traditional file system using a Venti hierarchy.
|
|
Each vac file is implemented as a Venti file;
|
|
each vac directory is implemented as a Venti
|
|
directory and a Venti file to provide traditional file system metadata.
|
|
The metadata is stored in a structure called a
|
|
.CW DirEntry :
|
|
.P1
|
|
DirEntry:
|
|
.ta +\w' 'u +\w' 'u
|
|
magic[4] \f(CW0x1c4d9072\fP (DirMagic)\fP
|
|
version[2] \f(CW9\fP
|
|
elem[s] \fRname (final path element only)\fP
|
|
entry[4] \fRentry number for Venti file or directory\fP
|
|
gen[4] \fRgeneration number\fP
|
|
mentry[4] \fRentry number for Venti file holding metadata\fP
|
|
mgen[4] \fRgeneration number\fP
|
|
qid[8] \fRunique file serial number\fP
|
|
uid[s] \fRowner\fP
|
|
gid[s] \fRgroup\fP
|
|
mid[s] \fRlast modified by\fP
|
|
mtime[4] \fRlast modification time\fP
|
|
ctime[4] \fRcreation time\fP
|
|
atime[4] \fRlast access time\fP
|
|
mode[4] \fRmode bits\fP
|
|
.P2
|
|
The notation
|
|
.CW name[s]
|
|
denotes a string stored as a two-byte length
|
|
and then that many bytes.
|
|
The above describes Version 9 of the
|
|
.CW DirEntry
|
|
format. Versions 7 and 8 are very similar; they can be
|
|
read by the current
|
|
.I vac
|
|
source code but are not written.
|
|
Earlier versions were not widespread.
|
|
A
|
|
.CW DirEntry
|
|
may be followed by optional extension sections, though none
|
|
are currently used.
|
|
The
|
|
.CW mode
|
|
bits include bits commonly used by
|
|
Unix and Windows, in addition to those used by Plan 9.
|
|
.PP
|
|
The
|
|
.CW entry
|
|
field is an index into the parallel Venti directory.
|
|
The
|
|
.CW gen
|
|
field must match the
|
|
.CW gen
|
|
field in the corresponding
|
|
.CW VtEntry
|
|
in the directory;
|
|
it is used to detect
|
|
stale indices.
|
|
Similarly,
|
|
.CW mentry
|
|
and
|
|
.CW mgen
|
|
are the index and generation number
|
|
for the metadata Venti file,
|
|
if the
|
|
.CW DirEntry
|
|
describes a vac directory.
|
|
.PP
|
|
The relation between Venti files and directories and
|
|
vac files and directories can be seen in this figure:
|
|
.PS
|
|
.ps 8
|
|
.vs 10
|
|
boxwid=0.1
|
|
boxht=0.1
|
|
f=0.9
|
|
mb=0.16
|
|
|
|
VtRoot: [
|
|
right
|
|
B1: box
|
|
move right 0.1
|
|
"\f(CWVtRoot\fP" ljust
|
|
]
|
|
|
|
SuperRoot: [
|
|
right
|
|
B1: box fill f
|
|
move right 0.1
|
|
"fs root block" ljust
|
|
] with .nw at VtRoot.sw + (0.2, -.2)
|
|
Root: [
|
|
right
|
|
B1: box fill f
|
|
B2: box fill f
|
|
B3: box fill f
|
|
move right 0.1
|
|
"root directory info block" ljust
|
|
] with .nw at SuperRoot.sw+(0.2, -.2)
|
|
Level1: [
|
|
RootMeta: [
|
|
box wid mb
|
|
move right 0.1
|
|
"root metadata" ljust
|
|
]
|
|
MetaSource: [
|
|
right
|
|
B1: box wid mb
|
|
B2: box wid mb
|
|
B3: box wid mb
|
|
B4: box wid mb
|
|
B5: box wid mb
|
|
] with .nw at RootMeta.sw+(0,-.2)
|
|
MB1: box wid mb invis at MetaSource.B1
|
|
MB2: box wid mb invis at MetaSource.B2
|
|
MB3: box wid mb invis at MetaSource.B3
|
|
MB4: box wid mb invis at MetaSource.B4
|
|
MB5: box wid mb invis at MetaSource.B5
|
|
|
|
Source: [
|
|
right
|
|
B1: box fill f
|
|
B2: box fill f
|
|
B3: box fill f
|
|
B4: box fill f
|
|
B5: box fill f
|
|
B6: box fill f
|
|
B7: box fill f
|
|
B8: box fill f
|
|
] with .nw at MetaSource.sw+(0,-.1)
|
|
SB1: box invis at Source.B1
|
|
SB2: box invis at Source.B2
|
|
SB3: box invis at Source.B3
|
|
SB4: box invis at Source.B4
|
|
SB5: box invis at Source.B5
|
|
SB6: box invis at Source.B6
|
|
SB7: box invis at Source.B7
|
|
SB8: box invis at Source.B8
|
|
] with .nw at Root.sw+(0.4,-.2)
|
|
Level2: [
|
|
MetaSource: [
|
|
right
|
|
B1: box wid mb
|
|
B2: box wid mb
|
|
B3: box wid mb
|
|
B4: box wid mb
|
|
B5: box wid mb
|
|
]
|
|
Source: [
|
|
right
|
|
B1: box fill f
|
|
B2: box fill f
|
|
B3: box fill f
|
|
B4: box fill f
|
|
B5: box fill f
|
|
B6: box fill f
|
|
B7: box fill f
|
|
B8: box fill f
|
|
] with .nw at MetaSource.sw+(0,-.1)
|
|
File: box wid 0.8 with .nw at Source.sw+(0,-.2)
|
|
] with .nw at Level1.sw+(0.6,-.2)
|
|
|
|
line -> from VtRoot.B1 down boxwid/2+0.2+boxwid/2 then to SuperRoot.w
|
|
line -> from SuperRoot.B1 down boxwid/2+0.2+boxwid/2 then to Root.w
|
|
line -> from Root.B3 down boxwid/2+0.2+boxwid/2 then to Level1.RootMeta.w
|
|
line -> from Root.B2 down boxwid/2+0.2+boxwid+0.2+boxwid/2 then to Level1.MetaSource.w
|
|
line -> from Root.B1 down boxwid/2+0.2+boxwid+0.1+boxwid+0.2+boxwid/2 then to Level1.Source.w
|
|
|
|
line -> from Level1.SB3 down boxwid/2+0.2+boxwid/2 then to Level2.MetaSource.w
|
|
line -> from Level1.SB2 down boxwid/2+0.2+boxwid+0.1+boxwid/2 then to Level2.Source.w
|
|
line -> from Level1.SB1 down boxwid/2+0.2+boxwid+0.1+boxwid+0.2+boxwid/2 then to Level2.File.w
|
|
|
|
arrowwid = arrowwid/2
|
|
arrowht = arrowht/2
|
|
line -> from Level1.MB1 to Level1.SB1.n
|
|
line -> from Level1.MB2 to Level1.SB2.n
|
|
line -> from Level1.MB2 to Level1.SB3.n
|
|
line -> from Level1.MB4 to Level1.SB7.n
|
|
line -> from Level1.MB5 to Level1.SB5.n
|
|
arrowwid = arrowwid * 2
|
|
arrowht = arrowht * 2
|
|
|
|
box dashed with .nw at Level1.MetaSource.nw+(-.05,.05) wid 0.8+.05*2 ht .3+.05*2
|
|
box dashed with .nw at Level2.MetaSource.nw+(-.05,.05) wid 0.8+.05*2 ht .3+.05*2
|
|
box dotted with .nw at Level2.File.nw+(-.05,.05) wid 0.8+0.05*2 ht .1+.05*2
|
|
|
|
[
|
|
KEY: box wid 1.5 invis "Key"
|
|
line from KEY.sw to KEY.se
|
|
k = -.1
|
|
kk=0.5
|
|
A: [
|
|
box wid 4*boxwid
|
|
"Venti file" ljust with .w at last box .w+(kk,0)
|
|
] with .nw at KEY.sw+(0,2*k)
|
|
B: [
|
|
box fill f
|
|
"Venti entry (\f(CWEntry\fP)" ljust with .w at last box .w+(kk,0)
|
|
] with .nw at A.sw+(0,k)
|
|
C: [
|
|
right
|
|
CC: box fill f
|
|
box fill f
|
|
box fill f
|
|
box fill f
|
|
"Venti directory" ljust with .w at CC.w+(kk,0)
|
|
] with .nw at B.sw+(0,k)
|
|
D: [
|
|
line -> right 3*boxwid
|
|
"Venti pointer (score)" ljust with .w at last line .w+(kk, 0)
|
|
] with .nw at C.sw+(0,k)
|
|
DD: [
|
|
box dotted wid 4*boxwid
|
|
"Vac file" ljust with .w at last box .w+(kk,0)
|
|
] with .nw at D.sw+(0,k)
|
|
E: [
|
|
box wid mb
|
|
"Vac entry (\f(CWDirEntry\fP)" ljust with .w at last box .w+(kk,0)
|
|
] with .nw at DD.sw+(0,k)
|
|
G: [
|
|
box dashed wid 4*boxwid
|
|
"Vac directory" ljust with .w at last box .w+(kk,0)
|
|
] with .nw at E.sw+(0,k)
|
|
H: [
|
|
arrowwid = arrowwid/2
|
|
arrowht = arrowht/2
|
|
line -> right 1.5*boxwid
|
|
"Vac pointer (integer index)" ljust with .w at last line .w+(kk, 0)
|
|
arrowwid = arrowwid * 2
|
|
arrowht = arrowht * 2
|
|
] with .nw at G.sw+(0,k)
|
|
] with .nw at VtRoot.nw+(3,0)
|
|
.PE
|
|
.LP
|
|
In reality, the story is slightly more complicated.
|
|
The metadata file in a Vac directory
|
|
is not just the concatenation of
|
|
.CW DirEntry
|
|
structures.
|
|
Instead, it is the concatenation of
|
|
.CW MetaBlocks .
|
|
A
|
|
.CW MetaBlock
|
|
contains some number of
|
|
.CW DirEntry
|
|
structures along with a sorted index to make it easy
|
|
to look for a particular
|
|
.CW DirEntry
|
|
by its
|
|
.CW elem
|
|
field.
|
|
The details are in the source code.
|
|
.PP
|
|
As shown in the diagram,
|
|
the root directory of the file system is summarized by
|
|
three
|
|
.CW VtEntry
|
|
structures describing
|
|
the Venti directory for the children of the root,
|
|
the Venti file for the metadata describing the children of the root,
|
|
and a Venti file holding metadata for the root directory itself.
|
|
These
|
|
.CW VtEntry
|
|
structures are placed in a Venti directory of their own,
|
|
described by the single
|
|
.CW VtEntry
|
|
in the
|
|
root block.
|
|
.NH 1
|
|
Fossil file system format
|
|
.HP
|
|
Fossil uses the vac format, with some small changes.
|
|
The changes only affect the data on the local disk; the data
|
|
archived to Venti is exactly in vac format.
|
|
.PP
|
|
Blocks stored on local disk may contain scores pointing at local disk
|
|
blocks or at Venti blocks.
|
|
Local block addresses are stored as 20-byte scores in which the first 16 bytes
|
|
are all zero and the last 4 bytes specify a block number in the disk.
|
|
Before a block is archived, all the
|
|
blocks it points to must be archived, and the local scores in the block
|
|
must be changed to Venti scores.
|
|
Using block addresses rather than content hashes for local data
|
|
makes the local file system easier to manage: if a local block's contents
|
|
change, the pointer to the block does not need to change.
|
|
.NH 2
|
|
Snapshots
|
|
.HP
|
|
Fossil is an archival file server.
|
|
It takes periodic snapshots of the file system,
|
|
which are made accessible through the file system.
|
|
Specifically, the active file system is presented in
|
|
.CW /active .
|
|
Ephemeral snapshots (those that are kept on local disk and eventually deleted)
|
|
are presented in
|
|
\f(CW/snapshot/\fIyyyy\f(CW/\fImmdd\f(CW/\fIhhmm\fR,
|
|
where
|
|
.I yyyy
|
|
is the full year,
|
|
.I mm
|
|
is the month number,
|
|
.I dd
|
|
is the day number,
|
|
.I hh
|
|
is the hour,
|
|
and
|
|
.I mm
|
|
is the minute.
|
|
Archival snapshots (those that are archived to Venti and persist forever)
|
|
are presented in
|
|
\f(CW/archive/\fIyyyy\f(CW/\fImmdds\fR,
|
|
where
|
|
.I yyyy ,
|
|
.I mm ,
|
|
and
|
|
.I dd
|
|
are year, month, and day as before,
|
|
and
|
|
.I s
|
|
is a sequence number if more than one
|
|
archival snapshot is done in a day.
|
|
For the first snapshot,
|
|
.I s
|
|
is null.
|
|
For the subsequent snapshots,
|
|
.I s
|
|
is
|
|
.CW .1 ,
|
|
.CW .2 ,
|
|
.CW .3 ,
|
|
etc.
|
|
.PP
|
|
To implement the snapshots, the file server maintains a
|
|
current
|
|
.I epoch
|
|
for the active file system.
|
|
Each local block has a label that records, among other things,
|
|
the epoch in which the block was allocated.
|
|
If a block was allocated in an epoch earlier than the current one,
|
|
it is immutable and treated as copy-on-write.
|
|
Taking a snapshot can be accomplished by
|
|
recording the address of the current root block and then
|
|
incrementing the epoch number.
|
|
Notice that the copy-on-write method makes
|
|
snapshots both time efficient and space efficient.
|
|
The only time cost is waiting for all current file system
|
|
requests to finish and then incrementing a counter.
|
|
After a snapshot, blocks only get copied when they are
|
|
next modified, so the per-snapshot
|
|
space requirement is proportional
|
|
to the amount of new data rather than the total
|
|
size of the file system.
|
|
.PP
|
|
The blocks in the archival snapshots are moved to Venti,
|
|
but the blocks in the ephemeral snapshots take up space
|
|
in the local disk file.
|
|
To allow reclamation of this disk space, the file system
|
|
maintains a
|
|
.I low
|
|
.I epoch ,
|
|
which is the epoch of the earliest ephemeral snapshot
|
|
still available.
|
|
Fossil only allows access to snapshots with epoch numbers
|
|
between the
|
|
low epoch and the current epoch
|
|
(also called the high epoch).
|
|
Incrementing the low epoch thus makes old
|
|
snapshots inaccessible.
|
|
The space required to store those snapshots can then
|
|
be reclaimed, as described below.
|
|
.NH 2
|
|
Local blocks
|
|
.HP
|
|
The bulk of the local disk file is the local blocks.
|
|
Each block has a 14-byte label associated with it, of the format:
|
|
.P1
|
|
Label:
|
|
.ta +\w' 'u +\w' 'u
|
|
state[1] \fRblock state\fP
|
|
type[1] \fRblock type\fP
|
|
epoch[4] \fRallocation epoch\fP
|
|
epochClose[4] \fRclose epoch\fP
|
|
tag[4] \fRrandom tag\fP
|
|
.P2
|
|
.LP
|
|
The
|
|
.CW type
|
|
is an analogue of the block types described earlier,
|
|
though different names are used, to distinguish between
|
|
pointers blocks in a hash tree for a data stream
|
|
and pointer blocks for a directory stream.
|
|
The
|
|
.CW epoch
|
|
was mentioned in the last section.
|
|
The other fields are explained below.
|
|
.PP
|
|
There are two distinguished blocks states
|
|
.CW BsFree
|
|
.CW 0x00 ) (
|
|
and
|
|
.CW BsBad
|
|
.CW 0xFF ), (
|
|
which mark blocks that are available for allocation
|
|
and blocks that are bad and should be avoided.
|
|
If
|
|
.CW state
|
|
is not one of these values, it is a bitwise
|
|
.I or ' `
|
|
of the following flags:
|
|
.P1
|
|
.ta +\w' 'u +\w' 'u
|
|
0x01 BsAlloc \fRblock is in use\fP
|
|
0x02 BsCopied \fRblock has been copied\fP
|
|
0x04 BsVenti \fRblock has been stored on Venti\fP
|
|
0x08 BsClosed \fRblock has been unlinked from active file system\fP
|
|
.P2
|
|
.LP
|
|
The flags are explained as they arise in the discussions below.
|
|
.PP
|
|
It is convenient to store some extra fields in the
|
|
.CW VtEntry
|
|
structure when it describes a Venti file or directory
|
|
stored on local disk.
|
|
Specifically, we set the
|
|
.CW VtEntryLocal
|
|
flag bit
|
|
and then use the bytes 7-16 of the score (which would
|
|
otherwise be zero, since it is a local score) to hold these fields:
|
|
.P1
|
|
.ta +\w' 'u +\w' 'u
|
|
archive[1] \fRboolean: this is an archival snapshot\fP
|
|
snap[4] \fRepoch number if root of snapshot\fP
|
|
tag[4] \fRrandom tag\fP
|
|
.P2
|
|
.LP
|
|
The extended
|
|
.CW VtEntry
|
|
structure is called an
|
|
.CW Entry .
|
|
The
|
|
.CW tag
|
|
field
|
|
in the
|
|
.CW Label
|
|
and the
|
|
.CW Entry
|
|
is used to identify dangling pointers or other file system corruption:
|
|
all the local blocks in a hash tree must
|
|
have tags matching the tag in the
|
|
.CW Entry .
|
|
If this
|
|
.CW Entry
|
|
points at the root of a snapshot,
|
|
the
|
|
.CW snap
|
|
field is the epoch of the snapshot.
|
|
If the snapshot is intended to be archived to Venti,
|
|
the
|
|
.CW archive
|
|
field is non-zero.
|
|
.NH 2
|
|
Block reclamation
|
|
.HP
|
|
The blocks in the active file system form a tree: each
|
|
block has only one parent.
|
|
Once a copy-on-write block
|
|
.I b
|
|
is replaced by its copy, it is no longer
|
|
needed by the active file system.
|
|
At this point,
|
|
.I b
|
|
is unlinked from the active file system.
|
|
We say that
|
|
.I b
|
|
is now
|
|
.I closed :
|
|
it is needed only for snapshots.
|
|
When a block is closed, the
|
|
.CW BsClosed
|
|
bit is set in its state, and the current epoch (called the block's closing epoch)
|
|
is stored in the
|
|
.CW epochClose
|
|
label field.
|
|
(Open blocks have an
|
|
.CW epochClose
|
|
of
|
|
.CW ~0 ).
|
|
.PP
|
|
A block is referenced by snapshots with epochs
|
|
between the block's allocation epoch and its closing epoch.
|
|
Once the file system's low epoch grows to be greater than or equal to the block's
|
|
closing epoch, the block is no longer needed for any snapshots
|
|
and can be reused.
|
|
.PP
|
|
In a typical configuration, where nightly archival snapshots
|
|
are taken and written to Venti, it is desirable to reclaim
|
|
the space occupied by now-archived blocks if possible.
|
|
To do this, Fossil keeps track of whether the pointers
|
|
in each block are unique to that block.
|
|
When a block
|
|
.I bb
|
|
is allocated, a pointer to
|
|
.I bb
|
|
is written into exactly one active block (say,
|
|
.I b ).
|
|
In the absence of snapshots, the pointer to
|
|
.I bb
|
|
will remain unique to
|
|
.I b ,
|
|
so that if the pointer is zeroed,
|
|
.I bb
|
|
can be immediately reused.
|
|
Snapshots complicate this invariant:
|
|
when
|
|
.I b
|
|
is copied-on-write, all its pointers
|
|
are no longer unique to it.
|
|
At time of the copy, the
|
|
.CW BsCopied
|
|
state bit in the block's label
|
|
is set to note the duplication of the pointers contained within.
|
|
.NH 2
|
|
Disk layout
|
|
.HP
|
|
The file system header describes the file system layout and has this format:
|
|
.P1
|
|
.ta +\w' 'u +\w' 'u
|
|
Header:
|
|
magic[4] \fR0x3776AE89 (HeaderMagic)\fP
|
|
version[2] \fR1 (HeaderVersion)\fP
|
|
blockSize[2] \fIfile system block size\fP
|
|
super[4] \fRblock offset of super block\fP
|
|
label[4] \fRblock offset of labels\fP
|
|
data[4] \fRdata blocks\fP
|
|
end[4] \fRend of file system\fP
|
|
.P2
|
|
.LP
|
|
The corresponding file system layout is:
|
|
.PS
|
|
.ps 8
|
|
.vs 9
|
|
boxwid=0.75
|
|
boxht=0.15
|
|
Empty: box "empty" ht 0.25
|
|
Header: box "header" with .n at Empty.s
|
|
Empty2: box "empty" with .n at Header.s
|
|
Super: box "super block" with .n at Empty2.s
|
|
Label: box "label" "blocks" with .n at Super.s ht 0.25
|
|
Data: box "data" "blocks" with .n at Label.s ht 0.3
|
|
" 0" ljust at Empty.ne
|
|
" 128kB" ljust at Header.ne
|
|
" \f5super\fP \(mu \f(CWblockSize\fP" ljust at Super.ne
|
|
" \f5label\fP \(mu \f(CWblockSize\fP" ljust at Label.ne
|
|
" \f5data\fP \(mu \f(CWblockSize\fP" ljust at Data.ne
|
|
" \f5end\fP \(mu \f(CWblockSize\fP" ljust at Data.se
|
|
"" at (-1,0)
|
|
"" at (6,0)
|
|
.PE
|
|
.LP
|
|
The numbers to the right of the blocks are byte offsets
|
|
of the boundaries.
|
|
.LP
|
|
The super block describes the file system itself and looks like:
|
|
.P1
|
|
.ta +\w' 'u +\w' 'u
|
|
Super:
|
|
magic[4] \fR0x2340A3B1 (SuperMagic)\fP
|
|
version[2] \fR1 (SuperVersion)\fP
|
|
epochLow[4] \fRfile system low epoch\fP
|
|
epochHigh[4] \fRfile system high (active) epoch\fP
|
|
qid[8] \fRnext qid to allocate\fP
|
|
active[4] \fRdata block number: root of active file system\fP
|
|
next[4] \fRdata block number: root of next file system to archive\fP
|
|
current[4] \fRdata block number: root of file system currently being archived\fP
|
|
last[20] \fRVenti score of last successful archive\fP
|
|
name[128] \fRname of file system (just a comment)\fP
|
|
.P2
|
|
.LP
|
|
.NH 1
|
|
Fossil server
|
|
.HP
|
|
The Fossil server is a user-space program that runs on a standard Plan 9 kernel.
|
|
.NH 2
|
|
Process structure
|
|
.PP
|
|
The file server is structured as a set of processes synchronizing
|
|
mostly through message passing along queues.
|
|
The processes are given names, which can be seen in the output of
|
|
.CW ps
|
|
.CW -a .
|
|
.PP
|
|
.CW Listen
|
|
processes announce on various network addresses.
|
|
A
|
|
.CW con
|
|
process handles each incoming connection, reading 9P requests
|
|
and adding them to a central message queue.
|
|
.CW Msg
|
|
processes remove 9P requests from the queue,
|
|
handle them, and write the responses to the appropriate
|
|
file descriptors.
|
|
.PP
|
|
The
|
|
.CW disk
|
|
process handles disk I/O requests made by the other processes.
|
|
The
|
|
.CW flush
|
|
process writes dirty blocks from the in-memory block cache to disk.
|
|
The
|
|
.CW unlink
|
|
process frees previously linked blocks once the blocks that point at them
|
|
have been written to disk.
|
|
.PP
|
|
A
|
|
.CW consI
|
|
reads from each console file (typically a pipe posted in
|
|
.CW /srv ),
|
|
adding the typed characters to the input queue.
|
|
The
|
|
.CW cons
|
|
process echoes input and runs the commands, saving
|
|
output in a ring buffer.
|
|
Because there is only one
|
|
.CW cons
|
|
process, only one console command may be executing at a time.
|
|
A
|
|
.CW consO
|
|
process copies this ring buffer to each console file.
|
|
.PP
|
|
The
|
|
.CW periodic
|
|
process runs periodic events, like
|
|
flushing the root metadata to disk or
|
|
taking snapshots of the file system.
|
|
.NH 2
|
|
Block cache
|
|
.HP
|
|
Fossil maintains an in-memory block cache which
|
|
holds both local disk blocks and Venti blocks.
|
|
Cache eviction follows a least recently used policy.
|
|
Dirty blocks are restricted to at most half the cache.
|
|
This can be changed by editing
|
|
.CW DirtyPercentage
|
|
in
|
|
.CW dat.h .
|
|
.PP
|
|
The block cache uses soft updates [1] to ensure that the on-disk
|
|
file system is always self-consistent.
|
|
Thus there is no
|
|
.I halt
|
|
console command
|
|
and no need to check a file system
|
|
that was shut down without halting.
|
|
.NH 2
|
|
Archiving
|
|
.HP
|
|
A background process writes blocks in archival snapshots to Venti.
|
|
Although
|
|
.CW /archive/\fIyyyy\fP/\fImmdds\fR
|
|
is a copy of only
|
|
.CW /active
|
|
at the time of the snapshot,
|
|
the archival process archives the
|
|
entire file tree rather than just
|
|
the subtree rooted at
|
|
.CW /active .
|
|
The snapshots
|
|
.CW /snapshot/\fIyyyy\fP/\fImmdd\fP/\fIhhmm
|
|
are stored as empty directories.
|
|
Once all the blocks have been archived,
|
|
a
|
|
.CW VtRoot
|
|
header for the file system is archived.
|
|
The score of that header is recorded in
|
|
.CW super.score
|
|
and also printed on the file server console.
|
|
The score can used by
|
|
.I flfmt
|
|
to restore a file system (see
|
|
.I fossil (4)).
|
|
.NH 2
|
|
Contrast with the old file server
|
|
.HP
|
|
The most obvious difference between Fossil and the
|
|
old Plan 9 file server [2] is that Fossil uses a Venti server as
|
|
its archival storage in place of a WORM juke box.
|
|
There are a few other architectural differences to be
|
|
aware of.
|
|
.PP
|
|
Fossil is a user-level program run on a standard kernel.
|
|
.PP
|
|
Fossil does not have any way to concatenate, stripe, or
|
|
mirror disk files. For functionality similar to the old file server's
|
|
configuration strings, use the experimental file stack device
|
|
(see
|
|
.I fs (3)).
|
|
.PP
|
|
Fossil speaks only 9P2000. Old 9P (aka 9P1) is not supported.
|
|
.PP
|
|
... XXX words about converting an old file system to fossil?
|
|
.NH 1
|
|
References
|
|
.LP
|
|
[1] Gregory R. Ganger, Marshall Kirk McKusick, Craig A. N. Soules,
|
|
and Yale N. Patt.
|
|
``Soft Updates: A Solution to the Metadata Update Problem
|
|
in File Systems,''
|
|
.I "ACM Transactions on Computer Systems" ,
|
|
Vol 18., No. 2, May 2000, pp. 127\-153.
|
|
.LP
|
|
[2] Sean Quinlan, ``A Cached WORM File System,''
|
|
.I "Software\(emPractice and Experience" ,
|
|
Vol 21., No 12., December 1991, pp. 1289\-1299.
|
|
.LP
|
|
[3] Sean Quinlan and Sean Dorward, ``Venti: A New Approach to Archival Storage,''
|
|
.I "Usenix Conference on File and Storage Technologies" ,
|
|
2002.
|