Imported Upstream version 2.4.4p3

[debian/amanda] / docs / DUMPER-API

AMANDA DUMPER API

Last modified $Date: 1998/10/06 17:17:00 $

by Alexandre Oliva <oliva@dcc.unicamp.br>

1. INTRODUCTION

This is a proposal of a mechanism for Amanda to support arbitrary
backup programs, that relies on a generic backup driver and scripts or
programs that interface with backup programs such as dump, tar,
smbclient, and others.  It can also be used to introduce pre- and
post-backup commands.

The interface is simple, but supports everything that is currently
supported by Amanda, and it can be consistently extended to support
new abstractions that may be introduced in the backup driver in the
future.

This proposal does not imply any modification in the Amanda protocol
or in Amanda servers; only Amanda clients have to be modified.  By
Amanda clients, we refer to hosts whose disks are to be backed up;
an Amanda server is a host connected to a tape unit.

Currently (as of release 2.4.1 of Amanda), Amanda clients support
three operations: selfcheck, estimate and backup.

Selfcheck is used by the server program amcheck, to check whether a
client is responding or if there are configuration or permission
problems in the client that might prevent the backup from taking
place.

Estimates are requested by the Amanda planner, that runs on the server
and collects information about the expected sizes of backups of each
disk at several levels.  Given this information and the amount of
available tape space, the planner can select which disks and which
levels it should tell dumper to run.

Dumper is yet another server-side program; it requests clients to
perform dumps, as determined by planner, and stores these dumps in
holding disks or sends them directly to the taper program.  The
interaction between dumper and taper is beyond the scope of this text.

We are going to focus on the interaction between the Amanda client
program and wrappers of dump programs.  These wrappers must implement
the DUMPER API.  The dumptype option `program' should name the wrapper
that will be used to back up filesystems of that dumptype.  One
wrapper may call another, so as to extend its functionality.

2. THE PROBLEM

Different backup programs present distinct requirements; some must be
run as super-user, whereas others can be run under other user-ids.
Some require a directory name, the root of the tree to be backed up;
others prefer a raw device name; some don't even refer to local disks
(SAMBA).  Some wrappers may need to know a filesystem type in order to
decide which particular backup program to use (dump, vdump, vxdump,
xfsdump, backup).

Some provide special options for estimates, whereas others must be
started as if a complete dump were to be performed, and must be killed
as soon as they print an estimate.

Furthermore, the output formats of these backup programs vary wildly.
Some will print estimates and total sizes in bytes, in 512-byte tape
blocks units, in Kbytes, Mbytes, Gbytes, and possibly Tbytes in the
near future.  Some will print a timestamp for the backup; some won't.

There are also restrictions related with possible scheduling policies.
For example, some backup programs only support full backups or
incrementals based on the last full backup (0-1).  Some support full
backups or incrementals based on the last backup, be it a full or an
incremental backup (0-inf++).  Some support incrementals based on a
timestamp (incr/date); whereas others are based on a limited number of
incremental levels, but incrementals of the same level can be
repeated, such as dump (0-9).

Amanda was originally built upon DUMP incremental levels, so this is
the only model it currently supports.  Backup programs that use other
incremental management mechanisms had to be adapted to this policy.
Wrapper scripts are responsible for this adaptation.

Another important issue has to do with index generation.  Some backup
programs can generate indexes, but each one lists files in its own
particular format, but they must be stored in a common format, so that 
the Amanda server can manipulate them.

The DUMPER API must accomodate for all these variations.

3. OVERVIEW OF THE API

We are going to define a standard format of argument lists that the
backup driver will provide to wrapper programs, and the expected
result of the execution of these wrappers.

The first argument to a wrapper should always be a command name.  If
no arguments are given, or an unsupported command is requested, an
error message should be printed to stderr, and the program should
terminate with exit status 1.

3.1.  The `support' command

As a general mechanism for Amanda to probe for features provided by a
backup program, a wrapper script must support at least the `support'
command.  Some features must be supported, and Amanda won't ever ask
about them.  Others will be considered as extensions, and Amanda will
ask the wrapper whether they are supported before issuing the
corresponding commands.

3.1.1. The `level-incrementals' subcommand

For example, before requesting for an incremental backup of a given
level, Amanda should ask the wrapper whether the backup program
supports level-based incrementals.  We don't currently support backup
programs that don't, but we may in the future, so it would be nice if
wrappers already implemented the command `support level-incrementals',
by returning a 0 exit status, printing, say, the maximum incremental
level it supports, i.e., 9.  A sample session would be:

% /usr/local/amanda/libexec/wrappers/DUMP support level-incrementals hda0
9

Note that the result of this support command may depend on filesystem
information, so the disklist filesystem entry should be specified as a
command line argument.  In the next examples, we are not going to use
full pathnames to wrapper scripts any more.

We could have defined a `support' command for full backups, but I
can't think of a backup program that does not support full backups...

3.1.2. The `index' subcommand

The ability to produce index files is also subject to an invocation of
`support' command.  When the support sub-command is `index', like in
the invocation below, the wrapper must print a list of valid indexing
mechanisms, one per line, most preferred first.  If indexing is not
supported, nothing should be printed, and the exit status should be 1.

        DUMP support index hda0

The currently known indexing mechanisms are:

output: implies that the command `index-from-output' generates an
index file from the output produced by the backup program (for
example, from `tar -cv').

image: implies that the command `index-from-image' generates an index
file from a backup image (for example, `tar -t').

direct: implies that the `backup' command can produce an index file as
it generates the backup image.

parse: implies that the `backup-parse' command can produce an index
file as it generates the backup formatted output .

The indexing mechanisms will be explicitly requested with the additionnal
option `index-<mode>' in the `backup' and `backup-parse' command invocation.

`index-from-image' should be supported, if possible, even if other
index commands are not, since it can be used in the future to create
index files from previously backed up filesystems.  

3.1.3. The `parse-estimate' subcommand

The `parse-estimate' support subcommand print a list of valid mechanisms to
parse the estimate output and write the estimate size to its output, the
two mechanisms are:

direct: implies that the `estimate' command can produce the estimate output.

parse: implies that the `estimate-parse' command can produce the estimate
output when fed with the `estimate' output.

The estimate parsing mechanisms will be explicitly requested with the 
additionnal option `estimate-<mode>' in the `estimate' and 
`estimate-parse' command invocation.

3.1.4. The `parse-backup' subcommand

The `parse-backup' support subcommand print a list of valid mechanisms to
parse the backup stderr, the two mechanisms are:

direct: implies that the `backup' command can produce the
backup-formatted-ouput.

parse: implies that the `backup-parse' command can produce the 
backup-formatted-ouput when fed with the `backup' stderr.

The backup parsing mechanisms will be explicitly requested with the 
additionnal option `backup-<mode>' in the `backup' and `backup-parse'
command invocation.

3.1.5. Others subcommands

Some other standard `support' sub-commands are `exclude' and
`exclude-list'.

3.1.6.

One may think (and several people did :-) that there should be only
one support command, that would print information about all supported
commands.  The main arguments against this proposal have to do with
extensibility:

1) the availability of commands might vary from filesystem to
filesystem.  No, I don't have an example, I just want to keep it as
open as possible :-)

2) one support subcommand may require command line arguments that
others don't, and we can't know in advance what these command line
arguments are going to be

3) the output format and exit status conventions of a support command
may vary from command to command; the only pre-defined convention is
that, if a wrapper does not know about a support subcommand, it should
return exit status 1, implying that the inquired feature is not
supported.

3.2. The `selfcheck' command

We should support commands to perform self-checks, run estimates,
backups and restores (for future extensions of the Amanda protocol
so as to support restores)

A selfcheck request would go like this:

        DUMP selfcheck hda0 option option=value ...

The options specified as command-line arguments are dumptype options
enabled for that disk, such as `index', `norecord', etc.  Unknown
options should be ignored.  For each successful check, a message such
as:

OK [/dev/hda0 is readable]
OK [/usr/sbin/dump is executable]

Errors should be printed as:

ERROR [/etc/dumpdates is not writable]

If selfcheck needs super-user (or some other user, for that matter)
access to perform some tests, it should print to the standard output
either:

USER root
GROUP operator

The backup driver should then arrange to re-run the script as the
specified user/group.  Security concerns may impose restrictions on
privileges that can be given to wrapper scripts.  For example, we may
require that, in order to run a wrapper script as any other user or
group, the wrapper script must be in a separate directory, say
/usr/local/amanda/libexec/wrappers-protected, and that the script, its
containing directory and all its parents must only be writable by
root.

The need for starting programs as other users requires amandad (that
will incorporate all the functionality from selfcheck, sendsize and
sendbackup) to be setuid-root.  However, it will fork a child process
and drop to the amanda user privileges as soon as possible.  This
child process will be driven through a pipe, and it will be able to
start services as other users, in a way that no other user, not even
the backup operator, will be able to run arbitrary commands.


A wrapper script will certainly have to figure out either the disk
device name or its mount point, given a filesystem name such as
`hda0', as specified in the disklist.  In order to help these scripts,
Amanda provides a helper program that can guess device names, mount
points and filesystem types, when given disklist entries.

The filesystem type can be useful on some operation systems, in which
more than one dump program is available; this information can help
automatically selecting the appropriate dump program.


The exit status of selfcheck and of this alternate script are probably
going to be disregarded.  Anyway, for consistency, selfcheck should
return exit status 0 for complete success, 1 if any failures have
occurred and 2 if it needs additional permissions (USER/GROUP).  Note
that, if the wrapper needs a special permission to perform a test, it
should not report a failure for that test.

3.3. The `estimate' and `estimate-parse' commands

Estimate requests can be on several different forms.  An estimate of a
full backup may be requested, or estimates for level- or
timestamp-based incrementals:

  DUMP estimate full hda0 option ...
  DUMP estimate level 1 hda0 option ...
  DUMP estimate diff 1998:09:24:01:02:03 hda0 option ...


If the backup program needs privileged access to obtain estimates, it
should just print:

USER root
GROUP operator

and exit, with exit status 2.  If requested estimate type is not
supported, exit status 3 should be returned.

If the option `estimate-direct' is set, then the `estimate' command
should write to stdout the estimated size, in bytes, a pair of numbers
that, multiplied by one another, yield the estimated size in bytes.

If the option `estimate-parse' is set, then the `estimate' command 
should write to stdout the informations needed by the 
`estimate-parse' command, that should extract from its input the 
estimated size.

The syntax of `estimate-parse' is identical to that of `estimate'.

Both `estimate' and `estimate-parse' can output the word `KILL', after
printing the estimate.  In this case, Amanda will send a SIGTERM
signal to the process group of the `estimate' process.  If it does not
die within a few seconds, a SIGKILL will be issued.

If `estimate' or `estimate-parse' succeed, they should exit 0,
otherwise exit 1, except for the already listed cases of exit status 2
and 3.

3.4. The `backup' and `backup-parse' commands

The syntax of `backup' is the same as that of `estimate'.  The backup
image should be written to standard output, whereas stderr should be
used for the user-oriented output of the backup program and other
messages.

If the option `backup-direct' is set, then the `backup' command should 
write to stderr a formatted-output-backup.

If the option `backup-parse' is set, then the `backup' command 
should write to stderr the informations needed by the `backup-parse'
command, that should edit its input so that it prints to standard
output a formatted-output-backup.

If the option `no-record' is set, then the `backup' command should
not modify its state file (ex. dump should not modify /etc/dumpdates).

The syntax of `backup-parse' is identical to that of `backup'.

The syntax of the formatted-output-backup is as follow:
All lines should start with either `| ' for normal output, `? ' for
strange output or `& ' for error output.  If the wrapper can determine
the total backup size from the output of the backup program, it should
print a line starting with `# ', followed by the total backup size in
bytes or by a pair of numbers that, multiplied, yield the total backup
size; this number will be used for consistency check.

The option `index-direct' should cause commands `backup' to output 
the index directly to file descriptor 3.  The option `index-parse' 
should cause commands `backup-parse' to output the index directly to
file descriptor 3.  The syntax of the index file is described in the 
next section.

3.5. The `index-from-output' and `index-from-image' commands

The syntax of the `index-from-output' and `index-from-image' commands
is identical to the one of `backup'.  They are fed the backup output
or image, and they must produce a list of files and directories, one
per line, to the standard output.  Directories must be identified by
the `/' termination.

After the file name and a blank space, any additional information
about the file or directory, such as permission data, size, etc, can
be added.  For this reason, blanks and backslashes within filenames
should be quoted with backslashes.  Linefeeds should be represented as
`\n', although it is not always possible to distinguish linefeeds in
the middle of filenames from ones that separate one file from another,
in the output of, say `restore -t'.  It is not clear whether we should
also support quoting mechanisms such as `\xHH', `\OOO' or `\uXXXX'.

3.6. The `restore' command

Yet to be specified.

3.7. The `print-command' command

This command must be followed by a valid backup or restore command,
and it should print a shell-command that would produce an equivalent
result, i.e., that would perform the backup to standard output, or
that would restore the whole filesystem reading from standard input.
This command is to be included in the header of backup images, to ease
crash-recovery.

4. Conclusion

Well, that's all.  Drop us a note at the amanda-hackers mailing list
if you have suggestions to improve this document and/or the API.  Some
help on its implementation would be welcome too.