2 * Copyright (c) 2009, 2010 Zmanda, Inc. All Rights Reserved.
4 * This program is free software; you can redistribute it and/or modify it
5 * under the terms of the GNU General Public License version 2 as published
6 * by the Free Software Foundation.
8 * This program is distributed in the hope that it will be useful, but
9 * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
10 * or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
13 * You should have received a copy of the GNU General Public License along
14 * with this program; if not, write to the Free Software Foundation, Inc.,
15 * 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
17 * Contact information: Zmanda Inc., 465 S. Mathilda Ave., Suite 300
18 * Sunnyvale, CA 94085, USA, or: http://www.zmanda.com
25 Amanda::Xfer - the transfer architecture
30 use Amanda::Xfer qw( :constants );
33 my $infd = POSIX::open("input", POSIX::O_RDONLY, 0);
34 my $outfd = POSIX::open("output", POSIX::O_CREAT|POSIX::O_WRONLY, 0640);
35 my $xfer = Amanda::Xfer->new([
36 Amanda::Xfer::Source::Fd->new($infd),
37 Amanda::Xfer::Dest::Fd->new($outfd)
40 my ($src, $xmsg, $xfer) = @_;
41 print "Message from $xfer: $xmsg\n"; # use stringify operations
42 if ($msg->{'type'} == $XMSG_DONE) {
43 Amanda::MainLoop::quit();
46 Amanda::MainLoop::run();
48 See L<http://wiki.zmanda.com/index.php/XFA> for background on the
49 transfer architecture.
51 =head1 Amanda::Xfer Objects
53 A new transfer is created with C<< Amanda::Xfer->new() >>, which takes
54 an arrayref giving the transfer elements which should compose the
57 The resulting object has the following methods:
63 Start this transfer. Processing takes place asynchronously, and messages will
64 begin queueing up immediately. If C<$cb> is given, then it is installed as the
65 callback for messages from this transfer. The callback receives three
66 arguments: the event source, the message, and a reference to the controlling
67 transfer. See the description of C<Amanda::Xfer::Msg>, below, for details.
69 There is no need to remove the source on completion of the transfer - that is
74 Stop transferring data. The transfer will send an C<XMSG_CANCEL>,
75 "drain" any buffered data as best it can, and then complete normally
80 Get the transfer's status. The result will be one of C<$XFER_INIT>,
81 C<$XFER_START>, C<$XFER_RUNNING>, or C<$XFER_DONE>. These symbols are
82 available for import with the tag C<:constants>.
86 Return a string representation of this transfer, suitable for use in
87 debugging messages. This method is automatically invoked when a
88 transfer is interpolated into a string:
90 print "Starting $xfer\n";
94 Get the L<Amanda::MainLoop> event source through which messages will
95 be delivered for this transfer. Use its C<set_callback> method to
96 connect a perl sub for processing events.
98 Use of this method is deprecated; instead, pass a callback to the C<start>
99 method. If you set a callback via C<get_source>, then you I<must> C<remove>
100 the source when the transfer is complete!
104 =head1 Amanda::Xfer::Element objects
106 The individual transfer elements that compose a transfer are instances
107 of subclasses of Amanda::Xfer::Element. All such objects have a
108 C<repr()> method, similar to that for transfers, and support a similar
109 kind of string interpolation.
111 Note that the names of these classes contain the words "Source",
112 "Filter", and "Dest". This is merely suggestive of their intended
113 purpose -- there are no such abstract classes.
115 =head2 Transfer Sources
117 =head3 Amanda::Xfer::Source::Device (SERVER ONLY)
119 Amanda::Xfer::Source::Device->new($device);
121 This source reads data from a device. The device should already be
122 queued up for reading (C<< $device->seek_file(..) >>). The element
123 will read until the end of the device file.
125 =head3 Amanda::Xfer::Source::Fd
127 Amanda::Xfer::Source::Fd->new(fileno($fh));
129 This source reads data from a file descriptor. It reads until EOF,
130 but does not close the descriptor. Be careful not to let Perl close
133 =head3 Amanda::Xfer::Source::Holding (SERVER-ONLY)
135 Amanda::Xfer::Source::Holding->new($filename);
137 This source reads data from a holding file (see L<Amanda::Holding>).
138 If the transfer only consists of a C<Amanda::Xfer::Source::Holding>
139 and an C<Amanda::Xfer::Dest::Taper::Splitter> (with no filters), then the source
140 will call the destination's C<cache_inform> method so that it can use
141 holding chunks for a split-part cache.
143 =head3 Amanda::Xfer::Source::Random
145 Amanda::Xfer::Source::Random->new($length, $seed);
147 This source provides I<length> bytes of random data (or an unlimited
148 amount of data if I<length> is zero). C<$seed> is the seed used to
149 generate the random numbers; this seed can be used in a destination to
150 check for correct output.
152 If you need to string multiple transfers together into a coherent sequence of
153 random numbers, for example when testing the re-assembly of spanned dumps, call
155 my $seed = $src->get_seed();
157 to get the finishing seed for the source, then pass this to the source
158 constructor for the next transfer. When concatenated, the bytestreams from the
159 transfers will verify correctly using the original random seed.
161 =head3 Amanda::Xfer::Source::Pattern
163 Amanda::Xfer::Source::Pattern->new($length, $pattern);
165 This source provides I<length> bytes containing copies of
166 I<pattern>. If I<length> is zero, the source provides an unlimited
169 =head3 Amanda::Xfer::Source::Recovery (SERVER ONLY)
171 Amanda::Xfer::Source::Recovery->new($first_device);
173 This source reads a datastream composed of on-device files. Its constructor
174 takes a pointer to the first device that will be read from; this is used
175 internally to determine whether DirectTCP is supported.
177 The element sense C<$XMSG_READY> when it is ready for the first C<start_part>
178 invocation. Don't do anything with the device between the start of the
179 transfer and when the element sends an C<$XMSG_READY>.
181 The element contains no logic to decide I<which> files to assemble into the
182 datastream; instead, it relies on the caller to supply pre-positioned devices:
184 $src->start_part($device);
186 Once C<start_part> is called, the source will read until C<$device> produces an
187 EOF. As each part is completed, the element sends an C<$XMSG_PART_DONE>
188 L<Amanda::Xfer::Msg>, with the following keys:
190 size bytes read from the device
191 duration time spent reading
192 fileno the on-media file number from which the part was read
194 Call C<start_part> with C<$device = undef> to indicate that there are no more
197 To switch to a new device in mid-transfer, use C<use_device>:
199 $dest->use_device($device);
201 This method must be called with a device that is not yet started, and thus must
202 be called before the C<start_part> method is called with a new device.
204 =head3 Amanda::Xfer::Source::DirectTCPListen
206 Amanda::Xfer::Source::DirectTCPListen->new();
208 This source is for use when the transfer data will come in via DirectTCP, with
209 the data's I<source> connecting to the data's I<destination>. Set up the
210 transfer, and after starting it, call its C<get_addrs> method to get an
211 arrayref of ip/port pairs, e.g., C<[ "192.168.4.5", 9924 ]>, all of which are
212 listening for an incoming data connection. Once a connection arrives, this
213 element will read data from it and send those data into the transfer.
215 my $addrs = $src->get_addrs();
217 =head3 Amanda::Xfer::Source::DirectTCPConnect
219 Amanda::Xfer::Source::DirectTCPConnect->new($addrs);
221 This source is for use when the transfer data will come in via DirectTCP, with
222 the data's I<destination> connecting to the the data's I<source>. The element
223 connects to C<$addrs> and reads the transfer data from the connection.
225 =head2 Transfer Filters
227 =head3 Amanda::Xfer::Filter:Process
229 Amanda::Xfer::Filter::Process->new([@args], $need_root);
231 This filter will pipe data through the standard file descriptors of the
232 subprocess specified by C<@args>. If C<$need_root> is true, it will attempt to
233 change to uid 0 before executing the process. Standard output from the process
234 is redirected to the debug log. Note that the process is invoked directly, not
235 via a shell, so shell metacharcters (e.g., C<< 2>&1 >>) will not function as
238 =head3 Amanda::Xfer::Filter:Xor
240 Amanda::Xfer::Filter::Xor->new($key);
242 This filter applies a bytewise XOR operation to the data flowing
245 =head2 Transfer Destinations
247 =head3 Amanda::Xfer::Dest::Device (SERVER ONLY)
249 Amanda::Xfer::Dest::Device->new($device, $max_memory);
251 This source writes data to a device. The device should be ready for
252 writing (C<< $device->start_file(..) >>). No more than C<$max_memory>
253 will be used for buffers. Use zero for the default buffer size. On
254 completion of the transfer, the file will be finished.
256 =head3 Amanda::Xfer::Dest::Buffer
258 Amanda::Xfer::Dest::Buffer->new($max_size);
260 This destination records data into an in-memory buffer which can grow up to
261 C<$max_size> bytes. The buffer is available with the C<get> method, which
262 returns a copy of the buffer as a perl scalar:
264 my $buf = $xdb->get();
266 =head3 Amanda::Xfer::Dest::DirectTCPListen
268 Amanda::Xfer::Dest::DirectTCPListen->new();
270 This destination is for use when the transfer data will come in via DirectTCP,
271 with the data's I<destination> connecting to the data's I<source>. Set up the
272 transfer, and after starting it, call this element's C<get_addrs> method to get
273 an arrayref of ip/port pairs, e.g., C<[ "192.168.4.5", 9924 ]>, all of which
274 are listening for an incoming data connection. Once a connection arrives, this
275 element will write the transfer data to it.
277 my $addrs = $src->get_addrs();
279 =head3 Amanda::Xfer::Dest::DirectTCPConnect
281 Amanda::Xfer::Dest::DirectTCPConnect->new($addrs);
283 This destination is for use when the transfer data will come in via DirectTCP,
284 with the data's I<source> connecting to the the data's I<destination>. The
285 element connects to C<$addrs> and writes the transfer data to the connection.
287 =head3 Amanda::Xfer::Dest::Fd
289 Amanda::Xfer::Dest::Fd->new(fileno($fh));
291 This destination writes data to a file descriptor. The file is not
292 closed after the transfer is completed. Be careful not to let Perl
293 close the file for you!
295 =head3 Amanda::Xfer::Dest::Null
297 Amanda::Xfer::Dest::Null->new($seed);
299 This destination discards the data it receives. If C<$seed> is
300 nonzero, then the element will validate that it receives the data that
301 C<Amanda::Xfer::Source::Random> produced with the same seed. No
302 validation is performed if C<$seed> is zero.
304 =head3 Amanda::Xfer::Dest::Taper (SERVER ONLY)
306 This is the parent class to C<Amanda::Xfer::Dest::Taper::Splitter> and
307 C<Amanda::Xfer::Dest::Taper::DirectTCP>. These subclasses allow a single
308 transfer to write to multiple files (parts) on a device, and even spread those
309 parts over multiple devices, without interrupting the transfer itself.
311 The subclass constructors all take a C<$first_device>, which should be
312 configured but not yet started; and a C<$part_size> giving the maximum size of
313 each part. Note that this value may be rounded up internally as necessary.
315 When a transfer using a taper destination element is first started, no data is
316 transfered until the element's C<start_part> method is called:
318 $dest->start_part($retry_part);
320 where C<$device> is the device to which the part should be written. The device
321 should have a file open and ready to write (that is,
322 C<< $device->start_file(..) >> has already been called). If C<$retry_part> is
323 true, then the previous, unsuccessful part will be retried.
325 As each part is completed, the element sends an C<$XMSG_PART_DONE>
326 C<Amanda::Xfer::Msg>, with the following keys:
328 successful true if the part was written successfully
329 eof recipient should not call start_part again
330 eom this volume is at EOM; a new volume is required
331 size bytes written to volume
332 duration time spent writing, not counting changer ops, etc.
333 partnum the zero-based number of this part in the overall dumpfile
334 fileno the on-media file number used for this part, or 0 if no file
337 If C<eom> is true, then the caller should find a new volume before
338 continuing. If C<eof> is not true, then C<start_part> should be called
339 again, with C<$retry_part = !successful>. Note that it is possible
340 for some destinations to write a portion of a part successfully,
341 but still stop at EOM. That is, C<eom> does not necessarily imply
344 To switch to a new device in mid-transfer, use C<use_device>:
346 $dest->use_device($device);
348 This method must be called with a device that is not yet started.
350 If neither the memory nor disk caches are in use, but the dumpfile is
351 available on disk, then the C<cache_inform> method allows the element
352 to use that on-disk data to support retries. This is intended to
353 support transfers from Amanda's holding disk (see
354 C<Amanda::Xfer::Source::Holding>), but may be useful for other
357 $dest->cache_inform($filename, $offset, $length);
359 This function indicates that C<$filename> contains C<$length> bytes of
360 data, beginning at offset C<$offset> from the beginning of the file.
361 These bytes are assumed to follow immediately after any bytes
362 previously specified to C<cache_inform>. That is, no gaps or overlaps
363 are allowed in the data stream described to C<cache_inform>.
364 Furthermore, the location of each byte must be specified to this
365 method I<before> it is sent through the transfer.
367 $dest->get_part_bytes_written();
369 This function returns the number of bytes written for the current part
372 =head3 Amanda::Xfer::Dest::Taper::Splitter
374 Amanda::Xfer::Dest::Taper::Splitter->new($first_device, $max_memory,
375 $part_size, $use_mem_cache, $disk_cache_dirname);
377 This class caches data from each part in one of a variety of ways, and supports
378 "rewinding" to retry a failed part (e.g., one that does not fit on a device).
379 It assumes that when a device reaches EOM while writing, the entire file is
380 corrupt. It does not (yet) support logical EOM, which can render this
381 assumption incorrect.
383 The C<$first_device> is used to calculate some internal constants, notably the
384 slab size, based on the device characteristics. Subsequent devices must have
385 the same block size. The C<$part_size> and C<$first_device> parameters are
388 If C<$use_mem_cache> is true, each part will be cached in memory (using
389 C<$part_size> bytes of memory; plan accordingly!). If C<$disk_cache_dirname>
390 is defined, then each part will be cached on-disk in a file in this directory.
391 It is an error to specify both in-memory and on-disk caching. If neither
392 option is specified, the element will operate successfully, but will not be
393 able to retry a part unless C<cache_inform> has been used properly (see above).
395 =head3 Amanda::Xfer::Dest::Taper::DirectTCP
397 Amanda::Xfer::Dest::Taper::DirectTCP->new($first_device, $part_size);
399 This class uses the Device API DirectTCP methods to write data to a device via
400 DirectTCP. Since all DirectTCP devices support logical EOM, this class does
401 not cache any data, and will never re-start an unsuccessful part.
403 As state above, C<$first_device> must not be started when C<new> is called.
404 Furthermore, no use of that device is allowed until the element sens an
405 C<$XMSG_READY> to indicate that it is finished with the device. The
406 C<start_part> method must not be called until this method is received either.
408 =head1 Amanda::Xfer::Msg objects
410 Messages are simple hashrefs, with a few convenience methods. Like
411 transfers, they have a C<repr()> method that formats the message
412 nicely, and is available through string interpolation:
414 print "Received message $msg\n";
416 The canonical description of the message types and keys is in
417 C<xfer-src/xmsg.h>, and is not duplicated here. Every message has the
418 following basic keys.
424 The message type -- one of the C<xmsg_type> constants available from
425 the import tag C<:constants>.
429 The transfer element that sent the message.
433 The version of the message. This is used to support extensibility of
438 Additional keys are described in the documentation for the elements
439 that use them. All keys are listed in C<xfer-src/xmsg.h>.