trAvis - MANAGER

Edit File: UUlib.pm

package Convert::UUlib;

use common::sense;

use Carp;

require Exporter;
require DynaLoader;

our $VERSION = 1.8;

our @ISA = qw(Exporter DynaLoader);

our @_consts = qw(
	ACT_COPYING ACT_DECODING ACT_ENCODING ACT_IDLE ACT_SCANNING

FILE_DECODED FILE_ERROR FILE_MISPART FILE_NOBEGIN FILE_NODATA
	FILE_NOEND FILE_OK FILE_READ FILE_TMPFILE

MSG_ERROR MSG_FATAL MSG_MESSAGE MSG_NOTE MSG_PANIC MSG_WARNING

OPT_RBUF OPT_WBUF
	OPT_BRACKPOL OPT_DEBUG OPT_DESPERATE OPT_DUMBNESS OPT_ENCEXT
	OPT_ERRNO OPT_FAST OPT_IGNMODE OPT_IGNREPLY OPT_OVERWRITE OPT_PREAMB
	OPT_PROGRESS OPT_SAVEPATH OPT_TINYB64 OPT_USETEXT OPT_VERBOSE
	OPT_VERSION OPT_REMOVE OPT_MOREMIME OPT_DOTDOT OPT_AUTOCHECK

RET_CANCEL RET_CONT RET_EXISTS RET_ILLVAL RET_IOERR RET_NODATA
	RET_NOEND RET_NOMEM RET_OK RET_UNSUP

B64_ENCODED BH_ENCODED PT_ENCODED QP_ENCODED
	XX_ENCODED UU_ENCODED YENC_ENCODED
);

our @_funcs = qw(
        Initialize CleanUp GetOption SetOption strerror SetMsgCallback
        SetBusyCallback SetFileCallback SetFNameFilter SetFileNameCallback
        FNameFilter LoadFile GetFileListItem GetFileList RenameFile DecodeToTemp
        RemoveTemp DecodeFile InfoFile Smerge QuickDecode EncodeMulti
        EncodePartial EncodeToStream EncodeToFile E_PrepSingle
        E_PrepPartial

straction strencoding strmsglevel
);

our @EXPORT = @_consts;
our @EXPORT_OK = @_funcs;
our %EXPORT_TAGS = (all => [@_consts,@_funcs], constants => \@_consts);

bootstrap Convert::UUlib $VERSION;

# dummy function for compatiiblity with pre-1.7 versions
sub Initialize { }

# action code -> string mapping
sub straction($) {
   return 'copying'	if $_[0] == &ACT_COPYING;
   return 'decoding'	if $_[0] == &ACT_DECODING;
   return 'encoding'	if $_[0] == &ACT_ENCODING;
   return 'idle'	if $_[0] == &ACT_IDLE;
   return 'scanning'	if $_[0] == &ACT_SCANNING;
   'unknown';
}

# encoding type -> string mapping
sub strencoding($) {
   return 'uuencode'		if $_[0] == &UU_ENCODED;
   return 'base64'		if $_[0] == &B64_ENCODED;
   return 'yenc'		if $_[0] == &YENC_ENCODED;
   return 'binhex'		if $_[0] == &BH_ENCODED;
   return 'plaintext'		if $_[0] == &PT_ENCODED;
   return 'quoted-printable'	if $_[0] == &QP_ENCODED;
   return 'xxencode'		if $_[0] == &XX_ENCODED;
   'unknown';
}

sub strmsglevel($) {
   return 'message'	if $_[0] == &MSG_MESSAGE;
   return 'note'	if $_[0] == &MSG_NOTE;
   return 'warning'	if $_[0] == &MSG_WARNING;
   return 'error'	if $_[0] == &MSG_ERROR;
   return 'panic'	if $_[0] == &MSG_PANIC;
   return 'fatal'	if $_[0] == &MSG_FATAL;
   'unknown';
}

1;
__END__

=head1 NAME

Convert::UUlib - decode uu/xx/b64/mime/yenc/etc-encoded data from a massive number of files

=head1 SYNOPSIS

use Convert::UUlib ':all';
 
 # read all the files named on the commandline and decode them
 # into the CURRENT directory. See below for a longer example.
 LoadFile $_ for @ARGV;

for my $uu (GetFileList) {
    if ($uu->state & FILE_OK) {
      $uu->decode;
      print $uu->filename, "\n";
    }
 }

=head1 DESCRIPTION

This module started as an interface to the uulib/uudeview library by Frank
Pilhofer that can be used to decode all kinds of usenet (and other)
binary messages.

After upstream abondoned the project, th library was continuously bugfixed
and improved in this module, with major focuses on security fixes,
correctness and speed (that does not mean that this library is considered
safe with untrusted data, but it surely is safer than the poriginal
uudeview).

Read the file doc/library.pdf from the distribution for in-depth
information about the C-library used in this interface, and the rest of
this document and especially the non-trivial decoder program at the end.

=head1 EXPORTED CONSTANTS

=head2 Action code constants

ACT_IDLE      we don't do anything
  ACT_SCANNING  scanning an input file
  ACT_DECODING  decoding into a temp file
  ACT_COPYING   copying temp to target
  ACT_ENCODING  encoding a file

=head2 Message severity levels

MSG_MESSAGE   just a message, nothing important
  MSG_NOTE      something that should be noticed
  MSG_WARNING   important msg, processing continues
  MSG_ERROR     processing has been terminated
  MSG_FATAL     decoder cannot process further requests
  MSG_PANIC     recovery impossible, app must terminate

=head2 Options

OPT_VERSION	version number MAJOR.MINORplPATCH (ro)
  OPT_FAST	assumes only one part per file
  OPT_DUMBNESS	switch off the program's intelligence
  OPT_BRACKPOL	give numbers in [] higher precendence
  OPT_VERBOSE	generate informative messages
  OPT_DESPERATE	try to decode incomplete files
  OPT_IGNREPLY	ignore RE:plies (off by default)
  OPT_OVERWRITE	whether it's OK to overwrite ex. files
  OPT_SAVEPATH	prefix to save-files on disk
  OPT_IGNMODE	ignore the original file mode
  OPT_DEBUG	print messages with FILE/LINE info
  OPT_ERRNO	get last error code for RET_IOERR (ro)
  OPT_PROGRESS	retrieve progress information
  OPT_USETEXT	handle text messages
  OPT_PREAMB	handle Mime preambles/epilogues
  OPT_TINYB64	detect short B64 outside of Mime
  OPT_ENCEXT	extension for single-part encoded files
  OPT_REMOVE	remove input files after decoding (dangerous)
  OPT_MOREMIME	strict MIME adherence
  OPT_DOTDOT	".."-unescaping has not yet been done on input files
  OPT_RBUF      set default read I/O buffer size in bytes
  OPT_WBUF      set default write I/O buffer size in bytes
  OPT_AUTOCHECK automatically check file list after every loadfile

=head2 Result/Error codes

RET_OK        everything went fine
  RET_IOERR     I/O Error - examine errno
  RET_NOMEM     not enough memory
  RET_ILLVAL    illegal value for operation
  RET_NODATA    decoder didn't find any data
  RET_NOEND     encoded data wasn't ended properly
  RET_UNSUP     unsupported function (encoding)
  RET_EXISTS    file exists (decoding)
  RET_CONT      continue -- special from ScanPart
  RET_CANCEL    operation canceled

=head2 File States

This code is zero, i.e. "false":

UUFILE_READ   Read in, but not further processed

The following state codes are or'ed together:

FILE_MISPART  Missing Part(s) detected
  FILE_NOBEGIN  No 'begin' found
  FILE_NOEND    No 'end' found
  FILE_NODATA   File does not contain valid uudata
  FILE_OK       All Parts found, ready to decode
  FILE_ERROR    Error while decoding
  FILE_DECODED  Successfully decoded
  FILE_TMPFILE  Temporary decoded file exists

=head2 Encoding types

UU_ENCODED    UUencoded data
  B64_ENCODED   Mime-Base64 data
  XX_ENCODED    XXencoded data
  BH_ENCODED    Binhex encoded
  PT_ENCODED    Plain-Text encoded (MIME)
  QP_ENCODED    Quoted-Printable (MIME)
  YENC_ENCODED  yEnc encoded (non-MIME)

=head1 EXPORTED FUNCTIONS

=head2 Initializing and cleanup

Initialize is automatically called when the module is loaded and allocates
quite a small amount of memory for todays machines ;) CleanUp releases that
again.

On my machine, a fairly complete decode with DBI backend needs about 10MB
RSS to decode 20000 files.

=over

=item CleanUp

Release memory, file items and clean up files. Should be called after a
decoidng run, if you want to start a new one.

=back

=head2 Setting and querying options

=over

=item $option = GetOption OPT_xxx

=item SetOption OPT_xxx, opt-value

=back

See the C<OPT_xxx> constants above to see which options exist.

=head2 Setting various callbacks

=over

=item SetMsgCallback [callback-function]

=item SetBusyCallback [callback-function]

=item SetFileCallback [callback-function]

=item SetFNameFilter [callback-function]

=back

=head2 Call the currently selected FNameFilter

=over

=item $file = FNameFilter $file

=back

=head2 Loading sourcefiles, optionally fuzzy merge and start decoding

=over

=item ($retval, $count) = LoadFile $fname, [$id, [$delflag, [$partno]]]

Load the given file and scan it for encoded contents. Optionally tag it
with the given id, and if C<$delflag> is true, delete the file after it
is no longer necessary. If you are certain of the part number, you can
specify it as the last argument.

A better (usually faster) way of doing this is using the C<SetFNameFilter>
functionality.

=item $retval = Smerge $pass

If you are desperate, try to call C<Smerge> with increasing C<$pass>
values, beginning at C<0>, to try to merge parts that usually would not
have been merged.

Most probably this will result in garbled files, so never do this by
default, except:

If the C<OPT_AUTOCHECK> option has been disabled (by default it is
enabled) to speed up file loading, then you I<have> to call C<Smerge -1>
after loading all files as an additional pre-pass (which is normally done
by C<LoadFile>).

=item $item = GetFileListItem $item_number

Return the C<$item> structure for the C<$item_number>'th found file, or
C<undef> of no file with that number exists.

The first file has number C<0>, and the series has no holes, so you can
iterate over all files by starting with zero and incrementing until you
hit C<undef>.

This function has to walk the linear list of fils on each access, so
if you want to iterate over all items, it is usually faster to use
C<GetFileList>.

=item @items = GetFileList

Similar to C<GetFileListItem>, but returns all files in one go, which is
very much faster for large number of items, and has no drawbacks when used
for a small number of items.

=back

=head2 Decoding files

=over

=item $retval = $item->rename ($newname)

Change the ondisk filename where the decoded file will be saved.

=item $retval = $item->decode_temp

Decode the file into a temporary location, use C<< $item->infile >> to
retrieve the temporary filename.

=item $retval = $item->remove_temp

Remove the temporarily decoded file again.

=item $retval = $item->decode ([$target_path])

Decode the file to its destination, or the given target path.

=item $retval = $item->info (callback-function)

=back

=head2 Querying (and setting) item attributes

=over

=item $state    = $item->state

=item $mode     = $item->mode ([newmode])

=item $uudet    = $item->uudet

=item $size     = $item->size

=item $filename = $item->filename ([newfilename})

=item $subfname = $item->subfname

=item $mimeid   = $item->mimeid

=item $mimetype = $item->mimetype

=item $binfile  = $item->binfile

=back

=head2 Information about source parts

=over

=item $parts = $item->parts

Return information about all parts (source files) used to decode the file
as a list of hashrefs with the following structure:

{
   partno   => <integer describing the part number, starting with 1>,
   # the following member sonly exist when they contain useful information
   sfname   => <local pathname of the file where this part is from>,
   filename => <the ondisk filename of the decoded file>,
   subfname => <used to cluster postings, possibly the posting filename>,
   subject  => <the subject of the posting/mail>,
   origin   => <the possible source (From) address>,
   mimetype => <the possible mimetype of the decoded file>,
   mimeid   => <the id part of the Content-Type>,
 }

Usually you are interested mostly the C<sfname> and possibly the C<partno>
and C<filename> members.

=back

=head2 Functions below are not documented and not very well tested - feedback welcome

QuickDecode
  EncodeMulti
  EncodePartial
  EncodeToStream
  EncodeToFile
  E_PrepSingle
  E_PrepPartial

=head2 EXTENSION FUNCTIONS

Functions found in this module but not documented in the uulib documentation:

=over

=item $msg = straction ACT_xxx

Return a human readable string representing the given action code.

=item $msg = strerror RET_xxx

Return a human readable string representing the given error code.

=item $str = strencoding xxx_ENCODED

Return the name of the encoding type as a string.

=item $str = strmsglevel MSG_xxx

Returns the message level as a string.

=item SetFileNameCallback $cb

Sets (or queries) the FileNameCallback, which is called whenever the
decoding library can't find a filename and wants to extract a filename
from the subject line of a posting. The callback will be called with
two arguments, the subject line and the current candidate for the
filename. The latter argument can be C<undef>, which means that no
filename could be found (and likely no one exists, so it is safe to also
return C<undef> in this case). If it doesn't return anything (not even
C<undef>!), then nothing happens, so this is a no-op callback:

sub cb {
      return ();
   }

If it returns C<undef>, then this indicates that no filename could be
found. In all other cases, the return value is taken to be the filename.

This is a slightly more useful callback:

sub cb {
     return unless $_[1]; # skip "Re:"-plies et al.
     my ($subject, $filename) = @_;
     # if we find some *.rar, take it
     return $1 if $subject =~ /(\w+\.rar)/;
     # otherwise just pass what we have
     return ();
  }

=back

=head1 LARGE EXAMPLE DECODER

The general workflow for decoding is like this:

=over

=item 1. Configure options with C<SetOption> or C<SetXXXCallback>.

=item 2. Load all source files with C<LoadFile>.

=item 3. Optionally C<Smerge>.

=item 4. Iterate over all C<GetFileList> items (i.e. result files).

=item 5. C<CleanUp> to delete files and free items.

=back

What follows is the file C<example-decoder> from the distribution that
illustrates the above worklfow in a non-trivial example.

#!/usr/bin/perl

# decode all the files in the directory uusrc/ and copy
   # the resulting files to uudst/

use Convert::UUlib ':all';

sub namefilter {
      my ($path) = @_;

$path=~s/^.*[\/\\]//;

$path
   }

sub busycb {
      my ($action, $curfile, $partno, $numparts, $percent, $fsize) = @_;
      $_[0]=straction($action);
      print "busy_callback(", (join ",",@_), ")\n";
      0
   }

SetOption OPT_RBUF, 128*1024;
   SetOption OPT_WBUF, 1024*1024;
   SetOption OPT_IGNMODE, 1;
   SetOption OPT_IGNMODE, 1;
   SetOption OPT_VERBOSE, 1;
   SetOption OPT_AUTOCHK, 0;

# show the three ways you can set callback functions. I normally
   # prefer the one with the sub inplace.
   SetFNameFilter \&namefilter;

SetBusyCallback "busycb", 333;

SetMsgCallback sub {
      my ($msg, $level) = @_;
      print uc strmsglevel $_[1], ": $msg\n";
   };

# the following non-trivial FileNameCallback takes care
   # of some subject lines not detected properly by uulib:
   SetFileNameCallback sub {
      return unless $_[1]; # skip "Re:"-plies et al.
      local $_ = $_[0];

# the following rules are rather effective on some newsgroups,
      # like alt.binaries.games.anime, where non-mime, uuencoded data
      # is very common

# if we find some *.rar, take it as the filename
      return $1 if /(\S{3,}\.(?:[rstuvwxyz]\d\d|rar))\s/i;

# one common subject format
      return $1 if /- "(.{2,}?\..+?)" (?:yenc )?$\d+\/\d+$/i;

# - filename.par (04/55)
      return $1 if /- "?(\S{3,}\.\S+?)"? (?:yenc )?$\d+\/\d+$/i;

# - (xxx) No. 1 sayuri81.jpg 756565 bytes
      # - (20 files) No.17 Roseanne.jpg [2/2]
      return $1 if /No\.[ 0-9]+ (\S+\....) (?:\d+ bytes )?\[/;

# try to detect some common forms of filenames
      return $1 if /([a-z0-9_\-+.]{3,}\.[a-z]{3,4}(?:.\d+))/i;

# otherwise just pass what we have
      ()
   };

# now read all files in the directory uusrc/*
   for (<uusrc/*>) {
      my ($retval, $count) = LoadFile ($_, $_, 1);
      print "file($_), status(", strerror $retval, ") parts($count)\n";
   }

Smerge -1;

SetOption OPT_SAVEPATH, "uudst/";

# now wade through all files and their source parts
   for my $uu (GetFileList) {
      print "file ", $uu->filename, "\n";
      print " state ", $uu->state, "\n";
      print " mode ", $uu->mode, "\n";
      print " uudet ", strencoding $uu->uudet, "\n";
      print " size ", $uu->size, "\n";
      print " subfname ", $uu->subfname, "\n";
      print " mimeid ", $uu->mimeid, "\n";
      print " mimetype ", $uu->mimetype, "\n";

# print additional info about all parts
      print " parts";
      for ($uu->parts) {
         for my $k (sort keys %$_) {
            print " $k=$_->{$k}";
         }
         print "\n";
      }

$uu->remove_temp;

if (my $err = $uu->decode) {
         print " ERROR ", strerror $err, "\n";
      } else {
         print " successfully saved as uudst/", $uu->filename, "\n";
      }
   }

print "cleanup...\n";

CleanUp;

=head1 PERLMULTICORE SUPPORT

This module supports the perlmulticore standard (see
L<http://perlmulticore.schmorp.de/> for more info) for the following
functions - generally these are functions accessing the disk and/or using
considerable CPU time:

LoadFile
   $item->decode
   $item->decode_temp
   $item->remove_temp
   $item->info

The perl interpreter will be reacquired/released on every callback
invocation, so for performance reasons, callbacks should be avoided if
that is costly.

Future versions might enable multicore support for more functions.

=head1 BUGS AND LIMITATIONS

The original uulib library this module uses was written at a time where
main memory of measured in megabytes and buffer overflows as a security
thign didn't exist. While a lot of security fixes have been applied over
the years (includign some defense in depth mechanism that can shield
against a lot of as-of-yet undetected bugs), using this library for
security purposes requires care.

Likewise, file sizes when the uulib library was written were tiny compared
to today, so do not expect this library to handle files larger than 2GB.

Lastly, this module uses a very "C-like" interface, which means it doesn't
protect you from invalid points as you might expect from "more perlish"
modules - for example, accessing a file item object after callinbg
C<CleanUp> will likely result in crashes, memory corruption, or worse.

=head1 AUTHOR

Marc Lehmann <schmorp@schmorp.de>, the original uulib library was written
by Frank Pilhofer <fp@informatik.uni-frankfurt.de>, and later heavily
bugfixed by Marc Lehmann.

=head1 SEE ALSO

perl(1), uudeview homepage at L<http://www.fpx.de/fp/Software/UUDeview/>.

=cut