w3mir's homepage

WARNING: A serious bug was discovered in w3mir 1.0.8 (it is probably present in some earlier versions too), which can cause removal of all your files, not just files in the mirror. To avoid this edit w3mir, search for 'sub rm_rf (line 2259 in 1.0.8) and put the statement 'die;' on the line below. After it should read:

sub rm_rf {
  die;

Instead of files being removed w3mir will now abort if the bug is triggered.

W3mir 1.0.10 has been released. Get it now.
Warning: 1.0.3, 1.0.7 and 1.0.8 had serious bugs and should not be used.

What is it?

w3mir is a all purpose HTTP copying and mirroring tool. The main focus of w3mir is to create and maintain a browseable copy of one, or several, remote WWW site(s). Used to the max w3mir can retreive the contents of several related sites and leave the mirror browseable via a local web server, or from a filesystem, such as directly from a CDROM.

w3mir's goal is to be able to make usefull mirrors of any reasonable WWW site. It specifically preserves link integrity within the mirrored documents as well as the integrety of links outside the mirror, following redirects as needed. If you want it to. w3mir has a powerfull ``multi scope'' mechanism enabeling the user to make mirrors of several related sites and have links between them refer to the mirrored documents rather than the original site. w3mir has several features directed at getting mirrors for CDROM burning and handling of some not too often seen problems when mirroring.

w3mir supports HTML4, and has partial support for CSS, Java and ActiveX. And it should work on Win32 machines.

Legalese

w3mir is free but it is Copyrighted by the various involved hackers. If you want to copy, hack or distribte w3mir you can do that providing you comply with the 'Artistic License' which acompanies all Perl distributions and is also bundled in the w3mir distribution.

Bugs and updates

If you find a problem with w3mir contact w3mir-core@usit.uio.no but please make sure you have the latest source first (get it from this page). If you submit a bug/problem report please include the output of 'w3mir -v' as well as the commandline and configuration file that causes the problem.

All notable changes and bugfixes will be announced on w3mir-info@usit.uio.no, which is a very low volume list. Mail a message consisting of subscribe to w3mir-info-request@usit.uio.no or surf on over to http://usit-lists.uio.no/mailman/listinfo/w3mir-info@usit.uio.no and fill out the form to get subscribed. If you use w3mir at all you should subscribe. You cannot post to this list unless you are subscribed. This is to cut down on spam.

Getting w3mir

w3mir is packaged and installed the standard way for perl packages. It uses a standard Makefile.PL, this means that it should be very easy to install and get working. Please read the INSTALL document first though. Also note that w3mir only works with Perl 5.002 and later versions.

Users of RedHat Linux 4.2 and 5.0 should note that some versions of redhats perl-5 rpms have been broken w.r.t. installing packages. Please get the latest rpm if you experience problems installing.

• The basic assumption is that you have perl5.002 or a later version installed. If you have a Unix machine see http://www.perl.com/CPAN/src/5.0/, if not see http://www.perl.com/CPAN/ports/ for a perl for your machine.

• w3mir installation instructions, explains how to install the prerequisites (for use with w3mir) as well as w3mir itself. For Win32 there are separate instructions.

• w3mir 1.0.* requires libwww-perl.

• And now also the URI module. Please note that w3mir 1.0.9 and later now works with all known versions of LWP and the URI module. 1.0.8 did not.

• You also need MIME-Base64 See the w3mir installation instructions on how to install it.

• And finaly w3mir itself (1.0.10).

Fixes in 1.0.10:

• Bug compatability with URI::URL: Replace some calls to path with epath calls.

Fixes in 1.0.9:

• Inserted the die; statement described on the top of this page. This is not exactly elegant, but it works.
• Compatible with the newest versions of LWP and the URI module.

Fixes in 1.0.8:

• W3mir 1.0.7 would remove all leading path elements to characters long, such as /../, as intended, but also /ab/ and /fo/.

Features/bugs in 1.0.7:

• As earlier, w3mir does not support cookies. I have received a patch for basic cookie support but have not yet incorporated it due to time-restraints.
• There actually exists web servers that break the HTTP standard rather thoroughly by not sending a Content-type header (this is mandatory in the standard). W3mir will not discover that a HTML file is HTML in this case and will not extract URLs from it.
• If you rewrite the ending filename URLs with Apply rules (as in index.htm -> index.html) and there are links to index.htm#foo the link will be incorrect after mirroring. The solution is to avoid those kinds of rules. This was meant to be fixed in 1.0.3 (which broke it as described above), and I will fix it when I get more time. The fix is to remove the #fragment part of URLs when they pass thru the apply machinery. Patches accepted.

Fixes in 1.0.7:

• W3mir can now handle URLs like http://some.host.com/../../some/path/ without remving all your files. Leading /../ sequences in the path component of URLs are now removed.
• The initial referer value is used in batch mode too.

Fixes in 1.0.6:

• -f works with binary files
• w3mir will no longer use 'Referer: (commandline)' when a URL is derived from the commandline/config file.
• More errormessage fixes
• Fixed install instructions to warn against the use of LWP 5.40.

Fixes in 1.0.5:

• Better error-messaged in -quiet mode
• Fixed quoting. Both " and ' quotes in documents are accepted.
• Fixed timeout handling.
• Fixed broken connection handling. Incomplete documents are deleted and retried.
• Fixed URL mangling problem. In 1.0.3 URLs like ``foo.html#bar'' would be rewritten to ``#bar''. This is quite bad and makes a lot of mirrors un-useable.

Fixes in 1.0.4:

1.0.4 was not publicaly released.

Fixes in 1.0.3:

• The -R/remove option to remove files is no longer more destructive than intended.
• Files with 'unsafe' characters in their filename is now saved as "foo bar" instead of "foo%20bar"
• The -B switch works once again.

Fixes in 1.0.2:

• W3mir now only newline converts HTML files, all other files are handled as binary. This should ensure file-integrety even when working with mis-configured servers.

Fixes in 1.0.1:

• -fa option and corresponding configuration file directive works again.
• Added -nnc option and a corresponding configuration file directive. It disables newline conversions, which could corrupt files the Webserver mis-declares. The HOWTO explains the uses of -nnc

FTP mirroring

W3mir cannot mirror FTP sites. So if you're looking for a way to mirror FTP sites take a look at fmirror (ftp://ftp.guardian.no/pub/free/ftp/fmirror/) or perhaps the original ftp mirror implementation for which I have the following information:

AVAILABILITY:
The latest version of mirror is available from:

        src.doc.ic.ac.uk [146.169.2.1]
                directory: computing/archiving/mirror
                (shortcut packages/mirror)
        
        ftp.th-darmstadt.de [130.83.55.75]
                directory: pub/networking/mirror

        ftp.sun.ac.za [146.232.212.2]
                directory: pub/packages/mirror

There is also 'omi', which I have never used but the documentation is promising.