Skip Menu |

This queue is for tickets about the File-Rsync-Mirror-Recent CPAN distribution.

Report information
The Basics
Id: 66491
Status: open
Priority: 0/
Queue: File-Rsync-Mirror-Recent
People
Owner: Nobody in particular
Requestors: ABH [...] cpan.org
Cc:
AdminCc:
Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)
Attachments
recent-rmirror-donelog.log
recent-rmirror-state.yml

History Show all quoted text
  Wed Mar 09 04:01:49 2011 ABH [...] cpan.org - Ticket created
Subject: Copies symlinks as files
I run with "links" => 1 and even "safe-links" => 1, but I still see this from time to time when starting a new mirror. I think what happens is that what should be the symlink gets copied first, but as the actual files. Later when it's updated (rrr-fsck adding it an extra time for no reason?), this error happens. I am attaching the rmirror-state and the donelog in case that's helpful. Warning: Wed Mar 9 01:01:12 2011, Error while rsyncing (1W): 'could not make way for new symlink: modules/by-authors could not make way for new symlink: modules/by- module/Date/ADAMK could not make way for new symlink: modules/by- module/Date/MIYAGAWA rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1506) [generator=3.0.4]', sleeping 36 at /home/mirrors/perl5/lib/perl5/File/Rsync/Mirror/Recentfile.pm line 1864 File::Rsync::Mirror::Recentfile::register_rsync_error('File::Rsync::Mirror::Recentfile=HASH(0xd8 6cb00)', 'could not make way for new symlink: modules/by-authors\x{a}', 'could not make way for new symlink: modules/by-module/Date/AD...', 'could not make way for new symlink: modules/by-module/Date/MI...', 'rsync error: some files/attrs were not transferred (see previ...') called at /home/mirrors/perl5/lib/perl5/File/Rsync/Mirror/Recentfile.pm line 1397 File::Rsync::Mirror::Recentfile::mirror_path('File::Rsync::Mirror::Recentfile=HASH(0xd86cb00)', 'ARRAY(0xe0c57d0)') called at /home/mirrors/perl5/lib/perl5/File/Rsync/Mirror/Recentfile.pm line 1265 File::Rsync::Mirror::Recentfile::_mirror_dlcollector('File::Rsync::Mirror::Recentfile=HASH(0xd86 cb00)', 'ARRAY(0xdafa2d0)', 'HASH(0xde0edf0)', 'ARRAY(0xde301e0)') called at /home/mirrors/perl5/lib/perl5/File/Rsync/Mirror/Recentfile.pm line 1125 eval {...} called at /home/mirrors/perl5/lib/perl5/File/Rsync/Mirror/Recentfile.pm line 1125 File::Rsync::Mirror::Recentfile::mirror('File::Rsync::Mirror::Recentfile=HASH(0xd86cb00)', 'piecemeal', 1, 'skip-deletes', 0, 'loop', 1) called at /home/mirrors/perl5/lib/perl5/File/Rsync/Mirror/Recent.pm line 751 File::Rsync::Mirror::Recent::_rmirror_mirror('File::Rsync::Mirror::Recent=HASH(0xde0ef70)', 4, 'HASH(0xdcf6e90)') called at /home/mirrors/perl5/lib/perl5/File/Rsync/Mirror/Recent.pm line 709 File::Rsync::Mirror::Recent::_rmirror_loop('File::Rsync::Mirror::Recent=HASH(0xdd35d30)', 20, 'HASH(0xdcf6e90)') called at /home/mirrors/perl5/lib/perl5/File/Rsync/Mirror/Recent.pm line 669 File::Rsync::Mirror::Recent::rmirror('File::Rsync::Mirror::Recent=HASH(0xdd35d30)', 'skip- deletes', 0, 'loop', 1) called at ./rrr-client line 77
Subject: recent-rmirror-state.yml
--- options: loop: 1 skip-deletes: 0 reduced_rfs: - -_current_tempfile: /mirrors/CPAN/.FRMRecent-RECENT-1h.json-qWDx.json -_done: !!perl/hash:File::Rsync::Mirror::Recentfile::Done -_logfile: recent-rmirror-donelog.log -_rfinterval: 1h -_interval: 1h -_localroot: /mirrors/CPAN -_merged: epoch: '1299661061.80973' into_interval: 6h time: '1299661061.81711' -_remoteroot: ask-mirrors2@cpan-rsync-master.perl.org::CPAN -_rfile: ~ -_seeded: 1 -_use_tempfile: 1 -aggregator: - 1h - 6h - 1d - 1W - 1M - 1Q - 1Y - Z -dirtymark: '1299417567.21358' -filenameroot: RECENT -ignore_link_stat_errors: 1 -is_slave: 1 -max_files_per_connection: 5000 -minmax: max: '1299661061.85054' min: '1299657714.39852' mtime: '1299661061' -protocol: 1 -rsync_options: &1 checksum: 0 compress: 1 links: 1 safe-links: 1 times: 1 -serializer_suffix: .json -tempdir: ~ -ttl: ~ -verbose: 1 -verboselog: 0 - -_done: !!perl/hash:File::Rsync::Mirror::Recentfile::Done -_logfile: recent-rmirror-donelog.log -_rfinterval: 1h -_interval: 1h -_localroot: /mirrors/CPAN -_remoteroot: ask-mirrors2@cpan-rsync-master.perl.org::CPAN -_rfile: ~ -_use_tempfile: 1 -aggregator: - 1h - 6h - 1d - 1W - 1M - 1Q - 1Y - Z -filenameroot: RECENT -have_mirrored: 0 -ignore_link_stat_errors: 1 -is_slave: 1 -max_files_per_connection: 5000 -protocol: 1 -rsync_options: checksum: 0 compress: 1 links: 1 safe-links: 1 times: 1 -serializer_suffix: .json -sleep_per_connection: ~ -tempdir: ~ -verbose: 1 - -_done: !!perl/hash:File::Rsync::Mirror::Recentfile::Done -_logfile: recent-rmirror-donelog.log -_rfinterval: 6h -_interval: 6h -_localroot: /mirrors/CPAN -_remoteroot: ask-mirrors2@cpan-rsync-master.perl.org::CPAN -_rfile: ~ -_use_tempfile: 1 -aggregator: - 1h - 6h - 1d - 1W - 1M - 1Q - 1Y - Z -filenameroot: RECENT -have_mirrored: 0 -ignore_link_stat_errors: 1 -is_slave: 1 -max_files_per_connection: 5000 -protocol: 1 -rsync_options: checksum: 0 compress: 1 links: 1 safe-links: 1 times: 1 -serializer_suffix: .json -sleep_per_connection: ~ -tempdir: ~ -verbose: 1 - -_done: !!perl/hash:File::Rsync::Mirror::Recentfile::Done -_logfile: recent-rmirror-donelog.log -_rfinterval: 1d -_interval: 1d -_localroot: /mirrors/CPAN -_remoteroot: ask-mirrors2@cpan-rsync-master.perl.org::CPAN -_rfile: ~ -_use_tempfile: 1 -aggregator: - 1h - 6h - 1d - 1W - 1M - 1Q - 1Y - Z -filenameroot: RECENT -have_mirrored: 0 -ignore_link_stat_errors: 1 -is_slave: 1 -max_files_per_connection: 5000 -protocol: 1 -rsync_options: checksum: 0 compress: 1 links: 1 safe-links: 1 times: 1 -serializer_suffix: .json -sleep_per_connection: ~ -tempdir: ~ -verbose: 1 - -_done: !!perl/hash:File::Rsync::Mirror::Recentfile::Done -_logfile: recent-rmirror-donelog.log -_rfinterval: 1W -_interval: 1W -_localroot: /mirrors/CPAN -_remoteroot: ask-mirrors2@cpan-rsync-master.perl.org::CPAN -_rfile: ~ -_use_tempfile: 1 -aggregator: - 1h - 6h - 1d - 1W - 1M - 1Q - 1Y - Z -filenameroot: RECENT -have_mirrored: 0 -ignore_link_stat_errors: 1 -is_slave: 1 -max_files_per_connection: 5000 -protocol: 1 -rsync_options: checksum: 0 compress: 1 links: 1 safe-links: 1 times: 1 -serializer_suffix: .json -sleep_per_connection: ~ -tempdir: ~ -verbose: 1 - -_done: !!perl/hash:File::Rsync::Mirror::Recentfile::Done -_logfile: recent-rmirror-donelog.log -_rfinterval: 1M -_interval: 1M -_localroot: /mirrors/CPAN -_remoteroot: ask-mirrors2@cpan-rsync-master.perl.org::CPAN -_rfile: ~ -_use_tempfile: 1 -aggregator: - 1h - 6h - 1d - 1W - 1M - 1Q - 1Y - Z -filenameroot: RECENT -have_mirrored: 0 -ignore_link_stat_errors: 1 -is_slave: 1 -max_files_per_connection: 5000 -protocol: 1 -rsync_options: checksum: 0 compress: 1 links: 1 safe-links: 1 times: 1 -serializer_suffix: .json -sleep_per_connection: ~ -tempdir: ~ -verbose: 1 - -_done: !!perl/hash:File::Rsync::Mirror::Recentfile::Done -_logfile: recent-rmirror-donelog.log -_rfinterval: 1Q -_interval: 1Q -_localroot: /mirrors/CPAN -_remoteroot: ask-mirrors2@cpan-rsync-master.perl.org::CPAN -_rfile: ~ -_use_tempfile: 1 -aggregator: - 1h - 6h - 1d - 1W - 1M - 1Q - 1Y - Z -filenameroot: RECENT -have_mirrored: 0 -ignore_link_stat_errors: 1 -is_slave: 1 -max_files_per_connection: 5000 -protocol: 1 -rsync_options: checksum: 0 compress: 1 links: 1 safe-links: 1 times: 1 -serializer_suffix: .json -sleep_per_connection: ~ -tempdir: ~ -verbose: 1 - -_done: !!perl/hash:File::Rsync::Mirror::Recentfile::Done -_logfile: recent-rmirror-donelog.log -_rfinterval: 1Y -_interval: 1Y -_localroot: /mirrors/CPAN -_remoteroot: ask-mirrors2@cpan-rsync-master.perl.org::CPAN -_rfile: ~ -_use_tempfile: 1 -aggregator: - 1h - 6h - 1d - 1W - 1M - 1Q - 1Y - Z -filenameroot: RECENT -have_mirrored: 0 -ignore_link_stat_errors: 1 -is_slave: 1 -max_files_per_connection: 5000 -protocol: 1 -rsync_options: checksum: 0 compress: 1 links: 1 safe-links: 1 times: 1 -serializer_suffix: .json -sleep_per_connection: ~ -tempdir: ~ -verbose: 1 - -_done: !!perl/hash:File::Rsync::Mirror::Recentfile::Done -_logfile: recent-rmirror-donelog.log -_rfinterval: Z -_interval: Z -_localroot: /mirrors/CPAN -_remoteroot: ask-mirrors2@cpan-rsync-master.perl.org::CPAN -_rfile: ~ -_use_tempfile: 1 -aggregator: - 1h - 6h - 1d - 1W - 1M - 1Q - 1Y - Z -filenameroot: RECENT -have_mirrored: 0 -ignore_link_stat_errors: 1 -is_slave: 1 -max_files_per_connection: 5000 -protocol: 1 -rsync_options: checksum: 0 compress: 1 links: 1 safe-links: 1 times: 1 -serializer_suffix: .json -sleep_per_connection: ~ -tempdir: ~ -verbose: 1 reduced_self: -__pathdb: {} -_dirtymark: '1299417567.21358' -_logfilefordone: recent-rmirror-donelog.log -_runstatusfile: recent-rmirror-state.yml -_verbose: 1 -_verboselog: 0 -ignore_link_stat_errors: 1 -localroot: /mirrors/CPAN/ -max_files_per_connection: 5000 -remote: ask-mirrors2@cpan-rsync-master.perl.org::CPAN/ -remoteroot: ask-mirrors2@cpan-rsync-master.perl.org::CPAN -rsync_options: *1 time: '1299661214'
Subject: recent-rmirror-donelog.log
Download recent-rmirror-donelog.log
application/octet-stream 742.6k

Message body not shown because it is not plain text.

  Wed Mar 09 12:23:28 2011 ABH [...] cpan.org - Correspondence added
I only just noticed that actually the modules/by-author directory doesn't successfully get any of the symlinks; only the PAUSE "base files". I think this is the big/only real show-stopper for using rrr for general CPAN.
  Wed Mar 09 12:23:28 2011 ABH [...] cpan.org - Status changed from 'new' to 'open'
  Thu Mar 10 19:17:41 2011 ABH [...] cpan.org - Correspondence added
Just in case someone else is interested in helping, here is what I think is happening: a/b/XYZ/c.tar.gz is the real file. (for example: authors/id/X/XY/XYZ/foo.tar.gz) d/e/XYZ is a symlink to a/b/XYZ (for example modules/by-authors/id/XYZ) The system ends up copying c.tar.gz into d/e/XYZ/c.tar.gz; so when it later gets around to making the symlink, it can't because the directory is already there.
  Tue Mar 15 06:15:33 2011 ABH [...] cpan.org - Correspondence added
This: https://github.com/abh/rersyncrecent/commit/0e31fa81c4ed14f7adc51860d964074f3a04375f and the small fix/change in rrr-fsck a couple commits later (stat => lstat) seems to help; if not fix it completely. https://github.com/abh/rersyncrecent/commits/