| Subject: | does not completely read block-compressed files |
These two commands should have the same output, but they don't:
wget -q -O - ftp://ftp.ncbi.nlm.nih.gov/snp/organisms/human_9606/VCF/00-All.vcf.gz | gunzip -c | head -20000 | tail -1
perl -MPerlIO::gzip -E 'open( my $f, "-|:gzip", "wget", "-qO", "-", "ftp://ftp.ncbi.nlm.nih.gov/snp/organisms/human_9606/VCF/00-All.vcf.gz"); print while <$f>' | head -20000 | tail -1
PerlIO::gzip seems to stop reading that file a little before line 500. The file in question was probably compressed with Heng Li's bgzip utility: http://samtools.sourceforge.net/tabix.shtml