Bug #105460 for Compress-Raw-Lzma: use of Compress::Raw::Lzma::RawDecoder fails with large amount of data

Hey Paul, thanks for the reply... didn't honestly expect a reply that quickly! ;) I'm just kiding... but I must honestly say I was desperately searching for a solution for this problem (since 2015!) and it seemed like (and I think it really was the case that) nobody could help me (didn't receive a reply here, nobody could help me on various perl-related IRC channels, I even tried to email you directly... but without success). That is also what makes me wondering... did you see the issue/problem in the past but you too were unsure what the problem could be ? If that was the case, it would suggest that it is somehow non-intuitive that we need to temporarily store the intermediate results etc... and finally it would mean that my code (also through your eyes) looked perfectly fine and it didn't trigger an alert (about the AppenOutput problem). What I am trying to say is, that I still think that it is non-intuitive that AppendOutput defaults to 0 for all Encoders. On the other side, I see your point about the risk of breaking perl scripts that are in the wild (on the other hand... one could also discuss that maybe some of them are exactly having the same problem like I did and maybe just the authors didn't test them well enough. I instead, tested my code and that was the reason that I found this problem. But I guess not everybody would notice the problem immediately if the scripts were not tested carefully with various amount of data and especially very large random data!). So one could in theory also speculate that by fixing the code (instead of the documentation) in some cases it would even fix those perl scripts (instead of breaking them). I'm not sure about the exact number of scripts that are broken (because they forgot to set AppendOutput => 1), nor the exact number of scripts that do store the intermediate results (or who force to set AppendOutput => 1). I have the feeling that many users (of course me included) who use this module would indeed take it for granted that the documentation matches the code and that AppendOutput always was set to 1. Having that exact number would probably help to understand which fix would be better (code fix or documentation fix?). I can perfectly deal with your decision to "never break existing code" (again: not sure if by "fixing" the code to suit the "old" documentation would do more harm than good or vice-versa). What I really ask (beg? ;) ) you to do is to change the documentation in a way that it will deal with all these following points: 1. Fix the SYNOPSIS section and all examples within the documentation to make it clear that one needs to either store the output of both code () and flush () or specify AppendOutput => 1 2. Add a paragraph to the documentation explaining what code () and flush () do and why one needs to either store the result of both or specify AppendOutput => 1 for ALL encoders. Aka warn the user that for large inputs the overall output of LZMA is the concatenation of the results of code () and flush () and that the users of this modules should make sure to not forget that flush () might override the variable (if only a single variable was used) if AppendOutput was not set to 1. 3. Whenever the AppendOutput option is explained (i.e. where we find this sub string: "AppendOutput => 0|1") in the documentation, clearly state that one either needs to force AppendOutput to 1 or "externally" deal with the intermediate results (and concatenate them to have the whole compressed data)... this is the case for ALL encoders. 4. Add an easy noticeable note (highlighted at the top?) to the documentation (and maybe to other files too, like the "Changes" file) to make it clear that the documentation changed with version > 2.071 and what those changes imply (and maybe also explain that some older perl scripts which "forgot" to set AppendOutput to 1 may be broken). Essentially, add a warning messages that some scripts that were implemented with older versions (< 2.071) might be broken or could return unexpected results. For #1 I suggest to include this extended and fixed example: SYNOPSIS ... # AppendOutput defaults to 0 my ($lz, $status) = new Compress::Raw::Lzma::RawEncoder or die "Cannot create lzma object: $status\n"; $status = $lz->code($input, $output_code_part); # ... # use the intermediate encoder's output stored in $output_code_part # ... $status = $lz->flush($output_flush_part); # ... # use the second/final part of the encoder's output # Note: the concatenation of $output_code_part and $output_flush_part is equal to the full output of the LZMA encoder # ... # same example with concatenation of the intermediate outputs within this script my ($lz, $status) = new Compress::Raw::Lzma::RawEncoder or die "Cannot create lzma object: $status\n"; $status = $lz->code($input, $output_code_part); $status = $lz->flush($output_flush_part); my $output = $output_code_part . $output_flush_part; # same example with AppendOutput set to 1 (AppendOutput defaults to 0) my ($lz, $status) = new Compress::Raw::Lzma::RawEncoder (AppendOutput => 1) or die "Cannot create lzma object: $status\n"; $status = $lz->code($input, $output); $status = $lz->flush($output); ... Thank you again Paul