For the most part this is working as intended. Let me explain:
To use streamscan, you clearly have to already have the data in ram. A file read in by perl typically uses 50% more data in memory than disk size. So a 150MB file will take >200MB if read into a perl variable.
Then you pass that data into streamscan, that creates yet another copy of the data in the screamscan function, now we are at >400MB. streamscan than passes the data to _send(), when creates another copy. So this puts mem usage above 600MB.
In perl 5.20+, the new copy on write feature (COW), should help here since the data of this file is not being modified, just being passed through functions. However, the call to join() on the data variable in streamscan was preventing this so I removed that. That cut the mem usage in half for perls 5.20+. However older perls had no effect. I therefore changed the way arguments were handled in _send(), just use the aliases/references in the function instead of copying the data into another variable.
The outcome in perl 5.20+ is mem usage does not grow beyond the initial size when the file was read into mem. However on perls < 5.20, the mem usage still doubles (but before it would triple so there is still some benefit here)
Committed to 1.93.
