| Subject: | Incremental Parsing of Large Objects |
When creating a non-blocking client/server I read data off the wire in chunks of 4096 bytes and pass them off to the incremental parser. Since I don't know how many, if any, JSON objects are contained in that 4096 bytes, I then attempt to read from the incremental parser. When the object is larger than 4096, and the end of the parsing occurs in the middle of a string, it croaks with the error:
unexpected end of string while parsing JSON string, at character offset 4096 (before "(end of string)")
Attached is a simple program demonstrating the issue. This problem does not appear with JSON::XS.
| Subject: | json.pl |
use strict;
use Test::More;
use IO::String;
use JSON::PP;
my $json = JSON::PP->new;
my $kb = 'a' x 1024;
my $hash = { map { $_ => $kb } (1..40) };
my $data = join ( '', ($json->encode($hash), $json->encode($hash) ) );
my $size = length($data);
print "Total size: [$size]\n";
my $fh = IO::String->new($data) or die "Failed to opne FH";
my $count = 0;
my $offset = 0;
while ($size) {
print "Bytes left [$size]\n";
my $bytes = sysread($fh, my $incr, 4096);
if ($bytes) {
$offset += $bytes;
$size -= $bytes;
$json->incr_parse($incr);
$count++;
while ( my $obj = $json->incr_parse ) {
print "Got JSON object\n";
}
}
elsif (defined $bytes) {
print "End of file\n";
close $fh;
last;
}
else {
die "Error occurred reading handle: $!\n";
}
}