Skip Menu |

This queue is for tickets about the RT-Client-REST CPAN distribution.

Report information
The Basics
Id: 127288
Status: resolved
Priority: 0/
Queue: RT-Client-REST

People
Owner: dean [...] fragfest.com.au
Requestors: wardmw [...] gmail.com
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



Subject: Problems downloading attachments
Date: Wed, 3 Oct 2018 18:07:57 +0100
To: bug-RT-Client-REST [...] rt.cpan.org
From: Martin Ward <wardmw [...] gmail.com>
Hi all, I am trying to write a Perl script that downloads file attachments from an old RT instance that needs to be decommissioned. The problem is that the RT modules seem to be doing something to the binary code such that the size is incorrect. When I download the attachment via the website it transfer fine. The example I man using is a zip file and I can unzip it fine afterwards; the file size is 460,749 bytes. If I retrieve the attachment by retrieving the ticket and iterating through the attachments then the size of the content file is 442,958 bytes, so bigger than expected. If I simply save this file then it is unreadable. I have tried encode_base64, decode_base64, encode_utf8 and decode_utf8 functions to en-decode the string to try and get it in to a reasonable state ( 460,749 bytes long and readable as a .zip file) but to no avail. I believe that it might be something to do with wide characters because if I do a hex dump of the start of the downloaded file I get: 0000000 50 4b 03 04 14 00 09 00 - 08 00 67 8d 25 46 00 00 0000010 00 00 00 00 00 00 00 00 - 00 00 1c 00 00 00 73 63 0000020 72 65 65 6e 73 68 6f 74 - 2d 31 37 32 20 32 31 20 0000030 32 34 32 20 36 34 2e 7a - 69 70 32 9e 8a fc b5 15 but if I hex dump the first few parts of the content I retrieve from RT then I get: 00000000 50 4B 03 04 14 00 09 00 - 08 00 67 FFFD 25 46 00 00 00000010 00 00 00 00 00 00 00 00 - 00 00 1C 00 00 00 73 63 00000020 72 65 65 6E 73 68 6F 74 - 2D 31 37 32 20 32 31 20 00000030 32 34 32 20 36 34 2E 7A - 69 70 32 FFFD FFFD FFFD FFFD 15 As you can see, there are a few wide characters in the attachment and I cannot explain their being there. I would appreciate any thoughts you have on this. The code I used to generate the hex dump is: #!/usr/bin/env perl # # wctest.pl - A test to see where the wide characters come from. use strict; use warnings; use RT::Client::REST; use RT::Client::REST::Ticket; use Data::HexDump; my $user='user'; my $pass='pass'; my $rt = RT::Client::REST->new( server => ('https://rt.local'), basic_auth_cb => ( sub { return ($user, $pass); } ) ); $rt->login( username=> $user, password=> $pass,); my $ticket_ptr = RT::Client::REST::Ticket->new(rt => $rt); my $results = $ticket_ptr->search( limits => [ { attribute => 'id', operator => '=', value => '51447' }, ],); my $iterator = $results->get_iterator; my ($ticket, $attachments); while ($ticket = &$iterator) { $attachments = $ticket->attachments; # Store attachments my $atch_ater = $attachments->get_iterator; while (my $att = &$atch_ater) { next if ($att->file_name eq ''); print HexDump substr($att->content, 0, 64); } } My environment is: RT instance: 4.0.23 Perl:5.16.3 RT Perl libraries: 0.52 O/S: inux rt50778.a.nominum.com 2.6.32-431.el6.x86_64 . |\/|artin
Patches gladly accepted :)
Im fairly certain the problem is that the client isnt taking care of RT's bizare API From https://rt-wiki.bestpractical.com/wiki/REST#Ticket_Attachment NOTE: RT returns the content indented with 9 spaces on each line, so that it lines up with the "Content:" header. Even if you strip this out with a regexp, the content is still UTF-8, which is probably not what you want. To get the original binary data back, strip out the 9 spaces with a regexp, strip off the 3 carriage returns at the end, and then convert the whole thing from UTF-8 to the native character encoding of the attachment, whatever that is. RT doesn't tell you, so you have know. If the attachments were uploaded by a U.S. Windows system, odds are that Windows-1252 is what you want. If you can't get the binary back intact, see the next method below. ***OR*** Ticket Attachment Content Gets the attachment data content without additional metadata or whitespace characters Request: /REST/1.0/ticket/<ticket-id>/attachments/<attachment-id>/content RT/3.8.0 200 Ok ... ... ... So to get the original content you still have to strip the first 2 lines of the response.
attaching some stuff so i can poke it
Subject: lipsum.txt
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas rhoncus gravida felis sed ultrices. In dui risus, interdum vel viverra et, molestie interdum risus. Praesent et molestie lectus. Nam viverra efficitur leo, non consequat velit iaculis sed. Vestibulum mattis neque a tempus hendrerit. Etiam non tincidunt nunc, id ullamcorper erat. Duis eget eros diam. Nullam sagittis justo eu dui consequat, at pharetra lacus gravida. Sed sodales urna congue suscipit maximus. Nullam quam diam, tempus vitae iaculis vitae, cursus vitae quam. Aliquam vitae enim eros. Vestibulum turpis lacus, sollicitudin ac pharetra quis, ullamcorper ac ante. In efficitur risus dui, et consequat nisi vulputate nec. Nunc aliquam, sem vitae lacinia luctus, diam nibh consequat lorem, a finibus felis justo ut est. Nulla semper interdum felis a dapibus. Nulla facilisi. Duis ut ultricies ipsum. Phasellus ante velit, cursus sollicitudin sem aliquam, laoreet fringilla mauris. Maecenas arcu neque, fringilla nec nulla sed, pharetra dapibus turpis. Aenean ultrices, urna eu laoreet bibendum, ante nisl aliquam neque, at dignissim sapien felis eget mi. Integer eleifend risus nec neque volutpat ullamcorper. In hac habitasse platea dictumst. Duis porttitor blandit sagittis. Phasellus gravida ultricies volutpat. Phasellus tincidunt ante eu nibh hendrerit, nec facilisis odio dignissim. Nam maximus, nisi sit amet egestas elementum, lorem sem varius urna, vel luctus augue nunc non risus. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nulla facilisi. Ut vitae neque in augue ornare pulvinar quis at mi. Aenean rutrum arcu tellus, molestie sagittis ipsum aliquam ac. Curabitur et pretium odio, sodales interdum felis. Pellentesque bibendum orci vel fringilla bibendum. Nam eget mauris libero. Nulla fringilla, nunc non varius rutrum, nulla neque aliquet magna, vitae convallis erat turpis a quam. Cras tempus orci sem, in vestibulum diam feugiat lobortis. Proin elementum nibh a nulla malesuada, sit amet aliquet ex ullamcorper. Fusce fringilla leo elit, quis porta ante ultrices sed. Ut et nunc ut elit egestas tincidunt. Maecenas fringilla varius lacus tincidunt mollis. Fusce lobortis luctus quam, eget mollis nulla sagittis in. Phasellus nec dignissim dolor, vitae lobortis ligula. Nam magna leo, sodales in commodo vel, feugiat sed quam. Maecenas suscipit elementum dui, sed bibendum diam gravida ut. Praesent ornare lacus non augue volutpat, vel consequat mauris ornare. Duis suscipit semper lacinia. Suspendisse egestas scelerisque sodales. In molestie ligula ligula, ac mattis elit aliquam a. Vestibulum luctus interdum tristique. Donec dolor purus, commodo accumsan luctus in, ultricies ut felis. Suspendisse porta, elit a fermentum lobortis, libero urna tristique massa, at aliquam lacus ligula sed metus. Etiam scelerisque massa vel est feugiat auctor. Nunc ultrices congue eleifend. Duis pellentesque orci pellentesque ullamcorper ullamcorper. Curabitur auctor pulvinar dui, ut blandit metus commodo et. Sed vestibulum rutrum erat, at fringilla turpis rutrum non. Curabitur et maximus tellus. Mauris dictum sit amet turpis efficitur condimentum. Mauris diam justo, luctus id leo sit amet, vestibulum varius nulla. Mauris eget blandit lacus, ac pulvinar tortor. Sed ornare ligula ac ante dignissim faucibus. Morbi eu ligula gravida, condimentum magna at, sollicitudin metus. Vestibulum id justo odio.
Subject: lipsum.txt.xz
Download lipsum.txt.xz
application/x-xz 1.3k

Message body not shown because it is not plain text.

another attachment
Subject: lipsum.txt.xz
Download lipsum.txt.xz
application/x-xz 1.3k

Message body not shown because it is not plain text.

with .txt files its chopping off the \n at the end.
Subject: Re: [rt.cpan.org #127288] Problems downloading attachments
Date: Thu, 8 Nov 2018 15:13:19 +0000
To: bug-RT-Client-REST [...] rt.cpan.org
From: Martin Ward <wardmw [...] gmail.com>
Chopping the \r is not necessarily a problem depending on whether it is translating that in to a \n, i.e. trying to translate Windows line endings in to unix ones. Of course it must keep some kind of line ending in there, it shouldn't just arbitrarily join up all th lines in a plain text file. However, I have resolved the problem that I was having, which was as you suggested, that "the client isnt taking care of RT's bizare API". The RT::Client::Rest Perl library that I am using has a function called "get_attachment" that requires an optional argument "undecoded" that needed to be set to 1 to have the script download the files successfully. It took me longer than necessary to resolve this issue because I had that written as uudecoded, which is not an unreasonable flag to have but which is invalid in this instance. I have submitted an RFE to that library developer to have it warn when an unrecognised flag is used... Thanks very much for your time and assistance, sorry to have wasted it trying to resolve a problem not of your creation. . |\/|artin On Thu, 8 Nov 2018 at 10:14, Dean Hamstead via RT < bug-RT-Client-REST@rt.cpan.org> wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=127288 > > > with .txt files its chopping off the \n at the end. >
ok good stuff
Subject: spaces.txt
just adding a file here for testing
Subject: newspaces.txt
closing