Bug #36558 for Image-ExifTool: metadata copying is broken

Sat Jun 07 22:28:36 2008 polettix [...] cpan.org - Ticket created

Subject:

metadata copying is broken

Hi, I noticed that metadata copying from one file to another is broken, i.e. metadata are not copied faithfully. In particular, the copy of the embedded preview image does not work (this does not apply to the thumbnail, anyway). The following command sequence shows what I mean; note that in the dscn4465-target.jpg file the Preview hasn't been copied properly. I attach the three files for your reference. # A sample photo file with both Thumbnail and Preview poletti@Polebian:~/sviluppo/perl/rotob$ exiftool -g dscn4465-mini.jpg Show quoted text

---- ExifTool ---- ExifTool Version Number : 7.30 ---- File ---- File Name : dscn4465-mini.jpg Directory : . File Size : 40 kB File Modification Date/Time : 2008:06:08 04:15:25 File Type : JPEG MIME Type : image/jpeg Exif Byte Order : Little-endian (Intel, II) Image Width : 488 Image Height : 380 Encoding Process : Baseline DCT, Huffman coding Bits Per Sample : 8 Color Components : 3 Y Cb Cr Sub Sampling : YCbCr4:2:0 (2 2) ---- JFIF ---- JFIF Version : 1.1 ---- EXIF ---- Components Configuration : YCbCr Flash : Auto, Fired Interoperability Index : R98 - DCF basic file (sRGB) Interoperability Version : 0100 Thumbnail Offset : 1992 Thumbnail Length : 2359 ---- MakerNotes ---- Maker Note Version : 2.00 Color Mode : Color Quality : Fine White Balance : Auto Focus Mode : AF-S Flash Setting : Normal ISO Selection : Auto Preview Image Start : 4352 Preview Image Length : 6797 Image Processing : Image Adjustment : Auto Auxiliary Lens : Off Manual Focus Distance : inf Digital Zoom : 0 AF Area Mode : Single Area AF Point : Center AF Points In Focus : Center Scene Mode : Noise Reduction : Off WB RBGG Levels : 0 0 0 0 Scene Assist : Retouch History : None Image Stabilization : VR-ON ---- XMP ---- Color Space : sRGB Compressed Bits Per Pixel : 4 Contrast : Normal Custom Rendered : Normal Date/Time Original : 2008:02:09 23:57:48+02:00 Digital Zoom Ratio : 0 Exif Version : 0220 Exposure Compensation : 0 Exposure Mode : Auto Exposure Program : Program AE Exposure Time : 1/59 F Number : 2.7 File Source : Digital Camera Flashpix Version : 0100 Focal Length : 7.5 mm Focal Length In 35mm Format : 36 mm Gain Control : Low gain up ISO : 64 Light Source : Unknown Max Aperture Value : 2.7 Metering Mode : Multi-segment Exif Image Width : 3264 Exif Image Height : 2448 Saturation : Normal Scene Capture Type : Standard Scene Type : Directly photographed Sharpness : Normal Subject Distance Range : Unknown User Comment : Compression : JPEG (old-style) Image Description : Make : NIKON Camera Model Name : COOLPIX P4 Orientation : Horizontal (normal) Resolution Unit : inches Software : COOLPIX P4V1.1 X Resolution : 300 Y Cb Cr Positioning : Co-sited Y Resolution : 300 Create Date : 2008:02:09 23:57:48+02:00 Modify Date : 2008:02:09 23:57:48+02:00 ---- Composite ---- Aperture : 2.7 Image Size : 488x380 Preview Image : (Binary data 6797 bytes, use -b option to extract) Scale Factor To 35 mm Equivalent: 4.8 Shutter Speed : 1/59 Thumbnail Image : (Binary data 2359 bytes, use -b option to extract) Circle Of Confusion : 0.006 mm Field Of View : 53.1 deg Focal Length : 7.5 mm (35 mm equivalent: 36.0 mm) Hyperfocal Distance : 3.33 m Light Value : 9.4 # A copy of that file, but without any metadata poletti@Polebian:~/sviluppo/perl/rotob$ exiftool -g dscn4465-bare.jpg

---- ExifTool ---- ExifTool Version Number : 7.30 ---- File ---- File Name : dscn4465-target.jpg Directory : . File Size : 24 kB File Modification Date/Time : 2008:06:08 04:18:25 File Type : JPEG MIME Type : image/jpeg Image Width : 488 Image Height : 380 Encoding Process : Baseline DCT, Huffman coding Bits Per Sample : 8 Color Components : 3 Y Cb Cr Sub Sampling : YCbCr4:2:0 (2 2) ---- Composite ---- Image Size : 488x380 # Make a copy of the "bare" file poletti@Polebian:~/sviluppo/perl/rotob$ cp dscn4465-bare.jpg dscn4465-target.jpg # Duplicate metadata in dscn4465-mini.jpg into the copied file poletti@Polebian:~/sviluppo/perl/rotob$ exiftool -TagsFromFile dscn4465-mini.jpg dscn4465-target.jpg 1 image files updated # Examine the target: the preview is incorrect poletti@Polebian:~/sviluppo/perl/rotob$ exiftool -g dscn4465-target.jpg

---- ExifTool ---- ExifTool Version Number : 7.30 ---- File ---- File Name : dscn4465-target.jpg Directory : . File Size : 31 kB File Modification Date/Time : 2008:06:08 04:19:50 File Type : JPEG MIME Type : image/jpeg Exif Byte Order : Little-endian (Intel, II) Image Width : 488 Image Height : 380 Encoding Process : Baseline DCT, Huffman coding Bits Per Sample : 8 Color Components : 3 Y Cb Cr Sub Sampling : YCbCr4:2:0 (2 2) ---- EXIF ---- Image Description : Make : NIKON Camera Model Name : COOLPIX P4 Orientation : Horizontal (normal) X Resolution : 300 Y Resolution : 300 Resolution Unit : inches Software : COOLPIX P4V1.1 Modify Date : 2008:02:09 23:57:48 Exposure Time : 1/59 F Number : 2.7 Exposure Program : Program AE ISO : 64 Exif Version : 0220 Date/Time Original : 2008:02:09 23:57:48 Create Date : 2008:02:09 23:57:48 Components Configuration : YCbCr Exposure Compensation : 0 Max Aperture Value : 2.7 Metering Mode : Multi-segment Light Source : Unknown Flash : Auto, Fired Focal Length : 7.5 mm User Comment : Flashpix Version : 0100 Color Space : sRGB Exif Image Width : 3264 Exif Image Height : 2448 File Source : Digital Camera Scene Type : Directly photographed Custom Rendered : Normal Exposure Mode : Auto Digital Zoom Ratio : 0 Focal Length In 35mm Format : 36 mm Scene Capture Type : Standard Gain Control : Low gain up Contrast : Normal Saturation : Normal Sharpness : Normal Subject Distance Range : Unknown Thumbnail Offset : 1910 Thumbnail Length : 2359 ---- MakerNotes ---- Maker Note Version : 2.00 Quality : Fine White Balance : Auto Focus Mode : AF-S Flash Setting : Normal ISO Selection : Auto Preview Image Start : 1680 Preview Image Length : 0 Image Processing : Image Adjustment : Auto Auxiliary Lens : Off Manual Focus Distance : inf Digital Zoom : 0 AF Area Mode : Single Area AF Point : Center AF Points In Focus : Center Scene Mode : Noise Reduction : Off WB RBGG Levels : 0 0 0 0 Scene Assist : Retouch History : None Image Stabilization : VR-ON ---- XMP ---- Compressed Bits Per Pixel : 4 Color Mode : Color Compression : JPEG (old-style) Y Cb Cr Positioning : Co-sited ---- Composite ---- Aperture : 2.7 Image Size : 488x380 Scale Factor To 35 mm Equivalent: 4.8 Shutter Speed : 1/59 Thumbnail Image : (Binary data 2359 bytes, use -b option to extract) Circle Of Confusion : 0.006 mm Field Of View : 53.1 deg Focal Length : 7.5 mm (35 mm equivalent: 36.0 mm) Hyperfocal Distance : 3.33 m Light Value : 9.4

Subject:

dscn4465-bare.jpg

Download dscn4465-bare.jpg
image/jpeg 23.6k

Subject:

dscn4465-mini.jpg

Download dscn4465-mini.jpg
image/jpeg 39.6k

Subject:

dscn4465-target.jpg

Download dscn4465-target.jpg
image/jpeg 31.1k

Sun Jun 08 08:03:18 2008 EXIFTOOL [...] cpan.org - Correspondence added

From:

EXIFTOOL [...] cpan.org

Thanks for the bug report and sample images. The behaviour has changed for 7.30, but unfortunately it was the previous versions that were buggy. The intent has been to not include the PreviewImage when copying all information. From the exiftool application documentation: 2) The maker note information is copied as a block, so it isn't affected like other information by subsequent tag assignments on the command line. Also, since the PreviewImage referenced from the maker notes may be rather large, it is not copied, and must be transferred separately if desired. I realize this is inconvenient if you really do want to copy the preview image, but when it is built into the makernotes by default it causes problems because it can make the makernotes too big to fit into a JPEG image when copying tags from a RAW to a JPEG. I hope this isn't too much of a problem for you. If so, I am open to suggestions.

Sun Jun 08 08:03:47 2008 The RT System itself - Status changed from 'new' to 'open'

Sun Jun 08 10:33:36 2008 polettix [...] cpan.org - Correspondence added

On Sun Jun 08 08:03:18 2008, EXIFTOOL wrote: Show quoted text

> Thanks for the bug report and sample images. > > The behaviour has changed for 7.30, but unfortunately it was the > previous versions that were buggy. The intent has been to not > include the PreviewImage when copying all information. From the > exiftool application documentation: > > 2) The maker note information is copied as a block, so it isn't > affected like other information by subsequent tag assignments > on the command line. Also, since the PreviewImage referenced > from the maker notes may be rather large, it is not copied, and > must be transferred separately if desired. > > I realize this is inconvenient if you really do want to copy the > preview image, but when it is built into the makernotes by default > it causes problems because it can make the makernotes too big > to fit into a JPEG image when copying tags from a RAW to a JPEG. > > I hope this isn't too much of a problem for you. If so, I am open > to suggestions.

Uhm, I admit that I did miss that in the docs, but mainly because I'm not actually using exiftool but the Image::ExifTool module from a program of mine. I saw that exiftool had the same "problem" so I decided it was better to refer to it. In my experiments, I saw that explicit copy of PreviewImage is toublesome as well. The final "recipe" by which I got it is the following: my ($in, $out) = @ARGV; { my $et = Image::ExifTool->new(); $et->Options(Binary => 1, Unknown => 1); $et->ExtractInfo($out); $et->SetNewValuesFromFile($in, '*:*'); $et->WriteInfo($out); } my $preview = do { my $et = Image::ExifTool->new(); $et->ExtractInfo($in); $et->Options(Binary => 1); $et->GetValue('PreviewImage'); }; { my $et = Image::ExifTool->new(); $et->Options(Binary => 1, Unknown => 1); $et->ExtractInfo($out); $et->SetNewValue(PreviewImage => $$preview); $et->WriteInfo($out); } It was not possible to use the $et in the first block to set the preview: it simply refused to do it properly. Moreover, I noticed that it's actually *hard* to add a PreviewImage to some image that has nothing into it (EXIF-wisely), but this may derive from my lack of understanding of how EXIF works. Regarding suggestions, the most simple thing that comes to mind is having an option by which one can decide if the preview image should be included or excluded. You could set exclusion by default, and let the user explicitly set if the preview is needed. In my case, I'm working on a little application to rotate images by 90 degrees, retain all EXIF info and rotate thumbnail/preview as well. If one has jpegtran it's easy (it allows keeping EXIF stuff, and only thumbnail and preview need to be rotated), but my application fall back to Image::Magick if jpegtran isn't available, and this is where Image::ExifTool's ability to copy all EXIF stuff comes handy. Another thing that could be handy is a "suggested" recipe to copy all the tags from one image to another. Cheers, Flavio.

Sun Jun 08 11:27:55 2008 EXIFTOOL [...] cpan.org - Correspondence added

Hi Flavio, On Sun Jun 08 10:33:36 2008, POLETTIX wrote: Show quoted text

> Uhm, I admit that I did miss that in the docs, but mainly because I'm > not actually using exiftool but the Image::ExifTool module from a > program of mine.

I apologize for this. I'll see if I can fix this. Show quoted text

> In my experiments, I saw that explicit copy of PreviewImage is > toublesome as well. The final "recipe" by which I got it is the following: > > my ($in, $out) = @ARGV; > { > my $et = Image::ExifTool->new(); > $et->Options(Binary => 1, Unknown => 1); > $et->ExtractInfo($out);

That ExtractInfo() call is not necessary for what you are doing. Also, the Options you set have no effect for SetNewValuesFromFile() or WriteInfo(), so this call can be removed too. Show quoted text

> $et->SetNewValuesFromFile($in, '*:*'); > $et->WriteInfo($out); > } > > my $preview = do { > my $et = Image::ExifTool->new(); > $et->ExtractInfo($in); > $et->Options(Binary => 1); > $et->GetValue('PreviewImage'); > }; > > { > my $et = Image::ExifTool->new(); > $et->Options(Binary => 1, Unknown => 1); > $et->ExtractInfo($out);

Again, you can remove the 2 lines above. Show quoted text

> $et->SetNewValue(PreviewImage => $$preview); > $et->WriteInfo($out); > } > > It was not possible to use the $et in the first block to set the > preview: it simply refused to do it properly.

This is because the PreviewImage is embedded in the MakerNotes, which are copied as a block. So attempting the copy the previewImage separately doesn't work because what needs to be done is that the makernotes have to be rebuilt to include the preview. Maybe I can check to see if PreviewImage is in the SetNewValuesFromFile() arguments, and if so build in the preview. I will look into this. On second thought, this will be very difficult to do for some models where the preview image is referenced from the makernotes but actually resides after the JPEG EOI. Show quoted text

> Moreover, I noticed that > it's actually *hard* to add a PreviewImage to some image that has > nothing into it (EXIF-wisely), but this may derive from my lack of > understanding of how EXIF works.

Unfortunately the PreviewImage is not part of the EXIF standard. So it may only be written with makernotes that support this. The PreviewImage is one of the MAJOR pains in handling meta information, since there is no proper standard and everyone does it differently, sometimes is it even found outside the EXIF segment (after the JPEG EOI as I mentioned above). Show quoted text

> Regarding suggestions, the most simple thing that comes to mind is > having an option by which one can decide if the preview image should be > included or excluded.

I will think about this, but as I mentioned it won't work for all makes. Show quoted text

> Another thing that could be handy is a "suggested" recipe to copy all > the tags from one image to another.

Basically, this exists in the SetNewValuesFromFile() examples. The recipe is this: $exifTool->SetNewValuesFromFile($srcFile, "*.*"); $exifTool->WriteInfo($dstFile); Note that the '*.*' is a good idea only if copying between like-typed files. Otherwise it is best to leave this out to allow information to be written to the appropriate groups.

Sun Jun 08 18:35:51 2008 flavio [...] polettix.it - Correspondence added

Subject:	Re: [rt.cpan.org #36558] metadata copying is broken
Date:	Mon, 9 Jun 2008 00:48:57 +0200 (CEST)
To:	bug-Image-ExifTool [...] rt.cpan.org
From:	"Flavio Poletti" <flavio [...] polettix.it>

Show quoted text

> <URL: http://rt.cpan.org/Ticket/Display.html?id=36558 >

>> Another thing that could be handy is a "suggested" recipe to copy all >> the tags from one image to another.

> > Basically, this exists in the SetNewValuesFromFile() examples. The > recipe is this: > > $exifTool->SetNewValuesFromFile($srcFile, "*.*"); > $exifTool->WriteInfo($dstFile); > > Note that the '*.*' is a good idea only if copying between like-typed > files. Otherwise it is best to leave this out to allow information to > be written to the appropriate groups. >

Maybe I didn't make myself clear, but I was suggesting to include a recipe to copy *all* the tags, including the preview. Something along these lines, thanks to your suggestions about unneeded stuff in my code: sub copy_all_metadata { my ($in, $out) = @_; # Copy all metadata - except PreviewImage which is special my $et = Image::ExifTool->new(); $et->SetNewValuesFromFile($in); $et->WriteInfo($out); # Grab PreviewImage from source, if present $et = Image::ExifTool->new(); $et->Options(Binary => 1); $et->ExtractInfo($in); my $preview = $et->GetValue('PreviewImage') or return; # Set PreviewImage in target image $et = Image::ExifTool->new(); $et->SetNewValue(PreviewImage => $$preview); $et->WriteInfo($out); return; } ## end sub copy_all_exif Anyway, it would be great to have an Option like StripPreview (or something like this) defaulting to true but that can be set to false to let SetNewValuesFromFile do its job including the preview. Cheers, Flavio.

Tue Jun 10 09:32:39 2008 EXIFTOOL [...] cpan.org - Correspondence added

From:

EXIFTOOL [...] cpan.org

Hi Flavio, On Sun Jun 08 11:27:55 2008, EXIFTOOL wrote: Show quoted text

> On Sun Jun 08 10:33:36 2008, POLETTIX wrote:

> > Uhm, I admit that I did miss that in the docs, but mainly because I'm > > not actually using exiftool but the Image::ExifTool module from a > > program of mine.

> > I apologize for this. I'll see if I can fix this.

Doh. The ExifTool.pod documentation already contains: "If a preview image exists, it is not copied. The preview image must be transferred separately if desired." On Sun Jun 08 18:35:51 2008, flavio@polettix.it wrote: Show quoted text

> Maybe I didn't make myself clear, but I was suggesting to include a recipe > to copy *all* the tags, including the preview. Something along these > lines, thanks to your suggestions about unneeded stuff in my code:

OK, I understand now, thanks. I agree that this would have been helpful in this case, but the documentation is cluttered enough already, and I try to avoid adding extra bulk unless necessary. I will wait and see if anyone else has a similar problem. Show quoted text

> Anyway, it would be great to have an Option like StripPreview (or > something like this) defaulting to true but that can be set to false to > let SetNewValuesFromFile do its job including the preview.

I have looked into this, but actually implementing the copy of the PreviewImage for the general case is very difficult. For Pentax specifically, it is not hard because the preview is small and can fit inside the MakerNotes, but for other makes it is actually separate and would be a real technical challenge to maintain the proper MakerNotes pointers to an image outside the MakerNotes when writing as separate tags to different images. Since my goal is to as much as possible provide a uniform interface, I want the behaviour to be the same across all makes. Maybe at some point down the road I will get enough requests to make this worth all the work. - Phil

Tue Jun 10 14:10:41 2008 flavio [...] polettix.it - Correspondence added

Subject:	Re: [rt.cpan.org #36558] metadata copying is broken
Date:	Tue, 10 Jun 2008 20:23:44 +0200 (CEST)
To:	bug-Image-ExifTool [...] rt.cpan.org
From:	"Flavio Poletti" <flavio [...] polettix.it>

Hi Phil, I don't want to be dense, only try to add some value :) Regarding the doc, I overlooked it in the notes, sorry for the traffic. I'd only object that the "separate transfer" of the preview image is not trivial and an example could help :D Regarding the feature, IMHO it's perfectly legal to just throw an exception if the user tries to do something that breaks the rules, document it and set the default to not copying the preview as it is right now. If you can give me some hints on where to look into the code I can try to figure out some patch/proof of concept. Thank you for your time and feel free to use it in better ways than reading and answering my emails :) Cheers, Flavio.

Tue Jun 10 14:30:34 2008 EXIFTOOL [...] cpan.org - Correspondence added

From:

EXIFTOOL [...] cpan.org

On Tue Jun 10 14:10:41 2008, flavio@polettix.it wrote: Show quoted text

> I don't want to be dense, only try to add some value :)

I appreciate this, and can see things from your perspective. Show quoted text

> Regarding the doc, I overlooked it in the notes, sorry for the traffic. > I'd only object that the "separate transfer" of the preview image is not > trivial and an example could help :D

This morning I released 7.31 in which I expanded that note to mention that you need to call WriteImage again for the preview. Show quoted text

> Regarding the feature, IMHO it's perfectly legal to just throw an > exception if the user tries to do something that breaks the rules, > document it and set the default to not copying the preview as it is right > now.

The problem is that if the EXIF doesn't fit inside the JPEG segment, then exiftool won't write it at all and let me tell you that many more people will complain that exiftool isn't writing EXIF than would complain of a missing preview (everyone seems to ignore the warnings -- I guess there are too many of them). Show quoted text

> If you can give me some hints on where to look into the code I can > try to figure out some patch/proof of concept.

Comment out line 937 of lib/Image/ExifTool/WriteExif.pl in ExifTool version 7.31. This line effectively prevents the PreviewImage from being added to the MakerNotes through the call to SetNewValue. As you know, at one point this code was functional, but it was changed back due to problems with too large EXIF segments when copying all tags from some types of RAW images. Show quoted text

> Thank you for your time and feel free to use it in better ways than > reading and answering my emails :)

LOL. It is my pleasure to help you. Glad we're converging here. - Phil

Wed Jun 18 09:01:16 2008 EXIFTOOL [...] cpan.org - Status changed from 'open' to 'resolved'

Wed Jun 18 09:01:24 2008 EXIFTOOL [...] cpan.org - Given to EXIFTOOL

Wed Jun 18 09:01:25 2008 EXIFTOOL [...] cpan.org - Fixed in 7.30 added