Sorry for the delay in responding. I was on vacation when you posted this and somehow it got lost in my backlog of mail.
Unfortunately the metadata is very often at the end of the file for video files, and in this case ExifTool must read the whole thing to find it. For videos with metadata at the start, ExifTool shouldn't have to read the whole thing. You can use the -v3 option to see the offsets of where the metadata is stored.
- Phil