Bug #12058 for CGI-Uploader: Preserve name of uploaded files / Directory mapping

Tue Mar 29 15:21:40 2005 Guest - Ticket created

Subject:

Preserve name of uploaded files / Directory mapping

Hello again Mark, It'd be really nice if the name of the file could be preserved when saving to the filesystem as well as stored into the uploads table. Some error checking would need to added to ensure that duplicates are not overwritten (or add a flag to allow overwrite). Along these lines, it'd be useful to add the feature of storing files into directory hierarchies like CPAN.pm does for authors. (I don't know what this technique is called). William

Tue Mar 29 15:56:53 2005 Guest - Correspondence added

From:

william [...] knowmad.com

Actually preserving the file_name turned out to be easier than I expected. I only had to add a line to extract_meta, tweak the call to store_file in store_upload and remove an assertion in store_file. I now have a working example. I've attached a patch so that you can review my modifications and consider adding this feature for future releases. Note that this version will overwrite any existing files with the same name. This patch includes the mods that I added to support Pg via ODBC. Enjoy! William

Download Uploader.pm.diff2
application/octet-stream 1.6k

Message body not shown because it is not plain text.

Tue Mar 29 15:59:27 2005 Guest - Correspondence added

Oh, and you need to include the file_name field in your uploads table: file_name character varying(255), Also, if you use up_table_map in your spec, you should add the following line: file_name => 'file_name', I think that's everything. William

Tue Mar 29 16:00:10 2005 mark [...] summersault.com - Correspondence added

Date:	Tue, 29 Mar 2005 16:00:45 -0500
From:	Mark Stosberg <mark [...] summersault.com>
To:	Guest via RT <bug-CGI-Uploader [...] rt.cpan.org>
Subject:	Re: [cpan #12058] Preserve name of uploaded files / Directory mapping
RT-Send-Cc:

On Tue, Mar 29, 2005 at 03:21:40PM -0500, Guest via RT wrote: Show quoted text

> > It'd be really nice if the name of the file could be preserved when > saving to the filesystem as well as stored into the uploads table. > Some error checking would need to added to ensure that duplicates are > not overwritten (or add a flag to allow overwrite).

I'm not interested in preserving the file name on the file system (although I would consider a patch). What's an example of when you would use the file name if it was stored in the database? I never use them. I suppose rt.cpan.org uses these to some affect-- to display the name of attachments to download. Some care would need to be taken that the file name was "safe"-- didn't include some evil XSS for example. :) Show quoted text

> Along these lines, it'd be useful to add the feature of storing files > into directory hierarchies like CPAN.pm does for authors. (I don't > know what this technique is called).

Did you find the option named 'file_scheme'? It does something like that, and I use it to store what I suspect is about 1,000,000 images. Mark

Tue Mar 29 16:02:08 2005 mark [...] summersault.com - Correspondence added

Date:	Tue, 29 Mar 2005 16:02:45 -0500
From:	Mark Stosberg <mark [...] summersault.com>
To:	Guest via RT <bug-CGI-Uploader [...] rt.cpan.org>
Subject:	Re: [cpan #12058] Preserve name of uploaded files / Directory mapping
RT-Send-Cc:

On Tue, Mar 29, 2005 at 03:59:27PM -0500, Guest via RT wrote: Show quoted text

> > This message about CGI-Uploader was sent to you by guest <> via rt.cpan.org > > Full context and any attached attachments can be found at: > <URL: https://rt.cpan.org/Ticket/Display.html?id=12058 > > > Oh, and you need to include the file_name field in your uploads table: > > file_name character varying(255), > > Also, if you use up_table_map in your spec, you should add the following > line: > > file_name => 'file_name', > > I think that's everything.

If you work out a few more details, this could turn into a real patch. :) ( Secretly, I'm hard at work on finishing a DFV 4.0 release with some nice interface enhancements. ) Mark

Tue Mar 29 16:32:40 2005 Guest - Correspondence added

Hi Mark, Glad you're warming up to the idea even if you don't need it personally. Here's the use case (which also has a placeholder in your cookbook): I need to create an anonymous uploader which can accept up to 5 files which are stored onto a web server. An email will be sent to a selected recipient. This message will have a hyperlink(s) to the uploaded file(s). I need to preserve file name so that any communication between the uploader and the recipient will make sense. BTW, I'd be glad to include the current project I'm building in the examples. Now for the gruesome details. Since you are more familiar with the API, can you suggest a way of indicating when to preserve filenames? I'd also need a flag to indicate whether to overwrite existing files. Perhaps a new spec option such as file_name which would default to "id" but could also be "filename" to support my proposal as well as future growth. I'll also need to fix fk_meta to return a proper path. Also, what happens if a field is in up_table_map but isn't available in the uploads database? Will that generate an error? I can probably delete the file_name field from the map if we are using id's as filenames. Thanks for the note about the file_scheme option. I suppose one option is to add a new file_scheme to support formats such as I had suggested: w/william.jpg (I've only built a one-level hierarchy; if you have code to support multiple levels, I'd be glad to include it). For now, I'll stick with the one directory. Cheers! William

Wed Mar 30 08:38:12 2005 Guest - Correspondence added

From:

william [...] knowwmad.com

OK, here it is. I was able to get all the tests to pass the original Uploader.pm then applied my changes. I've also included a new test file to ensure that my code is working as advertised. The attached archive contains patches for Upload.pm and create_uploader_table.Pg.sql (sorry, I don't have MySQL installed) as well as the new file. I think that should be enough to patch your version. Let me know how it goes. One issue that I'm not sure how to handle is not including the file_name field in the uploads database. I suspect there's a way to handle this but it would probably take some monkeying with the shared_meta. I noticed that you had an entry, bytes, commented out in the default up_table_map. Perhaps you already have a way of supporting extra fields that I can tap into. The problem I see with not putting file_name into up_table_map is that the field name cannot be easily changed by the user. If you incoporate these changes, I'll also send over my uploader cgi program that supports anonymous uploads of up to 5 files at a time and sends an email notification to any interested parties. Thanks, William PS: In the store_upload method, you are running extract_meta twice. The first one as it doesn't appear to be necessary.

Download CGI-Uploader.diff.tar.bz2
application/x-bzip2 3.5k

Message body not shown because it is not plain text.

Wed Mar 30 09:00:26 2005 mark [...] summersault.com - Correspondence added

Date:	Wed, 30 Mar 2005 09:01:04 -0500
From:	Mark Stosberg <mark [...] summersault.com>
To:	Guest via RT <bug-CGI-Uploader [...] rt.cpan.org>
Subject:	Re: [cpan #12058] Preserve name of uploaded files / Directory mapping
RT-Send-Cc:

On Wed, Mar 30, 2005 at 08:38:12AM -0500, Guest via RT wrote: Show quoted text

> > This message about CGI-Uploader was sent to you by guest <> via rt.cpan.org > > Full context and any attached attachments can be found at: > <URL: https://rt.cpan.org/Ticket/Display.html?id=12058 > > > OK, here it is.

Thanks a bunch. I hope to look at this in detail this weekend. Mark

Wed Mar 30 10:28:10 2005 Guest - Correspondence added

From:

william [...] knowwmad.com

[mark@summersault.com - Wed Mar 30 09:00:26 2005]: Show quoted text

> Thanks a bunch. I hope to look at this in detail this weekend.

Well let me toss out another version for you to review. This one does not automatically add the file_name to up_table_map. This means that I'm using a different sql file for generating the table when including the file_name. Also while working on this modification, I had Image::Magick go wacko on me. While debugging, I added some tests to the ImageMagick.pm module. I've included a patch in case you want to add them. It would have saved me a few minutes while tracking down the bug. All tests are passing for me using Perl 5.8.4 under Debian Sarge. I promise to stop thinking about this module now!! Thanks, William PS: I figured out why you called extract_meta twice in store_upload; it's being called with slightly different data the second time.

Download CGI-Uploader-v2.diff.tar.bz2
application/x-bzip2 4.1k

Message body not shown because it is not plain text.

Sat Apr 02 10:51:43 2005 MARKSTOS [...] cpan.org - Taken

Sat Apr 02 10:52:09 2005 MARKSTOS [...] cpan.org - Status changed from 'new' to 'open'

Sun Dec 18 22:26:55 2005 MARKSTOS [...] cpan.org - Correspondence added

Show quoted text

> > I need to create an anonymous uploader which can accept up to 5 files > which are stored onto a web server. An email will be sent to a selected > recipient. This message will have a hyperlink(s) to the uploaded > file(s). I need to preserve file name so that any communication between > the uploader and the recipient will make sense.

William, What about this idea to address this case. I think it solves your desire for clean URLs and my desire for unique IDs on the file system. Now we are storing the file name in the database. So a URL could be constructed that looks like this: images/2/PICT0001.jpg images/3/PICT0002.jpg images/4/PICT0001.jpg # a different one than the first! Using mod_rewrite or our own dispatcher (like CGI::Application::Dispatch) the file actually being accessed would be images/2.jpg. (The actual name would be just for show ). Avoiding file system conflicts seems important, especially for the common digital camera case (illustrated above), where the default names get reset after every roll, generating lots of conflicting file names. Within some photo editting programs, it's easy not to even notice a photo has a bad name, because the interface always shows you a thumbnail of the photo to identify it by. I have some other thoughts to share I'll follow up seperately. Mark