Skip Menu |

This queue is for tickets about the Business-OnlinePayment-AuthorizeNet CPAN distribution.

Report information
The Basics
Id: 15210
Status: resolved
Priority: 0/
Queue: Business-OnlinePayment-AuthorizeNet

People
Owner: ivan-pause [...] 420.am
Requestors: paul [...] booyaka.com
Cc: jshirley [...] cpan.org
AdminCc:

Bug Information
Severity: Important
Broken in:
  • 2.01
  • 3.00
  • 3.01
  • 3.10
  • 3.11
  • 3.12
  • 3.13
  • 3.14
  • 3.15
  • 3.16
  • 3.17
  • 3.18
  • 3.19
  • 3.20
Fixed in: 3.21



Subject: B:OP::AN reports a failure when fields contain the encapsulation character, even when the transaction succeeds
Business::OnlinePayment::AuthorizeNet 3.15 (and all earlier versions) will incorrectly report a transaction as failed when it actually has succeeded if the AuthorizeNet gateway response contains double quotes (") that are not used for encapsulation. This could happen if any user-supplied field (e.g., the "company" field, or the "description" field) contains the encapsulation character. For example, the following company name will trigger this behavior: "PerlShop" - for all your Perl needs When the string is passed back in the response data, the double-quotes will not be escaped, and Text::CSV_XS::parse will fail. Since B::OP::AN does not check parse()'s return status, the transaction will be reported as unsuccessful, even if the Authorize.Net backend succeeded in debiting or crediting the account. The Perl module returns an error_message value starting with: DEBUG: No x_response_code from server, (HTTPS response: HTTP/1.1 200 OK) The attached patch fixes this bug. It will check all of the supplied user data to ensure that no instances of the encapsulation character are submitted to Authorize.Net (outside of the x_encap_char field, that is). The transaction will be aborted before Authorize.Net is ever called. This is not an ideal solution -- ideally, we'd like Authorize.Net to escape instances of the encapsulation character in field data. However, Authorize.Net does not implement this. I also considered silently stripping out the encapsulation character from passed-in user data, but rejected this method on the grounds that some applications may depend on receiving the same data back that was originally submitted. Comments are welcome, - Paul
Business::OnlinePayment::AuthorizeNet 3.15 (and all earlier versions) will incorrectly report a transaction as failed when it actually has succeeded if the AuthorizeNet gateway response contains double quotes (") that are not used for encapsulation. This could happen if any user-supplied field (e.g., the "company" field, or the "description" field) contains the encapsulation character. For example, the following company name will trigger this behavior: "PerlShop" - for all your Perl needs When the string is passed back in the response data, the double-quotes will not be escaped, and Text::CSV_XS::parse will fail. Since B::OP::AN does not check parse()'s return status, the transaction will be reported as unsuccessful, even if the Authorize.Net backend succeeded in debiting or crediting the account. The Perl module returns an error_message value starting with: DEBUG: No x_response_code from server, (HTTPS response: HTTP/1.1 200 OK) The following patch fixes this bug. It will check all of the supplied user data to ensure that no instances of the encapsulation character are submitted to Authorize.Net (outside of the x_encap_char field, that is). The transaction will be aborted before Authorize.Net is ever called. This is not an ideal solution -- ideally, we'd like Authorize.Net to escape instances of the encapsulation character in field data. However, Authorize.Net does not implement this. I also considered silently stripping out the encapsulation character from passed-in user data, but rejected this method on the grounds that some applications may depend on receiving the same data back that was originally submitted. Comments are welcome, - Paul --- AuthorizeNet.pm.orig 2005-10-21 18:48:03.000000000 -0700 +++ AuthorizeNet.pm 2005-10-21 18:54:48.000000000 -0700 @@ -194,10 +194,33 @@ $post_data{'x_Test_Request'} = $self->test_transaction()?"TRUE":"FALSE"; $post_data{'x_ADC_Delim_Data'} = 'TRUE'; $post_data{'x_delim_char'} = ','; - $post_data{'x_encap_char'} = '"'; + $post_data{'x_encap_char'} = my $encap_char = '"'; $post_data{'x_ADC_URL'} = 'FALSE'; $post_data{'x_Version'} = '3.1'; + # Abort the transaction if any instances of the encapsulation + # character occur in the content parameters that the user passed + # in. This is necessary since Authorize.Net will not properly + # escape the encapsulation character if it occurs within field + # data when it sends back its response. This can cause + # Text::CSV_XS to fail the response parse, which in turn can cause + # the caller to believe that a transaction has failed, when it + # actually has succeeded. + # + while (my ($k, $v) = each %post_data) { + next if $k eq 'x_encap_char'; + + if (index ($v, $encap_char) > -1) { + # Fail the request + $self->is_success(0); + $self->error_message("This transaction cannot be processed, ". + "since the $k field contains at least ". + "one $encap_char character. Please ". + "remove this character and resubmit."); + return; + }; + }; + my $pd = make_form(%post_data); my $s = $self->server(); my $p = $self->port(); @@ -207,7 +230,7 @@ #escape NULL (binary 0x00) values $page =~ s/\x00/\^0/g; - my $csv = new Text::CSV_XS({ 'binary'=>1 }); + my $csv = new Text::CSV_XS({ 'binary'=>1, 'quote_char'=>$encap_char }); $csv->parse($page); my @col = $csv->fields();
On Fri Oct 21 22:18:55 2005, guest wrote: Show quoted text
> I also considered silently stripping out the encapsulation character > from passed-in user data, but rejected this method on the grounds
that Show quoted text
> some applications may depend on receiving the same data back that
was Show quoted text
> originally submitted.
Can you tell me more about why you rejected this approach? I'd be inclined to silently strip as the default, perhaps with an option to return an error instead like your patch, if an application has that need.
Subject: Re: [rt.cpan.org #15210] B:OP::AN reports a failure when fields contain the encapsulation character, even when the transaction succeeds
Date: Tue, 31 Oct 2006 10:59:59 -0700 (MST)
To: via RT <bug-Business-OnlinePayment-AuthorizeNet [...] rt.cpan.org>
From: Paul Walmsley <paul [...] booyaka.com>
On Sat, 21 Oct 2006, via RT wrote: Show quoted text
> <URL: http://rt.cpan.org/Ticket/Display.html?id=15210 > > > On Fri Oct 21 22:18:55 2005, guest wrote: >
>> I also considered silently stripping out the encapsulation character >> from passed-in user data, but rejected this method on the grounds
> that
>> some applications may depend on receiving the same data back that
> was
>> originally submitted.
> > Can you tell me more about why you rejected this approach? I'd be > inclined to silently strip as the default, perhaps with an option to > return an error instead like your patch, if an application has that > need.
I rejected that approach based on two basic principles: 1. Well-written code must not silently modify or corrupt user data -- especially code that handles payment processing. 2. Code should "fail fast" when something is wrong. An example of a problem that could be caused by a "silently-strip" approach: I've got a process that parses the Authorize.Net transaction log to reconcile its records with a local database. Suppose my process expects to exactly match the "Company" field in Authorize.Net with what's in the database. My process will fail if the encapsulation character is stripped. ... In any case, whichever fix you choose, I hope you incorporate it quickly. This bug report is over a year old, and it is a critical problem to fix. We've documented cases where customers' cards were charged, but B::OP::AN incorrectly reported to our system that the transaction failed. regards, - Paul
I'm not sure if either the "silently strip" or "throw error" solutions are really the right thing, and had the time to really think about what needs to be done here. If Authorize.Net can accept some sort of quoting or esacping, that would be ideal, but IIRC that's not possible. :/ Maybe we can check the passed-in data and set the delimiter character to something that we *know* won't be passed in?
I can't figure out how to use RT, but I'd love to add my support for this problem being fixed. It's quite serious and it's crazy how long it's been broken. Especially with a patch available, although I'd prefer the transaction to go through using the XML API, since that apparently doesn't have this problem. See http://www.authorize.net/kb.asp? page_id=169764&url=/authorize.net/consumer/search.asp chris@masto.com
From: ivan-pause [...] 420.am
On Tue May 20 14:49:13 2008, http://www.chrismasto.com/openid/ wrote: Show quoted text
> I can't figure out how to use RT, but I'd love to add my support for > this problem being fixed. It's > quite serious and it's crazy how long it's been broken.
I agree that it would be desirable to get the bug fixed. Support is great... help implementing a patch is even better. :) Show quoted text
> Especially with a patch available,
I don't think the initial patch is a good fix for the problem. Upon consideration, I think that the best way to fix the problem is to examine the passed-in data and choose an appropriate encapsulation character on-the-fly that isn't included in the passed-in data. Or, as you suggest, move to the XML API which does not have this problem. Help implementing either approach is more than welcome. I'd be more than happy to do a release given a working patch that applied against current code... Show quoted text
> although I'd prefer the transaction to go through using the XML API, > since that apparently > doesn't have this problem. See http://www.authorize.net/kb.asp? > page_id=169764&url=/authorize.net/consumer/search.asp
From: ivan-pause [...] 420.am
On Tue May 20 14:49:13 2008, http://www.chrismasto.com/openid/ wrote: Show quoted text
> It's too bad Authorize.net's CSV-like approach doesn't support any > kind of real escaping. It would be a lot quicker to work around it > that way.
Sure, if they just supported actual CSV there wouldn't be a problem in the first place. Show quoted text
> The only other choices, besides rewriting it all to use > XML, would seem to be to throw an error, remove invalid characters,
We've rejected these first two approaches. Both are unpleasant for various reasons. Show quoted text
> or scan all the data to try to find a non-conflicting delimeter.
This is the desired fix. We should try every possible legal encapsulation character before throwing an error. It is the best that can be done given the limitations of the API (without rewriting against the XML one).
Perhaps the better solution is to update the x_delim_char? The AIM documents suggest using |, which seems like a much better choice. If you look at their sample (from http://developer.authorize.net/samplecode/) you can see they do it this way. Testing with quotes in the fields works fine. Then, the split is a simple split(/\Q|/, $response->content); rather than relying on Text::CSV_XS which seems inappropriate given the response payload. Please see the attached patch. It would be great if this got fixed, this patch may not be 100% done but it passes tests and adds another with a quote. (Might I suggest putting the source on github or another public repos, to make it easier for submitting patches against?) Thanks, -Jay
diff -u -r Business-OnlinePayment-AuthorizeNet-3.20/AuthorizeNet/AIM.pm Business-OnlinePayment-AuthorizeNet-3.20-fix/AuthorizeNet/AIM.pm --- Business-OnlinePayment-AuthorizeNet-3.20/AuthorizeNet/AIM.pm 2008-02-02 17:35:05.000000000 -0800 +++ Business-OnlinePayment-AuthorizeNet-3.20-fix/AuthorizeNet/AIM.pm 2009-03-24 12:02:10.000000000 -0700 @@ -224,8 +224,9 @@ } $post_data{'x_ADC_Delim_Data'} = 'TRUE'; - $post_data{'x_delim_char'} = ','; - $post_data{'x_encap_char'} = '"'; + $post_data{'x_delim_char'} = '|'; + #$post_data{'x_encap_char'} = '"'; + $post_data{'x_encap_data'} = 'TRUE'; $post_data{'x_ADC_URL'} = 'FALSE'; $post_data{'x_Version'} = '3.1'; @@ -241,9 +242,7 @@ #trim 'ip_addr="1.2.3.4"' added by eProcessingNetwork Authorize.Net compat $page =~ s/,ip_addr="[\d\.]+"$//; - my $csv = new Text::CSV_XS({ binary=>1, escape_char=>'' }); - $csv->parse($page); - my @col = $csv->fields(); + my @col = split(/\Q|/, $page); $self->server_response($page); $self->avs_code($col[5]); diff -u -r Business-OnlinePayment-AuthorizeNet-3.20/t/credit_card.t Business-OnlinePayment-AuthorizeNet-3.20-fix/t/credit_card.t --- Business-OnlinePayment-AuthorizeNet-3.20/t/credit_card.t 2006-10-18 16:11:55.000000000 -0700 +++ Business-OnlinePayment-AuthorizeNet-3.20-fix/t/credit_card.t 2009-03-24 12:02:43.000000000 -0700 @@ -4,7 +4,7 @@ require "t/lib/test_account.pl"; my($login, $password) = test_account_or_skip(); -plan tests => 2; +plan tests => 3; use_ok 'Business::OnlinePayment'; @@ -32,3 +32,26 @@ $tx->submit(); ok($tx->is_success()) or diag $tx->error_message; + +$tx->content( + type => 'VISA', + login => $login, + password => $password, + action => 'Normal Authorization', + description => 'Business::OnlinePayment visa test for quotes', + amount => '59.95', + invoice_number => '100101', + customer_id => 'jsk', + first_name => 'Tofu "Quote"', + last_name => 'Beast', + address => '123 Anystreet', + city => 'Anywhere', + state => 'UT', + zip => '84058', + card_number => '4111111111111111', + expiration => expiration_date(), +); +$tx->test_transaction(1); # test, dont really charge +$tx->submit(); + +ok($tx->is_success()) or diag $tx->error_message;
On Tue Mar 24 15:04:44 2009, JSHIRLEY wrote: Show quoted text
> Perhaps the better solution is to update the x_delim_char? > > The AIM documents suggest using |, which seems like a much better > choice.
Swapping out one character for another does not seem like a useful improvement. We could break existing applications. The desired fix is: - Examine the passed-in data and choose an appropriate encapsulation character on-the-fly that isn't included in the passed-in data. or - Move to the XML API which does not have this problem. Help implementing either approach is more than welcome. I'd be more than happy to do a release given a working patch that applied against current code... Show quoted text
> (Might I suggest putting the source on github or another public > repos, to make it easier for submitting patches against?)
The code is currently available from our public repository: export CVSROOT=":pserver:anonymous@cvs.freeside.biz:/home/cvs/cvsroot" cvs login # The password for the user `anonymous' is `anonymous'. cvs checkout Business-OnlinePayment-AuthorizeNet or on the web: http://freeside.biz/cgi-bin/viewvc.cgi/Business-OnlinePayment-AuthorizeNet/ I'll add this information to the documentation.
On Tue Mar 24 15:32:05 2009, IVAN wrote: Show quoted text
> On Tue Mar 24 15:04:44 2009, JSHIRLEY wrote:
> > Perhaps the better solution is to update the x_delim_char? > > > > The AIM documents suggest using |, which seems like a much better > > choice.
> > Swapping out one character for another does not seem like a useful > improvement. We could break existing applications. >
No more than they are broken now, probably less so. This issue of encapsulating in quotes is pointless, which is a more important change. Changing the delimiter is to just go inline with what Authorize.NET themselves suggest. I see no value in using a ,. Show quoted text
> The desired fix is: > > - Examine the passed-in data and choose an appropriate encapsulation > character on-the-fly that isn't included in the passed-in data. > > or > > - Move to the XML API which does not have this problem. >
I only see the XML API in terms of ARB, do you have a link to the AIM implementation? Show quoted text
> Help implementing either approach is more than welcome. I'd be more > than happy to do a release given a working patch that applied against > current code... >
> > (Might I suggest putting the source on github or another public > > repos, to make it easier for submitting patches against?)
> > The code is currently available from our public repository: > > export CVSROOT=":pserver:anonymous@cvs.freeside.biz:/home/cvs/cvsroot" > cvs login > # The password for the user `anonymous' is `anonymous'. > cvs checkout Business-OnlinePayment-AuthorizeNet > > or on the web: > >
http://freeside.biz/cgi-bin/viewvc.cgi/Business-OnlinePayment-AuthorizeNet/ Show quoted text
> > I'll add this information to the documentation. >
Thanks.
RT is being a pain in the ass and not letting me watch this ticket. Can you add me as a watcher (jshirley@cpan.org) Thanks. PS., I'm happy to do the transition to XML, as I need this to work and not die on quotes. For now, I'm going to be running my patched version.
On Tue Mar 24 15:59:06 2009, JSHIRLEY wrote: Show quoted text
> On Tue Mar 24 15:32:05 2009, IVAN wrote:
> > On Tue Mar 24 15:04:44 2009, JSHIRLEY wrote:
> > > Perhaps the better solution is to update the x_delim_char? > > > > > > The AIM documents suggest using |, which seems like a much
better Show quoted text
> > > choice.
> > > > Swapping out one character for another does not seem like a useful > > improvement. We could break existing applications. > >
> > No more than they are broken now, probably less so. This issue of > encapsulating in quotes is pointless, which is a more important > change. Changing the delimiter is to just go inline with what > Authorize.NET themselves suggest. I see no value in using a ,.
Yes, it would probably have been a better choice to start with, but as I said, swapping out one character for another is not a desirable fix at this time. Show quoted text
> > The desired fix is: > > > > - Examine the passed-in data and choose an appropriate > > encapsulation character on-the-fly that isn't included in the > > passed-in data. > > > > or > > > > - Move to the XML API which does not have this problem.
> > I only see the XML API in terms of ARB, do you have a link to the > AIM implementation?
I do not. It is possible the suggestion was in error and no such API is available for AIM. Show quoted text
> RT is being a pain in the ass and not letting me watch this ticket. > Can you add me as a watcher (jshirley@cpan.org)
Sure, done. I find CPAN RT's setup a bit puzzling too sometimes. :)
On Tue Mar 24 16:41:07 2009, IVAN wrote: Show quoted text
> On Tue Mar 24 15:59:06 2009, JSHIRLEY wrote:
> > On Tue Mar 24 15:32:05 2009, IVAN wrote:
> > > On Tue Mar 24 15:04:44 2009, JSHIRLEY wrote:
> > > > Perhaps the better solution is to update the x_delim_char? > > > > > > > > The AIM documents suggest using |, which seems like a much
> better
> > > > choice.
> > > > > > Swapping out one character for another does not seem like a useful > > > improvement. We could break existing applications. > > >
> > > > No more than they are broken now, probably less so. This issue of > > encapsulating in quotes is pointless, which is a more important > > change. Changing the delimiter is to just go inline with what > > Authorize.NET themselves suggest. I see no value in using a ,.
> > Yes, it would probably have been a better choice to start with, but as > I said, swapping out one character for another is not a desirable fix > at this time. >
It isn't just swapping a character, it's swapping a character and removing the unnecessary encapsulation character. The quote problem goes away if you don't use the encapsulation, but then you have to fight with commas. At this point, to get a version out that will work, can we just have a configuration set for the x_delim_char, x_encap_data and x_encap_char? The defaults can maintain what they are now, and then people can simply enable the | and no encap_char if the have this issue? I can get that out with a pod patch today. Show quoted text
> > > The desired fix is: > > > > > > - Examine the passed-in data and choose an appropriate > > > encapsulation character on-the-fly that isn't included in the > > > passed-in data. > > > > > > or > > > > > > - Move to the XML API which does not have this problem.
> > > > I only see the XML API in terms of ARB, do you have a link to the > > AIM implementation?
> > I do not. It is possible the suggestion was in error and no such API > is available for AIM. >
> > RT is being a pain in the ass and not letting me watch this ticket. > > Can you add me as a watcher (jshirley@cpan.org)
> > Sure, done. I find CPAN RT's setup a bit puzzling too sometimes. :) >
It was a permissions problem with me, so thank you :)
Show quoted text
> It isn't just swapping a character, it's swapping a character and > removing the unnecessary encapsulation character. > > The quote problem goes away if you don't use the encapsulation, but > then you have to fight with commas.
Right. It introduces a different set of problems with different characters. This is an unacceptable change for existing applications. Show quoted text
> At this point, to get a version out that will work, can we just have > a configuration set for the x_delim_char, x_encap_data and > x_encap_char?
No. This is not Business::AuthorizeNet, it is Business::OnlinePayment::AuthorizeNet. Implementing a common, gateway-agnostic interface is the whole point. x_delim_char/x_encap_char are gateway-specific details that should not be exposed as part of the public interfaace. The module should be able to auto-select those values without manual configuration, as previously indicated. Show quoted text
> The defaults can maintain what they are now, and then people can > simply enable the | and no encap_char if the have this issue? > > I can get that out with a pod patch today.
As long as you're interested in contributing, how about just implementing the desired fix? It has the advantage of not requiring POD changes and not requiring folks to change anything if they have a specific issue... :) Show quoted text
> > > > The desired fix is: > > > > > > > > - Examine the passed-in data and choose an appropriate > > > > encapsulation character on-the-fly that isn't included in the > > > > passed-in data.
For what its worth, we subclassed B::OP::AuthorizeNet and created a MyAuthorizeNet driver that solves this problem by overloading get_fields() to simply replace '"' with '""' in every field. This way, the CSV response from authorizenet contains two double quotes .. e.g.: 'foo "test"' becomes 'foo ""test""'. This seems better than just failing to run the transaction if quotes are present, or changing the delimiter character. With the quotes escaped this way, Text::CSV_XS is happy and it fixes the problem. Here is our overloaded get_fields(): sub get_fields { my $self = shift; my %data = $self->SUPER::get_fields(@_); for my $key (keys %data) { my $value = $data{$key}; next unless defined $value; if ($value =~ /"/) { $value =~ s/"/""/g; $data{$key} = $value; } } return %data; }
Attached is a patch for BOP::AuthorizeNet version 3.20 that fixes this bug. Fix Method: As suggested by Ivan, this fix checks for the encapsulating character in our input. If it exists, we try and grab another encapsulating character. We have a bunch of various character options, but start with the ones given by AuthorizeNet in their example of x_encap_char here: http://developer.authorize.net/guides/AIM/Transaction_Response/Transaction_Response.htm If we are unable to get a usable encapsulating character we fail out gracefully, without ever having hit the Authorize.Net servers. Cheers, -- Josh
--- AIM.pm_3_20 2009-11-11 13:33:02.000000000 -0700 +++ AIM.pm 2009-11-11 13:33:23.000000000 -0700 @@ -223,9 +223,33 @@ $post_data{'x_Email_Customer'} = 'FALSE'; } + my $data_string = join("", values %post_data); + + my $encap_character; + # The first set of characters here are recommended by authorize.net in their + # encapsulating character example. + # The second set we made up hoping they will work if the first fail. + # The third chr(31) is the binary 'unit separator' and is our final last + # ditch effort to find something not in the input. + foreach my $char( qw( | " ' : ; / \ - * ), qw( # ^ + < > [ ] ~), chr(31) ){ + if( index($data_string, $char) == -1 ){ # found one. + $encap_character = $char; + last; + } + } + + if(!$encap_character){ + $self->is_success(0); + $self->error_message( + "DEBUG: Input contains all encapsulating characters." + . " Please remove | or ^ from your input if possible." + ); + return; + } + $post_data{'x_ADC_Delim_Data'} = 'TRUE'; $post_data{'x_delim_char'} = ','; - $post_data{'x_encap_char'} = '"'; + $post_data{'x_encap_char'} = $encap_character; $post_data{'x_ADC_URL'} = 'FALSE'; $post_data{'x_Version'} = '3.1'; @@ -241,7 +265,7 @@ #trim 'ip_addr="1.2.3.4"' added by eProcessingNetwork Authorize.Net compat $page =~ s/,ip_addr="[\d\.]+"$//; - my $csv = new Text::CSV_XS({ binary=>1, escape_char=>'' }); + my $csv = new Text::CSV_XS({ binary=>1, escape_char=>'', quote_char => $encap_character }); $csv->parse($page); my @col = $csv->fields();
RT-Send-CC: josh [...] infogears.com
On Wed Nov 11 20:21:56 2009, josh@infogears.com wrote: Show quoted text
> Attached is a patch for BOP::AuthorizeNet version 3.20 that fixes this > bug.
It doesn't seem like a good idea to join all the data into a giant string and then search it for each character. Can you use a hash to store which characters are in the post data instead? Then checking is much more straighforward and doesn't involve repeatedly sequentially scanning a large string for each possible encapsulation character. Also, why is the set of characters to check limited? Is there a reason not to try every ASCII character or at least 32-127 ?
On Fri Nov 13 15:08:22 2009, IVAN wrote: Show quoted text
> On Wed Nov 11 20:21:56 2009, josh@infogears.com wrote:
> > Attached is a patch for BOP::AuthorizeNet version 3.20 that fixes this > > bug.
> > It doesn't seem like a good idea to join all the data into a giant > string and then search it for each character. > > Can you use a hash to store which characters are in the post data > instead? Then checking is much more straighforward and doesn't involve > repeatedly sequentially scanning a large string for each possible > encapsulation character. > > Also, why is the set of characters to check limited? Is there a reason > not to try every ASCII character or at least 32-127 ?
Thanks for the reply Ivan. My first thought was to actually hash the string as you mentioned. However, I got to thinking about it more, and I figured the 99.9% majority use case here is going to be to hit the first encapsulating character. Because of that, I was thinking that using the hash method would be an unnecessary slow down for 99.9% of cases. (Since we have the added construction of a hash then. No matter what, we have to scan the string once.) For the 0.1% or lower cases that do have to scan the entire string, it really doesn't take all that long to do anyway. (Especially since they should hit one of the first couple of characters.) If you feel strongly about this, let me know and I can certainly modify the patch. Also, yes there is a reason the character check is limited. I initially thought the same as you there, as well, and thought to scan the entire range. However, the problem is, we are only scanning our input and the actual output from authorize.net could contain many things in that range. (I actually created the code to do this and found out I was wrong when it busted on me.) That means we can't just willy-nilly pick any random item there. The best items to use are the ones Authorize.Net recommends in their docs, since they might heed those and keep them out of their output. If we can't use those, I threw in a few extra that we can hope also aren't in the output given they are symbols.
On Fri Nov 13 15:08:22 2009, IVAN wrote: Show quoted text
> On Wed Nov 11 20:21:56 2009, josh@infogears.com wrote:
> > Attached is a patch for BOP::AuthorizeNet version 3.20 that fixes this > > bug.
> > It doesn't seem like a good idea to join all the data into a giant > string and then search it for each character. > > Can you use a hash to store which characters are in the post data > instead? Then checking is much more straighforward and doesn't involve > repeatedly sequentially scanning a large string for each possible > encapsulation character.
So, I was going to throw a patch in here to prevent some back-and-forth in case you really wanted this. I decided to do some benchmarks to check and see what impact this would really have. I was pretty shocked with what I found. The method I used is actually 10-26.6 times faster than the hash method on my sample data! (26.6 times faster when using the first encapsulating character '|' and 10 times faster when using the second to last '~'.) I tried all sorts of hash routine combinations that I could think of as well as using an array as a sort of hash based on the ord() of the character. I'm attaching my benchmark file (bench_character_check.pl) for perusal/running. At the bottom of it you will find some of my timings from my runs. Maybe someone can see some improvement that can be made or a possible mistake I made. I also tested with using larger values of data, but the difference between the two was still huge. (You can experiment at the bottom of the %post_data hash in the script. There are some comments there to help figure out what I mean.) One thing I did note was that qw() was complaining here about using the comment symbol '#' in the qw(). So I have modified my patch to take that symbol outside the qw to prevent this kind of warning. It is attached as patch_15210_fix_qw.diff. -- Josh

Message body is not shown because it is too large.

--- AIM.pm_3_20 2009-11-11 13:33:02.000000000 -0700 +++ AIM.pm 2009-11-11 13:33:23.000000000 -0700 @@ -223,9 +223,33 @@ $post_data{'x_Email_Customer'} = 'FALSE'; } + my $data_string = join("", values %post_data); + + my $encap_character; + # The first set of characters here are recommended by authorize.net in their + # encapsulating character example. + # The second set we made up hoping they will work if the first fail. + # The third chr(31) is the binary 'unit separator' and is our final last + # ditch effort to find something not in the input. + foreach my $char( qw( | " ' : ; / \ - * ), '#', qw( ^ + < > [ ] ~), chr(31) ){ + if( index($data_string, $char) == -1 ){ # found one. + $encap_character = $char; + last; + } + } + + if(!$encap_character){ + $self->is_success(0); + $self->error_message( + "DEBUG: Input contains all encapsulating characters." + . " Please remove | or ^ from your input if possible." + ); + return; + } + $post_data{'x_ADC_Delim_Data'} = 'TRUE'; $post_data{'x_delim_char'} = ','; - $post_data{'x_encap_char'} = '"'; + $post_data{'x_encap_char'} = $encap_character; $post_data{'x_ADC_URL'} = 'FALSE'; $post_data{'x_Version'} = '3.1'; @@ -241,7 +265,7 @@ #trim 'ip_addr="1.2.3.4"' added by eProcessingNetwork Authorize.Net compat $page =~ s/,ip_addr="[\d\.]+"$//; - my $csv = new Text::CSV_XS({ binary=>1, escape_char=>'' }); + my $csv = new Text::CSV_XS({ binary=>1, escape_char=>'', quote_char => $encap_character }); $csv->parse($page); my @col = $csv->fields();
RT-Send-CC: josh [...] infogears.com
Josh: Well, goes to show what they say about premature optimization! I was sure off-base here. Glad you've considered the issues. Patch applied, will be in 3.21.