Bug #75946 for App-SD: Error when cloning from github

Wed Mar 21 13:40:37 2012 mschwern [...] cpan.org - Ticket created

Subject:

Error when cloning from github

Attempting to clone http://github.com/schwern/test-more/issues I get the following error... $ git sd init Setting your SD email address to your git email: git config --get user.email sd config user.email-address schwern@pobox.com Cloning issues from git@github.com:schwern/test-more.git: sd clone --from github:git@github.com:schwern/test-more.git Username for http://github.com/: schwern GitHub API token for schwern (from http://github.com/account): Setting replica's username and token in the config file github:git@github.com:schwern/test-more.git: Finding matching tickets Not an ARRAY reference at /Users/schwern/perl5/perlbrew/perls/perl-5.14.1/lib/site_perl/5.14.1/App/SD/Replica/github/PullEncoder.pm line 80. Your Prophet database already exists. Line 80 is: @{ $self->sync_source->github->issue->comments( $args{ticket}->{number} ) }; Net::GitHub is at 0.30.

Wed Mar 21 13:44:47 2012 mschwern [...] cpan.org - Correspondence added

Adding in a debug statement before line 80, the offending ticket is https://github.com/schwern/test-more/issues/201

Wed Mar 21 13:44:48 2012 mschwern [...] cpan.org - Status changed from 'new' to 'open'

Wed Mar 21 13:55:33 2012 mschwern [...] cpan.org - Correspondence added

Subject:

Deal with Github API rate limiting

Sorry, my reply was hasty. It's not a particular ticket. What's happening is Github is rate limiting the issue API. DB<1> x $raw_txns 0 HASH(0x1032c7040) 'error' => 'HTTP/1.1 403 Forbidden Cache-Control: no-cache Connection: close Date: Wed, 21 Mar 2012 17:51:20 GMT Server: nginx/1.0.13 Content-Encoding: gzip Content-Length: 53 Content-Type: application/json; charset=utf-8 Client-Date: Wed, 21 Mar 2012 17:51:20 GMT Client-Peer: 207.97.227.239:80 Client-Response-Num: 1 Client-Transfer-Encoding: chunked Status: 403 Forbidden X-Content-Length: 73 X-Frame-Options: deny X-RateLimit-Limit: 60 X-RateLimit-Remaining: 0 X-Runtime: 3 {"error":["API Rate Limit Exceeded for 76.115.1.76"]} ' In other projects I've found one has to expect this from the Github API and have some sort of backoff and retry mechanism. Unfortunately Net::GitHub does not implement this. Here's an example of a work around in Gitpan. https://github.com/schwern/gitpan/blob/rewrite/lib/Gitpan/Github.pm

Wed Mar 21 14:55:37 2012 mschwern [...] cpan.org - Correspondence added

On Wed Mar 21 13:55:33 2012, MSCHWERN wrote: Show quoted text

> In other projects I've found one has to expect this from the Github API > and have some sort of backoff and retry mechanism. Unfortunately > Net::GitHub does not implement this.

Turns out they do, see "api_throttle" in Net::GitHub::V2, but only for authenticated requests. Getting an issue's comments is not authenticated. This seems to be an oversight and I've alerted Net::GitHub about it. https://github.com/fayland/perl-net-github/issues/30