Skip Menu |

This queue is for tickets about the Net-Duo CPAN distribution.

Report information
The Basics
Id: 127964
Status: resolved
Priority: 0/
Queue: Net-Duo
People
Owner: RRA [...] cpan.org
Requestors: das1 [...] lehigh.edu
iglesias [...] uci.edu
Cc:
AdminCc:
Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: 1.02

History Show all quoted text
  Fri Dec 07 17:48:12 2018 iglesias [...] uci.edu - Ticket created
Subject: Net::Duo and new pagination requirements for Duo API
Date: Fri, 7 Dec 2018 14:41:25 -0800
To: bug-Net-Duo [...] rt.cpan.org
From: Mike Iglesias <iglesias [...] uci.edu>
As noted here: https://help.duo.com/s/article/4744?language=en_US&mkt_tok=eyJpIjoiT1dFeE1ESmtNemMwTURrNCIsInQiOiJRN0ZQWmcrcXJ1WEk3aVA2Q2FIMlVZSThwVE44S0hvS1BxZ0gwTWdNdmRZdHB6U3hwR0RpRHRTbVlGSTMwSnVrVEh0UlBTV1BuRHJlMmhvTktTcFNcL0p0YkpYMFJoaUJmT2tvazdVZkZPdFJmNTRnTVdVTGV1R2o2UjB4NVhjWEcifQ%3D%3D Duo is changing their APIs to do "pagination", where you only get a certain number of records back when you do an API request, like to get all the user's information. Will Net::Duo be updated to handle this? We have several Perl scripts that produce reports on Duo users, integrations, etc., that would need to be rewritten if Net::Duo is not updated to handle pagination. Thank you for your time, and for Net::Duo. -- Mike Iglesias Email: iglesias@uci.edu University of California, Irvine phone: 949-824-6926 Office of Information Technology FAX: 949-824-2270
  Tue Jan 22 12:13:44 2019 das1 [...] lehigh.edu - Requestor das1@lehigh.edu added
  Thu Feb 28 10:09:54 2019 cjbrabec [...] gmail.com - Correspondence added
On Fri Dec 07 17:48:12 2018, iglesias@uci.edu wrote: Show quoted text
> Duo is changing their APIs to do "pagination", where you only get a > certain > number of records back when you do an API request, like to get all the > user's > information.
We also ran into this issue at NCSU. We're only using this API for the /admin/v1/users call in our reporting. I made a fork of the code on github and wrote a POC-quality patch to add pagination for that one call. fork is here: https://github.com/cjbnc/net-duo To implement pagination, I split the call_json() function into three pieces: - call_raw_json() = most of the original call_json code, except it returns the whole data result, not just the response piece. This is needed to get the metadata piece that contains the pagination offset - call_json() = a new wrapper function that calls call_raw_json() but still returns only the response piece, just like the original - call_paged_json() = a new wrapper function that makes repeated calls to call_raw_json() for each page of data and assembles the result into a single array And I updated Admin::users() to use the new call_paged_json() In practice, it works for us. Instead of making one API call for 40,700 users, now we make 136 calls for 300 users each. Wheee! Adding this to other calls may be as simple as updating call_json() to call_paged_json() in those calls. That assumes the pagination works the same and accepts the same max limit of 300 that I used. Another note, './Build test' fails on the Mock users call because it expects to see no Content, but instead sees that the paged call adds the limit and offset variables to the request. Hope this is useful. Charles
  Thu Feb 28 10:09:55 2019 The RT System itself - Status changed from 'new' to 'open'
  Thu Feb 28 13:02:21 2019 RRA [...] cpan.org - Correspondence added
Subject: Re: [rt.cpan.org #127964] Net::Duo and new pagination requirements for Duo API
Date: Thu, 28 Feb 2019 09:54:38 -0800
To: "Charles Brabec via RT" <bug-Net-Duo [...] rt.cpan.org>
From: Russ Allbery <rra [...] cpan.org>
Hi folks, My apologies for not having responded to this bug earlier. The reality is that this package is orphaned. It was something I wrote at my previous job, and I no longer use it personally (and probably won't in the future). I had thought I would be able to keep maintaining it after I changed jobs, but in retrospect that was unrealistic. That said, I feel bad about leaving people in the lurch, so I'm going to try to get out a new release that fixes this problem. I've set aside some time this weekend to work on that. Thank you very much for the analysis of the change that's needed! I may ask for a tester of a pre-release version on GitHub, since I don't personally have access to a Duo account with a sufficiently large number of users. Going forward, obviously this package should have an owner who is actively using it and interested in maintaining it. If anyone wants to take it over, let me know. -- #!/usr/bin/perl -- Russ Allbery, Just Another Perl Hacker $^=q;@!>~|{>krw>yn{u<$$<[~||<Juukn{=,<S~|}<Jwx}qn{<Yn{u<Qjltn{ > 0gFzD gD, 00Fz, 0,,( 0hF 0g)F/=, 0> "L$/GEIFewe{,$/ 0C$~> "@=,m,|,(e 0.), 01,pnn,y{ rw} >;,$0=q,$,,($_=$^)=~y,$/ C-~><@=\n\r,-~$:-u/ #y,d,s,(\$.),$1,gee,print
  Thu Feb 28 13:24:49 2019 iglesias [...] uci.edu - Correspondence added
Subject: Re: [rt.cpan.org #127964] Net::Duo and new pagination requirements for Duo API
Date: Thu, 28 Feb 2019 10:11:17 -0800
To: bug-Net-Duo [...] rt.cpan.org, das1 [...] lehigh.edu
From: Mike Iglesias <iglesias [...] uci.edu>
I switched our scripts over to using the Duo API calls, and their method of handling pagination is similar to what Charles did - they have a json_paging_api_call that sets up the paging and a "next" call that gives you the next object you asked for. The next call does the paging behind the scenes so you don't have to worry about it. I can help out with the testing if you need it as I still have copies of the Net::Duo scripts. Mike On 2/28/19 10:02 AM, Russ Allbery via RT wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=127964 > > > Hi folks, > > My apologies for not having responded to this bug earlier. The reality is > that this package is orphaned. It was something I wrote at my previous > job, and I no longer use it personally (and probably won't in the future). > I had thought I would be able to keep maintaining it after I changed jobs, > but in retrospect that was unrealistic. > > That said, I feel bad about leaving people in the lurch, so I'm going to > try to get out a new release that fixes this problem. I've set aside some > time this weekend to work on that. Thank you very much for the analysis > of the change that's needed! I may ask for a tester of a pre-release > version on GitHub, since I don't personally have access to a Duo account > with a sufficiently large number of users. > > Going forward, obviously this package should have an owner who is actively > using it and interested in maintaining it. If anyone wants to take it > over, let me know. >
  Thu Feb 28 14:49:33 2019 das1 [...] lehigh.edu - Correspondence added
On Thu Feb 28 10:09:54 2019, cjbrabec wrote: Show quoted text
> On Fri Dec 07 17:48:12 2018, iglesias@uci.edu wrote:
> > Duo is changing their APIs to do "pagination", where you only get a > > certain > > number of records back when you do an API request, like to get all > > the > > user's > > information.
> > We also ran into this issue at NCSU. We're only using this API for the > /admin/v1/users call in our reporting. I made a fork of the code on > github > and wrote a POC-quality patch to add pagination for that one call. > > fork is here: https://github.com/cjbnc/net-duo > > To implement pagination, I split the call_json() function into three > pieces: > - call_raw_json() = most of the original call_json code, except it > returns the > whole data result, not just the response piece. This is needed to > get the > metadata piece that contains the pagination offset > - call_json() = a new wrapper function that calls call_raw_json() but > still returns only the response piece, just like the original > - call_paged_json() = a new wrapper function that makes repeated calls > to call_raw_json() for each page of data and assembles the result > into > a single array > > And I updated Admin::users() to use the new call_paged_json() > > In practice, it works for us. Instead of making one API call for > 40,700 users, > now we make 136 calls for 300 users each. Wheee! > > Adding this to other calls may be as simple as updating call_json() to > call_paged_json() in those calls. That assumes the pagination works > the same > and accepts the same max limit of 300 that I used. > > Another note, './Build test' fails on the Mock users call because it > expects > to see no Content, but instead sees that the paged call adds the > limit and offset variables to the request. > > Hope this is useful. > > Charles
Thanks Charles - Very useful - I'm glad I didn't have to figure this out. - Dan
  Sat Mar 02 13:52:25 2019 RRA [...] cpan.org - Correspondence added
Subject: Re: [rt.cpan.org #127964] Net::Duo and new pagination requirements for Duo API
Date: Sat, 02 Mar 2019 10:52:10 -0800
To: "Mike Iglesias via RT" <bug-Net-Duo [...] rt.cpan.org>
From: Russ Allbery <rra [...] cpan.org>
"Mike Iglesias via RT" <bug-Net-Duo@rt.cpan.org> writes: Show quoted text
> I switched our scripts over to using the Duo API calls, and their method > of handling pagination is similar to what Charles did - they have a > json_paging_api_call that sets up the paging and a "next" call that > gives you the next object you asked for. The next call does the paging > behind the scenes so you don't have to worry about it.
Show quoted text
> I can help out with the testing if you need it as I still have copies of > the Net::Duo scripts.
Thank you! The code on the master branch at: https://github.com/rra/net-duo should hopefully do the right thing. I updated the test suite and switched to paginated calls for the integrations() and users() methods. Testing would be greatly appreciated, since I've only tested it against my understanding of the API documentation and not against the actual API (I no longer have access to the admin API for a Duo account). I have a bit more work to do before making a release (mostly to make it clear that the package is orphaned and is looking for a new maintainer), but will try to get a new release ASAP out once I have confirmation that this works. -- #!/usr/bin/perl -- Russ Allbery, Just Another Perl Hacker $^=q;@!>~|{>krw>yn{u<$$<[~||<Juukn{=,<S~|}<Jwx}qn{<Yn{u<Qjltn{ > 0gFzD gD, 00Fz, 0,,( 0hF 0g)F/=, 0> "L$/GEIFewe{,$/ 0C$~> "@=,m,|,(e 0.), 01,pnn,y{ rw} >;,$0=q,$,,($_=$^)=~y,$/ C-~><@=\n\r,-~$:-u/ #y,d,s,(\$.),$1,gee,print
  Sat Mar 02 20:28:09 2019 iglesias [...] uci.edu - Correspondence added
Subject: Re: [rt.cpan.org #127964] Net::Duo and new pagination requirements for Duo API
Date: Sat, 2 Mar 2019 17:27:36 -0800
To: bug-Net-Duo [...] rt.cpan.org, das1 [...] lehigh.edu
From: Mike Iglesias <iglesias [...] uci.edu>
I'm out of town this weekend but I can test Monday evening or Tuesday. Mike On 3/2/19 10:52 AM, Russ Allbery via RT wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=127964 > > > "Mike Iglesias via RT" <bug-Net-Duo@rt.cpan.org> writes: >
>> I switched our scripts over to using the Duo API calls, and their method >> of handling pagination is similar to what Charles did - they have a >> json_paging_api_call that sets up the paging and a "next" call that >> gives you the next object you asked for. The next call does the paging >> behind the scenes so you don't have to worry about it.
>
>> I can help out with the testing if you need it as I still have copies of >> the Net::Duo scripts.
> > Thank you! The code on the master branch at: > > https://github.com/rra/net-duo > > should hopefully do the right thing. I updated the test suite and > switched to paginated calls for the integrations() and users() methods. > Testing would be greatly appreciated, since I've only tested it against my > understanding of the API documentation and not against the actual API (I > no longer have access to the admin API for a Duo account). > > I have a bit more work to do before making a release (mostly to make it > clear that the package is orphaned and is looking for a new maintainer), > but will try to get a new release ASAP out once I have confirmation that > this works. >
  Tue Mar 05 10:50:50 2019 iglesias [...] uci.edu - Correspondence added
Subject: Re: [rt.cpan.org #127964] Net::Duo and new pagination requirements for Duo API
Date: Tue, 5 Mar 2019 07:50:21 -0800
To: bug-Net-Duo [...] rt.cpan.org, das1 [...] lehigh.edu
From: Mike Iglesias <iglesias [...] uci.edu>
I ran some tests using your new code with some scripts that produce listings of users and integrations, and they produce the same output as the scripts using Duo's perl API package. At least from my perspective your new code is working. Mike On 3/2/19 10:52 AM, Russ Allbery via RT wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=127964 > > > "Mike Iglesias via RT" <bug-Net-Duo@rt.cpan.org> writes: >
>> I switched our scripts over to using the Duo API calls, and their method >> of handling pagination is similar to what Charles did - they have a >> json_paging_api_call that sets up the paging and a "next" call that >> gives you the next object you asked for. The next call does the paging >> behind the scenes so you don't have to worry about it.
>
>> I can help out with the testing if you need it as I still have copies of >> the Net::Duo scripts.
> > Thank you! The code on the master branch at: > > https://github.com/rra/net-duo > > should hopefully do the right thing. I updated the test suite and > switched to paginated calls for the integrations() and users() methods. > Testing would be greatly appreciated, since I've only tested it against my > understanding of the API documentation and not against the actual API (I > no longer have access to the admin API for a Duo account). > > I have a bit more work to do before making a release (mostly to make it > clear that the package is orphaned and is looking for a new maintainer), > but will try to get a new release ASAP out once I have confirmation that > this works. >
  Tue Mar 05 17:43:37 2019 RRA [...] cpan.org - Correspondence added
Subject: Re: [rt.cpan.org #127964] Net::Duo and new pagination requirements for Duo API
Date: Tue, 05 Mar 2019 14:43:27 -0800
To: "Mike Iglesias via RT" <bug-Net-Duo [...] rt.cpan.org>
From: Russ Allbery <rra [...] cpan.org>
"Mike Iglesias via RT" <bug-Net-Duo@rt.cpan.org> writes: Show quoted text
> I ran some tests using your new code with some scripts that produce > listings of users and integrations, and they produce the same output as > the scripts using Duo's perl API package. At least from my perspective > your new code is working.
Awesome, thank you so much! I got a bit sidetracked by a bad cold, but the new release should be out today or tomorrow. -- #!/usr/bin/perl -- Russ Allbery, Just Another Perl Hacker $^=q;@!>~|{>krw>yn{u<$$<[~||<Juukn{=,<S~|}<Jwx}qn{<Yn{u<Qjltn{ > 0gFzD gD, 00Fz, 0,,( 0hF 0g)F/=, 0> "L$/GEIFewe{,$/ 0C$~> "@=,m,|,(e 0.), 01,pnn,y{ rw} >;,$0=q,$,,($_=$^)=~y,$/ C-~><@=\n\r,-~$:-u/ #y,d,s,(\$.),$1,gee,print
  Thu Mar 07 01:09:05 2019 RRA [...] cpan.org - Correspondence added
Subject: Re: [rt.cpan.org #127964] Net::Duo and new pagination requirements for Duo API
Date: Wed, 06 Mar 2019 22:08:56 -0800
To: "Russ Allbery via RT" <bug-Net-Duo [...] rt.cpan.org>
From: Russ Allbery <rra [...] cpan.org>
"Russ Allbery via RT" <bug-Net-Duo@rt.cpan.org> writes: Show quoted text
> Awesome, thank you so much! I got a bit sidetracked by a bad cold, but > the new release should be out today or tomorrow.
Net::Duo 1.02 has now been released with a fix for this problem. -- #!/usr/bin/perl -- Russ Allbery, Just Another Perl Hacker $^=q;@!>~|{>krw>yn{u<$$<[~||<Juukn{=,<S~|}<Jwx}qn{<Yn{u<Qjltn{ > 0gFzD gD, 00Fz, 0,,( 0hF 0g)F/=, 0> "L$/GEIFewe{,$/ 0C$~> "@=,m,|,(e 0.), 01,pnn,y{ rw} >;,$0=q,$,,($_=$^)=~y,$/ C-~><@=\n\r,-~$:-u/ #y,d,s,(\$.),$1,gee,print
  Sat Jan 04 23:18:46 2020 RRA [...] cpan.org - Fixed in 1.02 added
  Sat Jan 04 23:18:46 2020 RRA [...] cpan.org - Correspondence added
This was fixed in 1.02 but was never closed. Doing that belatedly.
  Sat Jan 04 23:18:47 2020 RRA [...] cpan.org - Status changed from 'open' to 'resolved'
  Sat Jan 04 23:18:47 2020 RRA [...] cpan.org - Taken