Bug #124091 for Text-TFIDF: IDF computation improvement

Mon Jan 15 14:57:26 2018 gene [...] cpan.org - Ticket created

Subject:

IDF computation improvement

According to https://en.wikipedia.org/wiki/Tf%E2%80%93idf#Inverse_document_frequency_2 the IDF can be computed without the addition of 1 to the count. So, https://metacpan.org/source/LMETCALF/Text-TFIDF-0.03/lib/Text/TFIDF.pm#L70 could be changed to this: return - log( $count / scalar( keys %{ $t->{file} } ) ) / log(10); This makes the wikipedia examples work out nicely. :-) -Gene

Mon Jan 15 14:58:43 2018 gene [...] cpan.org - Correspondence added

Whoops. That $t should of course be $self. On Mon Jan 15 11:57:26 2018, GENE wrote: Show quoted text

> return - log( $count / scalar( keys %{ $t->{file} } ) ) / log(10);

Tue Jan 16 20:23:08 2018 LMETCALF [...] cpan.org - Correspondence added

On Mon Jan 15 14:57:26 2018, GENE wrote: Show quoted text

> According to > https://en.wikipedia.org/wiki/Tf%E2%80%93idf#Inverse_document_frequency_2 > the IDF can be computed without the addition of 1 to the count. So, > https://metacpan.org/source/LMETCALF/Text-TFIDF- > 0.03/lib/Text/TFIDF.pm#L70 could be changed to this: > > return - log( $count / scalar( keys %{ $t->{file} } ) ) / log(10); > > This makes the wikipedia examples work out nicely. :-) > > -Gene

The new version of TF-IDF should be correct. The Changes file cites the URLs with the computational source

Tue Jan 16 20:23:08 2018 The RT System itself - Status changed from 'new' to 'open'

Tue Jan 16 20:23:09 2018 LMETCALF [...] cpan.org - Status changed from 'open' to 'resolved'