Bug #105008 for TAP: TAP changes

Fri Jun 05 18:53:22 2015 aschwarz1309 [...] att.net - Ticket created

Subject:	TAP changes
Date:	Fri, 5 Jun 2015 15:52:59 -0700
To:	<bug-TAP [...] rt.cpan.org>
From:	"Arthur Schwarz" <aschwarz1309 [...] att.net>

These comments are made with respect to the TAP version 13 Specification Please don't take the comments amiss. I intend to state them succinctly without apology or caveat. This will, unfortunately, make them appear as a criticism and that my view should prevail. Neither viewpoints are correct. All of the comments can be viewed as "I'd like to see ...", or "gosh darn, it would be nice if ...". A clear exposition of the syntax for TAP statements should precede their use and explanation. This will reduce ambiguity in interpretation. That having been said, I think an interpretation of the current syntax is: ^1..N[ # comment] ^1..0[ # comment] ^ok[ [N [comment ][# directiver [comment]] ^not ok[ [N [comment ][# directiver [comment]] ^Bail out! ^#[ Diagnostic] ^ +--- ^ *... ^ YAML comment where ^ means the beginning of the line ' ' is exactly on blank [ ] are optional entries + one or more N a test line number test line comments can not have an embedded # I would be nice if this syntax was broadened ^<ws>*1..N[<ws>+comment] ^<ws>*1..0[<ws>+comment] ^<ws>*ok[<ws>+N[[<ws>+comment] | [<ws>+#[<ws>*comment]]] ^<ws>*not ok[<ws>+N[<ws>+comment]] ^<ws>*Bail out![<ws>*comment] ^<ws>*#<ws>*Diagnostic ^<ws>*--- ^<ws>*... ^<ws>*YAML comment where <ws> is ' ' or \t * is 0 or more + one or more | is alternation (this or that) (I can provide syntax equations if this suits your purposes). There are several reasons this alternate may be desirable: 1: Checking for TAP Protocol compliance is burdensome. Every violation of a spacing requirement makes the statement incorrect. This is needless if the standard is changed so that if it says that if one <ws> is valid then many can be used. 2: Requiring that TAP statement begin in column one is Burdensome to check and has no useful functionality. 3: Allowing both a test line and a Directive comment seems Counter-intuitive. It seems more sensible to either have a comment followed by a Directive, or a Directive followed by a comment, but not both. 3: The requirement that one or more <ws> characters follow a Directive prefix is unnecessary (but attractive). In the 'new' syntax the following tests are acceptable. 1..N ok not ok 123 # Diagnostic # Diagnostic ok comment ok 123 # SKIP comment --- jabbering YAML YAML ... ok N comment # SKIP is not acceptable The use and placement of 1..N needs to be better addressed. 1: If 1..N occurs before the test lines then ok 123 can be verified. 2: If 1..N follows all test line then ok 123 can not be check using a single pass analyzer. Output results have to be deferred. 3: If 1..N is missing then no checks are possible. This can't be determined until all test lines are input since 1..N may be at the end of the TAP Protocol file. Building an analyzer becomes more difficult 1..N can be missing or at the end of the file. If deference is to be given to the TAP output process then maybe 1..N should be removed as a requirement. Some words should be stated concering the placement of Diagnostics. For example: Can zero or more Diagnostics precede the first test line Can zero or more Diagnostics follow the last test line Is there a relationship between a Diagnosic and a preceding or following test line which has to be observed. The same comments apply to YAML code. There are no examples of an inline YAML block. In particular the syntax specifies that the only format for a YAML block is: ssss--- and .../ Do you mean: /^[[:blank:]]+---/ and /^[[:blank:]]+.../ The meaning of TODO has not been stated. My interpretation is that it is a SKIP. The use of blank lines is not stated. I think that the current specification is very clear (except TODO) and simple. I am building a little TAP converter and, for the most part, have little difficulty. But, protocol checking would be easier if some of the 'start on column' one conditions where lifted, and a better explanation of 1..N was included. thanks art The failure of the past is the challenge of the present and the success of the future.

Fri Jun 05 19:17:45 2015 ether [...] cpan.org - Correspondence added

If you're asking for clarifications to the documentation, patches are welcome. However, with regards to changing the specification itself: Show quoted text

> 1: Checking for TAP Protocol compliance is burdensome.

Yes, well, that's a specification for you. Show quoted text

> 2: Requiring that TAP statement begin in column one is > Burdensome to check and has no useful functionality.

Not so. Consider subtests. Show quoted text

> 3: Allowing both a test line and a Directive comment seems > Counter-intuitive. It seems more sensible to either have > a comment followed by a Directive, or a Directive followed > by a comment, but not both.

How so?

Fri Jun 05 19:17:45 2015 The RT System itself - Status changed from 'new' to 'open'

Sun Jun 07 18:52:05 2015 aschwarz1309 [...] att.net - Correspondence added

CC:	"'Arthur Schwarz'" <aschwarz1309 [...] att.net>
Subject:	RE: [rt.cpan.org #105008] TAP changes
Date:	Sun, 7 Jun 2015 15:51:25 -0700
To:	<bug-TAP [...] rt.cpan.org>
From:	"Arthur Schwarz" <aschwarz1309 [...] att.net>

Take anything I say seriously and we are in serious difficulty. Some of the responses are whimsical (but accurate maybe) and some are pointed (maybe not so accurate). Read and enjoy. If you're asking for clarifications to the documentation, patches are welcome. Show quoted text

> > However, with regards to changing the specification itself: >

> > 1: Checking for TAP Protocol compliance is burdensome.

> > Yes, well, that's a specification for you.

Taking the gloves off are we? Specifications have a difficult job in saying 'exactly what is meant' and paying attention to the implementers needs. This specification does not say 'exactly what is meant' and does not cater to its audience. It leaves unsaid the exact meaning of most of its constructs and then provides an example which is misleading as being the specification. To be specific, there are no syntax diagrams or Backus Naur Form (BNF) equations or any other formal description of this language. What there is is contradictory seeming to be a meld of vim and regular expressions. There are four different style comments represented (only some are described): 1: Comments preceded with a # on a line, called Diagnostics 2: Comments following not ok/ok which must not have a leading number (Descriptor) 3: Comments following a Directives which can have a leading number 4: Comments following a test plan which must have a '#' prefix There is no indication whether a separator is required nor what that separator should be. Without that specification the following is legal okis a fine day#TODO anything Other than a number prefix being reprehensible, there is no indication of what a Descriptor comment can have. The following is legal. okis a fi##ne day##TODO anything The use of \s to mean ' ' is not standard and (I believe) is vim specific. The exclusion of a tab, \t, is inexplicable and doesn't allow [:blank:]' or [ \t] to be used. The use of SKIP\S is clearly wrong. The escape character, '\', is not explained and options are not represented in this way. What this Directive says is that SKIP is written as: ok #SKIP\S The specification clearly specifies that the test plan format is 1..N but if there are k test cases you have not clearly specified how to treat k < N and k > N. Your statements of an abrupt test termination is not sufficient to describe the exact meaning of k < N. The specification uses a regular expression to define ok, /^ok/, and not ok, /^not ok/ without specifying either that it is a regular expression or where to find an explanation of regular expressions, or to have defined the regular expressions used. And this definition is in conflict with the use of \s+ in describing Directives and YAML initiators and terminators. The specification specifies that ok and not ok are case sensitive, and TODO and SKIP are not. There is no specification of form or format of the Descriptor following the keyword. The following are legal: Bail Out!# you scurvy rat Bail Out!1 fine day to go fishing "#, which the harness must ignore" a semantic meaning expressed here but missing through most of the document {test line, test plan errors, Directives, and so on}. Errors are not explained. If an error is detected does it invalidate all tests? What is an 'error'? Does a test line with a SKIP or TODO Directive count toward the N in the test plan? The YAML block initiators can not start in column one. By the by, the regular expression written to describe the YAML block closing operator should be /\s+.../ in the formalism chosen, not \s+.../. The following appears legal. --- --- this begins in column 1 ... this begins in column 1 ... Since there is no specification of YAML contents except for the wish (not specification) that "The YAML document is indented ...". The statement "make it easier for the parser to recover if the trailing ..." is demonstrably incorrect for a lot of reasons. The statement concerning the PERL test harness is clearly incorrect and unexplained. There is a big difference between a specification of content and the processing of content and the document nowhere addresses this. TODO: "Should a todo test point begin succeeding..." What mechanism is provided in the specification to identify that TODO has succeeded or failed? Directives are only associated with a 'not ok'. TODO: "...the harness should report it as a bonus" does this mean that TODO is not counted as a test line with respect to either the test plan nor counting each test line as seen? How does the test harness report a bonus? SKIP: "... emitted if the test file is skipped completely". Test files retain artifacts of test execution. If the test file is skipped then no artifacts are reported and this document is moot. Do you mean to say that no test is reported? Suppose the test plan is 1..0 and there are k test lines, each one with a SKIP directive. Is this legal? Suppose the test plan is 1..k and there are k test lines each one with a SKIP directive. Is this different from a test plan of 1..0? Bail Out! There is no regular expression defining its 'formal' format and the separator between 'Bail' and 'Out!'. Bail Out! "As an emergency measure a test script ..." do you mean Harness? If not, what is a test script and what are its requirements. Is there a scripting language to be used? Can a PERL program be a test script (as in #!/bin/perl)? Bail Out! "... to standard output" What does this mean? Is this document describing the TAP language, the TAP Producer, the TAP Consumer, and how does this relate to Bail Out!? Are you saying that when a Bail Out is seen by the TAP Consumer the following ... is done? If so, can't Bail Out! be in a generated test file? If it is in a generated test file, does Bail Out" have to be in column 1 of a line? Bail Out! "... interpreter ..." What interpreter? Are you talking about the semantics of Bail Out! with respect to a particular TAP Consumer? Bail Out! There is no requirement that Bail Out! be put in column 1 (mentioned above) but there is an indication that Bail Out! be output to stdout. Does this mean that all statements can be output to stdout or only Bail Out!? If all statements can be output to stdout then are they required to be put into column 1, and what does column 1 mean in this context? Diagnostics: " ... which the harness must ignore ...". Suppose my harness chooses not to ignore the comment, what happens? Since the TAP Specification specifies the contents of a test file (although note the comments in Bail Out!) does this preclude 'structured Diagnostic comments" suitable for analysis? Diagnostics: "TAP13's YAML blocks are intended to replace diagnostics ...". Is there any particular subset of YAML which is interesting? Suppose I just wanted to make a comment: # this is a wonderful day How do I say this in YAML? Diagnostics: Does the TAP Specification require that a full YAML parser be provided? How would I do this in YAML? What part of a YAML block should the harness be responsible for processing? Suppose test files are (just) handed around, and I get a test file and am interested in comments. How do I treat YAML? Anything else: "How a harness handles the incorrect line is undefined." Shouldn't the specification tell about correct and incorrect handling of all statements? If an incorrect statement is seen, does this invalidate all test results in the file (one error leads to total failure) because test intent can not be established? Anything else: a little example ok starts in a column > 1 This statement is not TAP compliant. It belongs in the Anything else category. What should be done with it? Show quoted text

>

> > 2: Requiring that TAP statement begin in column one is > > Burdensome to check and has no useful functionality.

> > Not so. Consider subtests.

What is a subtest? Where is it defined? My current understanding is that test results are reported in test lines w/wo Directives, and that all tests have equal merit. What is a subtest and how is it distinguished? Show quoted text

>

> > 3: Allowing both a test line and a Directive comment seems > > Counter-intuitive. It seems more sensible to either have > > a comment followed by a Directive, or a Directive followed > > by a comment, but not both.

> > How so?

Because there are two comments describing the same event. Consider two cases: 1: A computer generated file, and 2: A human generated file. A computer has to be 'told' what to do. That being said, when a test is skipped all information concerning the test can be described after the Descriptive. That is, "This is a test. It has been skipped because". A human has to insert a skip and comment. If makes some sense that his job is made easier if the SKIP and comment are just appended. Now, does the specification mean to address manually generated/modified test files? Show quoted text

> >

Tue Jun 09 12:18:48 2015 aschwarz1309 [...] att.net - Correspondence added

Subject:	RE: [rt.cpan.org #105008] TAP changes
Date:	Tue, 9 Jun 2015 09:18:07 -0700
To:	"'Arthur Schwarz'" <aschwarz1309 [...] att.net>, <bug-TAP [...] rt.cpan.org>
From:	"Arthur Schwarz" <aschwarz1309 [...] att.net>

Do you want me to rewrite the TAP Specification? art Show quoted text

-----Original Message----- Take anything I say seriously and we are in serious difficulty. Some of the responses are whimsical (but accurate maybe) and some are pointed (maybe not so accurate). Read and enjoy. If you're asking for clarifications to the documentation, patches are welcome.

> > However, with regards to changing the specification itself: >

> > 1: Checking for TAP Protocol compliance is burdensome.

> > Yes, well, that's a specification for you.

Taking the gloves off are we? Specifications have a difficult job in saying 'exactly what is meant' and paying attention to the implementers needs. This specification does not say 'exactly what is meant' and does not cater to its audience. It leaves unsaid the exact meaning of most of its constructs and then provides an example which is misleading as being the specification. To be specific, there are no syntax diagrams or Backus Naur Form (BNF) equations or any other formal description of this language. What there is is contradictory seeming to be a meld of vim and regular expressions. There are four different style comments represented (only some are described): 1: Comments preceded with a # on a line, called Diagnostics 2: Comments following not ok/ok which must not have a leading number (Descriptor) 3: Comments following a Directives which can have a leading number 4: Comments following a test plan which must have a '#' prefix There is no indication whether a separator is required nor what that separator should be. Without that specification the following is legal okis a fine day#TODO anything Other than a number prefix being reprehensible, there is no indication of what a Descriptor comment can have. The following is legal. okis a fi##ne day##TODO anything The use of \s to mean ' ' is not standard and (I believe) is vim specific. The exclusion of a tab, \t, is inexplicable and doesn't allow [:blank:]' or [ \t] to be used. The use of SKIP\S is clearly wrong. The escape character, '\', is not explained and options are not represented in this way. What this Directive says is that SKIP is written as: ok #SKIP\S The specification clearly specifies that the test plan format is 1..N but if there are k test cases you have not clearly specified how to treat k < N and k > N. Your statements of an abrupt test termination is not sufficient to describe the exact meaning of k < N. The specification uses a regular expression to define ok, /^ok/, and not ok, /^not ok/ without specifying either that it is a regular expression or where to find an explanation of regular expressions, or to have defined the regular expressions used. And this definition is in conflict with the use of \s+ in describing Directives and YAML initiators and terminators. The specification specifies that ok and not ok are case sensitive, and TODO and SKIP are not. There is no specification of form or format of the Descriptor following the keyword. The following are legal: Bail Out!# you scurvy rat Bail Out!1 fine day to go fishing "#, which the harness must ignore" a semantic meaning expressed here but missing through most of the document {test line, test plan errors, Directives, and so on}. Errors are not explained. If an error is detected does it invalidate all tests? What is an 'error'? Does a test line with a SKIP or TODO Directive count toward the N in the test plan? The YAML block initiators can not start in column one. By the by, the regular expression written to describe the YAML block closing operator should be /\s+.../ in the formalism chosen, not \s+.../. The following appears legal. --- --- this begins in column 1 ... this begins in column 1 ... Since there is no specification of YAML contents except for the wish (not specification) that "The YAML document is indented ...". The statement "make it easier for the parser to recover if the trailing ..." is demonstrably incorrect for a lot of reasons. The statement concerning the PERL test harness is clearly incorrect and unexplained. There is a big difference between a specification of content and the processing of content and the document nowhere addresses this. TODO: "Should a todo test point begin succeeding..." What mechanism is provided in the specification to identify that TODO has succeeded or failed? Directives are only associated with a 'not ok'. TODO: "...the harness should report it as a bonus" does this mean that TODO is not counted as a test line with respect to either the test plan nor counting each test line as seen? How does the test harness report a bonus? SKIP: "... emitted if the test file is skipped completely". Test files retain artifacts of test execution. If the test file is skipped then no artifacts are reported and this document is moot. Do you mean to say that no test is reported? Suppose the test plan is 1..0 and there are k test lines, each one with a SKIP directive. Is this legal? Suppose the test plan is 1..k and there are k test lines each one with a SKIP directive. Is this different from a test plan of 1..0? Bail Out! There is no regular expression defining its 'formal' format and the separator between 'Bail' and 'Out!'. Bail Out! "As an emergency measure a test script ..." do you mean Harness? If not, what is a test script and what are its requirements. Is there a scripting language to be used? Can a PERL program be a test script (as in #!/bin/perl)? Bail Out! "... to standard output" What does this mean? Is this document describing the TAP language, the TAP Producer, the TAP Consumer, and how does this relate to Bail Out!? Are you saying that when a Bail Out is seen by the TAP Consumer the following ... is done? If so, can't Bail Out! be in a generated test file? If it is in a generated test file, does Bail Out" have to be in column 1 of a line? Bail Out! "... interpreter ..." What interpreter? Are you talking about the semantics of Bail Out! with respect to a particular TAP Consumer? Bail Out! There is no requirement that Bail Out! be put in column 1 (mentioned above) but there is an indication that Bail Out! be output to stdout. Does this mean that all statements can be output to stdout or only Bail Out!? If all statements can be output to stdout then are they required to be put into column 1, and what does column 1 mean in this context? Diagnostics: " ... which the harness must ignore ...". Suppose my harness chooses not to ignore the comment, what happens? Since the TAP Specification specifies the contents of a test file (although note the comments in Bail Out!) does this preclude 'structured Diagnostic comments" suitable for analysis? Diagnostics: "TAP13's YAML blocks are intended to replace diagnostics ...". Is there any particular subset of YAML which is interesting? Suppose I just wanted to make a comment: # this is a wonderful day How do I say this in YAML? Diagnostics: Does the TAP Specification require that a full YAML parser be provided? How would I do this in YAML? What part of a YAML block should the harness be responsible for processing? Suppose test files are (just) handed around, and I get a test file and am interested in comments. How do I treat YAML? Anything else: "How a harness handles the incorrect line is undefined." Shouldn't the specification tell about correct and incorrect handling of all statements? If an incorrect statement is seen, does this invalidate all test results in the file (one error leads to total failure) because test intent can not be established? Anything else: a little example ok starts in a column > 1 This statement is not TAP compliant. It belongs in the Anything else category. What should be done with it?

>

> > 2: Requiring that TAP statement begin in column one is > > Burdensome to check and has no useful functionality.

> > Not so. Consider subtests.

What is a subtest? Where is it defined? My current understanding is that test results are reported in test lines w/wo Directives, and that all tests have equal merit. What is a subtest and how is it distinguished?

>

> > 3: Allowing both a test line and a Directive comment seems > > Counter-intuitive. It seems more sensible to either have > > a comment followed by a Directive, or a Directive followed > > by a comment, but not both.

> > How so?

Because there are two comments describing the same event. Consider two cases: 1: A computer generated file, and 2: A human generated file. A computer has to be 'told' what to do. That being said, when a test is skipped all information concerning the test can be described after the Descriptive. That is, "This is a test. It has been skipped because". A human has to insert a skip and comment. If makes some sense that his job is made easier if the SKIP and comment are just appended. Now, does the specification mean to address manually generated/modified test files?

> >