TCK Coverage Document for Jakarta Authentication:
(version 3.0)


TCK Components:
---------------
- User's Guide
- Compatibility Rules
- Configuration Instructions
- Test Suite
- API Tests 
- Signature Test
- End-to-End Tests
- Ant Task Tests
- Framework Code
- JavaTest TM Harness


Terminology of Metrics
----------------------
- Assertion: A specific statement of functionality or behavior derived from a specification.
  A testable assertion is one that can be validated in an implementation by testing.
- Test: A binary application (or script) comprised of one or more Test Cases.
- Test Case: A single set of test inputs, execution conditions, and expected results
  developed to verify an implementation's conformance with a specific assertion.
- Specification Assertion Coverage: Ratio of all assertions tested by at least one test
  case to the total number of testable assertions defined by the specification.
- API Coverage: Ratio of methods directly exercised by test cases to the total number
  of methods defined by the specification.


Coverage
--------
- Total testable assertions derived from Jakarta Authentication Specifications and Javadoc:
       157 specification assertions 
     + 107 API assertions 
     ------------------------------
       264 testable assertions


- Total testable assertions tested
       90 specification assertions tested
     + 47 API assertions tested (only include servlet profile not soap)
     --------------------------------------
      137 testable assertions tested


  - 57.32% of testable specification assertions tested
  - 43.92% of testable API assertions tested
  - 51.89% of all (specification and API) assertions tested

- Assertions were identified through the use of CTS Tools and hand markup
  (hand markup was required for the Jakarta Authentication specifications)

- API Signature Coverage: 100% for all defined public and protected members and validated
  by the signature test included with the Jakarta Authentication TCK.

- See the following HTML reports in this bundle for Jakarta Authentication specification and API coverage metrics:
    * coverage/jaspic/api/summary.html
    * coverage/jaspic/spec/summary.html


Quality Assurance
-----------------
- TCK was run using representative configurations of the Reference Implementation on
  the following platforms:
    * CentOS Linux 7 / Java SE 8
    * Alpine Linux v3.12 / Java SE 8
- Code quality was demonstrated through use of code reviews and inspections
- User's Guide was constructed from the standard CTS User's Guide template
- Documentation instructions were verified and tested


Justification of Adequacy
-------------------------
The Jakarta Authentication CTS provides a set of tests to ensure all implementations of
the Jakarta Authentication specifications are compatible.  As with all TCKs it is 
impossible to provide tests for 100% assertion coverage.  Note, while
the CTS may not have tests for all assertions in Jakarta Authentication, all
implementations of Jakarta Authentication must be compatible with the specifications.

Untested assertions are due to the effects of availability and cost of 
test development resources.

