Conformance discussion

[Lynne Rosenthal <lynne.rosenthal@nist.gov>]

Hello All

At last weeks meeting (in Brussels) there was a preliminary discussion of
Conformance.  This email is a continuation of that discussion,
reviewing some of what was said and identifying some of the issues. 

CONFORMANCE IN GENERAL
Conformance is usually defined as testing to see if an implementation
faithfully meets the requirements of a standard or
specification.   The requirements or criteria for conformance
must be specified in the standard or specification. This is usually in a
conformance clause or conformance statement, but sometimes some of the
criteria can be found in the body of the specification.  If the
criteria or requirements for conformance are not specified there can be
no conformance testing.  It also follows, that claims of compliant
products would be meaningless.

An implementation's conformance to a specification is necessary, but not
sufficient, to guarantee interoperability with other
implementations.

Conformance tests should be used both by implementors early on in
development to improve the quality of their implementations, and by
industry associations wishing to administer a testing and certification
service. 

THE CONFORMANCE CLAUSE
The conformance clause of a standard specification is a high-level
description of what is required of implementors and application
developers.  It, in turn, refers to other parts of the
standard.  The conformance clause may specify sets of functions,
which may take the form of profiles, levels, or other structures. 
The conformance clause may specify minimal requirements for certain
functions and minimal requirements for implementation-dependent values.
Additionally it may specify the permissibility of extensions, options,
and alternative approaches and how they are to be handled. 

The Conformance clause deals with such issues as:
Profiles and/or Levels - Applications often do not require all the
features within a standard.  It is also possible that
implementations may not be able to implement all the features.  In
these cases, it may be desirable to partition the specifications into
subsets of functionality.  A profile is a subset of the overall
specifications that includes all of the functionality necessary to
satisfy the requirements of a particular community of users. Levels are
nested subsets of the specifications

Extensions - There are two main approaches to handling implementation
specific extensions to a standard.  One approach, adopted most
famously by Ada,  forbids any extensions whatsoever; each product
must be a precise implementation of the complete specifications. 
This is called strict conformance.The other approach adopts the weaker
overall requirement that, while extensions are allowed, an implementation
must perform all the functionality in the specifications exactly as
specified.  This more common approach usually includes some
additional, more specific, requirements in the conformance clause, along
the lines of:

–       Extensions shall not re-define semantics for existing functions;

–       Extensions shall not cause standard-conforming functions (i.e., functions that do not use the extensions) to execute incorrectly;

–       The mechanism for determining application conformance and the extensions shall be clearly described in the documentation, and the extensions shall be marked as such;

–       Extensions shall follow the principles and guidelines of the standard they extend, that is, the specifications must be extended in a standard manner.

One approach that has been used successfully is to have a register
for extensions.  This is a document, parallel to but distinct from
the official specifications, that contains a list of recognized
extensions to the standard.  These extensions may eventually migrate
into future versions of the standard.

Options within profiles/levels -  Even within a profile/level
structure, a standard may classify features as "mandatory" vs.
"optional," leading to a table of perhaps hundreds or thousands
of features so classified

Alternate approaches -   Specifications may describe several
different ways to accomplish its operation (e.g., a choice of file
formats or codes).  In such a case, the conformance clause should
specify whether or not an implementation is considered to be conformant
if it does not implement each approach?  (If  the
specifications don't describe the different approaches in the first
place, then it's an implementor detail irrelevant to conformance.)

APPROPRIATE WORDING OF THE TEXT OF THE SPECIFICATION ITSELF
This refers to two concepts:
Statements indicating that an implementation "shall,"
"should," or "may" implement a certain feature. 
The meaning of the words "shall," "should," and
"may," in the context of the given standard, would be defined
in the conformance clause.

Some recent standards include, as official parts of the standard, lists
of assertions.  These assertions are statements of functionality or
behavior derived from the standard, and which are true for conforming
implementations.  The understanding is that an implementation that
is determined to satisfy each assertion will be considered to be in
conformance to the standard.  Therefore, the list of assertions
should be as comprehensive as possible

TEST METRICS (test suites or tools)
The test metrics are the means for measuring or assessing whether an
implementation is conformant.  Generally, these test suites are not
part of the specification or developed by the specification-developers
(ebXML committee).  However, organizations like NIST and OASIS do
develop these test suites (or tools) - so that developers can identify
those areas in which they are implementing the specification correctly as
well as those areas in which they haven't implemented the specifications.
Additionally users can use these test to assess for themselves if an
implementation is compliant - or how 'good' the implelmentation is with
respect to implementing the specification.  OASIS/NIST currently
have test suites for XML processors, DOM, and are working on XSLT and
Schema. NIST, in addition to developing a prototype of the Reg/Rep
specification, is also thinking about developing conformance tests for
Reg/Rep. 
  
Generically, a conformance test suite is a collection of combinations of
legal and illegal inputs to the implementation being tested, together
with a corresponding collection of  "expected
results."  If such a list is provided, the starting point for
the development of the test suite is the list of assertions for the
standard.  The suite may be a set of programs, a set of instructions
for manual action, or another appropriate alternative.  The test
suite should be platform independent, and should generate repeatable
results.  

CONFORMANCE TESTING PROGRAM
A Testing Program (certification or branding) is probably outside the
scope of this group, but may be something that ebXML committee is
interested in pursuing.  A testing program cannot exist without a
test tool or test suite, but a test tool/suite can exist without a
testing program.  Not all specifications or standards need a testing
program.  Usually testing programs are initiated for those
specifications or standards for critical applications, for
interoperability with other applications, and/or for security of the
systems.  The decision to establish a program is based on the risk
of nonconformance versus the costs of creating and running a
program.    A conformance testing program usually
includes: Standard or specification;Test tool (e.g., tool, suite, and/or
reference implementation; Procedures for testing; Organization(s) to do
testing, issue certificates of validation, and arbitrate disputes