OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

ebxml-architecture message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [Elist Home]

Subject: Conformance discussion

Hello All

At last weeks meeting (in Brussels) there was a preliminary discussion of Conformance.  This email is a continuation of that discussion, reviewing some of what was said and identifying some of the issues. 

Conformance is usually defined as testing to see if an implementation faithfully meets the requirements of a standard or specification.   The requirements or criteria for conformance must be specified in the standard or specification. This is usually in a conformance clause or conformance statement, but sometimes some of the criteria can be found in the body of the specification.  If the criteria or requirements for conformance are not specified there can be no conformance testing.  It also follows, that claims of compliant products would be meaningless.

An implementation's conformance to a specification is necessary, but not sufficient, to guarantee interoperability with other implementations.

Conformance tests should be used both by implementors early on in development to improve the quality of their implementations, and by industry associations wishing to administer a testing and certification service.

The conformance clause of a standard specification is a high-level description of what is required of implementors and application developers.  It, in turn, refers to other parts of the standard.  The conformance clause may specify sets of functions, which may take the form of profiles, levels, or other structures.  The conformance clause may specify minimal requirements for certain functions and minimal requirements for implementation-dependent values. Additionally it may specify the permissibility of extensions, options, and alternative approaches and how they are to be handled.

The Conformance clause deals with such issues as:
Profiles and/or Levels - Applications often do not require all the features within a standard.  It is also possible that implementations may not be able to implement all the features.  In these cases, it may be desirable to partition the specifications into subsets of functionality.  A profile is a subset of the overall specifications that includes all of the functionality necessary to satisfy the requirements of a particular community of users. Levels are nested subsets of the specifications

Extensions - There are two main approaches to handling implementation specific extensions to a standard.  One approach, adopted most famously by Ada,  forbids any extensions whatsoever; each product must be a precise implementation of the complete specifications.  This is called strict conformance.The other approach adopts the weaker overall requirement that, while extensions are allowed, an implementation must perform all the functionality in the specifications exactly as specified.  This more common approach usually includes some additional, more specific, requirements in the conformance clause, along the lines of:
       Extensions shall not re-define semantics for existing functions;
       Extensions shall not cause standard-conforming functions (i.e., functions that do not use the extensions) to execute incorrectly;
       The mechanism for determining application conformance and the extensions shall be clearly described in the documentation, and the extensions shall be marked as such;
       Extensions shall follow the principles and guidelines of the standard they extend, that is, the specifications must be extended in a standard manner.

One approach that has been used successfully is to have a register for extensions.  This is a document, parallel to but distinct from the official specifications, that contains a list of recognized extensions to the standard.  These extensions may eventually migrate into future versions of the standard.

Options within profiles/levels -  Even within a profile/level structure, a standard may classify features as "mandatory" vs. "optional," leading to a table of perhaps hundreds or thousands of features so classified

Alternate approaches -   Specifications may describe several different ways to accomplish its operation (e.g., a choice of file formats or codes).  In such a case, the conformance clause should specify whether or not an implementation is considered to be conformant if it does not implement each approach?  (If  the specifications don't describe the different approaches in the first place, then it's an implementor detail irrelevant to conformance.)

This refers to two concepts:
Statements indicating that an implementation "shall," "should," or "may" implement a certain feature.  The meaning of the words "shall," "should," and "may," in the context of the given standard, would be defined in the conformance clause.

Some recent standards include, as official parts of the standard, lists of assertions.  These assertions are statements of functionality or behavior derived from the standard, and which are true for conforming implementations.  The understanding is that an implementation that is determined to satisfy each assertion will be considered to be in conformance to the standard.  Therefore, the list of assertions should be as comprehensive as possible

TEST METRICS (test suites or tools)
The test metrics are the means for measuring or assessing whether an implementation is conformant.  Generally, these test suites are not part of the specification or developed by the specification-developers (ebXML committee).  However, organizations like NIST and OASIS do develop these test suites (or tools) - so that developers can identify those areas in which they are implementing the specification correctly as well as those areas in which they haven't implemented the specifications. Additionally users can use these test to assess for themselves if an implementation is compliant - or how 'good' the implelmentation is with respect to implementing the specification.  OASIS/NIST currently have test suites for XML processors, DOM, and are working on XSLT and Schema. NIST, in addition to developing a prototype of the Reg/Rep specification, is also thinking about developing conformance tests for Reg/Rep.
Generically, a conformance test suite is a collection of combinations of legal and illegal inputs to the implementation being tested, together with a corresponding collection of  "expected results."  If such a list is provided, the starting point for the development of the test suite is the list of assertions for the standard.  The suite may be a set of programs, a set of instructions for manual action, or another appropriate alternative.  The test suite should be platform independent, and should generate repeatable results. 

A Testing Program (certification or branding) is probably outside the scope of this group, but may be something that ebXML committee is interested in pursuing.  A testing program cannot exist without a test tool or test suite, but a test tool/suite can exist without a testing program.  Not all specifications or standards need a testing program.  Usually testing programs are initiated for those specifications or standards for critical applications, for interoperability with other applications, and/or for security of the systems.  The decision to establish a program is based on the risk of nonconformance versus the costs of creating and running a program.    A conformance testing program usually includes: Standard or specification;Test tool (e.g., tool, suite, and/or reference implementation; Procedures for testing; Organization(s) to do testing, issue certificates of validation, and arbitrate disputes

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [Elist Home]

Search: Match: Sort by:
Words: | Help

Powered by eList eXpress LLC