RE: Yet more issues to resolve

["Burdett, David" <david.burdett@commerceone.com>]

Marty

I think that with your suggestion below, we are getting to close into the
internal design of  a MSH. However the spec does say in section 10.1.1 on
Persistent Storage and System Failure ...

>>>
This specification recognizes that different degrees of resilience may be
realized depending on the technology that is used to persist the data.
However, as a minimum, persistent storage that has the resilience
characteristics of a hard disk (or equivalent) SHOULD be used. It is
strongly RECOMMENDED though that implementers of this specification use
technology that is resilient to the failure of any single hardware or
software component. 

Even after a system interruption or failure, a MSH MUST ensure that messages
in persistent storage are processed in the same way as if the system failure
or interruption had not occurred. How this is done is an implementation
decision.
>>>

Does this meet your requirement?

David

-----Original Message-----
From: Martin W Sachs [mailto:mwsachs@us.ibm.com]
Sent: Thursday, January 04, 2001 7:18 AM
To: Burdett, David
Cc: 'Prasad Yendluri'; 'Stefano POGLIANI';
ebxml-transport@lists.ebxml.org; ebxml-tp@lists.ebxml.org
Subject: RE: Yet more issues to resolve



Again, let's be careful about implications.  While the messaging service
may only have to persist the message ID, the business process may need the
whole message to be persisted at least for a time.  This is another
candidate for a non-normative comment.

One could duck the whole issue by assuming that the business process will
use ACID semantics to move each message from the messaging service domain
to its own domain (or vice versa) and deal only with the messaging
service's persistence requirements in the messaging service specification.
Persistence, of course, would have to be at least until the business
process captures the message. This approach would certain require a
non-normative comment since it has implications for the messaging service
implementation.

Regards,
Marty

****************************************************************************
*********

Martin W. Sachs
IBM T. J. Watson Research Center
P. O. B. 704
Yorktown Hts, NY 10598
914-784-7287;  IBM tie line 863-7287
Notes address:  Martin W Sachs/Watson/IBM
Internet address:  mwsachs @ us.ibm.com
****************************************************************************
*********



"Burdett, David" <david.burdett@commerceone.com> on 01/04/2001 12:56:13 AM

To:   "'Prasad Yendluri'" <pyendluri@webmethods.com>
cc:   "'Stefano POGLIANI'" <stefano.pogliani@sun.com>, Martin W
      Sachs/Watson/IBM@IBMUS, ebxml-transport@lists.ebxml.org,
      ebxml-tp@lists.ebxml.org
Subject:  RE: Yet more issues to resolve




Prasad

See  comments below.

Regards

David
-----Original Message-----
From: Prasad Yendluri  [mailto:pyendluri@webmethods.com]
Sent: Wednesday, January 03, 2001  7:21 PM
To: Burdett, David
Cc: 'Stefano POGLIANI'; Martin  W Sachs; ebxml-transport@lists.ebxml.org;
ebxml-tp@lists.ebxml.org
Subject: Re: Yet more issues to  resolve

David,

The description for the "persistDuration" parameter reads as follows:
PersistDuration is the minimum length of time in days that a  Message that
is sent reliably is kept in Persistent Storage by a MSH. The  value used
for PersistDuration is an implementation decision although it  MUST be
greater than the value of the TimeToLive parameter for any message  that is
sent.If a duplicate message (i.e. with the same MessageId ) is  received
before the PersistDuration has passed, then the MSH that receives  it MUST
process it as a duplicate message as described  in sections  10.2.1.3 and
Error! Reference source not found..

If a duplicate message is received after the PersistDuration has  passed,
then although it may be treated as a duplicate, the sender must  realize
that it will probably be treated by the MSH as if the message were a  new
message that had not been received before.
1. If the purpose of this is to simply detect duplicates why require the
entire message to be stored? Isn't saving the value of the messageId
enough?
<DB>
Yes,  the MessageId is enough. It was not my intention to REQUIRE that a
whole  message should be kept. Note that I did not say that "
PersistDuration is  the minimum length of time in days that a **complete**
Message that is sent  reliably **MUST be** kept in Persistent Storage by a
MSH. Also lines 1213-1216 in Reliable messaging,  that describe the
behavior of a MSH that receives a message, explicitly say  ...
>> 1)        If the message is not a duplicate then do the following:
     a)        Save the MessageId of the received message in  persistent
storage. As an implementation decision, the whole message  MAY be stored if
there are other reasons for doing  so.<<
Another factor is that, if you want to support a Message Status  Inquiry,
then you will also need to record the time a message was received. I  agree
though that the description of persist duration could be clarified to
indicate that only the MessageId needs to be saved.
</DB>
 If any, I would think it is  the sender MSH that needs to persist the
message, so that a retry would a  simple retransmission.
<DB>
I  agree the sender MSH does need to persist the message. Line 1200 in the
Reliable Messaging specification that is describing the behavior of a
sender  of the message says ...
>> 1)        Save the message in persistent storage (see section 10.1.1)<<
However how long the sender saves the message for is  not relevant to
Reliable Messaging. You need to know it for the receiver since  you need to
know if you resend a message whether or not duplicate filtering  will work
which it won't if the reciever has not persisted the message. The  same
does not apply to the sender since the persisting of the message only
applies to resending.
</DB>

2. Why the value limited to "days"?
<DB>A good point.  My original thinking was that if the message is being
placed in some type of  persistent storage such as a disk, then you
probably would not want to clear  it out too often. So setting it to whole
days makes it a simpler value than  the years, months, weeks, days, hours,
minutes, millisecs?? that you might  otherwise need. Another approach is to
specify a single integer and then put  the units in separately such as
<persistDuration  units="seconds">20</persistDuration> in the CPA. Would
that work  better?</DB>

3. The last paragraph seems to imply that sender and receiver could be in
an inconsistent state without clear knowledge of, if retries on a message
are  still permitted or not. That is sender thinks retries are still
permitted but,  receiver does not know it is a re-send, as the message was
deleted from the  persistent store. If we got the Reliable Messaging
specification right, this  should never be the case. At least one of the
parties should always  know.  The receiver must always know if the received
message is duplicate  or not, if the message was received once
successfully, irrespective of, if the  entire message is (still) persisted
or not (based on the MessageId alone). We  should never retire the
MessageId either from sender/receiver (or  intermediary) perspective. It is
a GUID isn't it?
<DB>I  agree that ideally the Message Ids should never be "retired".
However what I  think you are suggesting is that the MessageIds are
**never** deleted by the  receiver. I honestly don't think this is
practical, since if you want to  forever to be able to do a duplicate
check: a) the MessageIds received have to  be immediately available in
order to provide a rapid response to the duplicate  check, and b) this
would imply ever increasing persistent storage which would  prove costly to
maintain. This means that the recipient of a message will have  to, at some
point, delete old message ids. The point is that the sender needs  to know
whether or not duplicate filtering will work. By specifying a
persistDuration for received Message Ids, the Sender will knowthat the
duplicate filtering will work, especially if the persist duration is
significantly longer than any period of time over which a message might be
resent. However the sender can never know EXACTLY when the sender will
delete  a message id unless they do a query first.

Perhaps it would be better  if the last paragraph said ... "A MSH SHOULD
NOT resend a message to a  receiver if the receiver has probably deleted
the message since the time  indicated by PersistDuration has passed since
the sender first sent the  message." You could even require that a sender
report a delivery failure if  the time has passed.
</DB>

4. I think, how long a message should be persisted in the MSH should be
based on acknowledgment and retries and associated timeout constraints of
the  business process choreography and not based on a separate parameter
that tells  how long the message should be kept.  Any parameter that
dictates how long a  message is persisted should probably be driven by
legal requirements such as  non-repdudiation.
<DB>There has been recent discussion  on this topic on the list. See the
thread at ...
http://lists.ebxml.org/archives/ebxml-transport/200012/msg00211.html
I disagree that  basing persistence duration solely on acknowledgement and
retries is  sufficient. This can work for a sender if there are no legal or
other  reasons for saving the message. However it is insufficient for a
receiver  of a message.

I completely agree though that the value set on message  persistence by a
recipient of a message is governed by several factors  including:
a) the need for a sender to know if reliable messaging will  work
b) the requirements of a business process choreography
c) other  needs such as legal requirements.
This is why persistDuration is a  parameter that can only be set in the
CPP/CPA (see the table at line 1482). It  can't be set in the header.
Therefore persistDuration can be, as you  suggest, controlled by the needs
of the business process as the CPA much of  the information in the CPA is
controlled by business process needs. However,  the information still needs
to be known by the MSH that is doing reliable  messaging and hence the need
to include as a CPA parameter that is used by the  MSH.
</DB>

Regards, Prasad

"Burdett, David" wrote:  Stefano

Version 0.91 of the spec has the concept of a "persistDuration" parameter
that would be in the CPA/CPP that identifies the number of days a  message
that is sent persistently should be kept by the receiving MSH.

The spec is not specific about how long messages that are sent are
persisted. However this should be as I think you are saying, until the
sender knows that the sent message has arrived since an acknowledgement
message has been returned.

Does this make sense?

David

-----Original Message-----
From: Stefano POGLIANI [mailto:stefano.pogliani@sun.com]
Sent: Wednesday, January 03, 2001 3:29 PM
To: Martin W Sachs;  Burdett, David
Cc: ebxml-transport@lists.ebxml.org;  ebxml-tp@lists.ebxml.org
Subject: RE: Yet more issues to resolve

Marty, David,

a little clarification w.r.t. "I also don't like the idea of placing a
restriction on the MSH to persist a message until the transaction is
complete, ..."

I think that an option would be to allow the MSH to be "controlled" by
outside w.r.t. this part and by defining a "default behaviour". "Not
saying
anything" (as mentioned by Marty) would mean, IMHO, that the  "default
behaviour" is to remove the message as soon as it reliably  reaches the
target MSH.
Giving the option to "control from outside"  would allow the layer between
the MSH and the real application to take  care (if it will be capable) of a
more sophisticated behaviour.

/Stefano

» -----Original Message-----
» From: Martin W Sachs [mailto:mwsachs@us.ibm.com]
» Sent:  Saturday, December 30, 2000 2:23 AM
» To: Burdett, David
» Cc:  ebxml-transport@lists.ebxml.org; ebxml-tp@lists.ebxml.org
» Subject: RE:  Yet more issues to resolve
»
»
»
» David,
»
» The  definition of persistDuration in your email looks OK.
»
» My point  about decoupling persistence from reliable messaging was
» a CPA/CPP
» point, not a MSH point.  I am simply suggesting that two parties
» might have
» reasons to come to agreement on persistence that have  nothing to do with
» the MSH in addition to its use with RM.
»
»  Regarding "I also don't like the idea of placing a restriction on the
MSH
» to persist a message until the transaction is complete, ..."  I  don't
like
» placing that restriction either.  I was asking for the  opposite - that
the
» MSH specification should not say anything which  either prescribes or
» suggests that a message can be deleted from the  persistent store
» as soon as
» it reliably reaches the destination  MSH.  Earlier versions of the RM
» specification included such  words.
»
» Regards,
» Marty
»  ******************************************************************
»  *******************
»
» Martin W. Sachs
» IBM T. J. Watson  Research Center
» P. O. B. 704
» Yorktown Hts, NY 10598
»  914-784-7287;  IBM tie line 863-7287
» Notes address:  Martin  W Sachs/Watson/IBM
» Internet address:  mwsachs @ us.ibm.com
»  ******************************************************************
»  *******************
»
»
»
» "Burdett, David"  <david.burdett@commerceone.com> on 12/27/2000 11:20:24
AM
»
»  To:   Martin W Sachs/Watson/IBM@IBMUS
» cc:    ebxml-transport@lists.ebxml.org, ebxml-tp@lists.ebxml.org
»  Subject:  RE: Yet more issues to resolve
»
»
»
» Marty
»
» In version 0.9a of the spec, a parameter called "pesistDuration"
» is defined
» that is described as follows:
»
»  "PersistDuration is the minimum length of time in days that a Message
that
» is sent reliably is kept in Persistent Storage by a MSH. The
»  value used for
» PersistDuration is an implementation decision although  it MUST be
greater
» than the value of the TimeToLive parameter for any  message that is sent.
»
» If a duplicate message (i.e. with the same  MessageId) is received before
» the
» PersistDuration has passed,  then the MSH that receives it MUST process
it
» as
» a duplicate  message as described in sections 10.2.1.1 and 10.2.1.2.
»
» If a  duplicate message is received after the PersistDuration has passed,
»  then although it may be treated as a duplicate, the sender must realize
» that
» it will probably be treated by the MSH as if the message  were a
» new message
» that had not been received before."
»
» So as it stands, the persistDuration only applies to Relilable  Messaging
» behavior which I think is right. I know that there are  other, e.g.
legal,
» reasons why a messages might be persisted beyond  this time. However from
a
» MSH perspective, I think we can only specify  the behavior of the MSH and
» nothing else.
»
» I also don't  like the idea of placing a restriction on the MSH to
» persist a
»  message until the transaction is complete, since the MSH will
» probably  need
» to be told by the application that the transaction is complete  and it's
» possible that some backend systems will not be able to easily  do this.
»
» Do you think the existing definition is OK?
»
»  David
»
» -----Original Message-----
» From: Martin W Sachs [mailto:mwsachs@us.ibm.com]
» Sent:  Tuesday, December 19, 2000 9:56 AM
» To: Christopher Ferris
» Cc:  ebxml-transport@lists.ebxml.org; ebxml-tp@lists.ebxml.org
» Subject: Re:  Yet more issues to resolve
»
»
»
» Chris,
»
»  Regarding issue 19:  I agree that there should be nothing in the TRP
spec
» about how long a message must be persisted since higher layers  are
» involved.  Perhaps the messaging spec should say that  persistence must
be
» at least until the reliable messaging procedure  has been
» completed for that
» message transmission.  I  suggest a non-normative note explaining the
» issues.
»
» In my  opinion, at the higher levels, persistence should be at least
until
»  the business transaction is concluded.  Persistence beyond that
»  is probably
» not a matter that has to be agreed to.  It is  probably an internal
matter
» for each party.
»
» If the  persistence element will be in your upcoming CPP/CPA DTD, please
»  supply some explanatory text with it.  I suggest not coupling the
»  persistence element in the CPA to reliable messaging.  While it is
» essential to reliable messaging, it is also something that might be
needed
» without ebXML Reliable Messaging.  For example, if the  parties use a
» transport that is reliable in and of itself, they would  probably not
want
» to invoke ebXML Reliable Messaging on top of it but  they might want to
» specify something about persistence.
»
»  Regards,
» Marty
»
»
»
»
»
»  ******************************************************************
»  **********
»
» *********
»
» Martin W. Sachs
» IBM T. J.  Watson Research Center
» P. O. B. 704
» Yorktown Hts, NY 10598
»  914-784-7287;  IBM tie line 863-7287
» Notes address:  Martin  W Sachs/Watson/IBM
» Internet address:  mwsachs @ us.ibm.com
»  ******************************************************************
»  **********
»
» *********
»
»
»
» Christopher Ferris  <chris.ferris@east.sun.com>@east.sun.com on
12/19/2000
» 12:23:17  PM
»
» Sent by:  Chris.Ferris@east.sun.com
»
»
»  To:   ebxml-transport@lists.ebxml.org
» cc:
»  Subject:  Re: Yet more issues to resolve
»
»
»
» Ian,
»
» Thanks for sending these out. Some comments below.
»
»  Cheers,
»
» Chris
» ian.c.jones@bt.com wrote:
» >
»  > TR&P people,
» >
»  >         Sorry this is late -  the real world gets in the way
» occasionally.
» >
» > The  following issues may now have been overtaken by events but
» we need to
» at
» > least acknowledge this so that I can get the database of  issue and
» changes
» > update by the New Year.
» >
»  > ####
» > Issue 15
» > Title - RM Info
» > Detail -  7.9.4  - RM Info: why isn't this in routing header?
» >
»  > Proposal -  Move to Routing Header
» > Resolution -
»  > Response By - December 21
»
» I have to disagree with this. The  ReliableMessagingInfo
» element applies to the message, not to a  particular hop
» along the way. Note that David has crafted a revised  section
» on RM in the draft we've been collaborating on which we
»  hope to get to the list today (although, we recognise that
» it still  needs work).
»
» >
» > ####
» > Issue 16
» >  Title - TPAInfo
» > Detail - Add new section containing all TPA(CPA)  information
» >
» > Proposal - see above
» > Resolution  -
» > Response By -
»
» This one I'm not sure I understand. I  thought that we agreed
» in Tokyo to flatten the TPAInfo section by  moving all of
» the child elements to be direct decendants of Header.
»
» What information was missing from TPAInfo anyway? I'm not
»  clear on what the proposal really suggests.
»
» >
» > ####
» > Issue 17
» > Title - Sequence number
» > Detail -  7.10 SequenceNumber as defined adds little or no value over
the
» >  MessageID element required in MessageData.  Recommend removing this
» section
» > and the corresponding references to SequenceNumber  in later  examples,
» the
» > DTD and the schema.
» >  If this section remains, must define the exact content of the
»  SequenceNumber
» > element when the numeric value is greater than  999.  Is the next value
» > "1000" or "1,000"?  Is the text  for the maximum value  "999999999" or
» > "999,999,999" or are  both allowed (and equivalent)?  Side note:
Numeric
» > formats  vary by locale and a comma is not always the thousands
» > If this  section remains, "long time" should be defined.  At least,
refer
»  to
» > a TPA which makes this concrete for a specific trading  relationship or
» > mention later text on this subject (if any).
» >
» > Proposal -
» > Resolution -
» > Response  By -
»
» Since we're going to spec the schema using XMLSchema, we  can
» be more precise in the typing of elements and attributes.
» I  agree that the spec needs to be tightened up for this
» particular  element.
»
» I'll have to review David's revision of RM to see what  that
» has to offer.
»
» >
» > ####
» > Issue  18
» > Title - URI definitions in RM
» > Detail - 7.10 -  SenderURI, ReceiverURI and ErrorURI are not
» defined.  Are
»  > they identifiers or service locations?  From later discussion in  this
» > document, it appears that ErrorURI is a service location at  which
error
» > responses may be received.  The other two aren't  discussed much.
» However,
» > I'd recommend they be defined  similarly to the PartyId elements in the
» > Header (i.e. the parties  participating in the current leg of the
» journey).
» > That  requires adding a context attribute to SenderURI and
» ReceiverURI and
» > (potentially) renaming the elements.  Alternatively,  SenderURI and
» > ReceiverURI may somehow identify the service  location used in this
» > particular leg (this is probably redundant  information for ReceiverURI
» with
» > the transport destination  and SenderURI may not exist -- not all
senders
» are
» >  reachable in return).
» >
» > Proposal -
» > Resolution  -
» > Response By -
»
» I assume that the subject refers to  URI elements in RoutingHeader
» not RM. Agree that these need to be more  precisely defined.
» As to defining them similarly to the PartyId  element...
» I had actually proposed to David in our discussions that
» instead of defining different elements, that we re-use
» To and  From but with precise wording that the PartyId
» MUST be an URL form of  URI (e.g. something addressable).
»
» This also brings to mind a  security issue (possibly) as
» if an intermediate signs a header (with  the added RoutingHeader
» element) then it MAY be important to be able  to identify
» the intermediary with the specific RoutingHeader. I hadn't
» thought of this before, but it MAY be important to consider.
»
» >
» > ####
» > Issue 19
» > Title - Persistent  storage
» > Detail - 7.11 - "Persistent storage" actually lasts how  long?  Is the
» > original message deleted by the Messaging  Service immediately after
» > receiving an acknowledgement?  In  the case of an error, which
» is reported
» to
» > the  sending application (party), does that report also result
» in deletion
» of
» > the original message from the persistent storage?
»  > What information does the Receiver store persistently in this step?
» > Recommend storing the MessageID of the original message and the  entire
» > response document.  To check for "exactly the same as  the original
» message"
» > as described on line 828, may be  necessary to store entire original
» message
» > or its checksum  as well.
» >
» > Proposal - This is out of scope of the  TR&P if required it should be
in
» the
» > CPP/CPA
» >  Resolution -
» > Response By -
»
» David has added a section  on persistence in his RM draft
» proposal. It doesn't get into "how  long" as this is correctly
» identified as being something that the  parties MUST agree
» upon and have reflected in the CPA. A new element  is being added
» to the CPA for this purpose (e.g. to capture the  persistenceDuration
» with which a message is to be archived).
»
» >
» > Ian Jones
» > E-Commerce Engineer
» >
» >
»