RE: Yet more issues to resolve

["Burdett, David" <david.burdett@commerceone.com>]

Marty

I honestly don't think that it violates anything in the spec since resending
over a different transport has long been thought of as a good idea. I agree
though that an implementation decision could be to design/implement a
solution that has different MSH for each transport protocol.

In my view, it is more of a CPA/CPP issue, in that if you define alternate
transport channels for a particular message in a business process, then you
need to make sure that you have the technology to support the alternative
transport within the same message. If this is not the case then don't put it
in the CPP/CPA.

Another way you might be able to do it is by sending the same payload again
using a different messageId.

Thoughts?

David

-----Original Message-----
From: Martin W Sachs [mailto:mwsachs@us.ibm.com]
Sent: Friday, January 05, 2001 7:25 AM
To: Burdett, David
Cc: Doug Bunting; 'Prasad Yendluri'; 'Stefano POGLIANI';
ebxml-transport@lists.ebxml.org; ebxml-tp@lists.ebxml.org
Subject: RE: Yet more issues to resolve



David,

>>>It can't always be "exactly  the same stream of bytes on the wire since,
if you resend a message using a  different transport protocol, then you
need to use a different Routing Header  with different URIs.<<<

MWS:  We need to be very careful about resending a message using a
different transport protocol.  A different transport
protocol may mean a different physical port on the sending machine and
possibly a different instance of the MSH.  So I agree that in this case, it
won't be exactly the same stream of bytes.  But if it is a different
instance of the MSH, does this violate any assumptions that are behind the
specification?  Do we have to beef up the specification to ensure that
multiple instances of the MSH can be involved in the protocol for the same
message?


Regards,
Marty

****************************************************************************
*********

Martin W. Sachs
IBM T. J. Watson Research Center
P. O. B. 704
Yorktown Hts, NY 10598
914-784-7287;  IBM tie line 863-7287
Notes address:  Martin W Sachs/Watson/IBM
Internet address:  mwsachs @ us.ibm.com
****************************************************************************
*********



"Burdett, David" <david.burdett@commerceone.com> on 01/05/2001 02:47:07 AM

To:   Doug Bunting <Doug@ariba.com>, "'Prasad Yendluri'"
      <pyendluri@webmethods.com>
cc:   "'Stefano POGLIANI'" <stefano.pogliani@sun.com>, Martin W
      Sachs/Watson/IBM@IBMUS, ebxml-transport@lists.ebxml.org,
      ebxml-tp@lists.ebxml.org
Subject:  RE: Yet more issues to resolve




Doug

See  comments below.

David
-----Original Message-----
From: Doug Bunting  [mailto:Doug@ariba.com]
Sent: Thursday, January 04, 2001 11:29  AM
To: Burdett, David; 'Prasad Yendluri'
Cc: 'Stefano  POGLIANI'; Martin W Sachs; ebxml-transport@lists.ebxml.org;
ebxml-tp@lists.ebxml.org
Subject: RE: Yet more issues to  resolve


Two additional comments related to persistent storage with respect  only to
implementing Reliable Messaging.  Again, legal or business  reasons for
additional storage must be considered and described separately, I  think
that's the purpose of the persistDuration parameter.

   Persistent storage at the receiving side MUST (in RFC 2119
   terms) include both the messageID value and the corresponding
   intermediate ACK (if any).  The exact same answer should go back to the
   sender no matter how many times they send a duplicate  message.

>>>I agree, that MessageId needs to be  persisted. However persisting
the intermediate ack is a function of the  sending process not the
receiving process. See also section  10.2.1.2<<<

   If it isn't  already clear in the specification (and, I don't think it
   is), we should  make sure the messageID corresponds to a particular set
   of bytes.  I  believe we've said that retries must use exactly the same
   message as  before.  I don't think we've made "exactly" quite clear
   enough.   Is it "semantically identical content" or "matching canonical
   form" or  (my choice) "exactly the same stream of bytes on the wire"?

>>>It can't always be "exactly  the same stream of bytes on the wire since,
if you resend a message using a  different transport protocol, then you
need to use a different Routing Header  with different URIs.<<<
 We may  also want to describe a possible error (detected by the receiver)
when  the checksum of the message doesn't match that (optionally) stored
along  with a messageID.
>>>One of the  problems of doing checksums, is that it has to be done over
a canonical  version of the document. Sending the same bytes might result
in a different  set of bytes being received, especially if you are using a
transport such as  SMTP that can alter the message content. So I think that
if this is important,  then you should use XML Dsig to make sure that the
message is the same as  it includes transformations that allow a canonical
form to be generated.  <<
 I  definitely agree that the sender needs to persist the entire request
until  it receives the intermediate or final acknowledgement.  The receiver
should not be required to persist the request for reliable  messaging
purposes, though they could.  (Of course, a receiver that is  forwarding
the message in turn must persist the request as any other sender  would.
Since we distinguish between the inbound and outbound MSH  instances at
intermediate nodes, this might not need to be described in our
specification.)  The intermediate choice of storing a hash or
checksum would be OPTIONAL (also in RFC 2119 terms).
>>>Please see  comments above, and the revised definition of
PersistDuration<<<

thanx,
    doug

-----Original Message-----
From: Burdett, David  [mailto:david.burdett@commerceone.com]
Sent: January 3, 2001  21:56
To: 'Prasad Yendluri'
Cc: 'Stefano POGLIANI'; Martin  W Sachs; ebxml-transport@lists.ebxml.org;
ebxml-tp@lists.ebxml.org
Subject: RE: Yet more issues to  resolve


Prasad

See  comments below.

Regards

David
-----Original Message-----
From: Prasad Yendluri  [mailto:pyendluri@webmethods.com]
Sent: Wednesday, January 03,  2001 7:21 PM
To: Burdett, David
Cc: 'Stefano POGLIANI';  Martin W Sachs; ebxml-transport@lists.ebxml.org;
ebxml-tp@lists.ebxml.org
Subject: Re: Yet more issues to  resolve

David,

The description for the "persistDuration" parameter reads as follows:
PersistDuration is the minimum length of time in days that  a Message that
is sent reliably is kept in Persistent Storage by a MSH.  The value used
for PersistDuration is an implementation decision although  it MUST be
greater than the value of the TimeToLive parameter for any  message that is
sent.If a duplicate message (i.e. with the same MessageId ) is  received
before the PersistDuration has passed, then the MSH that receives  it MUST
process it as a duplicate message as described  in sections  10.2.1.3 and
Error! Reference source not found..

If a duplicate message is received after the PersistDuration has  passed,
then although it may be treated as a duplicate, the sender must  realize
that it will probably be treated by the MSH as if the message were  a new
message that had not been received before.
1. If the purpose of this is to simply detect duplicates why require  the
entire message to be stored? Isn't saving the value of the messageId
enough?
<DB>
Yes, the MessageId is enough. It was not my  intention to REQUIRE that a
whole message should be kept. Note that I did  not say that "
PersistDuration is the minimum length of time in days that  a **complete**
Message that is sent reliably **MUST be** kept in Persistent  Storage by a
MSH. Also  lines 1213-1216 in Reliable messaging, that describe the
behavior of a MSH  that receives a message, explicitly say ...
>> 1)        If the message is not a duplicate then do the following:
     a)        Save the MessageId of the received message in  persistent
storage. As an implementation decision, the whole message  MAY be stored if
there are other reasons for doing  so.<<
Another factor is that, if you want to support a Message Status  Inquiry,
then you will also need to record the time a message was received.  I agree
though that the description of persist duration could be clarified  to
indicate that only the MessageId needs to be saved.
</DB>
 If any, I would think it is  the sender MSH that needs to persist the
message, so that a retry would a  simple retransmission.
<DB>
I  agree the sender MSH does need to persist the message. Line 1200 in the
Reliable Messaging specification that is describing the behavior of a
sender  of the message says ...
>> 1)        Save the message in persistent storage (see section 10.1.1)<<
However how long the sender saves the message for  is not relevant to
Reliable Messaging. You need to know it for the receiver  since you need to
know if you resend a message whether or not duplicate  filtering will work
which it won't if the reciever has not persisted the  message. The same
does not apply to the sender since the persisting of the  message only
applies to resending.
</DB>

2. Why the value limited to "days"?
<DB>A good point. My original thinking  was that if the message is being
placed in some type of persistent storage  such as a disk, then you
probably would not want to clear it out too often.  So setting it to whole
days makes it a simpler value than the years, months,  weeks, days, hours,
minutes, millisecs?? that you might otherwise need.  Another approach is to
specify a single integer and then put the units in  separately such as
<persistDuration  units="seconds">20</persistDuration> in the CPA. Would
that work  better?</DB>

3. The last paragraph seems to imply that sender and receiver could be in
an inconsistent state without clear knowledge of, if retries on a message
are still permitted or not. That is sender thinks retries are still
permitted but, receiver does not know it is a re-send, as the message was
deleted from the persistent store. If we got the Reliable Messaging
specification right, this should never be the case. At least one of the
parties should always know.  The receiver must always know if the  received
message is duplicate or not, if the message was received once
successfully, irrespective of, if the entire message is (still) persisted
or  not (based on the MessageId alone). We should never retire the
MessageId  either from sender/receiver (or intermediary) perspective. It is
a GUID  isn't it?
<DB>I agree that ideally the  Message Ids should never be "retired".
However what I think you are  suggesting is that the MessageIds are
**never** deleted by the receiver. I  honestly don't think this is
practical, since if you want to forever to be  able to do a duplicate
check: a) the MessageIds received have to be  immediately available in
order to provide a rapid response to the duplicate  check, and b) this
would imply ever increasing persistent storage which  would prove costly to
maintain. This means that the recipient of a message  will have to, at some
point, delete old message ids. The point is that the  sender needs to know
whether or not duplicate filtering will work. By  specifying a
persistDuration for received Message Ids, the Sender will  knowthat the
duplicate filtering will work, especially if the persist  duration is
significantly longer than any period of time over which a  message might be
resent. However the sender can never know EXACTLY when the  sender will
delete a message id unless they do a query first.

Perhaps it would be better if the last paragraph said ... "A MSH  SHOULD
NOT resend a message to a receiver if the receiver has probably  deleted
the message since the time indicated by PersistDuration has passed  since
the sender first sent the message." You could even require that a  sender
report a delivery failure if the time has  passed.
</DB>

4. I think, how long a message should be persisted in the MSH should be
based on acknowledgment and retries and associated timeout constraints of
the business process choreography and not based on a separate parameter
that  tells how long the message should be kept.  Any parameter that
dictates how long a  message is persisted should probably be driven by
legal requirements such as  non-repdudiation.
<DB>There has been recent  discussion on this topic on the list. See the
thread at ...
http://lists.ebxml.org/archives/ebxml-transport/200012/msg00211.html
I disagree  that basing persistence duration solely on acknowledgement and
retries is  sufficient. This can work for a sender if there are no legal or
other  reasons for saving the message. However it is insufficient for a
receiver of a message.

I completely agree though that the value set  on message persistence by a
recipient of a message is governed by several  factors including:
a) the need for a sender to know if reliable messaging  will work
b) the requirements of a business process choreography
c)  other needs such as legal requirements.
This is why persistDuration is a  parameter that can only be set in the
CPP/CPA (see the table at line 1482).  It can't be set in the header.
Therefore persistDuration can be, as you  suggest, controlled by the needs
of the business process as the CPA much of  the information in the CPA is
controlled by business process needs. However,  the information still needs
to be known by the MSH that is doing reliable  messaging and hence the need
to include as a CPA parameter that is used by  the MSH.
</DB>

Regards, Prasad

"Burdett, David" wrote:  Stefano

Version 0.91 of the spec has the concept of a "persistDuration"  parameter
that would be in the CPA/CPP that identifies the number of  days a message
that is sent persistently should be kept by the  receiving MSH.

The spec is not specific about how long messages that are sent are
persisted. However this should be as I think you are saying, until the
sender knows that the sent message has arrived since an  acknowledgement
message has been returned.

Does this make sense?

David

-----Original Message-----
From: Stefano POGLIANI [mailto:stefano.pogliani@sun.com]
Sent: Wednesday, January 03, 2001 3:29 PM
To: Martin W Sachs;  Burdett, David
Cc: ebxml-transport@lists.ebxml.org;  ebxml-tp@lists.ebxml.org
Subject: RE: Yet more issues to resolve

Marty, David,

a little clarification w.r.t. "I also don't like the idea of placing a
restriction on the MSH to persist a message until the transaction is
complete, ..."

I think that an option would be to allow the MSH to be "controlled" by
outside w.r.t. this part and by defining a "default behaviour". "Not
saying
anything" (as mentioned by Marty) would mean, IMHO, that the  "default
behaviour" is to remove the message as soon as it reliably  reaches the
target MSH.
Giving the option to "control from  outside" would allow the layer between
the MSH and the real  application to take care (if it will be capable) of a
more  sophisticated behaviour.

/Stefano

» -----Original Message-----
» From: Martin W Sachs [mailto:mwsachs@us.ibm.com]
»  Sent: Saturday, December 30, 2000 2:23 AM
» To: Burdett, David
»  Cc: ebxml-transport@lists.ebxml.org; ebxml-tp@lists.ebxml.org
»  Subject: RE: Yet more issues to resolve
»
»
»
» David,
»
» The definition of persistDuration in your email looks OK.
»
» My point about decoupling persistence from reliable messaging  was
» a CPA/CPP
» point, not a MSH point.  I am simply  suggesting that two parties
» might have
» reasons to come to  agreement on persistence that have nothing to do with
» the MSH in  addition to its use with RM.
»
» Regarding "I also don't like the  idea of placing a restriction on the
MSH
» to persist a message until  the transaction is complete, ..."  I don't
like
» placing that  restriction either.  I was asking for the opposite - that
the
»  MSH specification should not say anything which either prescribes or
»  suggests that a message can be deleted from the persistent store
» as  soon as
» it reliably reaches the destination MSH.  Earlier  versions of the RM
» specification included such words.
»
»  Regards,
» Marty
»  ******************************************************************
»  *******************
»
» Martin W. Sachs
» IBM T. J. Watson  Research Center
» P. O. B. 704
» Yorktown Hts, NY 10598
»  914-784-7287;  IBM tie line 863-7287
» Notes address:   Martin W Sachs/Watson/IBM
» Internet address:  mwsachs @  us.ibm.com
»  ******************************************************************
»  *******************
»
»
»
» "Burdett, David"  <david.burdett@commerceone.com> on 12/27/2000 11:20:24
AM
»
» To:   Martin W Sachs/Watson/IBM@IBMUS
»  cc:   ebxml-transport@lists.ebxml.org, ebxml-tp@lists.ebxml.org
» Subject:  RE: Yet more issues to resolve
»
»
»
» Marty
»
» In version 0.9a of the spec, a parameter called  "pesistDuration"
» is defined
» that is described as follows:
»
» "PersistDuration is the minimum length of time in days that a  Message
that
» is sent reliably is kept in Persistent Storage by a  MSH. The
» value used for
» PersistDuration is an implementation  decision although it MUST be
greater
» than the value of the  TimeToLive parameter for any message that is sent.
»
» If a  duplicate message (i.e. with the same MessageId) is received before
»  the
» PersistDuration has passed, then the MSH that receives it MUST  process
it
» as
» a duplicate message as described in sections  10.2.1.1 and 10.2.1.2.
»
» If a duplicate message is received  after the PersistDuration has passed,
» then although it may be  treated as a duplicate, the sender must realize
» that
» it will  probably be treated by the MSH as if the message were a
» new message
» that had not been received before."
»
» So as it stands, the  persistDuration only applies to Relilable Messaging
» behavior which I  think is right. I know that there are other, e.g.
legal,
» reasons why  a messages might be persisted beyond this time. However from
a
» MSH  perspective, I think we can only specify the behavior of the MSH and
»  nothing else.
»
» I also don't like the idea of placing a  restriction on the MSH to
» persist a
» message until the  transaction is complete, since the MSH will
» probably need
» to  be told by the application that the transaction is complete and it's
»  possible that some backend systems will not be able to easily do this.
»
» Do you think the existing definition is OK?
»
» David
»
» -----Original Message-----
» From: Martin W Sachs [mailto:mwsachs@us.ibm.com]
»  Sent: Tuesday, December 19, 2000 9:56 AM
» To: Christopher Ferris
» Cc: ebxml-transport@lists.ebxml.org; ebxml-tp@lists.ebxml.org
»  Subject: Re: Yet more issues to resolve
»
»
»
» Chris,
»
» Regarding issue 19:  I agree that there should be nothing  in the TRP
spec
» about how long a message must be persisted since  higher layers are
» involved.  Perhaps the messaging spec should  say that persistence must
be
» at least until the reliable messaging  procedure has been
» completed for that
» message  transmission.  I suggest a non-normative note explaining the
»  issues.
»
» In my opinion, at the higher levels, persistence  should be at least
until
» the business transaction is  concluded.  Persistence beyond that
» is probably
» not a  matter that has to be agreed to.  It is probably an internal
matter
» for each party.
»
» If the persistence element will be in  your upcoming CPP/CPA DTD, please
» supply some explanatory text with  it.  I suggest not coupling the
» persistence element in the CPA  to reliable messaging.  While it is
» essential to reliable  messaging, it is also something that might be
needed
» without ebXML  Reliable Messaging.  For example, if the parties use a
»  transport that is reliable in and of itself, they would probably not
want
» to invoke ebXML Reliable Messaging on top of it but they might want  to
» specify something about persistence.
»
» Regards,
»  Marty
»
»
»
»
»
»  ******************************************************************
»  **********
»
» *********
»
» Martin W. Sachs
» IBM T.  J. Watson Research Center
» P. O. B. 704
» Yorktown Hts, NY 10598
» 914-784-7287;  IBM tie line 863-7287
» Notes address:   Martin W Sachs/Watson/IBM
» Internet address:  mwsachs @  us.ibm.com
»  ******************************************************************
»  **********
»
» *********
»
»
»
» Christopher  Ferris <chris.ferris@east.sun.com>@east.sun.com on
12/19/2000
»  12:23:17 PM
»
» Sent by:  Chris.Ferris@east.sun.com
»
»
» To:   ebxml-transport@lists.ebxml.org
» cc:
» Subject:  Re: Yet more issues to resolve
»
»
»
» Ian,
»
» Thanks for sending these out. Some comments below.
»
» Cheers,
»
» Chris
» ian.c.jones@bt.com wrote:
» >
» > TR&P people,
» >
»  >         Sorry this is late -  the real world gets in the way
» occasionally.
» >
» >  The following issues may now have been overtaken by events but
» we  need to
» at
» > least acknowledge this so that I can get the  database of issue and
» changes
» > update by the New Year.
» >
» > ####
» > Issue 15
» > Title - RM Info
» > Detail - 7.9.4  - RM Info: why isn't this in routing  header?
» >
» > Proposal -  Move to Routing Header
» > Resolution -
» > Response By - December 21
»
» I  have to disagree with this. The ReliableMessagingInfo
» element  applies to the message, not to a particular hop
» along the way. Note  that David has crafted a revised section
» on RM in the draft we've  been collaborating on which we
» hope to get to the list today  (although, we recognise that
» it still needs work).
»
» >
» > ####
» > Issue 16
» > Title - TPAInfo
» >  Detail - Add new section containing all TPA(CPA) information
» >
» > Proposal - see above
» > Resolution -
» >  Response By -
»
» This one I'm not sure I understand. I thought  that we agreed
» in Tokyo to flatten the TPAInfo section by moving all  of
» the child elements to be direct decendants of Header.
»
»  What information was missing from TPAInfo anyway? I'm not
» clear on  what the proposal really suggests.
»
» >
» > ####
»  > Issue 17
» > Title - Sequence number
» > Detail - 7.10  SequenceNumber as defined adds little or no value over
the
» >  MessageID element required in MessageData.  Recommend removing this
» section
» > and the corresponding references to  SequenceNumber in later  examples,
» the
» > DTD and the  schema.
» > If this section remains, must define the exact content  of the
» SequenceNumber
» > element when the numeric value is  greater than 999.  Is the next value
» > "1000" or  "1,000"?  Is the text for the maximum value  "999999999" or
» > "999,999,999" or are both allowed (and equivalent)?  Side  note:
Numeric
» > formats vary by locale and a comma is not always  the thousands
» > If this section remains, "long time" should be  defined.  At least,
refer
» to
» > a TPA which makes this  concrete for a specific trading relationship or
» > mention later  text on this subject (if any).
» >
» > Proposal -
» >  Resolution -
» > Response By -
»
» Since we're going to  spec the schema using XMLSchema, we can
» be more precise in the  typing of elements and attributes.
» I agree that the spec needs to be  tightened up for this
» particular element.
»
» I'll have to  review David's revision of RM to see what that
» has to offer.
»
» >
» > ####
» > Issue 18
» > Title - URI  definitions in RM
» > Detail - 7.10 - SenderURI, ReceiverURI and  ErrorURI are not
» defined.  Are
» > they identifiers or  service locations?  From later discussion in this
» >  document, it appears that ErrorURI is a service location at which
error
» > responses may be received.  The other two aren't discussed  much.
» However,
» > I'd recommend they be defined similarly to  the PartyId elements in the
» > Header (i.e. the parties  participating in the current leg of the
» journey).
» > That  requires adding a context attribute to SenderURI and
» ReceiverURI and
» > (potentially) renaming the elements.  Alternatively,  SenderURI and
» > ReceiverURI may somehow identify the service  location used in this
» > particular leg (this is probably  redundant information for ReceiverURI
» with
» > the transport  destination and SenderURI may not exist -- not all
senders
» are
»  > reachable in return).
» >
» > Proposal -
» >  Resolution -
» > Response By -
»
» I assume that the  subject refers to URI elements in RoutingHeader
» not RM. Agree that  these need to be more precisely defined.
» As to defining them  similarly to the PartyId element...
» I had actually proposed to David  in our discussions that
» instead of defining different elements, that  we re-use
» To and From but with precise wording that the PartyId
» MUST be an URL form of URI (e.g. something addressable).
»
»  This also brings to mind a security issue (possibly) as
» if an  intermediate signs a header (with the added RoutingHeader
» element)  then it MAY be important to be able to identify
» the intermediary  with the specific RoutingHeader. I hadn't
» thought of this before,  but it MAY be important to consider.
»
» >
» > ####
» > Issue 19
» > Title - Persistent storage
» >  Detail - 7.11 - "Persistent storage" actually lasts how long?  Is the
» > original message deleted by the Messaging Service immediately  after
» > receiving an acknowledgement?  In the case of an  error, which
» is reported
» to
» > the sending application  (party), does that report also result
» in deletion
» of
»  > the original message from the persistent storage?
» > What  information does the Receiver store persistently in this step?
» >  Recommend storing the MessageID of the original message and the entire
» > response document.  To check for "exactly the same as the  original
» message"
» > as described on line 828, may be  necessary to store entire original
» message
» > or its  checksum as well.
» >
» > Proposal - This is out of scope of  the TR&P if required it should be
in
» the
» > CPP/CPA
» > Resolution -
» > Response By -
»
» David has  added a section on persistence in his RM draft
» proposal. It doesn't  get into "how long" as this is correctly
» identified as being  something that the parties MUST agree
» upon and have reflected in the  CPA. A new element is being added
» to the CPA for this purpose (e.g.  to capture the persistenceDuration
» with which a message is to be  archived).
»
» >
» > Ian Jones
» > E-Commerce  Engineer
» >
» >
»