Discussion:
How to avoid file corruption transferring members from one file to another
(too old to reply)
Gary Thompson
2014-10-21 18:40:02 UTC
Permalink
Raw Message
I have program FTP_ASN_1 sending EDI 856 ASN documents to a remote server.
If an FTP connection can't be made to the remote, the ASN document is
copied to a member file WAIT_Q, and the ASN program continues with the
next ASN as our warehouse completes the next shipment.

We may not be able to connect to the remote FTP server for a few minutes
or some hours, possibly a day.

I have program FTP_ASN_2 pulling ASN documents that have not yet been sent
from members in a file I call SEND_Q. A separate file is used to avoid contention
with FTP_ASN_1.

When FTP_ASN_2 finds no member in SEND_Q, it stops its FTP process and looks
to copy any members from WAIT_Q to SEND_Q and then remove all _copied_
members from SEND_Q.

My question is how to avoid 'collisions' between FTP_ASN_1 and FTP_ASN_2 during
the time members are being 'moved' from WAIT_Q to SEND_Q ?

My ideas have so far included:

Use a single data area and require both FTP_ASN_1 and FTP_ASN_2 to lock the data
area before adding new or copying/deleting WAIT_Q members.

Have both FTP_ASN_1 and FTP_ASN_2 get exclusive control of WAIT_Q with ALCOBJ.

Looking for any advice or comments about design pitfalls . . .
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To post a message email: MIDRANGE-L-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org
Before posting, please take a moment to review the archives
at http://archive.midrange.com/midrange-l.
Darryl Freinkel
2014-10-21 20:40:50 UTC
Permalink
Raw Message
I don't remember how I did this at other sites, but you could write a
trailer file to the target system folder when you have completed sending
the file. The target system could monitor for the file and then start
processing the file when its done.

Sometimes the target system is too quick for the sending system to complete
it processing and the file may not be done by the time the target reads it
and as a result misses a bunch of records.
Post by Gary Thompson
I have program FTP_ASN_1 sending EDI 856 ASN documents to a remote server.
If an FTP connection can't be made to the remote, the ASN document is
copied to a member file WAIT_Q, and the ASN program continues with the
next ASN as our warehouse completes the next shipment.
We may not be able to connect to the remote FTP server for a few minutes
or some hours, possibly a day.
I have program FTP_ASN_2 pulling ASN documents that have not yet been sent
from members in a file I call SEND_Q. A separate file is used to avoid contention
with FTP_ASN_1.
When FTP_ASN_2 finds no member in SEND_Q, it stops its FTP process and looks
to copy any members from WAIT_Q to SEND_Q and then remove all _copied_
members from SEND_Q.
My question is how to avoid 'collisions' between FTP_ASN_1 and FTP_ASN_2 during
the time members are being 'moved' from WAIT_Q to SEND_Q ?
Use a single data area and require both FTP_ASN_1 and FTP_ASN_2 to lock the data
area before adding new or copying/deleting WAIT_Q members.
Have both FTP_ASN_1 and FTP_ASN_2 get exclusive control of WAIT_Q with ALCOBJ.
Looking for any advice or comments about design pitfalls . . .
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
Before posting, please take a moment to review the archives
at http://archive.midrange.com/midrange-l.
--
Darryl Freinkel
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To post a message email: MIDRANGE-L-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org
Before posting, please take a moment to review the archives
at http://archive.midrange.com/midrange-l.
Gary Thompson
2014-10-21 22:03:03 UTC
Permalink
Raw Message
Thanks Darryl,
I can see how the trailer record could help in that case,
but, thankfully, I've not yet had that problem.

I'm trying to sync separate programs in separate jobs
so file members made by job 1 can be transferred to
job 2 without corrupted data or dead-locks.

Each job has its 'own file' where members are stored
until they can be processed, but at some point file 1
members must get copied to file 2 and that is the
point where I want to avoid contention/corruption.

My first try will be to use a data area each program
reads/writes so each program 'sees' when the
other needs exclusive use of the WAIT_Q file.

-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org] On Behalf Of Darryl Freinkel
Sent: Tuesday, October 21, 2014 2:41 PM
To: Midrange Systems Technical Discussion
Subject: Re: How to avoid file corruption transferring members from one file to another

I don't remember how I did this at other sites, but you could write a trailer file to the target system folder when you have completed sending the file. The target system could monitor for the file and then start processing the file when its done.

Sometimes the target system is too quick for the sending system to complete it processing and the file may not be done by the time the target reads it and as a result misses a bunch of records.
Post by Gary Thompson
I have program FTP_ASN_1 sending EDI 856 ASN documents to a remote server.
If an FTP connection can't be made to the remote, the ASN document is
copied to a member file WAIT_Q, and the ASN program continues with the
next ASN as our warehouse completes the next shipment.
We may not be able to connect to the remote FTP server for a few
minutes or some hours, possibly a day.
I have program FTP_ASN_2 pulling ASN documents that have not yet been sent
from members in a file I call SEND_Q. A separate file is used to avoid contention
with FTP_ASN_1.
When FTP_ASN_2 finds no member in SEND_Q, it stops its FTP process and
looks to copy any members from WAIT_Q to SEND_Q and then remove all
_copied_ members from SEND_Q.
My question is how to avoid 'collisions' between FTP_ASN_1 and
FTP_ASN_2 during the time members are being 'moved' from WAIT_Q to
SEND_Q ?
Use a single data area and require both FTP_ASN_1 and FTP_ASN_2 to
lock the data area before adding new or copying/deleting WAIT_Q
members.
Have both FTP_ASN_1 and FTP_ASN_2 get exclusive control of WAIT_Q with ALCOBJ.
Looking for any advice or comments about design pitfalls . . .
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
a moment to review the archives at
http://archive.midrange.com/midrange-l.
--
Darryl Freinkel
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list To post a message email: MIDRANGE-L-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org Before posting, please take a moment to review the archives at http://archive.midrange.com/midrange-l.
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To post a message email: MIDRANGE-L-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org
Before posting, please take a moment to review the archives
at http://archive.midrange.com/midrange-l.
Alan Campin
2014-10-21 22:47:12 UTC
Permalink
Raw Message
Not sure I see the problem. If you have an exclusive lock on the member
when you writing to it how can the other job get to it?

The other thing you might so is name the member by some name like T_xxxxx
and never transfer anything with a T_ name. When you finished writing,
rename to a regular name.
Post by Gary Thompson
Thanks Darryl,
I can see how the trailer record could help in that case,
but, thankfully, I've not yet had that problem.
I'm trying to sync separate programs in separate jobs
so file members made by job 1 can be transferred to
job 2 without corrupted data or dead-locks.
Each job has its 'own file' where members are stored
until they can be processed, but at some point file 1
members must get copied to file 2 and that is the
point where I want to avoid contention/corruption.
My first try will be to use a data area each program
reads/writes so each program 'sees' when the
other needs exclusive use of the WAIT_Q file.
-----Original Message-----
Darryl Freinkel
Sent: Tuesday, October 21, 2014 2:41 PM
To: Midrange Systems Technical Discussion
Subject: Re: How to avoid file corruption transferring members from one file to another
I don't remember how I did this at other sites, but you could write a
trailer file to the target system folder when you have completed sending
the file. The target system could monitor for the file and then start
processing the file when its done.
Sometimes the target system is too quick for the sending system to
complete it processing and the file may not be done by the time the target
reads it and as a result misses a bunch of records.
Post by Gary Thompson
I have program FTP_ASN_1 sending EDI 856 ASN documents to a remote
server.
Post by Gary Thompson
If an FTP connection can't be made to the remote, the ASN document is
copied to a member file WAIT_Q, and the ASN program continues with the
next ASN as our warehouse completes the next shipment.
We may not be able to connect to the remote FTP server for a few
minutes or some hours, possibly a day.
I have program FTP_ASN_2 pulling ASN documents that have not yet been
sent
Post by Gary Thompson
from members in a file I call SEND_Q. A separate file is used to avoid contention
with FTP_ASN_1.
When FTP_ASN_2 finds no member in SEND_Q, it stops its FTP process and
looks to copy any members from WAIT_Q to SEND_Q and then remove all
_copied_ members from SEND_Q.
My question is how to avoid 'collisions' between FTP_ASN_1 and
FTP_ASN_2 during the time members are being 'moved' from WAIT_Q to
SEND_Q ?
Use a single data area and require both FTP_ASN_1 and FTP_ASN_2 to
lock the data area before adding new or copying/deleting WAIT_Q
members.
Have both FTP_ASN_1 and FTP_ASN_2 get exclusive control of WAIT_Q with ALCOBJ.
Looking for any advice or comments about design pitfalls . . .
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
a moment to review the archives at
http://archive.midrange.com/midrange-l.
--
Darryl Freinkel
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
moment to review the archives at http://archive.midrange.com/midrange-l.
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
Before posting, please take a moment to review the archives
at http://archive.midrange.com/midrange-l.
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To post a message email: MIDRANGE-L-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org
Before posting, please take a moment to review the archives
at http://archive.midrange.com/midrange-l.
Gary Thompson
2014-10-22 13:41:46 UTC
Permalink
Raw Message
Thanks Alan,
I want JOB_2 to copy all members from file WAIT_Q to SEND_Q.

The copy all members step is the "point of contention/collision" I want to control.

Each WAIT_Q member has a document JOB_1 was not able to FTP to remote server MOSTLY_UP, which may not be available for minutes or hours.

My current plan uses data area SYNC_DA that both jobs must lock/read/update/release.

JOB_2 must open SYNC_DA, write "COPY_MBRS" and then close SYNC_DA before copying WAIT_Q members to file SEND_Q.

After a WAIT_Q member is copied, JOB_2 removes it from WAIT_Q.
After all WAIT_Q members are copied/removed, JOB_2 writes "ADD_MBRS" to SYNC_DA.

At this point, JOB_2 tries to FTP any/all SEND_Q members to MOSTLY_UP, removing SEND_Q members after successful FTP.

JOB_1 waits for documents to be created and when received, attempts to FTP to MOSTLY_UP.
If FTP works, JOB_1 waits for the next document.
If FTP faild, JOB_1 must read SYNC_DA, looking for "ADD_MBRS".
If "ADD_MBRS" is found, JOB_1 determines the next member name for WAIT_Q, copies the document to the next WAIT_Q member, then releases SYNC_DA.
If JOB_1 does not find "ADD_MBRS", it releases SYNC_DA, and tries later.

Problems like JOB_1 not able to add to WAIT_Q or JOB_2 not able to FTP SEND_Q members will be handled by setting limits and having each job SOS to QSYSOPR.
This problem is a big issue for my company and this project is maybe a general model to automate a problem that has, to date, been manual,
or, in the case of EDI 856 ASN, not handled because the ASN is very time-critical.

Again, thanks to Alan and all on this list (especially Bruce, Doug and Ron for APIs at work, and Jamief for example RPGLE from CODE400 ) .


-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org] On Behalf Of Alan Campin
Sent: Tuesday, October 21, 2014 4:47 PM
To: Midrange Systems Technical Discussion
Subject: Re: How to avoid file corruption transferring members from one file to another

Not sure I see the problem. If you have an exclusive lock on the member when you writing to it how can the other job get to it?

The other thing you might so is name the member by some name like T_xxxxx and never transfer anything with a T_ name. When you finished writing, rename to a regular name.
Post by Gary Thompson
Thanks Darryl,
I can see how the trailer record could help in that case, but,
thankfully, I've not yet had that problem.
I'm trying to sync separate programs in separate jobs so file members
made by job 1 can be transferred to job 2 without corrupted data or
dead-locks.
Each job has its 'own file' where members are stored until they can be
processed, but at some point file 1 members must get copied to file 2
and that is the point where I want to avoid contention/corruption.
My first try will be to use a data area each program reads/writes so
each program 'sees' when the other needs exclusive use of the WAIT_Q
file.
-----Original Message-----
Darryl Freinkel
Sent: Tuesday, October 21, 2014 2:41 PM
To: Midrange Systems Technical Discussion
Subject: Re: How to avoid file corruption transferring members from one file to another
I don't remember how I did this at other sites, but you could write a
trailer file to the target system folder when you have completed
sending the file. The target system could monitor for the file and
then start processing the file when its done.
Sometimes the target system is too quick for the sending system to
complete it processing and the file may not be done by the time the
target reads it and as a result misses a bunch of records.
Post by Gary Thompson
I have program FTP_ASN_1 sending EDI 856 ASN documents to a remote
server.
Post by Gary Thompson
If an FTP connection can't be made to the remote, the ASN document
is copied to a member file WAIT_Q, and the ASN program continues
with the next ASN as our warehouse completes the next shipment.
We may not be able to connect to the remote FTP server for a few
minutes or some hours, possibly a day.
I have program FTP_ASN_2 pulling ASN documents that have not yet been
sent
Post by Gary Thompson
from members in a file I call SEND_Q. A separate file is used to avoid contention
with FTP_ASN_1.
When FTP_ASN_2 finds no member in SEND_Q, it stops its FTP process
and looks to copy any members from WAIT_Q to SEND_Q and then remove
all _copied_ members from SEND_Q.
My question is how to avoid 'collisions' between FTP_ASN_1 and
FTP_ASN_2 during the time members are being 'moved' from WAIT_Q to
SEND_Q ?
Use a single data area and require both FTP_ASN_1 and FTP_ASN_2 to
lock the data area before adding new or copying/deleting WAIT_Q
members.
Have both FTP_ASN_1 and FTP_ASN_2 get exclusive control of WAIT_Q with ALCOBJ.
Looking for any advice or comments about design pitfalls . . .
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L)
subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
take a moment to review the archives at
http://archive.midrange.com/midrange-l.
--
Darryl Freinkel
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
a moment to review the archives at http://archive.midrange.com/midrange-l.
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing
unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
a moment to review the archives at
http://archive.midrange.com/midrange-l.
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list To post a message email: MIDRANGE-L-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org Before posting, please take a moment to review the archives at http://archive.midrange.com/midrange-l.
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To post a message email: MIDRANGE-L-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org
Before posting, please take a moment to review the archives
at http://archive.midrange.com/midrange-l.
Buck Calabro
2014-10-22 14:34:11 UTC
Permalink
Raw Message
Post by Gary Thompson
I have program FTP_ASN_1 sending EDI 856 ASN documents to a remote server.
If an FTP connection can't be made to the remote, the ASN document is
copied to a member file WAIT_Q, and the ASN program continues with the
next ASN as our warehouse completes the next shipment.
We may not be able to connect to the remote FTP server for a few minutes
or some hours, possibly a day.
I have program FTP_ASN_2 pulling ASN documents that have not yet been sent
from members in a file I call SEND_Q. A separate file is used to avoid contention
with FTP_ASN_1.
When FTP_ASN_2 finds no member in SEND_Q, it stops its FTP process and looks
to copy any members from WAIT_Q to SEND_Q and then remove all _copied_
members from SEND_Q.
My question is how to avoid 'collisions' between FTP_ASN_1 and FTP_ASN_2 during
the time members are being 'moved' from WAIT_Q to SEND_Q ?
Use a single data area and require both FTP_ASN_1 and FTP_ASN_2 to lock the data
area before adding new or copying/deleting WAIT_Q members.
Have both FTP_ASN_1 and FTP_ASN_2 get exclusive control of WAIT_Q with ALCOBJ.
Looking for any advice or comments about design pitfalls . . .
The multi-member file was a good solution for transitioning from card to
disk. In the days of very smart database optimisers, putting a key on
the record and keeping all the records together in one table / member is
more useful. Rather than copy / move records from place to place,
normalise the data into a header (with status) and detail. Instead of
moving from WAIT_Q to SEND_Q, just change the status of the header from
WAIT to SEND.

I fully realise that this advice is easy to give and not so easy to
follow, as I have had to do this very thing several times, and it's
always been a struggle. Start out by creating the header file and use
that to maintain the status of each FTP session. When the send process
runs out of orders in SEND state, it can call the procedure to move the
WAIT orders to SEND; for now, that procedure will do what it always did
(copy from WAIT_Q to SEND_Q) but it'll also do one more thing: after the
copy is complete, it will change the status to SEND, at which time the
send process knows it can safely handle the order.
--
--buck

'I had nothing to offer anybody except my own confusion' - Jack Kerouac
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To post a message email: MIDRANGE-L-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org
Before posting, please take a moment to review the archives
at http://archive.midrange.com/midrange-l.
Gary Thompson
2014-10-22 15:06:40 UTC
Permalink
Raw Message
Buck,
Thank you, your suggestion for a normalized DB is spot on, and where I want to get to, but time is scarce.
Today, I'm creating one data area both FTP_ASN_1 and FTP_ASN_2 must lock/read/update/release before changing WAIT_Q member(s).
I'm depending on that data area to prevent member corruption.

-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org] On Behalf Of Buck Calabro
Sent: Wednesday, October 22, 2014 8:34 AM
To: midrange-l-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org
Subject: Re: How to avoid file corruption transferring members from one file to another
Post by Gary Thompson
I have program FTP_ASN_1 sending EDI 856 ASN documents to a remote server.
If an FTP connection can't be made to the remote, the ASN document is
copied to a member file WAIT_Q, and the ASN program continues with the
next ASN as our warehouse completes the next shipment.
We may not be able to connect to the remote FTP server for a few
minutes or some hours, possibly a day.
I have program FTP_ASN_2 pulling ASN documents that have not yet been sent
from members in a file I call SEND_Q. A separate file is used to avoid contention
with FTP_ASN_1.
When FTP_ASN_2 finds no member in SEND_Q, it stops its FTP process and
looks to copy any members from WAIT_Q to SEND_Q and then remove all
_copied_ members from SEND_Q.
My question is how to avoid 'collisions' between FTP_ASN_1 and
FTP_ASN_2 during the time members are being 'moved' from WAIT_Q to SEND_Q ?
Use a single data area and require both FTP_ASN_1 and FTP_ASN_2 to
lock the data area before adding new or copying/deleting WAIT_Q members.
Have both FTP_ASN_1 and FTP_ASN_2 get exclusive control of WAIT_Q with ALCOBJ.
Looking for any advice or comments about design pitfalls . . .
The multi-member file was a good solution for transitioning from card to disk. In the days of very smart database optimisers, putting a key on the record and keeping all the records together in one table / member is more useful. Rather than copy / move records from place to place, normalise the data into a header (with status) and detail. Instead of moving from WAIT_Q to SEND_Q, just change the status of the header from WAIT to SEND.

I fully realise that this advice is easy to give and not so easy to follow, as I have had to do this very thing several times, and it's always been a struggle. Start out by creating the header file and use that to maintain the status of each FTP session. When the send process runs out of orders in SEND state, it can call the procedure to move the WAIT orders to SEND; for now, that procedure will do what it always did (copy from WAIT_Q to SEND_Q) but it'll also do one more thing: after the copy is complete, it will change the status to SEND, at which time the send process knows it can safely handle the order.

--
--buck

'I had nothing to offer anybody except my own confusion' - Jack Kerouac
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list To post a message email: MIDRANGE-L-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org Before posting, please take a moment to review the archives at http://archive.midrange.com/midrange-l.
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To post a message email: MIDRANGE-L-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org
Before posting, please take a moment to review the archives
at http://archive.midrange.com/midrange-l.
CRPence
2014-10-22 19:52:40 UTC
Permalink
Raw Message
Post by Gary Thompson
I have program FTP_ASN_1 sending EDI 856 ASN documents to a remote
server. If an FTP connection can't be made to the remote, the ASN
document is copied to a member file WAIT_Q, and the ASN program
continues with the next ASN as our warehouse completes the next
shipment.
We may not be able to connect to the remote FTP server for a few
minutes or some hours, possibly a day.
I have program FTP_ASN_2 pulling ASN documents that have not yet been
sent from members in a file I call SEND_Q. A separate file is used
to avoid contention with FTP_ASN_1.
When FTP_ASN_2 finds no member in SEND_Q, it stops its FTP process
and looks to copy any members from WAIT_Q to SEND_Q and then remove
all _copied_ members from SEND_Q.
My question is how to avoid 'collisions' between FTP_ASN_1 and
FTP_ASN_2 during the time members are being 'moved' from WAIT_Q to
SEND_Q ?
The most conspicuous solution would be to *not* move them. Of course
they are not actually being _moved_, they are actually copied; that they
are copied instead of moved complicates the issue, per likely performing
a clearly non-atomic operation that should appear atomic to
participants, to avoid any requirement for the alluded external
synchronization. Avoiding yet another _copy_ is also a benefit.

The FTP_ASN_2 job could process only members in WAIT_Q added *prior*
to that deviation [away from the SEND_Q file], and could even operate
directly against the WAIT_Q file; send the member data, and then remove
that member when sent. If the extra _copy_ was required, that same
algorithm [using FIFO] should prevent collisions as well. The Retrieve
Member Description (RTVMBRD) enables traversing the member list by
date-order [FIFO], as well the Display File Description (DSPFD) for the
Type Of Information (TYPE) of *MBR when output is directed to an Output
File (OUTFILE).
Post by Gary Thompson
Use a single data area and require both FTP_ASN_1 and FTP_ASN_2 to
lock the data area before adding new or copying/deleting WAIT_Q
members.
As a resource external-to\separate-from the actual resource being
copied, there is a level of indirection; implicitly adding specific
complexity. However as a single entity, that might be utilized with a
lower cost, both in coding and run-time.
Post by Gary Thompson
Have both FTP_ASN_1 and FTP_ASN_2 get exclusive control of WAIT_Q
with ALCOBJ.
The Allocate Object (ALCOBJ) can only exclusively lock the data of a
*specific member*, not the file; the file and member objects will have
an *SHRRD lock, and only the data [the dataspace actually] will have an
*EXCL lock. The methods available to lock exclusively the actual
database *FILE object, are incompatible with also enabling effective
atomicity of the multi-step process; i.e. the processing would be unable
to effect Remove Member (RMVM) while that lock is held, regardless the
lock is held within the same thread\process.

However the move\copy process could lock each member [the dataspace]
separately, exclusively, and then release those locks implicitly by the
eventual RMVM request, only after all of the members had been copied.
Thus somewhat more complicated than locking a third-party object, but
the mutex is established directly on each entity\resource rather than
something elsewhere.
Post by Gary Thompson
Looking for any advice or comments about design pitfalls . . .
The idea of using members for logical separation of the data is often
a poor choice when there are concerns for concurrency\synchronization;
database members, by design, are best treated as something to persist.
As Buck suggests, the use of data _records_ is a much better approach;
logical separation by a key. Note: while keeping the multi-member
implementation, the mutex could be implemented on the first record of
data or all rows of data in any one member being processed, and could
include commitment control for an easier means to backout [rollback]
work; deleting all of the records could serve as a semaphore for
allowance of removing the member.

Given the Stream File (STMF) can similarly be both processed as data
[possibly as /records/ or as just a image\binary] by the FTP, and
processed as /objects/ operated against by the system with effective
atomicity, the STMF might be a better choice; e.g. one might expect the
operations like move and rename should appear atomic to participants
accessing the STMF. That eliminates the travails with accomplishing
what is an effect atomic member\data copy and removal of the original.

I can not vouch for the efficacy of, with regard to the effective
atomicity of, the Move Object (MOV) [aka MOVE] command, but that would
be theoretically the more appropriate choice than a copy. That feature
is implemented with a copy [effectively Copy File (CPYF); just what
might be the choice for a user data copy], but ideally would be
implemented such that concurrent operations were unable to interfere
with the /from-objects/. And even if the individual object /moved/ is
seemingly atomic, and while there is the ability to move a generic [i.e.
effectively *ALL], there is no capability to perform the multi-object
operation under commitment control, so the issue with a partially
complete operation would exist if using that function. IIRC the
documentation [help text] for the MOV command says there is no support
for /moving database members/, but my recollection was that the feature
worked as long as the CPYF feature could operate on the files, and
specifically to operate having specified *NONE for the Format Option
(FMTOPT) parameter; i.e. the record formats were identical.
--
Regards, Chuck
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To post a message email: MIDRANGE-L-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org
Before posting, please take a moment to review the archives
at http://archive.midrange.com/midrange-l.
Gary Thompson
2014-10-22 21:58:37 UTC
Permalink
Raw Message
Thank you Chuck,
I learned several things I'd no/little idea of, (happens often reading your replies).
I'm now building/testing code based on a data area that FTP_ASN_1 and FTP_ASN_2 use to avoid contention for WAIT_Q.
The member choice is typical of the many programs in our house that transfer data using FTP, so a solution using data records may come later.
I really like your MOV suggestion, especially doing something similar with STMF on the IFS, which, likewise, may come later.
Your insights into the issues with the copy/delete of members makes believe it's important to avoid contention for WAIT_Q members
between FTP_ASN_1 and FTP_ASN_2.
At one time I had the thought that the right member naming convention could avoid the need for two files . . .
Thanks again!

-----Original Message-----
From: MIDRANGE-L [mailto:midrange-l-bounces-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org] On Behalf Of CRPence
Sent: Wednesday, October 22, 2014 1:53 PM
To: midrange-l-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org
Subject: Re: How to avoid file corruption transferring members from one file to another
Post by Gary Thompson
I have program FTP_ASN_1 sending EDI 856 ASN documents to a remote
server. If an FTP connection can't be made to the remote, the ASN
document is copied to a member file WAIT_Q, and the ASN program
continues with the next ASN as our warehouse completes the next
shipment.
We may not be able to connect to the remote FTP server for a few
minutes or some hours, possibly a day.
I have program FTP_ASN_2 pulling ASN documents that have not yet been
sent from members in a file I call SEND_Q. A separate file is used to
avoid contention with FTP_ASN_1.
When FTP_ASN_2 finds no member in SEND_Q, it stops its FTP process and
looks to copy any members from WAIT_Q to SEND_Q and then remove all
_copied_ members from SEND_Q.
My question is how to avoid 'collisions' between FTP_ASN_1 and
FTP_ASN_2 during the time members are being 'moved' from WAIT_Q to
SEND_Q ?
The most conspicuous solution would be to *not* move them. Of course they are not actually being _moved_, they are actually copied; that they are copied instead of moved complicates the issue, per likely performing a clearly non-atomic operation that should appear atomic to participants, to avoid any requirement for the alluded external synchronization. Avoiding yet another _copy_ is also a benefit.

The FTP_ASN_2 job could process only members in WAIT_Q added *prior* to that deviation [away from the SEND_Q file], and could even operate directly against the WAIT_Q file; send the member data, and then remove that member when sent. If the extra _copy_ was required, that same algorithm [using FIFO] should prevent collisions as well. The Retrieve Member Description (RTVMBRD) enables traversing the member list by date-order [FIFO], as well the Display File Description (DSPFD) for the Type Of Information (TYPE) of *MBR when output is directed to an Output File (OUTFILE).
Post by Gary Thompson
Use a single data area and require both FTP_ASN_1 and FTP_ASN_2 to
lock the data area before adding new or copying/deleting WAIT_Q
members.
As a resource external-to\separate-from the actual resource being copied, there is a level of indirection; implicitly adding specific complexity. However as a single entity, that might be utilized with a lower cost, both in coding and run-time.
Post by Gary Thompson
Have both FTP_ASN_1 and FTP_ASN_2 get exclusive control of WAIT_Q
with ALCOBJ.
The Allocate Object (ALCOBJ) can only exclusively lock the data of a *specific member*, not the file; the file and member objects will have an *SHRRD lock, and only the data [the dataspace actually] will have an *EXCL lock. The methods available to lock exclusively the actual database *FILE object, are incompatible with also enabling effective atomicity of the multi-step process; i.e. the processing would be unable to effect Remove Member (RMVM) while that lock is held, regardless the lock is held within the same thread\process.

However the move\copy process could lock each member [the dataspace] separately, exclusively, and then release those locks implicitly by the eventual RMVM request, only after all of the members had been copied.
Thus somewhat more complicated than locking a third-party object, but the mutex is established directly on each entity\resource rather than something elsewhere.
Post by Gary Thompson
Looking for any advice or comments about design pitfalls . . .
The idea of using members for logical separation of the data is often a poor choice when there are concerns for concurrency\synchronization; database members, by design, are best treated as something to persist.
As Buck suggests, the use of data _records_ is a much better approach; logical separation by a key. Note: while keeping the multi-member implementation, the mutex could be implemented on the first record of data or all rows of data in any one member being processed, and could include commitment control for an easier means to backout [rollback] work; deleting all of the records could serve as a semaphore for allowance of removing the member.

Given the Stream File (STMF) can similarly be both processed as data [possibly as /records/ or as just a image\binary] by the FTP, and processed as /objects/ operated against by the system with effective atomicity, the STMF might be a better choice; e.g. one might expect the operations like move and rename should appear atomic to participants accessing the STMF. That eliminates the travails with accomplishing what is an effect atomic member\data copy and removal of the original.

I can not vouch for the efficacy of, with regard to the effective atomicity of, the Move Object (MOV) [aka MOVE] command, but that would be theoretically the more appropriate choice than a copy. That feature is implemented with a copy [effectively Copy File (CPYF); just what might be the choice for a user data copy], but ideally would be implemented such that concurrent operations were unable to interfere with the /from-objects/. And even if the individual object /moved/ is seemingly atomic, and while there is the ability to move a generic [i.e.
effectively *ALL], there is no capability to perform the multi-object operation under commitment control, so the issue with a partially complete operation would exist if using that function. IIRC the documentation [help text] for the MOV command says there is no support for /moving database members/, but my recollection was that the feature worked as long as the CPYF feature could operate on the files, and specifically to operate having specified *NONE for the Format Option
(FMTOPT) parameter; i.e. the record formats were identical.

--
Regards, Chuck
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list To post a message email: MIDRANGE-L-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org Before posting, please take a moment to review the archives at http://archive.midrange.com/midrange-l.
--
This is the Midrange Systems Technical Discussion (MIDRANGE-L) mailing list
To post a message email: MIDRANGE-L-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org
To subscribe, unsubscribe, or change list options,
visit: http://lists.midrange.com/mailman/listinfo/midrange-l
or email: MIDRANGE-L-request-Zwy7GipZuJhWk0Htik3J/***@public.gmane.org
Before posting, please take a moment to review the archives
at http://archive.midrange.com/midrange-l.
Loading...