Production Issue: Drop ship order line workflow issue

 Recently I closed a case of troubleshooting work flow issue in a drop ship order line. The order went through an obsolete workflow and get invoiced and closed while some holds exist, which is absolutely impossible.

 

In wf_activity_status WF_ITEM_ACTIVITY_STATUSES, the 'eligiblity' is in 'COMPLETE' status while it should return on_hold. after checking the hold table, the exact hold is there. Later, we found out it's the standard purge and archive program that purge the holds but restored holds 6 hours later. It's just in this 6 hours, these Drop ship orders got processed and workflow got pushed forward. The trap in the  troubleshooting is the wf_item_activity_stauses table, the wf engine always update the last 'eligibility' activity and records the latest timestamp , thus it's difficult to know whether it's triggered hundreds of times in the past and completed at last.