Triggered Batch Processing: Automate Conditional Workflows
Not every workflow should process files individually as they arrive. Triggered batch processing waits for specific conditions — time schedules, file count thresholds, or sentinel marker files — before executing operations on accumulated batches. FileWatcher supports multiple trigger types that coordinate batch operations across data sources and processing stages.
When to Use Batch Processing Over Real-Time
Individual file processing works well when each file is independent and can be handled in isolation. But many business scenarios require batch processing — operating on groups of files together. The choice between real-time and batch processing depends on several factors:
- Downstream system requirements — Some target systems expect consolidated batches rather than individual records. Database bulk loads, report generation, and accounting system imports often work in batch mode
- Processing efficiency — Aggregating files before processing reduces connection overhead, transaction costs, and resource contention compared to processing each file individually
- Data completeness — Some operations require all files from a set before proceeding. A daily reconciliation cannot run until all branch files have arrived
- Resource constraints — Processing during off-peak hours reduces impact on production systems and avoids resource contention with interactive users
- Coordination requirements — Multi-source workflows must wait for all inputs before combining them into unified outputs
FileWatcher bridges both approaches — its folder monitoring provides real-time detection, while its trigger mechanisms hold processing until batch conditions are satisfied.
Time-Based Scheduling
The simplest trigger type is time-based scheduling — executing batch operations at predetermined times regardless of how many files have accumulated:
Fixed Schedule Triggers
Configure FileWatcher to execute processing at specific times: daily at 2 AM, hourly on the half-hour, every Monday at 6 AM, or on the last business day of each month. Fixed schedules are appropriate when downstream systems expect deliveries at specific times, when processing windows are defined by operational agreements, or when resource availability is time-dependent.
Interval-Based Triggers
Rather than fixed clock times, interval triggers fire after a specified duration since the last execution: every 15 minutes, every 4 hours, or every 30 seconds. Interval triggers are useful when you want regular processing without aligning to specific clock times, or when the processing cadence needs to adapt to changing volumes.
Business Calendar Awareness
Sophisticated batch schedules account for business calendars — skip weekends, adjust for public holidays, or align with financial periods. FileWatcher's scheduling can accommodate these requirements through configurable day-of-week restrictions and exception dates that suppress or reschedule triggers on non-business days.
File Count Thresholds
File count triggers execute batch processing when a specified number of files accumulate in a monitored folder. This approach is useful when processing efficiency improves with larger batches or when downstream systems work best with consolidated inputs:
- Minimum count triggers — Wait until at least N files are present before executing. Ensures batches contain enough data to justify the processing overhead
- Maximum count limits — Trigger processing when N files accumulate regardless of other conditions, preventing unbounded growth in input folders
- Combined with time — Process when N files are ready OR when a maximum wait time expires, whichever comes first. This ensures both efficiency (batching) and timeliness (no file waits indefinitely)
FileWatcher monitors file counts in real time through its folder monitoring infrastructure, triggering batch operations the moment threshold conditions are met. The count evaluation considers file filters, so only relevant files contribute to the threshold.
Sentinel and Marker File Triggers
Sentinel files (also called marker files, trigger files, or flag files) are a coordination mechanism widely used in enterprise data processing. The appearance of a specific file signals that a set of data files is complete and ready for processing:
How Sentinel Files Work
A sending system deposits data files into a folder throughout a processing cycle. Once all files for that cycle are complete, it writes a sentinel file — typically a small or empty file with a recognisable name like READY.flag, COMPLETE.txt, or a filename containing a timestamp. FileWatcher monitors for the sentinel file rather than the data files, ensuring processing only begins when the complete set is available.
Sentinel File Patterns
Common sentinel file conventions include:
- Static name — A fixed filename like "GO" or "READY" that signals processing should begin. Simple but limited to one active batch at a time
- Timestamped name — Files like "BATCH_20250115.done" that identify which specific batch is ready. Supports multiple concurrent batches
- Content-bearing — Sentinel files containing metadata: expected file count, checksums, batch identifier, or processing instructions
- Extension change — Data files arrive with a temporary extension (.partial), and the sending system renames to the final extension (.csv) when complete. The rename event serves as the trigger
Post-Trigger Sentinel Handling
After processing triggers, the sentinel file itself needs handling. FileWatcher can delete it, rename it, move it to an archive folder, or replace it with a response sentinel that signals completion back to the sending system. Proper sentinel cleanup prevents re-triggering on subsequent polling cycles.
Multi-Source Coordination
Complex batch workflows depend on inputs from multiple sources that arrive independently. Processing cannot begin until all sources have delivered their contributions. FileWatcher coordinates multi-source batches through several mechanisms:
Multi-Folder Watching
Monitor multiple input folders simultaneously, tracking arrival status for each source. Only trigger processing when all expected sources have delivered. This pattern supports daily reconciliation workflows where branch offices, partner systems, or departmental feeds must all be present before consolidation begins.
Dependency Chaining
Batch stages can depend on completion of prior stages. FileWatcher chains batch operations so that stage 2 only executes after stage 1 completes successfully. This builds reliable multi-step batch pipelines where each stage processes the output of the previous stage.
Timeout and Escalation
When coordinating multiple sources, what happens if one source fails to deliver? FileWatcher supports timeout configurations that escalate after maximum wait periods — sending alerts, processing available files without the missing source, or executing a fallback workflow. This prevents a single missing input from blocking an entire batch pipeline indefinitely.
Batch Processing with TextPipe
Once batch conditions are met, the processing itself often involves data transformation. TextPipe integrates naturally into batch workflows — FileWatcher passes accumulated files through TextPipe filter lists for transformation, consolidation, or format conversion before delivering batch output to target systems. TextPipe's ability to process files of unlimited size makes it suitable for large consolidated batches.
Error Handling in Batch Workflows
Batch processing introduces unique error handling considerations:
- Partial batch failures — When one file in a batch fails processing, decide whether to process the remaining files or fail the entire batch. FileWatcher supports both approaches
- Trigger misfire recovery — If the system is offline when a time trigger fires, FileWatcher detects the missed trigger on restart and executes the batch
- Incomplete batch timeout — When file count thresholds are not met within expected timeframes, escalate or process the partial batch
- Idempotent processing — Design batch operations that can safely be re-executed if failures require retry, preventing duplicate processing
Industry Applications
Triggered batch processing serves numerous industry scenarios:
- Financial services — End-of-day batch processing for transaction reconciliation, statement generation, and regulatory reporting
- Healthcare — Batch claims processing triggered by file count thresholds or daily schedule cutoffs
- Retail — Overnight price updates, inventory consolidation from multiple stores, and sales reporting aggregation
- Manufacturing — Shift-end production data collection, quality metrics consolidation, and supply chain feed processing
- Logistics — Route optimisation batch runs triggered by order count thresholds or dispatch schedule times
Connecting to the Broader Workflow
Triggered batch processing works alongside other workflow automation components. Folder monitoring detects arriving files. Automated file processing handles the operations within each batch. FTP automation delivers batch output to remote destinations. And file-based business workflows show how these components combine into complete enterprise solutions.
Get Started with Triggered Batch Processing
FileWatcher makes it straightforward to configure time-based, count-based, and sentinel-based batch triggers through its visual interface. Start with a simple scheduled trigger, test your batch workflow with sample data, and progressively add sophistication as your requirements demand. For the complete guide to setting up FileWatcher, see file watcher Windows.