FileWatcher Automation Setup | DataMystic Developer

FileWatcher Automation Setup

Configure FileWatcher for event-driven file processing. Monitor folders for new or changed files and automatically trigger action chains — copy, move, rename, execute external programs, FTP upload, and integrate with TextPipe, WordPipe, or ExcelPipe for automated transformations.

Prerequisites: FileWatcher installed and licensed. For service mode, administrator privileges are required for installation. Network folder monitoring requires appropriate share permissions.

Folder Monitoring Setup

FileWatcher monitors one or more folders for file system events (creation, modification, deletion, rename). Configure watched folders, file masks, and whether to monitor subdirectories.

<!-- FileWatcher configuration: Basic folder monitoring -->
<FileWatcher>
  <Watch>
    <Folder>C:\Data\Incoming</Folder>
    <Filter>*.csv;*.txt;*.dat</Filter>
    <IncludeSubfolders>true</IncludeSubfolders>
    <Events>
      <Created>true</Created>
      <Modified>true</Modified>
      <Deleted>false</Deleted>
      <Renamed>false</Renamed>
    </Events>
    <StabilityDelay>5</StabilityDelay>
    <!-- Wait 5 seconds after last write before triggering -->
  </Watch>

  <Watch>
    <Folder>\\FileServer\Departments\Finance\Drop</Folder>
    <Filter>*.xlsx;*.xlsm</Filter>
    <IncludeSubfolders>false</IncludeSubfolders>
    <Events>
      <Created>true</Created>
      <Modified>false</Modified>
      <Deleted>false</Deleted>
      <Renamed>false</Renamed>
    </Events>
    <StabilityDelay>10</StabilityDelay>
  </Watch>
</FileWatcher>

Command-Line Setup

REM Add a folder watch via command line
FileWatcher.exe /addwatch /folder:"C:\Data\Incoming" /filter:"*.csv;*.txt" /subfolders /events:create,modify /delay:5

REM Add a network folder watch
FileWatcher.exe /addwatch /folder:"\\Server\Share\DropZone" /filter:"*.xlsx" /events:create /delay:10

REM List current watches
FileWatcher.exe /listwatches

REM Remove a watch
FileWatcher.exe /removewatch /folder:"C:\Data\Incoming"

Trigger Conditions

Define conditions that must be met before actions execute. Conditions include file size thresholds, name patterns, time-of-day restrictions, and file stability checks to ensure files are fully written before processing.

<!-- FileWatcher trigger conditions -->
<TriggerConditions>
  <!-- Only process files larger than 1KB (skip empty/incomplete files) -->
  <MinFileSize>1024</MinFileSize>

  <!-- Only process during business hours -->
  <TimeWindow>
    <StartTime>06:00</StartTime>
    <EndTime>22:00</EndTime>
    <DaysOfWeek>Mon,Tue,Wed,Thu,Fri</DaysOfWeek>
  </TimeWindow>

  <!-- File must be stable (no writes) for at least 10 seconds -->
  <StabilityCheck>
    <Seconds>10</Seconds>
    <RetryInterval>2</RetryInterval>
    <MaxRetries>30</MaxRetries>
  </StabilityCheck>

  <!-- Only trigger for files matching pattern -->
  <FileNamePattern>^(DAILY|WEEKLY)_.*\.csv$</FileNamePattern>

  <!-- Skip files still being written (locked) -->
  <SkipLocked>true</SkipLocked>
</TriggerConditions>

Action Chains

Define sequences of actions that execute when triggers fire. Actions include copy, move, rename, delete, execute an external program, FTP/SFTP upload, email notification, and logging. Actions execute in order and can use variables for the triggered file path.

<!-- FileWatcher action chain: Process, archive, and notify -->
<ActionChain name="Process Incoming CSV">
  <!-- Step 1: Copy to processing folder -->
  <Action type="Copy">
    <Source>{FilePath}</Source>
    <Destination>C:\Data\Processing\{FileName}</Destination>
    <Overwrite>true</Overwrite>
  </Action>

  <!-- Step 2: Execute TextPipe transformation -->
  <Action type="Execute">
    <Program>C:\Program Files\DataMystic\TextPipe\TextPipe.exe</Program>
    <Arguments>/filter:"C:\Filters\csv_cleanup.fll" /input:"{FilePath}" /output:"C:\Data\Processed\{FileName}" /silent /overwrite</Arguments>
    <WaitForExit>true</WaitForExit>
    <Timeout>300</Timeout>
  </Action>

  <!-- Step 3: Move original to archive -->
  <Action type="Move">
    <Source>{FilePath}</Source>
    <Destination>C:\Data\Archive\{Year}\{Month}\{FileName}</Destination>
  </Action>

  <!-- Step 4: FTP upload processed file -->
  <Action type="FTP">
    <Server>ftp.partner.com</Server>
    <Port>21</Port>
    <Username>upload_user</Username>
    <Password>{ENV:FTP_PASSWORD}</Password>
    <RemotePath>/incoming/{FileName}</RemotePath>
    <LocalFile>C:\Data\Processed\{FileName}</LocalFile>
    <Mode>Binary</Mode>
  </Action>

  <!-- Step 5: Log the operation -->
  <Action type="Log">
    <File>C:\Logs\filewatcher_{Date}.log</File>
    <Message>{Timestamp} - Processed: {FileName} ({FileSize} bytes)</Message>
  </Action>
</ActionChain>

Rename Action with Pattern Variables

<!-- Rename incoming files with timestamp for deduplication -->
<ActionChain name="Timestamp and Route">
  <Action type="Rename">
    <Source>{FilePath}</Source>
    <Pattern>{FileBaseName}_{Timestamp:yyyyMMdd_HHmmss}{FileExtension}</Pattern>
  </Action>

  <Action type="Move">
    <Source>{NewFilePath}</Source>
    <Destination>C:\Data\Queued\{NewFileName}</Destination>
  </Action>
</ActionChain>

Integration with TextPipe/WordPipe/ExcelPipe

FileWatcher works as an event-driven orchestrator for other DataMystic products. When files arrive in monitored folders, FileWatcher triggers TextPipe, WordPipe, or ExcelPipe to process them automatically.

TextPipe Integration (CSV/Text Processing)

REM FileWatcher triggers TextPipe when new CSV files arrive
REM This batch file is called by FileWatcher with %1 = triggered file path

@echo off
set INPUT_FILE=%~1
set OUTPUT_DIR=C:\Data\Processed
set FILTER=C:\Filters\standardize_csv.fll
set LOG=C:\Logs\filewatcher_textpipe.log

echo [%date% %time%] Processing: %INPUT_FILE% >> "%LOG%"

TextPipe.exe /filter:"%FILTER%" /input:"%INPUT_FILE%" /output:"%OUTPUT_DIR%\%~nx1" /silent /overwrite

if %ERRORLEVEL% EQU 0 (
    echo [%date% %time%] SUCCESS: %~nx1 >> "%LOG%"
    move "%INPUT_FILE%" "C:\Data\Archive\%~nx1"
) else (
    echo [%date% %time%] FAILED: %~nx1 - Error %ERRORLEVEL% >> "%LOG%"
    move "%INPUT_FILE%" "C:\Data\Failed\%~nx1"
)

WordPipe Integration (Document Compliance)

REM FileWatcher triggers WordPipe when Word documents are dropped into compliance folder
REM Applies standard disclaimer and updates branding automatically

@echo off
set INPUT_FILE=%~1
set LOG=C:\Logs\filewatcher_wordpipe.log

echo [%date% %time%] Compliance check: %INPUT_FILE% >> "%LOG%"

REM Apply compliance disclaimer
WordPipe.exe /find:"{INSERT_DISCLAIMER}" /replace:"This document is confidential and intended for internal use only." /folder:"%~dp1" /include:"%~nx1"

REM Update branding to current standard
WordPipe.exe /find:"Old Company Name" /replace:"Current Company Name" /folder:"%~dp1" /include:"%~nx1" /headers /footers

echo [%date% %time%] Compliance applied: %~nx1 >> "%LOG%"

ExcelPipe Integration (Spreadsheet Standardization)

REM FileWatcher triggers ExcelPipe when new Excel files arrive
REM Fixes data source connections and standardizes headers

@echo off
set INPUT_FILE=%~1
set LOG=C:\Logs\filewatcher_excelpipe.log

echo [%date% %time%] Standardizing: %INPUT_FILE% >> "%LOG%"

REM Update data source connections to production server
ExcelPipe.exe /find:"Server=DEV-SQL01" /replace:"Server=PROD-SQL01" /folder:"%~dp1" /include:"%~nx1" /formulas /cells

REM Standardize department code references
ExcelPipe.exe /find:"DEPT_OLD" /replace:"DEPT_2025" /folder:"%~dp1" /include:"%~nx1" /cells

echo [%date% %time%] Standardization complete: %~nx1 >> "%LOG%"
move "%INPUT_FILE%" "C:\Data\Standardized\%~nx1"

Service Mode Installation

Install FileWatcher as a Windows service for always-on monitoring that starts automatically with the system. Service mode ensures file processing continues even when no user is logged in.

Windows Service Installation

REM Install FileWatcher as a Windows service (run as Administrator)
FileWatcher.exe /installservice

REM Install with a specific configuration file
FileWatcher.exe /installservice /config:"C:\Config\filewatcher_production.xml"

REM Start the service
net start "DataMystic FileWatcher"

REM Check service status
sc query "DataMystic FileWatcher"

REM Stop the service
net stop "DataMystic FileWatcher"

REM Uninstall the service
FileWatcher.exe /uninstallservice

PowerShell Service Management

# Install and configure FileWatcher as a Windows service
# Run as Administrator

# Install the service
& "C:\Program Files\DataMystic\FileWatcher\FileWatcher.exe" /installservice /config:"C:\Config\filewatcher_prod.xml"

# Configure service recovery options (restart on failure)
sc.exe failure "DataMystic FileWatcher" reset= 86400 actions= restart/60000/restart/60000/restart/60000

# Set service to auto-start with delayed start
Set-Service -Name "DataMystic FileWatcher" -StartupType AutomaticDelayedStart

# Start the service
Start-Service -Name "DataMystic FileWatcher"

# Verify service is running
Get-Service -Name "DataMystic FileWatcher" | Format-List Name, Status, StartType

# Monitor service health
$svc = Get-Service -Name "DataMystic FileWatcher"
if ($svc.Status -ne 'Running') {
    Write-Warning "FileWatcher service is not running! Status: $($svc.Status)"
    Start-Service -Name "DataMystic FileWatcher"
    Write-Host "Service restarted."
}

Configuration File Reference

FileWatcher uses an XML configuration file to define all watches, conditions, and actions. Below is a complete production configuration example that monitors multiple folders and triggers different processing pipelines.

<?xml version="1.0" encoding="utf-8"?>
<FileWatcherConfig>
  <Settings>
    <LogFile>C:\Logs\filewatcher_service.log</LogFile>
    <LogLevel>Info</LogLevel>
    <MaxConcurrent>4</MaxConcurrent>
    <RetryOnError>true</RetryOnError>
    <RetryCount>3</RetryCount>
    <RetryDelay>30</RetryDelay>
  </Settings>

  <!-- Watch 1: Incoming CSV data feeds -->
  <Watch name="CSV Data Feeds">
    <Folder>C:\Data\Incoming\CSV</Folder>
    <Filter>*.csv</Filter>
    <IncludeSubfolders>true</IncludeSubfolders>
    <Events>Created</Events>
    <StabilityDelay>5</StabilityDelay>
    <Actions>
      <Execute program="TextPipe.exe" args='/filter:"C:\Filters\csv_clean.fll" /input:"{FilePath}" /output:"C:\Data\Processed\{FileName}" /silent /overwrite' wait="true" timeout="120" />
      <Move source="{FilePath}" dest="C:\Data\Archive\{Year}-{Month}\{FileName}" />
      <Log message="{Timestamp} CSV processed: {FileName}" />
    </Actions>
  </Watch>

  <!-- Watch 2: Word documents for compliance -->
  <Watch name="Word Compliance">
    <Folder>\\Server\Compliance\Incoming</Folder>
    <Filter>*.docx;*.doc</Filter>
    <IncludeSubfolders>false</IncludeSubfolders>
    <Events>Created</Events>
    <StabilityDelay>10</StabilityDelay>
    <Actions>
      <Execute program="WordPipe.exe" args='/find:"DRAFT" /replace:"APPROVED" /folder:"{FileDir}" /include:"{FileName}" /headers /footers' wait="true" timeout="180" />
      <Move source="{FilePath}" dest="\\Server\Compliance\Approved\{FileName}" />
      <Log message="{Timestamp} Word compliance: {FileName}" />
    </Actions>
  </Watch>

  <!-- Watch 3: Excel reports for data source update -->
  <Watch name="Excel Reports">
    <Folder>C:\Reports\Drop</Folder>
    <Filter>*.xlsx</Filter>
    <IncludeSubfolders>false</IncludeSubfolders>
    <Events>Created</Events>
    <StabilityDelay>15</StabilityDelay>
    <Actions>
      <Execute program="ExcelPipe.exe" args='/find:"Server=DEV" /replace:"Server=PROD" /folder:"{FileDir}" /include:"{FileName}" /formulas /cells' wait="true" timeout="180" />
      <Copy source="{FilePath}" dest="C:\Reports\Published\{FileName}" overwrite="true" />
      <Log message="{Timestamp} Excel updated: {FileName}" />
    </Actions>
  </Watch>

  <!-- Watch 4: FTP upload on completion -->
  <Watch name="FTP Upload Queue">
    <Folder>C:\Data\OutboundQueue</Folder>
    <Filter>*.*</Filter>
    <IncludeSubfolders>false</IncludeSubfolders>
    <Events>Created</Events>
    <StabilityDelay>3</StabilityDelay>
    <Actions>
      <FTP server="sftp.partner.com" port="22" user="upload" keyfile="C:\Keys\partner.ppk" remotepath="/incoming/{FileName}" localfile="{FilePath}" mode="SFTP" />
      <Move source="{FilePath}" dest="C:\Data\Sent\{FileName}" />
      <Log message="{Timestamp} FTP uploaded: {FileName}" />
    </Actions>
  </Watch>
</FileWatcherConfig>

Loading Configuration at Runtime

REM Load a specific configuration file
FileWatcher.exe /config:"C:\Config\filewatcher_production.xml"

REM Validate configuration without starting monitoring
FileWatcher.exe /config:"C:\Config\filewatcher_production.xml" /validate

REM Reload configuration while service is running
FileWatcher.exe /reload

REM Export current configuration to file
FileWatcher.exe /exportconfig:"C:\Config\filewatcher_backup.xml"

Next Steps

FileWatcher Configuration Reference — Full XML schema documentation
TextPipe Batch Processing — Command-line batch automation
WordPipe Batch Processing — Document batch operations
CI/CD Integration — Pipeline automation with DataMystic tools