Synchronization Timing and Performance Monitoring

The OpenConnector SynchronizationService now includes comprehensive timing measurements for external-to-internal synchronization operations. This feature provides detailed insights into performance bottlenecks and helps optimize synchronization processes.

Overview

The timing system tracks the duration of each stage in the synchronization process, providing:

• Stage-by-stage timing: Detailed breakdown of time spent in each synchronization phase
• Object-level metrics: Individual processing times for each synchronized object
• Performance analytics: Statistical analysis including averages, medians, and efficiency ratios
• Bottleneck identification: Automatic detection of the slowest stages
• Rate limiting awareness: Tracking of rate limit impacts on performance

Timing Stages

1. Configuration and Validation

• Purpose: Loading synchronization configuration and validating source settings
• Includes:
  • Configuration dot notation processing
  • Source mapper operations
  • Validation checks
• Typical Duration: 1-10ms

2. Fetch Objects

• Purpose: Retrieving objects from the external source
• Includes:
  • API calls to external sources
  • Pagination handling
  • Rate limit management
  • Response parsing
• Typical Duration: 100-5000ms (varies by source and object count)

3. Object Preparation

• Purpose: Preparing the fetched object list for processing
• Includes:
  • Object counting
  • Result position handling
  • Data structure preparation
• Typical Duration: 1-5ms

4. Process Objects

• Purpose: Core synchronization logic for individual objects
• Includes:
  • Object validation
  • Mapping execution
  • Contract synchronization
  • Target updates
• Typical Duration: 50-500ms per object

5. Cleanup Invalid

• Purpose: Removing objects that no longer exist in the source
• Includes:
  • Contract comparison
  • Orphaned object deletion
  • Database cleanup
• Typical Duration: 10-100ms

6. Follow-ups

• Purpose: Executing follow-up synchronizations
• Includes:
  • Follow-up synchronization execution
  • Chained synchronization handling
• Typical Duration: Varies by follow-up complexity

Timing Data Structure

The timing information is stored in the synchronization log result under the timing key:

{
  'timing': {
    'stages': {
      'configuration_validation': {
        'duration_ms': 2.45,
        'description': 'Configuration loading and source validation'
      },
      'fetch_objects': {
        'duration_ms': 1250.67,
        'description': 'Fetching objects from external source',
        'objects_fetched': 25,
        'rate_limited': false
      },
      'object_preparation': {
        'duration_ms': 1.23,
        'description': 'Object list preparation and counting',
        'final_object_count': 25
      },
      'process_objects': {
        'duration_ms': 3456.78,
        'description': 'Processing and synchronizing individual objects',
        'objects_processed': 25,
        'average_per_object_ms': 138.27,
        'min_object_ms': 45.12,
        'max_object_ms': 567.89,
        'median_object_ms': 125.45
      },
      'cleanup_invalid': {
        'duration_ms': 45.67,
        'description': 'Deleting invalid/orphaned objects',
        'objects_deleted': 3
      },
      'follow_ups': {
        'duration_ms': 234.56,
        'description': 'Executing follow-up synchronizations',
        'follow_ups_executed': 2
      }
    },
    'total_ms': 4990.36,
    'summary': {
      'slowest_stage': {
        'name': 'process_objects',
        'duration_ms': 3456.78,
        'description': 'Processing and synchronizing individual objects'
      },
      'efficiency_ratio': 0.6925,
      'objects_per_second': 5.01
    }
  }
}

Performance Metrics

Object Processing Statistics

For the core object processing stage, detailed statistics are provided:

• Average per object: Mean processing time per object
• Minimum time: Fastest object processing time
• Maximum time: Slowest object processing time
• Median time: Middle value of all processing times (less affected by outliers)

Efficiency Ratio

The efficiency ratio indicates how much time was spent on actual object processing versus overhead:

Efficiency Ratio = Process Objects Duration / Total Duration

• 0.8-1.0: Highly efficient, minimal overhead
• 0.6-0.8: Good efficiency, reasonable overhead
• 0.4-0.6: Moderate efficiency, some optimization possible
• 0.0-0.4: Low efficiency, significant overhead

Objects per Second

Throughput metric showing how many objects are processed per second:

Objects per Second = Total Objects / (Total Duration in seconds)

Performance Analysis

Identifying Bottlenecks

1. Check the slowest stage: The slowest_stage in the summary identifies the primary bottleneck
2. Analyze object processing variance: Large differences between min/max object times indicate inconsistent performance
3. Review efficiency ratio: Low ratios suggest too much overhead relative to actual processing

Common Performance Patterns

API-Bound Synchronization

{
  'fetch_objects': { 'duration_ms': 4000 },
  'process_objects': { 'duration_ms': 500 },
  'efficiency_ratio': 0.11
}

Solution: Optimize API calls, implement caching, or use pagination

Processing-Bound Synchronization

{
  'fetch_objects': { 'duration_ms': 200 },
  'process_objects': { 'duration_ms': 3000 },
  'efficiency_ratio': 0.88
}

Solution: Optimize mapping logic, reduce database queries, or implement async processing

Rate-Limited Synchronization

{
  'fetch_objects': { 
    'duration_ms': 10000,
    'rate_limited': true 
  }
}

Solution: Implement exponential backoff, reduce request frequency, or use multiple sources

Monitoring and Alerting

Performance Thresholds

Set up monitoring based on these recommended thresholds:

• Total Duration: Alert if > 30 seconds for typical synchronizations
• Objects per Second: Alert if < 1 object/second for simple synchronizations
• Efficiency Ratio: Alert if < 0.3 for processing-heavy synchronizations
• Individual Object Time: Alert if max > 5 seconds for single objects

Log Analysis

Use the timing data for:

1. Trend Analysis: Track performance over time
2. Capacity Planning: Estimate resource needs for larger datasets
3. Optimization Validation: Measure improvement after changes
4. Error Correlation: Link performance degradation to specific issues

Optimization Strategies

Based on Timing Data

High Fetch Time

• Implement request batching
• Use compression
• Optimize query parameters
• Consider caching strategies

High Processing Time

• Optimize mapping logic
• Reduce database queries
• Implement parallel processing
• Use more efficient data structures

High Cleanup Time

• Optimize deletion queries
• Batch delete operations
• Index relevant database columns
• Consider soft deletes

Inconsistent Object Times

• Identify problematic object patterns
• Optimize conditional logic
• Implement object-specific optimizations
• Consider async processing for complex objects

Example Analysis

Sample Timing Output

{
  'timing': {
    'total_ms': 2500,
    'stages': {
      'fetch_objects': { 'duration_ms': 800 },
      'process_objects': { 
        'duration_ms': 1500,
        'average_per_object_ms': 75,
        'max_object_ms': 200
      }
    },
    'summary': {
      'efficiency_ratio': 0.6,
      'objects_per_second': 8
    }
  }
}

Analysis

• Good throughput: 8 objects/second is reasonable
• Balanced load: 60% efficiency shows good balance between fetch and processing
• Potential outlier: Max object time of 200ms vs average of 75ms suggests some objects are more complex

Recommendations

1. Investigate objects taking >150ms to process
2. Consider optimizing the fetch stage (32% of total time)
3. Monitor for consistency in future runs

Integration with Monitoring Tools

Logging Integration

The timing data can be integrated with logging systems:

// Example: Log performance metrics
$timingData = $log->getResult()['timing'];
$logger->info('Synchronization completed', [
    'total_ms' => $timingData['total_ms'],
    'objects_processed' => $timingData['stages']['process_objects']['objects_processed'],
    'efficiency_ratio' => $timingData['summary']['efficiency_ratio'],
    'slowest_stage' => $timingData['summary']['slowest_stage']['name']
]);

Metrics Export

Export key metrics to monitoring systems:

• synchronization.duration.total
• synchronization.duration.fetch
• synchronization.duration.process
• synchronization.efficiency_ratio
• synchronization.objects_per_second

Best Practices

Performance Monitoring

1. Establish Baselines: Record typical performance for different synchronization types
2. Set Realistic Thresholds: Base alerts on historical data and business requirements
3. Monitor Trends: Look for gradual performance degradation over time
4. Correlate with Changes: Link performance changes to code or configuration updates

Optimization Workflow

1. Measure First: Always collect timing data before optimizing
2. Focus on Bottlenecks: Optimize the slowest stages first
3. Test Incrementally: Make small changes and measure impact
4. Validate Improvements: Ensure optimizations actually improve performance

Documentation

1. Record Optimizations: Document what changes improved performance
2. Share Insights: Communicate performance patterns across the team
3. Update Thresholds: Adjust monitoring thresholds as performance improves
4. Plan Capacity: Use timing data for infrastructure planning