Sync Execution Engine

The sync execution engine is responsible for moving data between Otesse and connected third-party providers. It supports inbound sync (provider to Otesse), outbound sync (Otesse to provider), and bidirectional sync with configurable conflict resolution. Each sync run is tracked as a SyncExecution record with detailed counts of processed, created, updated, skipped, and failed records.

Sync Jobs

A sync job defines what data to sync, in which direction, and on what schedule. Each job is tied to a specific integration connection and entity type.

Creating a Sync Job

From the connected app detail view, click "Add Sync Job" and configure:

Field	Description	Options
Name	Descriptive name for the job	Free text (e.g., "Stripe Customer Sync")
Entity Type	What kind of data to sync	Depends on provider (customer, invoice, product, etc.)
Direction	Which way data flows	Inbound, Outbound, or Bidirectional
Mode	Full sync or incremental	Full (all records) or Incremental (changed since last run)
Schedule	When the job runs	Manual only, or a cron schedule
Retry on Failure	Whether to automatically retry	Yes/No (default: Yes)
Max Retries	Maximum retry attempts	1-10 (default: 3)
Timeout	Maximum execution time	1-1440 minutes (default: 60)

Sync Directions

Direction	Behavior
Inbound	Reads data from the provider and creates/updates records in Otesse
Outbound	Reads records from Otesse and creates/updates records at the provider
Bidirectional	Syncs in both directions. Requires a conflict resolution strategy

Sync Modes

Mode	Behavior	Best For
Full	Processes all records regardless of modification date	Initial sync, reconciliation after errors
Incremental	Only processes records modified since the last successful run	Regular scheduled syncs, efficiency

Execution Flow

When a sync job runs (manually, on schedule, or triggered by a webhook), the engine executes the following steps:

1. Create Execution Record

A SyncExecution record is created with status "running" and all counters initialized to zero. This record tracks the progress and outcome of the run.

2. Fetch Source Records

For inbound sync:

Decrypt stored credentials
Call the provider's API to list records of the configured entity type
Apply incremental filter if mode is incremental (e.g., ?updated_since={lastRunAt})
Page through results using the provider's pagination (default page size: 100)

For outbound sync:

Query the local Otesse database for records of the configured entity type
Apply incremental filter: records modified since lastRunAt
Batch records (default batch size: 100)

3. Process Each Record

For each source record in the batch:

Look up external reference — Check if an ExternalReference exists linking this entity to the provider. If found, this is an update. If not, this is a create.

Apply field mappings — For each FieldMappingRule, read the source field, apply any transformation, apply default values for null fields, and write to the target field.

Detect conflicts (bidirectional only) — If both sides modified the same record since the last sync, apply the configured conflict resolution strategy.

Execute the write — Create or update the target record. For inbound creates, also create an ExternalReference linking the local and provider records.

Increment counters — Track created, updated, skipped, and failed counts.

4. Complete Execution

After all records are processed:

Set the execution status based on results
Record completedAt and durationSeconds
Update the sync job's lastRunAt and calculate nextRunAt (if scheduled)

Execution Status Resolution

Condition	Status	Meaning
Zero failures	`completed`	All records processed successfully
Some failures, some successes	`completedwitherrors`	Partial success
All records failed	`failed`	Complete failure
Circuit breaker triggered	`failed`	Halted due to >50% failure rate (after 10+ records)
Timeout exceeded	`timed_out`	Execution exceeded the configured timeout
User or system cancelled	`cancelled`	Manually stopped or connection lost

Trigger Types

Sync jobs can be triggered three ways:

Trigger	How It Works
Manual	User clicks "Run Now" on the sync job. Executes immediately.
Scheduled	Cron scheduler fires at the configured interval (e.g., every 15 minutes, daily at midnight).
Webhook	An inbound webhook event matches the sync job's entity type. Only the affected record is synced (single-record sync for near-real-time updates).

Circuit Breaker

The engine includes a circuit breaker to prevent runaway failures:

If more than 50% of processed records fail AND at least 10 records have been processed, the execution is halted early
The status is set to "failed" with the circuit breaker reason noted
This prevents a bad API key or schema mismatch from generating thousands of error records

Monitoring Sync Jobs

The sync job detail view shows:

Section	Content
Configuration	Name, entity type, direction, mode, schedule, retry settings
Execution History	Table of past runs with status, duration, record counts, and trigger type
Execution Detail	Expandable view with error logs, record breakdown pie chart, and individual failure details
Dead Letter Queue	Records that have failed across multiple execution attempts (failed 3+ times)