Sync Bridge

Overview & Problem

The platform ecosystem consists of a legacy monolith and a new full-stack JavaScript platform built as micro-services.

Both systems are temporarily running in parallel in production until we deprecate the legacy monolith. Without a sync mechanism, data between the systems would quickly become inconsistent, leading to operational issues and increased maintenance overhead.

Solution

I built Sync Bridge, an ETL system in Node.js, using Koa, that keeps both systems in sync automatically. It handles bi-directional data flow, ensures transactional integrity, and leverages Datadog observability for production reliability.

Sync Bridge runs on a scheduled CronJob that checks for data changes every minute using the updated_at field in both databases to determine the most recent change.

System Architecture

• Sync Bridge sits at the centre, handling ETL, data transformation, and conflict resolution.
• Cronjobs trigger scheduled syncs, ensuring up-to-date consistency.
• Datadog observability provides metrics, alerting, and dashboards for system health and reliability.

Technical Challenges & Solutions

• Bi-directional sync: prevented overwriting newer updates via timestamp-based conflict resolution.
• Schema differences: transformation layer reconciles legacy legacy monolith fields with new micro-service schemas.
• Transactional safety: database transactions ensure atomic updates and prevent partial writes.
• Production reliability: observability in Datadog ensures failures are detected, tracked, and alerted immediately.

Code Snippets

// Cursor-based paginator for customer data. Each page is transformed concurrently.
// Flow is intentionally staged: identities -> groupings -> geo entities -> reconciliation.
// e.g. for (const stage of stages) await runStage(stage)

type FetchPage<T> = (cursor: string | null, size: number) => Promise<{
  rows: T[];
  nextCursor: string | null;
}>;

async function drainInPages<T, R>(
  fetchPage: FetchPage<T>,
  transform: (row: T) => Promise<R>,
  pageSize = 100,
): Promise<R[]> {
  const output: R[] = [];
  let cursor: string | null = null;

  while (true) {
    const { rows, nextCursor } = await fetchPage(cursor, pageSize);
    output.push(...(await Promise.all(rows.map((row) => transform(row)))));

    if (!nextCursor || rows.length === 0) break;
    cursor = nextCursor;
  }

  return output;
}

// Build one CronJob manifest per customer from runtime config.
// History is kept small, runs are single-shot, and the sync entrypoint is executed directly.
function makeCustomerSyncCronJob(config: CustomerConfig) {
  return {
    apiVersion: 'batch/v1',
    kind: 'CronJob',
    metadata: {
      name: \`sync-\${config.customerSlug}\`,
      labels: { /* business, customer, component: sync */ },
    },
    spec: {
      schedule: config.schedule,
      concurrencyPolicy: 'Forbid',
      failedJobsHistoryLimit: 0,
      successfulJobsHistoryLimit: 1,
      jobTemplate: {
        spec: {
          template: {
            spec: {
              restartPolicy: 'OnFailure',
              containers: [
                {
                  name: 'sync-worker',
                  image: config.image,
                  args: ['--require=dd-trace/init', 'dist/job.js'],
                  env: [ /* CUSTOMER_*, DB_*, upstream/downstream hosts */ ],
                },
              ],
            },
          },
        },
      },
    },
  };
}

Impact & Metrics

• Syncs thousands of records per run with zero data inconsistencies.
• Reduces manual reconciliation across production systems, saving hundreds of man-hours.
• Minimises urgency to remove legacy system, ensuring that we can continue to deliver new features and de-prioritise rebuilding existing services.
• Datadog dashboards provide real-time monitoring of ETL health, sync latency, and error rates.

Skills Demonstrated

• Backend system design with Node.js and Koa.
• ETL pipeline architecture.
• Database design, transactions, and indexing.
• Scheduling, automation, and failure recovery.
• Observability and production monitoring using Datadog.
• Bi-directional data synchronisation and conflict resolution.