# Building Custom Integrations Without Losing Your Mind The integration demo always looks easy. "Just connect the API, map some fields, and data flows automatically." Then you start building and discover that their date format doesn't match yours, null values crash your mapper, and rate limits mean your 10,000-record sync takes four hours. Custom integrations are where optimistic architecture meets messy reality. Here's how to navigate the mess. ## The Integration Spectrum Not all integrations are equal in complexity. Understanding where yours falls helps set expectations: ### Level 1: Read-Only Data Pull (1-3 days) Fetch data from an external API, transform it, and store it. No data goes back. Examples: pulling exchange rates, importing product catalogs, syncing calendar events. ### Level 2: Bidirectional Sync (1-3 weeks) Data flows both directions. Changes in your system push to the external system, and vice versa. Conflict resolution becomes a real problem. Examples: CRM sync, inventory management, order processing. ### Level 3: Real-Time Event Processing (2-4 weeks) Events from the external system trigger immediate actions in yours, and vice versa. Requires webhook handling, event queuing, and retry logic. Examples: payment processing, shipping notifications, real-time collaboration. ### Level 4: Deep Workflow Integration (1-3 months) The external system becomes part of your business workflow. Users interact with it through your interface. Errors in the external system affect your user experience. Examples: embedded payment processing, integrated document signing, white-labeled third-party features. ## The Seven Rules of Integration Engineering ### Rule 1: Never Trust External Data Every field from an external API can be null, empty, wrong type, or missing entirely — regardless of their documentation. Validate everything at the boundary. Use runtime validation (Zod, Joi) to catch schema violations before they reach your business logic. The cost of adding input validation is measured in minutes. The cost of debugging a production crash caused by an unexpected null is measured in hours. ### Rule 2: Design for Downtime External APIs go down. Not might — will. Your integration must handle unavailability gracefully: - Queue outgoing requests and retry when the service recovers - Serve cached data when fresh data is unavailable - Degrade gracefully instead of crashing — show "data unavailable" rather than a 500 error - Alert operators when downtime exceeds expected thresholds ### Rule 3: Respect Rate Limits Most APIs enforce rate limits. Exceeding them gets your integration blocked — sometimes permanently. Implement: - Token bucket or sliding window rate limiting - Exponential backoff on 429 responses - Batch operations where the API supports them - Off-peak scheduling for bulk operations ### Rule 4: Handle Pagination APIs that return lists paginate results. Some use cursor-based pagination, some use offset-based, some use keyset pagination. Your integration must handle all of these correctly, including: - Following next-page tokens until no more results - Handling pages that return zero results (empty page isn't always the last page) - Resuming from the last successful page if the sync is interrupted ### Rule 5: Idempotent Operations Every operation in your integration should be safe to retry. If a sync runs twice, the result should be the same as running once. This means: - Use upserts instead of inserts (create or update based on external ID) - De-duplicate events based on event IDs - Track sync state per record, not per batch ### Rule 6: Log Everything, Store Context When an integration fails at 3am, your logs are the only witness. Log: - Every external API call with request/response (redacting sensitive data) - Every data transformation with before/after values - Every decision point (why was this record skipped? matched? created?) - Every error with full stack trace and context ### Rule 7: Monitor Business Metrics, Not Just Technical Ones Monitoring "API response time: 200ms" is necessary but insufficient. Also monitor: - "Records synced in the last hour: 0" (sync is broken) - "Records with sync errors: increasing" (data quality issue) - "Last successful full sync: 3 days ago" (something is stuck) ## The Connector Pattern For platforms that need multiple integrations, the connector pattern provides a consistent architecture: **Connector interface:** Every integration implements the same interface — connect, disconnect, sync, and handle events. This standardization means your team can build new integrations faster because the pattern is familiar. **Configuration over code:** Connection details (API keys, URLs, field mappings) live in configuration, not code. This lets non-developers set up new connections and modify existing ones without deploying code. **Transformation layer:** Raw external data passes through a transformation layer that normalizes it into your internal schema. The transformation rules are explicit and auditable. **Sync state management:** Track the sync state per connection — last sync timestamp, cursor position, error count. This enables resumable syncs, incremental updates, and sync health monitoring. ## Common Integration Anti-Patterns **Tight coupling to external schemas.** If your internal data model mirrors the external API's structure, a change on their end requires changes throughout your system. Transform external data into your canonical model at the boundary. **Synchronous external calls in user requests.** If a page load requires calling an external API, the user waits for that API's response time. Sync data in the background; serve from your local copy. **Ignoring webhooks.** Polling an API every 5 minutes to detect changes is wasteful. Most modern APIs offer webhooks — use them. Poll as a fallback for APIs that don't support webhooks. **Manual field mapping.** For integrations with 50+ fields, manual mapping in code is unmaintainable. Build a field mapping configuration that business users can adjust without code changes. ## Testing Integrations Integration tests are different from unit tests. You need: **Contract tests.** Verify that the external API still matches your expectations. Run against their sandbox or a recorded response set. Alert when the schema changes. **Failure simulation.** Test with timeouts, 500 errors, rate limit responses, and malformed data. Your integration should handle all of these gracefully. **Load tests.** Sync 10,000 records, not 10. Performance problems only appear at scale. **End-to-end flow tests.** Create a record in the external system, verify it appears in yours. Update it. Delete it. Verify each state change propagates correctly. ## The Realistic Timeline Whatever you estimate for an integration, multiply by 2.5. The API documentation covers 80% of the cases. The remaining 20% — edge cases, undocumented behavior, production quirks — take 60% of the time. Plan for it. Budget for it. And build the architecture that makes the next integration easier, because there's always a next integration.