Skip to content

Instantly share code, notes, and snippets.

@harimohanraj89

harimohanraj89/heroku-connect-integration.md

Last active November 25, 2018 10:11
Show Gist options
  • Save harimohanraj89/d9c5f31220993b207a48 to your computer and use it in GitHub Desktop.
Save harimohanraj89/d9c5f31220993b207a48 to your computer and use it in GitHub Desktop.
Download ZIP
Getting to the Core of Heroku Connect
Raw
heroku-connect-integration.md

Getting to the C.O.R.E of

Heroku Connect

Hari Mohanraj

  • SFDC Data Mover vs Heroku Connect
  • Design and Implementation
  • Hurdles and Learnings
  • What's Next

Goal

Admissions producers need access to up-to-date data from Core to sell our long-form courses to prospective students.

Do It Better Than We Currently Are

  • Accuracy and Currency
  • Maintainability
  • Bi-directional Data Movement

Team

  • Michelle
  • Hopkins, Hari ft. Cory
  • Dan Ross, John Walgren
  • GetStuffDone (?)
  • Salesforce (??)
  • Authors of Salesforce plugins (???)

The Old Days - SFDC Data Mover

How It Works:

  • Executes periodic jobs
  • Reaches into Core's tables
  • Interacts directly with Salesforce API

Issues:

  • Tight coupling to Core database
  • ETL code is confusing
    • Limited understanding of the application
  • Fatal errors are a regular occurrence
    • And debugging is difficult
  • Sub-par performance

The New Days - Heroku Connect

"...bi-directional synchronization between Salesforce and Heroku Postgres..."

Heroku Connect Dashboard

Heroku Connect

  • Authorize Heroku Connect (HC) to hook into a Salesforce instance
  • Use HC to create staging tables in your app's DB to mirror Salesforce tables
  • Put data into your staging tables, and HC will send it to Salesforce
  • Change data in Salesforce, and HC will pull it down to your staging tables
  • Last update wins

Wins (with diagram)

  • Staging tables reduce coupling with Core's database
  • Brand new code is less confusing (for now)
  • HC abstracts away interaction with Salesforce
  • Logs, metrics, errors are easier to access through HC interface

Heroku Connect Tables

Core Model Table Name
Metro sfdc.metro__c
Campus sfdc.campus__c
User sfdc.account
Lead sfdc.interest__c
Program sfdc.product2
Instance sfdc.date__c
Invoice sfdc.invoice__c
Enrollment sfdc.enrollment__c

Design

Heroku Connect Integration Design

Code

"I just changed, can I be synced to Salesforce please?"

Heroku Connect Core Model

Code

"Worry not. I'll ask the right service object to do so."

Heroku Connect SFDC Manager

Code

"You're a Core Metro, right? Let me turn you into a Salesforce Metro and load you into our staging table."

Heroku Connect SFDC Mapper

Deployment

  • Synchronized switch-flipping with SFDC Data Mover
  • One table at a time

Hurdles

  • External dependencies and influences
  • Heroku Connect is still young
  • Lose control of your schema
  • Data parity between production and staging

External dependencies and influences

  • Salesforce Data Model
  • GSD - an external team
  • Salesforce - Business interfering with technology
  • Salesforce plugin authors

Fun Anecdote: Update Hell

Heroku Connect is still young

  • Cleaning up after itself
  • Don't trust the UI

Lose control of your schema

  • Heroku Connect takes the liberty of modifying your database schema
  • No artifact of these changes in codebase

Fun Anectode: Core DB Clone Wars

Data parity between production and staging

  • Core Staging maintains data parity with Core Production
  • Salesforce Staging (sandbox) does not
  • Selective parity between Core Staging and Core Production

Other Hurdles

  • Technical debt in our SFDC data model = 45k Honeybadger Errors
  • after_commit plus double-tapping = race condition

What's Next

  • Shame!
    • ETL in one object
    • after_commit
    • Metrics
  • A separate service
  • Two-way data flow

That's All Folks

References

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment