← All docs

Onboarding: connect your data

shaft runs the ingestion agent on our infrastructure. To let it read from your database and write to your BigQuery, you grant it access from your side — no software to install, and no secret keys ever leave your environment. You can revoke everything at any time.

There are two grants: (1) let our agent reach your source database, and (2) let our service account write to your BigQuery. Both are below.

1. Allow shaft's IP to reach your database

Our agent connects outbound from a fixed IP address:

34.185.215.106

Add it to your database's allowlist (/32 = just this one address). We'll notify you in advance if it ever changes.

Cloud SQL (Postgres / MySQL) Instance → Connections → Networking → Authorized networks → add:

34.185.215.106/32

Self-managed Postgres / MySQL

Prefer not to expose the database to the internet? shaft also supports an SSH tunnel or the shaft proxy — see those docs and skip this step.

2. Create a database user for shaft

Create a dedicated login with read-only access to the tables you want to sync:

CREATE USER shaft WITH PASSWORD 'a-strong-password';
GRANT CONNECT ON DATABASE yourdb TO shaft;
GRANT USAGE ON SCHEMA public TO shaft;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO shaft;

For change-data-capture (CDC / streaming) sources, the user also needs replication and logical decoding:

You'll enter the connection URL and this user/password when you create the source connector in the portal — or authenticate with shaft's service account instead of a password (see the next section).

Keyless option: authenticate with shaft's service account (IAM)

For Cloud SQL Postgres you can skip the password entirely: shaft logs in with its own Google service account using a short-lived IAM token — nothing secret is stored. Choose Service account identity (IAM) as the auth method on the source connector, then set this up on your instance:

  1. Enable IAM authentication — set the instance flag cloudsql.iam_authentication=on.
  2. Grant shaft's service account the Cloud SQL roles in your project: 
    gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member=serviceAccount:shaft-agent@coreshaft-prod.iam.gserviceaccount.com \
  --role=roles/cloudsql.client
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member=serviceAccount:shaft-agent@coreshaft-prod.iam.gserviceaccount.com \
  --role=roles/cloudsql.instanceUser
  3. Create the IAM database user for it, then grant it read access: 
    gcloud sql users create shaft-agent@coreshaft-prod.iam.gserviceaccount.com \
  --instance=YOUR_INSTANCE --type=cloud_iam_service_account
    Cloud SQL registers the DB role without the .gserviceaccount.com suffix, so connect to the database and grant that name (add REPLICATION too if you'll use streaming/CDC): 
    GRANT CONNECT ON DATABASE yourdb TO "shaft-agent@coreshaft-prod.iam";
GRANT USAGE ON SCHEMA public TO "shaft-agent@coreshaft-prod.iam";
GRANT SELECT ON ALL TABLES IN SCHEMA public TO "shaft-agent@coreshaft-prod.iam";
ALTER ROLE "shaft-agent@coreshaft-prod.iam" WITH REPLICATION;   -- for CDC
  4. In the connector, enter shaft-agent@coreshaft-prod.iam as the Database user / service account. No password is asked for.

IAM works for both query-based sync and CDC — verified on Cloud SQL Postgres with cloudsql.logical_decoding + cloudsql.iam_authentication on. For streaming (CDC), grant the IAM role the REPLICATION attribute (the ALTER ROLE … WITH REPLICATION line above). You still authorize the IP (section 1).

3. Grant shaft access to your BigQuery

shaft writes to BigQuery as its own Google service account:

shaft-agent@coreshaft-prod.iam.gserviceaccount.com

Grant that service account two roles in your project — nothing is shared with us, you're only authorizing our identity:

| Role | Where | Why | |------|-------|-----| | roles/bigquery.jobUser | project | run the load/insert jobs | | roles/bigquery.dataEditor | the target dataset | write rows into your tables |

With gcloud:

# 1) job runner (project level)
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:shaft-agent@coreshaft-prod.iam.gserviceaccount.com" \
  --role="roles/bigquery.jobUser"

# 2) write access (dataset level — least privilege)
bq add-iam-policy-binding \
  --member="serviceAccount:shaft-agent@coreshaft-prod.iam.gserviceaccount.com" \
  --role="roles/bigquery.dataEditor" \
  YOUR_PROJECT_ID:YOUR_DATASET

In the Cloud Console: IAM & Admin → Grant access → paste the service account email → add the two roles above.

Also make sure the BigQuery API is enabled in your project (bigquery.googleapis.com — on by default for most projects) and the target dataset exists. shaft creates the target table for you if it's missing:

Prefer stricter isolation? (impersonation)

If your policy doesn't allow granting an external service account directly, you can keep the identity entirely on your side:

  1. Create a service account in your project and give it the two BigQuery roles above.

  2. Grant our service account permission to impersonate it:

    gcloud iam service-accounts add-iam-policy-binding \
  YOUR_SA@YOUR_PROJECT_ID.iam.gserviceaccount.com \
  --member="serviceAccount:shaft-agent@coreshaft-prod.iam.gserviceaccount.com" \
  --role="roles/iam.serviceAccountTokenCreator"

  3. Tell us that service account's email — we set it on your BigQuery connector, and shaft writes as your own identity.

Checklist

Once these are in place, create your source and BigQuery connectors in the portal, pair them into a pipeline, and shaft starts moving your data. To stop access, remove the IP from the allowlist and the IAM bindings — nothing else lives in your environment.

More guides