Shopify Source Plugin

Latest: v3.1.4

The CloudQuery Shopify plugin pulls data from Shopify and loads it into any supported CloudQuery destination (e.g. PostgreSQL, BigQuery, Snowflake, and more).

Authentication

In order to fetch information from Shopify, cloudquery needs to be authenticated. Either an API key and password (in the case of basic custom/private apps) or an access token (for OAuth apps) is required for authentication.

Refer to the Shopify Help Center article on Custom apps (opens in a new tab) and create a custom app. Follow Get the API credentials for a custom app section to get the credentials for Admin API and put them in your plugin configuration as api_key and api_secret.

If you have a large or busy store, API key/secret type credentials might not be enough due to the heavy rate limiting. In this case, you can use OAuth in your custom app to get an access token which allow many more requests a second. To use that token in your plugin configuration instead, just set it in access_token and remove api_key and api_secret sections. For more information, refer to Shopify.dev (opens in a new tab) on the subject.

Incremental Syncing

The Shopify plugin supports incremental syncing. This means that only new data will be fetched from Shopify and loaded into your destination for supported tables (support depending on API endpoint). This is done by keeping track of the last item fetched and only fetching data that has been created since then. To enable this, backend option must be set in the spec (as shown below). This is documented in the Managing Incremental Tables section.

Example

This example syncs from Shopify to a Postgres destination. The (top level) source spec section is described in the Source Spec Reference. Incremental syncing is enabled and will be saved to a .cq/state/ directory by default.

kind: source
# Common source-plugin configuration
spec:
  name: shopify
  path: cloudquery/shopify
  version: "v3.1.4"
  tables: ["*"]
  destinations: ["postgresql"]
  # Shopify specific configuration
  spec:
    backend_options:
      table_name: "test_state_table"
      connection: "@@plugins.postgresql.connection"
    api_key: "<YOUR_API_KEY_HERE>"
    api_secret: "<YOUR_API_SECRET_HERE>"
    shop_url: "https://<YOUR_SHOP>.myshopify.com"
#    concurrency: 1000

The Shopify plugin supports incremental syncing for event data. This means that only new events will be fetched from Shopify and loaded into your destination. This is done by keeping track of the last event fetched and only fetching events that has been created since then. To enable this, backend option must be set in the spec (as shown in the example). This is documented in the Managing Incremental Tables section.

Configuration Reference

This is the (nested) spec used by the Shopify source plugin:

api_key (string, required*): The API Key for your custom app in your store.
api_secret (string, required*): The API Secret for your custom app in your store.
access_token (string, required if api_key/secret is not used): An access token for your Shopify custom app. This is an alternative way of authenticating, use either this or the ones above.
shop_url (string, required): The URL of your Shopify store. Must start with https:// and end with .myshopify.com.
api_version (string, optional. Default: 2023-01): The Shopify Admin API version to use. See here (opens in a new tab) for more information.
timeout_secs (integer in seconds, optional. Default: 10): Timeout for requests against the Shopify Admin API.
max_retries (integer, optional. Default: 30): Number of retries if a request was rate limited.
page_size (integer, optional. Default: 50): Maximum number of items queried each request. Find an optimum value to balance amount of data fetched and requests timing out. Maximum value 250.
concurrency (integer, optional. Default: 1000): Maximum number of concurrent requests to the Shopify Admin API.

preview backend_options (object) (default: not used)

Allowed properties are table_name and connection. Use this configuration to enable incremental syncs for supported tables. See more here. Example

kind: source
spec:
  name: shopify
  path: cloudquery/shopify
  version: "v3.1.4"
  destinations: ["postgresql"]
  spec:
    backend_options:
      table_name: "test_state_table"
      connection: "@@plugins.postgresql.connection"

Query Examples

Get all your active products with a specific tag

SELECT * FROM shopify_products WHERE status='active' AND 'your-tag' = ANY(tags);

Tables Tables