Add Klaviyo Ingestion

⚠️ CRITICAL: This command generates ALL files at ONCE using exact templates

I'll create a complete Klaviyo ingestion setup based on proven templates from docs/sources/klaviyo.md.

---

What I'll Generate

MANDATORY: All files created in ONE response

I will generate ALL of the following files in a SINGLE response using multiple Write tool calls:

Workflow Files

1. ingestion/klaviyo_ingest_inc.dig - Incremental ingestion workflow
2. ingestion/klaviyo_ingest_hist.dig - Historical backfill workflow

Configuration Files (in ingestion/config/)

3. klaviyo_datasources.yml - Datasource definitions for all objects
4. klaviyo_profiles_load.yml - Profiles configuration
5. klaviyo_events_load.yml - Events configuration
6. klaviyo_campaigns_load.yml - Campaigns configuration
7. klaviyo_lists_load.yml - Lists configuration
8. klaviyo_email_templates_load.yml - Email templates configuration
9. klaviyo_metrics_load.yml - Metrics configuration

Credentials Template

10. Updated credentials_ingestion.json with Klaviyo credentials section

Total: 10 files created in ONE response

---

Prerequisites

Please provide the following information:

Required

1. Klaviyo API Key: Your Klaviyo private API key (will be stored as secret)
2. TD Authentication ID: Treasure Data authentication ID for Klaviyo connector (e.g., klaviyo_auth_default)
3. Default Start Date: Initial historical load start date
   • Format: YYYY-MM-DDTHH:MM:SS.000000
   • Example: 2023-09-01T00:00:00.000000

Optional

4. Target Database: Default is client_src (leave blank to use default)

---

Process I'll Follow

Step 1: Read Klaviyo Documentation (MANDATORY)

I will READ these documentation files BEFORE generating ANY code:

• docs/sources/klaviyo.md - Klaviyo exact templates
• docs/patterns/workflow-patterns.md - Workflow patterns
• docs/patterns/logging-patterns.md - Logging templates
• docs/patterns/timestamp-formats.md - Klaviyo timestamp format (.000000)
• docs/patterns/incremental-patterns.md - Dual field names for campaigns

Step 2: Generate ALL 10 Files in ONE Response

Using multiple Write tool calls in a SINGLE message:

• Write workflow files (2 files)
• Write datasource config (1 file)
• Write load configs (6 files)
• Write credentials template update (1 file)

Step 3: Copy Exact Templates

Templates will be copied character-for-character from documentation:

• Klaviyo-specific timestamp format: .000000 (6 decimals, NO Z)
• Dual incremental fields for campaigns: updated_at in table, updated in API
• Events with NO incremental field parameter
• Exact SQL logging blocks
• Exact error handling blocks

Step 4: Verify Quality Gates

Before delivering, I will verify: ✅ All 10 files created ✅ Klaviyo timestamp format: .000000 (6 decimals, NO Z) ✅ Campaigns dual field names correct ✅ Events config has NO incremental_field parameter ✅ All logging blocks present (start, success, error) ✅ All error handling blocks present ✅ Parallel processing with limit: 3 ✅ COALESCE fallback to historical table

---

Klaviyo-Specific Configuration

Objects Included

1. Profiles: Customer profiles (incremental: updated)
2. Events: Customer events (NO incremental field)
3. Campaigns: Email campaigns (incremental: updated_at in table, updated in API)
4. Lists: Email lists (incremental: updated)
5. Email Templates: Campaign templates (incremental: updated)
6. Metrics: Event metrics (incremental: updated)

Key Features

• Dual incremental fields: Campaigns use different field names in table vs API
• Events handling: No incremental parameter in config
• Timestamp format: .000000 (6 decimals, NO Z suffix)
• Parallel processing: Limit of 3 for API rate limits
• Fallback logic: COALESCE from incremental → historical → default

---

After Generation

1. Upload Credentials

# Navigate to your ingestion directory
cd ingestion/
td wf secrets --project ingestion --set @credentials_ingestion.json

2. Test Syntax

td wf check klaviyo_ingest_inc.dig
td wf check klaviyo_ingest_hist.dig

3. Run Historical Backfill (First Time)

td wf run klaviyo_ingest_hist.dig

4. Run Incremental (Ongoing)

td wf run klaviyo_ingest_inc.dig

5. Monitor Ingestion

SELECT * FROM client_src.ingestion_log
WHERE source_name LIKE 'klaviyo%'
ORDER BY time DESC
LIMIT 20

---

Production-Ready Guarantee

All generated code will:

• ✅ Follow exact Klaviyo templates from docs/sources/klaviyo.md
• ✅ Use correct timestamp format (.000000)
• ✅ Handle dual incremental fields correctly
• ✅ Include all 6 data objects
• ✅ Include comprehensive logging and error handling
• ✅ Work the first time without modifications

---

Ready to proceed? Provide the required information (API key, TD auth ID, start date) and I'll generate all 10 files in ONE response using exact templates from documentation.