From shipshitdev-library
Builds resilient data ingestion pipelines for paginated REST APIs like Twitter or exchanges, tracking progress with watermarks, avoiding duplicates, handling rate limits, and enabling incremental updates plus historical backfills.
npx claudepluginhub shipshitdev/skillsThis skill uses the workspace's default tool permissions.
Build data pipelines that never lose progress and never re-fetch existing data.
Provides UI/UX resources: 50+ styles, color palettes, font pairings, guidelines, charts for web/mobile across React, Next.js, Vue, Svelte, Tailwind, React Native, Flutter. Aids planning, building, reviewing interfaces.
Fetches up-to-date documentation from Context7 for libraries and frameworks like React, Next.js, Prisma. Use for setup questions, API references, and code examples.
Analyzes competition with Porter's Five Forces, Blue Ocean Strategy, and positioning maps to identify differentiation opportunities and market positioning for startups and pitches.
Build data pipelines that never lose progress and never re-fetch existing data.
Track TWO cursors to support both forward and backward fetching:
| Watermark | Purpose | API Parameter |
|---|---|---|
newest_id | Fetch new data since last run | since_id |
oldest_id | Backfill older data | until_id |
A single watermark only fetches forward. Two watermarks enable:
newest_id)oldest_id)These are different operations with different timing:
| What | When to Save | Why |
|---|---|---|
| Data records | After EACH page | Resilience: interrupted on page 47? Keep 46 pages |
| Watermarks | ONCE at end of run | Correctness: only commit progress after full success |
fetch page 1 → save records → fetch page 2 → save records → ... → update watermarks
First run (no watermarks)?
├── YES → Full fetch (no since_id, no until_id)
└── NO → Backfill flag set?
├── YES → Backfill mode (until_id = oldest_id)
└── NO → Update mode (since_id = newest_id)
This pattern works best with ID-based pagination (numeric IDs that can be compared). For other pagination types:
| Type | Adaptation |
|---|---|
| Cursor/token | Store cursor string instead of ID; can't compare numerically |
| Timestamp | Use last_timestamp column; compare as dates |
| Offset/limit | Store page number; resume from last saved page |
See references/patterns.md for schemas and code examples.