palantir-ci-integration

Palantir CI Integration

Overview

Set up GitHub Actions CI pipelines for Foundry integrations. Covers running transform unit tests with PySpark, SDK integration tests with mocked APIs, and linting Foundry-specific patterns.

Prerequisites

• GitHub repository with Foundry integration code
• foundry-platform-sdk in requirements
• pytest test suite

Instructions

Step 1: GitHub Actions Workflow

# .github/workflows/foundry-ci.yml
name: Foundry CI
on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"
          cache: pip

      - name: Set up Java (for PySpark)
        uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: "11"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run unit tests
        run: pytest tests/ -v --tb=short --junitxml=test-results.xml

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: test-results
          path: test-results.xml

  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install ruff
      - run: ruff check src/ tests/

  integration:
    runs-on: ubuntu-latest
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    needs: [test, lint]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install -r requirements.txt
      - name: Run integration smoke test
        env:
          FOUNDRY_HOSTNAME: ${{ secrets.FOUNDRY_HOSTNAME }}
          FOUNDRY_CLIENT_ID: ${{ secrets.FOUNDRY_CLIENT_ID }}
          FOUNDRY_CLIENT_SECRET: ${{ secrets.FOUNDRY_CLIENT_SECRET }}
        run: python scripts/smoke_test.py

Step 2: Secret Configuration

# Add secrets to GitHub repository
gh secret set FOUNDRY_HOSTNAME --body "mycompany.palantirfoundry.com"
gh secret set FOUNDRY_CLIENT_ID --body "your-client-id"
gh secret set FOUNDRY_CLIENT_SECRET --body "your-client-secret"

Step 3: Custom Linting Rules for Foundry

# scripts/lint_foundry.py — catch common Foundry mistakes
import ast, sys

class FoundryLinter(ast.NodeVisitor):
    def visit_Str(self, node):
        # Flag hardcoded Foundry hostnames
        if "palantirfoundry.com" in node.s:
            print(f"  Line {node.lineno}: Hardcoded Foundry hostname — use env var")
        # Flag hardcoded RIDs
        if node.s.startswith("ri.foundry.main"):
            print(f"  Line {node.lineno}: Hardcoded RID — use config/env var")

for path in sys.argv[1:]:
    tree = ast.parse(open(path).read())
    FoundryLinter().visit(tree)

Output

• GitHub Actions workflow with unit tests, linting, and integration tests
• PySpark tests running in CI with JDK setup
• Secrets configured securely in GitHub
• Custom linting for Foundry-specific patterns

Error Handling

CI Issue	Cause	Fix
PySpark tests fail	No JDK	Add setup-java step
Integration test 401	Bad secrets	Re-set gh secret set
Slow tests	Full Spark startup	Use local[1] master
Import errors	Missing deps	Pin all deps in requirements.txt

Resources

• GitHub Actions
• PySpark Testing

Next Steps

For deployment pipelines, see palantir-deploy-integration.