Senior Data Engineer

About Shadow & Darkroom: Shadow is a premier, hyper-growth artificial intelligence pioneer, marketing software innovator, and digital orchestration leader purpose-built to empower small commercial teams to market and scale their businesses efficiently. Built alongside Darkroom—a highly prominent performance marketing agency with 10 years of successful industry heritage, a 100+ member digital workforce, and a diverse portfolio of over 1,000 partner consumer brands—Shadow serves as the world core AI coordination layer. The platform seamlessly unifies shared AI memory arrays, centralized autonomous agent controls, and multi-model orchestration frameworks under a single integrated dashboard. Leveraging real, high-volume marketing data from day one, Shadow enables fast-growing challenger brands doing $80M–$500M in revenue to shift from legacy workflow automation into high-performance agent management confidently across global channels.

Position Overview

We are seeking a highly analytical, systems-minded Senior Data Engineer to join our core global Shadow product team in a full-time remote capacity within Pakistan, Brazil, or India. In this hands-on, build-heavy infrastructure seat, you will step up to claim individual engineering accountability over the entire high-scale data ingestion and processing layer that pulls the world marketing information directly into Shadow. Moving entirely away from passive maintenance loops or synthetic demo building, you will engineer multi-tenant pipelines that aggregate, normalize, and synchronize extensive data flows from prominent advertising, analytics, and e-commerce nodes. This high-agency role demands a seasoned data craftsman who has operated large-scale database clusters serving 1,000+ active users, maintains uncompromising data quality standards, and designs robust structures that expose clean datasets for AI agents to reason over seamlessly.

Key Responsibilities

• Multi-Tenant Ingestion Layer Governance: Build, scale, and maintain our high-throughput data ingestion pipelines across third-party marketing and e-commerce APIs, explicitly coding for OAuth authentication, data extraction, rate-limit handling, and backfill patterns.
• Hands-On Python and SQL Modeling: Program and refactor highly optimized analytical data flows, transformation models, and custom extraction triggers natively utilizing Python scripts and advanced SQL query scripts.
• Enterprise Warehouse Schema Design: Architect, optimize, and maintain multi-tenant database systems and centralized shared schemas natively inside contemporary enterprise environments such as BigQuery, Snowflake, or Redshift.
• Data Freshness and Observability Triage: Own data pipeline reliability at scale, engineering proactive monitoring systems and tracing frameworks to detect data drift, connection dropouts, or sync inaccuracies before consumers do.
• AI Agent Retrieval Engineering: Partner alongside core AI and machine learning engineering cells to expose pristine, well-modeled data matrices that autonomous agents and retrieval mechanisms can query fluidly.
• API Integration Hardening: Integrate, test, and reinforce structural data connectors against popular enterprise platforms, including Meta, Google, TikTok, GA4, Shopify, and Klaviyo, managing pagination and schema variations.
• SOC 2 Compliance Safeguarding: Enforce enterprise-grade data security, strict multi-tenant customer isolation barriers, audit trails, and encryption standards to protect sensitive data under rigid SOC 2 compliance rules.
• Orchestration and Tooling Optimization: Configure and manage stable automation pipelines and transformation dependencies natively leveraging tools like dbt, Apache Airflow, Dagster, or PostgreSQL + pgvector databases.

Required Skills & Qualifications

• Proven professional history running advanced data engineering, enterprise pipeline architecture, backend systems development, or full-stack big data consulting.
• Demonstrated commercial experience building, operating, and cost-optimizing large data systems serving 1,000+ concurrent multi-tenant users or equivalent data storage volumes.
• Deep, authoritative technical command of relational schema composition, ETL/ELT development paradigms, and database query tuning natively using SQL and Python languages.
• Hands-on experience developing customized API connectors, explicitly handling OAuth tokens, high-volume pagination, rate-limit thresholds, and automated schema mutations.
• Practical operational familiarity managing, mapping, or partitioning sensitive data assets inside production-grade environments under real compliance guidelines like SOC 2 tracking.
• Outstanding verbal and written communication mechanics in fluent English, showcasing an absolute capacity to coordinate architectural decisions and explain technical trade-offs to senior leadership cells, CEOs, and CMOs.
• Location Context: Parameters open exclusively to qualified senior data engineers based permanently and resident within Pakistan, Brazil, or India to execute 100% remotely from home.

Preferred Strategic Indicators (Nice to Have)

• Prior professional exposure operating within martech, adtech, or an adjacent marketing domain, navigating attribution models, currency conversions, and cross-platform deduplication messes.
• Practical production experience configuring cloud-native microservices within GCP (Google Cloud Platform, Cloud Run) or utilizing tracing tools in an LLM context (e.g., Langfuse).
• An outcome-driven, entrepreneurial personal mindset that runs as a power AI user, embedding automated systems and repeatable structures into every aspect of software engineering.

What We Offer

• The exceptional professional canvas to directly direct, code-shape, and deploy the central data ingestion layers and multi-tenant schemas powering the AI coordination framework for leading consumer brands globally.
• Highly attractive, capability-benchmarked full-time baseline compensation packages calibrated to reward your data architecture authority and processing velocity.
• Profound work-from-home remote parameters offering 100% remote location options, absolute schedule trust, and zero physical office geographical commuting friction.
• Immediate immersion into a flat, fast-moving software engineering environment that rejects rigid corporate layers to grant you absolute product ownership and direct technical decision-making space.
• The unique opportunity to work with real, high-volume performance marketing records from day one, expanding your technical mastery across cutting-edge agentic AI retrieval systems.