Data Engineer (Part-Time) - Unstructured Data Pipeline Specialist

Join Neurons Lab as a Data Engineer (part-time) on a flagship engagement with a European private investment group — a holding company with a C-level executive team, an investment/portfolio function, and an affiliated family office.

Project Overview

The programme builds one private, access-scoped context layer over the group's data, then AI skills and agents on top of it. You will build the plumbing underneath:

• Phase 1 (Capture) — turn on ingestion across calls, email, Slack, and messengers, board protocols, decks, and portfolio updates, both live and historical;
• Phase 2 (Connect) — land it all in the context layer with identity resolved, access scope attached, and lineage intact.

This is deliberately an unstructured-first data-engineering role. There is no clean warehouse to model: the raw material is transcripts, threads, attachments, and years of archives. The hard problems include entity resolution across people and entities, deduplication, incremental sync, PII handling, and keeping costs manageable at scale.

Roughly eight to ten two-week sprints overall, with your workload front-weighted to the first four or five. Allocation may flex above 0.5 FTE during Capture and Connect and settle back afterward.

Key Responsibilities

• Stand up capture by default: notetaker on every call with speaker attribution, plus ingestion from mail, Slack, and messengers — designed as opt-out, not opt-in, and reversible if the client changes their mind.
• Backfill the archive: years of historical email, Slack, board protocols, decks, and portfolio updates — parsed, deduplicated, and dated correctly.
• Build document parsing for the awkward long tail: PDFs, scanned board packs, spreadsheets, slide decks, and forwarded attachments.
• Implement identity/entity resolution: the same person across Slack handle, mail alias, and calendar invite; the same portfolio company across a deck, a mail thread, and a CRM record.
• Build chunking and embedding pipelines and load the vector + graph stores behind the ontology the architect defines.
• Implement incremental sync through the connector layer (MCP/Composio-class) — no full re-crawls, no silent drift, clear handling of edits and deletions.
• Attach access scope and provenance to every record at ingestion, ensuring permission-aware retrieval and audit downstream.
• Run PII detection, redaction, and retention logic; provide evidence to the client's security function regarding what is stored, where, and for how long.
• Orchestrate with Airflow/Step Functions; build repeatable, monitored pipelines rather than scripts, with alerting when a source stops flowing.
• Keep cost and latency under control at volume — batching, incremental embedding, storage tiering — and report the unit economics.
• Write runbooks so the client's own team can operate this after handover.