AI Systems & Core Experience Engineer

What You’ll Do

• Manage member-facing AI behavior across prompt-to-create, agentic editing, and future jobs such as segmentation, scheduling, analytics, and recommendations.
• Create, test, version, and document prompts and agent instructions, with a clear record of what changed, why it changed, and how behavior improved.
• Define how the system should use context, member data, tools, and product state. Build and prototype the agent loops, tools, and structured interfaces needed to make those behaviors work.
• Own Flodesk’s AI evaluation practice, including representative datasets, behavioral scenarios, scoring rubrics, regression suites, and human review.
• Build and operate eval harnesses, monitoring, and quality dashboards so the team can measure quality without relying on one-off manual checks.
• Diagnose failures across prompts, context, orchestration, models, tools, data, and product code, then fix the right layer or partner with the engineer who owns it.
• Set quality baselines and release gates for prompt, model, and agent changes. Turn production failures, traces, and member feedback into durable regression cases.
• Partner with product, design, marketing, and copy experts to encode Flodesk’s point of view into the system, while bringing your own judgment about how the product should behave.

What You Bring

• A track record of building or meaningfully improving production LLM or agentic products, not just prototypes.
• Strong software engineering skills in Python, TypeScript, or a similar language that let you build reliable internal tools, evaluation systems, and prototypes.
• The interaction between prompts, context, tools, agent loops, orchestration, and product state is clear to you.
• LLM evaluation, observability, and experimental design are areas where you have direct experience, and you know how to combine automated checks with human judgment.
• A strong product sense that helps you translate ambiguous member feedback into testable quality definitions and concrete system changes.
• Systems thinking and reusable patterns guide your work, so today’s email experience gets better without building in assumptions that make tomorrow’s surfaces harder to build.
• Across technical and nontechnical disciplines, your communication is clear, and you can explain tradeoffs involving quality, latency, reliability, and cost.

Bonus points if you’ve:

• Built content-generation, creative, or marketing tools.
• Built multi-turn agents that use tools and preserve user intent across edits.
• Used Langfuse, Braintrust, or comparable evaluation and observability platforms.
• Worked on products for small businesses, creators, or marketers.