Data Engineer & AI Specialist (Databricks & Claude AI Integration)

Core Responsibilities:

Design and build pipelines for ingesting data into Databricks from various sources like SAP and website scraping. Core technical skills required include PySpark and PySQL.

Write and conduct scenario-based testing, including edge cases, and collaborate with stakeholders to finalize necessary code changes and acceptance criteria.

Understand the Databricks Unity Catalog permissions model, including the use of OBO tokens and configuration of Service Principals for Databricks Agents, Genie Spaces, Machine Learning, and Foundation Models.

Possess knowledge of Databricks Apps. Ability to build and deploy Databricks Apps using Visual Studio Code.

Send custom notifications from Databricks using APIs to custom Team Channel Webhooks.

AI Specialization:

Build minimal Single Page Applications (SPAs) using FastAPI and React front-end, integrating Chat UI interfaces with Claude Models/APIs.

Develop skills in prompt engineering, including the use of defined System Prompts and User Prompts.

Possess basic knowledge of Token Economics and the ability to enforce rules in prompts, in addition to standard NLP language processing.

Reengineer code to align with user requirement specifications.

Use Graphs to detect interdependencies between rules and circular references.

Leverage both the Claude Vision API and standard text processing.

Develop Retrieval Augmented Generation specifically for Databricks-like AI Search indexes and Hierarchical Chunking.

Implement basic guard rails related to AI safety.

Understand various caching mechanisms.

Tools:

Ability to work with Claude Code and Genie.