Data Engineer & AI Specialist (Databricks & Claude Integration)

Design and build pipelines for ingesting data into Databricks from various sources like SAP, website scraping. Core technical skill will be PySpark and PySQL.

Write and conduct scenario-based testing, including edge cases, and collaborate with stakeholders to finalize necessary code changes and acceptance criteria.

Understand the Databricks Unity Catalog permissions model and configure OBO tokens and Service Principals for Databricks Agents, Genie Spaces, Machine Learning, and Foundation Models.

Possess knowledge of Databricks Apps. Ability to build and deploy Databricks Apps using Visual Studio Code.

Send custom notifications from Databricks using APIs to custom team channel webhooks.

AI:

• Ability to use FastAPI and React front-end to build minimal Single Page Applications (SPAs) using Chat UI interfaces and Claude Models/APIs.
• Ability to use defined System Prompts, User Prompts, etc., including skills in prompt engineering.
• Basic knowledge of token economics and ability to use rule enforcement in prompts, in addition to standard NLP language.
• Ability to reengineer code and map it to user requirement specifications.
• Ability to use Graphs to detect interdependencies between rules and circular references.
• Ability to use both Claude Vision API in addition to standard text processing.
• Understand and develop Retrieval Augmented Generation specifically for Databricks like AI Search indexes, Hierarchical Chunking, etc.
• Knowledge of implementing basic guardrails related to AI safety.
• Understand various caching mechanisms.

Tools:

• Ability to work with Claude Code and Genie.