Solution Architecture/Data Architecture & AI Engineer

Solution Architecture/Data Architecture

• Modify source code to accommodate architectural considerations such as transitioning from MSAL-based authentication to JWT-based authentication while using Databricks X-forwarded headers within Apps.
• Conduct technical reviews for configuring necessary Azure resources for deploying applications.
• Prepare and submit technical/network architecture documentation for cloud and security team reviews and approvals before deployment.

Databricks

• Design and build data ingestion pipelines into Databricks from sources like SAP and website scraping, leveraging core skills in PySpark and PySQL.
• Write and perform scenario-based testing, including edge cases, and collaborate with stakeholders to finalize code changes and acceptance criteria.
• Understand Databricks Unity Catalog permissions, OBO tokens, and configuration of Service Principals for Databricks Agents, Genie Spaces, Machine Learning, and Foundation Models.
• Possess knowledge of Databricks Apps and the ability to build and deploy them using Visual Studio Code.
• Send custom notifications from Databricks using APIs to custom Team Channel Webhooks.

AI:

• Develop minimal Single Page Applications (SPAs) using FastAPI and React frontends with Chat UI interfaces and Claude Models/APIs.
• Apply prompt engineering skills, including the use of defined System Prompts and User Prompts.
• Have basic knowledge of token economics and rule enforcement in prompts, in addition to standard NLP language.
• Reengineer code to align with user requirement specifications.
• Use Graphs to detect interdependencies between rules and circular references.
• Utilize both Claude Vision API and standard text processing.
• Develop Retrieval Augmented Generation specifically tailored for Databricks-like AI Search indexes and Hierarchical Chunking.
• Implement basic guardrails related to AI safety.
• Understand various caching mechanisms.

Tools:

• Work with Claude Code and Genie.

Full Stack Development:

• Understand and troubleshoot full-stack architecture, including frontend, backend, middleware, data layers, and routes.
• Correct and rewrite code based on security findings, in addition to standard development practices.
• Document code using techniques like TDD, FDD, DFD, and SDs.

DevOps:

• Create and automate deployment pipelines (CI/CD) for Teams Bot Deployment, Web Application Deployments, and Databricks Bundle/Notebook Deployments through ADO.
• Add validation steps such as Snyk Scans and linting.
• Configure various environments based on CI/CD pipelines.

Communication Skills and Expectations:

• Collaborate closely with Toxicologists and Project Managers from the client team based on concise requirement descriptions.
• Deliver work as an Individual Contributor in short sprint cycles.