Automated Protection Pipeline Engineer

CloudLinux is a global remote-first company driven by principles: do the right thing, employees first, remote-first, and delivering high-volume, low-cost Linux infrastructure and security products to help companies increase the efficiency of their operations.

Imunify360 Security Suite is an innovative security solution designed specifically for shared and VPS/Dedicated servers, delivering comprehensive and complete attack prevention with a six-layer approach.

We are looking for an experienced engineer to own the automated pipelines that turn threat intelligence into shipped protection for tens of millions of websites.

When a new vulnerability appears, a chain of automated systems must notice it, obtain the vulnerable source, produce WAF rules, generate tests, prove the rules work and don't break legitimate traffic, and roll it out across tens of millions of websites — then monitor production to pull it back automatically if it misbehaves. This chain exists and runs 24/7, and a single unreliable link could cost customers their protection. We need a strong Engineer to expand it rapidly while keeping it reliable and transparent, solving hard engineering problems at scale.

This is not an analyst or research role, but additional experience in this domain is welcome. We need someone who builds systems that keep working: complex logic underneath, boring and reliable on the outside.

The position is fully remote with flexible hours, allowing you to plan your day and work from anywhere in the world.

What you would own

Systems that are in production today and need to grow considerably:

• The automated protection pipeline — the chain from threat intelligence to a validated, deployed rule. Multi-stage, largely autonomous, and required to finish inside a fixed time window every day.
• Progressive release automation — rolling protection out across the fleet in controlled stages, with automated guardrails that hold or roll back a stage without a human in the loop.
• Quality gates — deciding, from live production signal, whether something shipped is doing harm, and acting on that before it reaches the next stage. Errors are costly in both directions: miss a problem and customer sites break; over-correct and protection is silently removed.
• CI at scale — validation that stands up real, disposable environments across a large matrix of software versions and configurations, and returns a trustworthy verdict fast enough to stay inside the release window.
• LLM orchestration and cost control — several subsystems are driven by AI agents inside purpose-built harnesses, with evaluation, budgets, and spend accounting as load-bearing components.
• Observability and alerting across all of it — the pipelines are expected to report their own condition, prove they are healthy, and escalate on their own.

This runs on top of a petabyte-scale threat-intelligence store and live telemetry from more than 60 million websites, under strict end-to-end latency budgets measured in hours. The failure mode we care most about eliminating is a budget missed silently.

We will go into the specifics during the interview process.

Key responsibilities

• Designing, building, and operating the automated pipelines described above, end to end;
• Turning fragile multi-stage batch jobs into resumable, idempotent, observable systems with explicit state machines and recovery paths;
• Defining and enforcing latency budgets and SLOs per stage, and making violations visible and actionable rather than silent;
• Building the observability layer — metrics, dashboards, alerting, and health gates — so the pipeline reports its own condition instead of needing someone to go and look;
• Designing and implementing guardrails: automatic hold and rollback, blast-radius limits, kill switches, and safe-by-default behavior when an upstream dependency is unavailable;
• Making the systems low-maintenance: eliminating manual steps, removing standing human babysitting, and reducing the operational surface rather than adding to it;
• Writing and maintaining unit and integration tests for logic that is genuinely hard to test — concurrency, partial failure, external API flakiness, multi-stage state;
• Investigating and resolving complex issues across ClickHouse, GitLab CI, S3/object storage, Prometheus/Grafana, and third-party APIs;
• Collaborating with the security analysts and the Server team on architecture, and pushing back when a proposed design will not survive contact with production.

Requirements

• 5+ years of professional backend / platform / infrastructure engineering experience;
• Demonstrable experience building and operating multi-stage data or automation pipelines — CI/CD systems, ETL/ELT, build and release automation, job orchestration, ML/data platforms, or similar. This is the single most important requirement;
• Real depth in at least one of Python, Go, or Rust. We use all three, and depth is how we verify the experience behind it;
• Systems design judgement, more than raw coding throughput. The difficult part of this role is deciding what to build, working out where it will break, and making it prove its own correctness;
• Practical experience with workflow orchestration and job scheduling (Airflow, Temporal, Prefect, Dagster, Argo, custom schedulers — whatever you have run in production);
• A working instinct for reliability engineering: idempotency, retries with backoff, exactly-once vs at-least-once, checkpointing and resumability, graceful degradation, backpressure, and safe handling of partial failure;
• Hands-on observability experience — Prometheus/Grafana, LGTM stack, or equivalent — including designing metrics rather than only consuming dashboards;
• Deep CI/CD experience, ideally GitLab CI including dynamic/child pipelines and self-hosted runners; comfort with Docker and container-based test environments;
• Experience with object storage (S3/Ceph or equivalent) and with large-scale analytical stores — ClickHouse or another columnar database;
• Comfort designing state machines and long-running processes that survive restarts, and reasoning about concurrency across multiple in-flight rollouts;
• Excellent debugging skills across system, network, and data layers;
• Strong communication skills and comfort working in a distributed team;
• Proficiency in spoken and written English.

Nice to have

• Experience with progressive delivery — canary and percentage-based rollouts, feature flags, automated rollback, blast-radius control;
• Experience running AI/LLM systems in production, particularly cost control, token accounting, evaluation harnesses, and dealing with non-deterministic components inside a deterministic pipeline;
• Experience with fleet-scale telemetry and with building quality gates on top of noisy production signal;
• Familiarity with WordPress, PHP, or WAF/ModSecurity concepts;
• Experience with configuration management (Ansible, Puppet, Salt) and with Linux service operations.

You do not need a cybersecurity background. Most of the hard problems here are orchestration, reliability, correctness under concurrency, and observability. Domain knowledge is learnable.

We value engineers who are

• Curious and fearless problem solvers — not afraid to dig into existing systems, investigate root causes, and propose improvements;
• Sceptical by default — who ask what would falsify a conclusion before acting on it, and who trust measurements over plausible reasoning;
• Pragmatic and detail-oriented — focused on building reliable, maintainable systems, and allergic to solutions that require a human to remember something;
• Owners — comfortable being the person accountable for whether a pipeline ran correctly last night;
• Effective communicators — able to articulate ideas clearly, exchange feedback constructively, and foster collaboration across teams;
• Engaging and proactive — contributing energy, initiative, and a positive presence that strengthens team culture.

Benefits

What's in it for you?

• A focus on professional development.
• Interesting and challenging projects.
• Fully remote work with flexible working hours, allowing you to schedule your day and work from any location worldwide.
• Paid 24 days of vacation per year, 10 days of national holidays, and unlimited sick leaves.
• Compensation for private medical insurance.
• Co-working and gym/sports reimbursement.
• Budget for education.
• The opportunity to receive a reward for the most innovative idea that the company can patent.