Research Scientist (Protocol Learning) at Pluralis Research

Pluralis Research is pioneering Protocol Learning, a method for training and serving large models in a fully decentralized way on consumer-grade devices connected via the internet. Despite initial skepticism, they’ve achieved significant progress, including the permissionless run of Agora, which pretrained an 8B model from scratch on consumer GPUs without any single participant holding the full weights. Explore their mission in full via A Third Path: Protocol Learning.

This approach challenges traditional datacenter training assumptions, addressing challenges like communication-efficient training across parallelism axes, fault tolerance during node churn, heterogeneous compute and networks, and robustness to malicious participants. Their published methods include Subspace Networks, Factored Gossip DiLoCo, AsyncMesh, and Sentinel.

As a Research Scientist, you’ll develop the training system to scale Protocol Learning from 8B models to frontier-scale models, enabling large models to run on heterogeneous hardware across physically distinct regions, connected only by the internet.

Key Responsibilities

• Distributed pretraining: Implement and optimize model-parallel training, including data, pipeline, and tensor parallelism for large models on heterogeneous GPUs under low-bandwidth, high-latency links.

• Performance optimization: Develop techniques to reduce communication overhead while maintaining model convergence in challenging network environments.

• Elasticity and fault tolerance: Ensure runs survive node churn with robust checkpointing, state synchronization, and recovery as participants join and leave.

• Run instrumentation: Build monitoring systems to track throughput, bottlenecks, and model quality across hundreds of devices.

What We're Looking For

• Hands-on distributed training (required): Experience training models across multiple devices using PyTorch with FSDP, DeepSpeed, Megatron, or custom implementations. Understanding of data, tensor, and pipeline parallelism is essential.

• Strong engineering: Proficiency in production-quality Python, concurrency, failure handling, and profiling before optimization.

• Evidence of execution: Track record of shipping systems, contributing to research code, open-source projects, or significant personal projects.

• Mission alignment: Belief that Protocol Learning represents a viable third path for collective, trustless, and sovereign AI.

Nice to Have

• Hands-on experience training or serving large language models such as Nemotron, Qwen, or OLMo.

• Experience with peer-to-peer (P2P) networking and NAT traversal.

• Experience with post-training and reinforcement learning (RL).

• Experience with inference and serving systems.

• Experience at proprietary, open-weight, or open-source AI labs.