工作经历 / Experience Experience
上海人工智能实验室 · 开源决策智能实验室 (OpenDILab) · 研究工程师 Shanghai AI Laboratory · OpenDILab · Research Engineer
2023.04 — 至今 2023.04 — Present
商汤科技 · 研究院 · 强化学习算法研究员 SenseTime · Research Institute · RL Algorithm Researcher
2021.07 — 2023.04
核心项目与研究 / Projects & Research Projects & Research
AgenticRL 探究：面向智能体任务的统一 RL 后训练框架 AgenticRL Exploration: Unified RL Post-training Framework for Agentic Tasks
2026.05 — 至今 2026.05 — Present
  • 建设适配不同 harness、不同大小与类型模型、不同 RL 算法的训练框架，形成 Harness × Model × Algorithm 三维可插拔能力，支撑通用能力训练与安全能力训练的协同提升。
  • 围绕 Agentic-RL 中关键的探索-利用机制、world-model 机制等方向展开算法探究，提升复杂多轮交互任务中的训练效率、稳定性与可迁移性。
  • Building an RL training framework that supports different harnesses, model sizes and types, and RL algorithms through pluggable Harness × Model × Algorithm abstractions, enabling coordinated improvement of general capabilities and safety capabilities.
  • Conducting algorithmic research on key Agentic-RL mechanisms, including exploration-exploitation, reward signal modeling, and world-model-based training, to improve efficiency, stability, and transferability in complex multi-turn interaction tasks.
PriorZero：语言先验增强的世界模型决策框架 PriorZero: Language-Prior-Augmented World Model Decision Framework
2025.06 — 2026.04 2025.06 — 2026.04
  • arXiv 2026 PriorZero：针对 LLM 知识迁移到长程稀疏奖励 RL 中的 prior-dynamics mismatch，提出解耦的 rollout-training 框架。通过 Root-Prior Injection 将 LLM 语义先验仅注入 MCTS 根节点，引导搜索聚焦语义上更有潜力的动作，同时保留世界模型在隐空间中的多步前瞻能力。
  • 训练阶段解耦世界模型学习与 LLM 适配：世界模型基于交互数据持续优化 dynamics、policy 与 value 预测，并利用其价值估计构造低方差、细粒度的信用分配信号进行交替强化学习微调；在 Jericho 与 BabyAI 等基准上提升探索效率与最终性能。
  • arXiv 2026 PriorZero: addresses the prior-dynamics mismatch when transferring LLM knowledge to long-horizon, sparse-reward RL through a decoupled rollout-training framework. Root-Prior Injection incorporates LLM semantic priors only at the MCTS root node, focusing search on semantically promising actions while preserving the world model's latent multi-step lookahead.
  • During training, PriorZero decouples world-model learning from LLM adaptation: the world model continually improves its dynamics, policy, and value predictions on interaction data, and its value estimates provide low-variance, fine-grained credit assignment signals for alternating RL fine-tuning. Experiments on Jericho and BabyAI show improved exploration efficiency and final performance.
LightRFT：轻量化、全模态及奖励模型驱动的强化学习微调框架 LightRFT: Lightweight, All-Modality & Reward-Model-Driven RL Fine-Tuning Framework
2025.03 — 2025.09 2025.03 — 2025.09
  • 作为核心成员，为百卡级多模态大模型安全对齐项目 arXiv 2025 SafeWork-R1 构建底层训练设施。设计了统一的 Strategy 抽象层，实现了训练（DeepSpeed/FSDP）、推理（vLLM/SGLang）与验证（规则/模型）等异构任务在统一控制平面下的灵活切换与高效共置，显著提升了多模态基础模型与奖励模型协同训练的资源利用率和易用性。
  • 开展 Off-policy 设置下的算法机制研究，致力于解决强化学习微调中的样本效率与训练稳定性难题；同时深入探索基于奖励模型的强化微调算法，以提升模型对齐效果。
  • As a core member, built the underlying training infrastructure for the 100+ GPU multimodal LLM safety alignment project arXiv 2025 SafeWork-R1. Designed a unified Strategy abstraction layer enabling flexible switching and efficient co-location of heterogeneous tasks (training via DeepSpeed/FSDP, inference via vLLM/SGLang, verification via rules/models) under a unified control plane, significantly improving resource utilization and usability for joint training of multimodal foundation models and reward models.
  • Leading research on off-policy algorithms for RL fine-tuning, addressing sample efficiency and training stability challenges; exploring Reward Model RFT algorithms to improve model alignment.
LightZero：通用序列决策的 MCTS+RL 统一基准 LightZero: Unified MCTS+RL Benchmark for General Sequential Decision
2023.04 — 至今 2023.04 — Present
  • NeurIPS 2023 Spotlight 项目负责人。该框架旨在标准化 MCTS+RL 算法族，集成了 AlphaZero、MuZero 等 10 余种主流算法，是目前最全面的开源 MCTS 算法库。
  • 采用 Python/C++ 混合编程（Core MCTS in C++），设计了轻量化、模块化的系统架构。通过多进程并行数据采集与 DDP 分布式训练支持，显著提升了在 Atari、DMC 及棋类博弈等复杂环境下的训练效率与扩展性。
  • NeurIPS 2023 Spotlight Project lead. The framework standardizes the MCTS+RL algorithm family, integrating 10+ mainstream algorithms including AlphaZero and MuZero — the most comprehensive open-source MCTS library to date.
  • Designed a lightweight, modular architecture using Python/C++ hybrid programming (Core MCTS in C++). Multi-process parallel data collection and DDP distributed training significantly improved efficiency and scalability across Atari, DMC, and board game environments.
ScaleZero：可扩展的异构多任务学习 ScaleZero: Scalable Heterogeneous Multi-Task Learning
2024.05 — 2026.03
  • ICLR 2026 ScaleZero：针对大规模异构多任务学习中的可塑性（Plasticity）与效率瓶颈，系统性优化了世界模型设计。仅依托在线隐空间规划，在 Atari、DMC、Jericho 等多领域基准上实现了超越专用单任务模型的性能。设计了基于 LoRA 的在线动态参数扩展策略，能够根据任务训练进程自适应分配计算资源，在 DMC 任务上仅需约 75% 的交互步数即可达到单任务基线的性能水平。
  • ICLR 2026 ScaleZero: Systematically optimized world model design for plasticity and efficiency bottlenecks in large-scale heterogeneous multi-task learning. Achieved performance surpassing dedicated single-task models on Atari, DMC, and Jericho benchmarks using only online latent-space planning. Designed a LoRA-based online dynamic parameter expansion strategy that adaptively allocates compute resources, reaching single-task baseline performance with only ~75% of interaction steps on DMC tasks.
通用高效的 MCTS+RL 算法研究 General & Efficient MCTS+RL Algorithm Research
2023.09 — 2024.09
  • TMLR 2025 UniZero：独立提出基于 Transformer 的模块化世界模型架构。通过构建共享隐空间并联合优化长时序动态预测与决策目标，有效解决了 MCTS 算法在异构状态动作空间及长程依赖任务中的泛化难题，在单/多任务基准上均取得优异性能。
  • CoRL 2025 RemembeRL Workshop ReZero：合作提出反向重分析（Backward-view Reanalyze）机制，在保持样本效率的同时，将 MCTS 类算法的训练时间效率提升 1.5-2 倍。
  • TMLR 2025 UniZero: Independently proposed a modular Transformer-based world model architecture. By constructing a shared latent space and jointly optimizing long-horizon dynamics prediction and decision objectives, effectively addressed generalization challenges of MCTS in heterogeneous state-action spaces and long-horizon tasks, achieving strong performance on both single-task and multi-task benchmarks.
  • CoRL 2025 RemembeRL Workshop ReZero: Co-proposed the Backward-view Reanalyze mechanism, improving training time efficiency of MCTS-based algorithms by 1.5-2x while maintaining sample efficiency.
DI-engine 强化学习开源框架与 PPO x Family 公开课 DI-engine RL Framework & PPO x Family Course
2021.07 — 2023.04
  • 作为 Top-2 贡献者参与 DI-engine 研发（3.6k+ Stars）。该框架通过算法与系统协同设计和抽象，灵活支持了从 DQN、PPO 到 DreamerV3、NGU 等全栈算法。重点负责了探索机制（Exploration）、POMDP 支持及 Model-based RL 模块的架构设计与实现。
  • 深度参与决策智能入门公开课 PPO x Family 制作（2.5k+ Stars）。负责多模态观测、复杂动作空间及多智能体协作等核心章节的实验设计与教程编写，帮助社区初学者从理论到实践全面掌握 PPO 算法家族。
  • Top-2 contributor to DI-engine (3.6k+ Stars). The framework supports a full stack of algorithms from DQN, PPO to DreamerV3, NGU through co-designed algorithm-system abstractions. Led the architecture design and implementation of exploration mechanisms, POMDP support, and model-based RL modules.
  • Deeply involved in the PPO x Family introductory course on decision intelligence (2.5k+ Stars). Designed experiments and wrote tutorials for core chapters on multimodal observations, complex action spaces, and multi-agent cooperation, helping community beginners learn the PPO algorithm family from theory to practice.
代表性论文 / Publications Publications
ICLR 2026
Pu, Y., et al. "One Model for All Tasks: Leveraging Efficient World Models in Multi-Task Planning." arXiv preprint arXiv:2509.07945 (2025).
arXiv 2026
Xiong, J., Pu, Y., Tang, J., Niu, Y. "PriorZero: Bridging Language Priors and World Models for Decision Making." arXiv preprint arXiv:2605.12289 (2026).
SafeWork-R1
Bao, Y., ..., Pu, Y., et al. "SafeWork-R1: Coevolving Safety and Intelligence under the AI-45° Law." arXiv preprint.
TMLR 2024
Pu, Y., Niu, Y., et al. "UniZero: Generalized and efficient planning with scalable latent world models." Transactions on Machine Learning Research.
NeurIPS 2023 Spotlight
Niu, Y., Pu, Y., et al. "LightZero: A unified benchmark for Monte Carlo tree search in general sequential decision scenarios." Advances in Neural Information Processing Systems, 36.
CoRL 2025 RemembeRL Workshop
Xuan, C., Niu, Y., Pu, Y., et al. "ReZero: Boosting MCTS-based algorithms by backward-view and entire-buffer reanalyze." OpenReview.
Preprint 2024
Niu, Y., Pu, Y., et al. "Unifying diverse decision-making scenarios with learned discrete actions." arXiv preprint.
Preprint 2021
Pu, Y., Wang, S., et al. "Decomposed soft actor-critic method for cooperative multi-agent reinforcement learning." arXiv preprint.
教育背景 / Education Education
中国科学技术大学 · 硕士，电子工程与信息科学 University of Science and Technology of China · M.S. in Electronic Engineering & Information Science
2018 — 2021
哈尔滨工业大学 · 学士，通信工程，专业排名 3/91 Harbin Institute of Technology · B.E. in Communication Engineering, Rank: 3/91
2014 — 2018
技能 / Skills Skills
强化学习 (RL) Reinforcement Learning (RL)
熟悉 Python/PyTorch，拥有丰富的 RL 算法设计开发与性能优化经验。在基于世界模型的 RL (World Models)、蒙特卡洛树搜索 (MCTS)、高效探索 (Exploration)、Agentic-RL 及多任务学习等方向有深入研究与顶会发表记录。
Proficient in Python/PyTorch with extensive experience in RL algorithm design, development, and performance optimization. In-depth research and top-venue publications in world model-based RL, Monte Carlo Tree Search (MCTS), efficient exploration, Agentic-RL, and multi-task learning.
大语言模型 (LLM) Large Language Models (LLM)
具备百亿参数级多模态大模型的后训练 (Post-training) 实战经验。熟练掌握 PPO、GRPO 等强化学习微调技术，并参与面向工具交互、可验证反馈和安全对齐的统一 RL 后训练框架建设。密切关注 RLHF/RLVR/AgenticRL 与世界模型相融合的前沿技术。
Hands-on post-training experience with 10B+ parameter multimodal LLMs. Proficient in PPO, GRPO and other RL fine-tuning techniques, with experience building unified RL post-training workflows for tool interaction, verifiable feedback, and safety alignment. Closely tracking the frontier of RLHF/RLVR/AgenticRL integrated with world models.
技术影响力 Community Impact
长期维护知乎专栏 《决策智能与强化学习》、《MCTS+RL》 及 《基础模型和多模态交互》。创建并维护前沿主题资源库 awesome-exploration-rl 与 awesome-RLVR/RLHF 等，持续追踪领域最新动态。
Long-term maintainer of Zhihu columns on Decision Intelligence & RL, MCTS+RL, and Foundation Models & Multimodal Interaction. Also created and maintains curated resource repositories including awesome-exploration-rl, awesome-RLVR/RLHF, continuously tracking the latest developments in the field.
其他信息 / Misc Misc
学术服务：担任 ICLR 2026、NeurIPS 2026、NeurIPS 2025 Workshop 审稿人。连续 3 年受邀于清华大学深圳研究生院《大数据分析》课程讲授《强化学习与 MCTS 基础》。作为核心负责人参与《世界模型》专业书籍编写 (WIP)。
荣誉奖项：本科期间连续三年获国家励志奖学金，荣获山东省优秀毕业生称号。
综合素质：通过全国计算机四级、英语六级 (CET-6)。具备对前沿科技的好奇心与技术攻关精神，热爱羽毛球与登山。
Academic Service: Reviewer for ICLR 2026, NeurIPS 2026, and NeurIPS 2025 Workshop. Guest lecturer on RL & MCTS Fundamentals at Tsinghua University Shenzhen for 3 consecutive years. Core contributor to a professional book on World Models (WIP).
Honors: National Encouragement Scholarship for 3 consecutive years during undergraduate studies. Outstanding Graduate of Shandong Province.
Misc: National Computer Rank Examination Level 4, CET-6. Curious about frontier technologies, committed to technical problem solving, and passionate about badminton and hiking.