feat: LLM-synthesized hints for failed trajectories

[dzorlu]

Summary

• Replace static verifier-feedback hints (0% recovery in 35B parity run) with Claude Sonnet-powered hint synthesis that analyzes the full failed trajectory + verifier errors
• New hint_synthesizer.py module with batch async synthesis (semaphore-controlled concurrency, 30s timeout, automatic static fallback)
• Track hint_category (llm_synthesized vs static_fallback) with per-category success rate metrics in WandB

Test plan

• Verify Hint [llm_synthesized] entries appear in logs with actionable text
• Check WandB for hint/category_llm_synthesized_success_rate > 0% within first 10 steps
• Confirm static fallback works when ANTHROPIC_API_KEY is missing
• Compare eval pass@3 at step 20 vs parity run baseline (62.1%)

🤖 Generated with Claude Code

---

[gemini-code-assist]

Code Review

This pull request introduces SkyRL-v2, a fork optimized for multi-node FSDP2 training with features like synchronous model offloading, chunked logit computation, and hint-augmented advantage computation. It also adds Vision-Language support, S3 checkpoint management, and a task generation environment. Feedback identifies critical bugs in advantage computation where torch.tensor() is used on CUDA tensors instead of torch.stack(), and an invalid model identifier in the hint synthesizer. Additionally, the reviewer suggests moving imports to the top of the file for PEP 8 compliance and documenting environment variables required for the Supabase fallback mechanism.

[gemini-code-assist]

Using torch.tensor([raw]) where raw is a list of scalar tensors located on a GPU will trigger a TypeError because torch.tensor() attempts to convert the input to a NumPy array first, which is not allowed for CUDA tensors. Additionally, creating a tensor from a list of tensors is inefficient as it involves multiple host-device synchronizations. Use torch.stack(raw) instead, which is faster and correctly handles tensors on any device.

                else:
                    raw_tensor = torch.stack(raw)
                    id2mean[idx] = raw_tensor.mean()
                    id2std[idx] = raw_tensor.std()

[gemini-code-assist]

Similar to the issue in the is_hinted path, torch.tensor([id2score[idx]]) will crash if the tensors in the list are on a GPU. Use torch.stack() for better performance and compatibility with CUDA tensors.

Suggested change

	id2mean[idx] = torch.mean(torch.tensor(id2score[idx]))
	id2std[idx] = torch.std(torch.tensor([id2score[idx]]))
	elif len(id2score[idx]) > 1:
	group_scores = torch.stack(id2score[idx])
	id2mean[idx] = group_scores.mean()
	id2std[idx] = group_scores.std()

[gemini-code-assist]

The model name claude-sonnet-4-20250514 appears to be invalid or a future-dated placeholder. Anthropic's current Claude 3.5 Sonnet model identifier is typically claude-3-5-sonnet-20241022 (or claude-3-5-sonnet-latest). Using a non-existent model name will cause the Anthropic API to return a 404 error, breaking the hint synthesis feature.

Suggested change

	model: str = "claude-sonnet-4-20250514",
	model: str = "claude-3-5-sonnet-20241022",

[gemini-code-assist]

Standard Python practice (PEP 8) recommends placing all imports at the top of the file. Moving ast and re to the top-level imports improves readability and ensures they are only imported once.

[gemini-code-assist]

The _query_supabase_scores method relies on SUPABASE_URL and SUPABASE_KEY environment variables. These should be documented in the module docstring or the script headers (e.g., in scripts/fleet-task-gen-run.sh) to ensure users are aware of the requirements for this fallback mechanism.

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 9 additional findings.