Skip to main content
Back

Google Cloud's PaperOrchestra Turns Experiment Logs Into Conference-Ready Papers

Google Cloud AI Research has open-sourced a multi-agent system that turns experiment logs and raw ideas into submission-ready LaTeX manuscripts with real citations.

  • Google Cloud AI Research released PaperOrchestra, an Apache 2.0 multi-agent paper writer.
  • Specialized agents handle outline, literature review, section writing, refinement, and plotting in parallel.
  • Semantic Scholar API grounds citations to reduce hallucinated references in the generated lit review.
  • Outputs compilable LaTeX targeting CVPR or ICLR templates from idea summaries and experiment logs.
  • Human evaluation shows 14 to 38 percent overall win margin against autonomous baselines, still trailing humans.
  • Companion PaperWritingBench benchmarks 200 reverse-engineered CVPR and ICLR 2025 papers.

Google Cloud AI Research has released PaperOrchestra, an Apache 2.0 project that converts research notes, experimental results, and venue templates into structured LaTeX manuscripts. The system targets a difficult stage of research: assembling scattered source material into a coherent first draft with citations, figures, and conference formatting.

The repository includes a command-line pipeline, a Streamlit interface, and templates for venues such as CVPR and ICLR. It supports Gemini, OpenAI, and Vertex AI models, while Semantic Scholar supplies publication metadata for citation checks. Users run the software themselves and cover their own model API and compute costs; Google does not provide a hosted service.

Five agents, one manuscript

PaperOrchestra divides the writing process among specialized agents that can work in parallel, exchange outputs, and critique the evolving draft.

Agent Responsibility
Outline Organizes the research question, method, evidence, and planned sections.
Plotting Creates statistical plots and conceptual diagrams from supplied data.
Literature Review Searches for relevant work and checks candidate papers through Semantic Scholar.
Section Writing Drafts the manuscript in the selected LaTeX template.
Content Refinement Revises the draft using simulated peer-review feedback.

A run begins with three main inputs:

  • Idea Summary: the research problem, proposed method, and theoretical basis.
  • Experimental Log: numerical results from tables, along with factual observations extracted from figures.
  • Venue template: the LaTeX files and formatting rules for the target conference.

Users must prepare those inputs carefully, including converting important figure findings into explicit observations. The agents then build the outline, review related work, generate figures, draft sections, and refine the manuscript.

Two venues, 200 reconstructed papers

The project evaluates its pipeline with PaperWritingBench, a benchmark built from 200 accepted papers: 100 from CVPR 2025 and 100 from ICLR 2025. The team worked backward from the completed papers to reconstruct the kinds of notes and experimental records that a writing system might receive. Using two venues also tests whether the pipeline can handle CVPR’s double-column format and ICLR’s single-column layout.

Eleven AI researchers conducted blind, side-by-side comparisons involving PaperOrchestra, a single-agent baseline, and AI Scientist-v2. The project reports the following absolute win-rate margins across those comparisons:

Evaluation area Reported margin
Literature-review quality 50 to 68 percentage points
Overall manuscript quality 14 to 38 percentage points

Human-written source papers still received stronger overall assessments. The repository also says the full benchmark dataset will be released separately, which currently limits independent reproduction of the reported results.

Where the pipeline holds up

  • Citation discovery: Semantic Scholar checks whether candidate papers exist and appear relevant, reducing fabricated references.
  • Format control: the pipeline writes into a conference template and produces a LaTeX project instead of an unformatted text draft.
  • Repeatable structure: labs can apply the same outline and formatting process across related experiments or ablation studies.
  • Input sensitivity: vague method descriptions and incomplete logs propagate into weak claims, missing context, and unreliable figures.
  • Human-level quality: the benchmark results leave a measurable gap between generated drafts and the original manuscripts.

Semantic Scholar grounding confirms publication metadata and broad relevance. It does not prove that a cited paper supports a particular sentence, so every citation still requires source-level review. Researchers also need to verify equations, numerical claims, generated plots, and LaTeX output before circulation.

From input folder to LaTeX

The documented setup uses Python 3.11 in a Conda environment, with provider credentials stored in a .env file. A typical CLI run looks like this:

python paper_writing_cli.py \
  --raw_materials_dir /path/to/raw_materials/ \
  --latex_template_dir templates/cvpr2025 \
  --output_dir /path/to/output/ \
  --use_plotting true

Separate --writer_model_name and --reflection_model_name flags allow different models for drafting and critique. That configuration can assign routine section generation to a lower-cost model while reserving a stronger model for review passes. The bundled Streamlit application provides an interactive alternative to the CLI.

Research material sent to Gemini, OpenAI, or Vertex AI is subject to the selected provider’s data-handling terms. Teams working with confidential results should review those terms before uploading notes or experiment logs. The Apache 2.0 license covers PaperOrchestra’s code; model services and third-party templates retain their own terms.

The useful lane

PaperOrchestra fits projects whose experiments and central claims are already settled. Practical uses include:

  1. Turning completed experiments into a workshop or conference draft.
  2. Preparing a related-work section for manual verification and revision.
  3. Generating baseline plots and conceptual diagrams from structured results.
  4. Creating consistent LaTeX scaffolds for related studies from the same lab.

The system accelerates assembly and revision while leaving scientific responsibility with the researchers. Authors remain accountable for the claims, mathematics, evidence, citations, disclosures, and final submission.