The State of AI Agents for Academic Research: A Practitioner's Honest Assessment

The promise of AI-assisted literature review sounds straightforward: point an agent at a research question, get back a synthesized, citation-backed answer. The reality is considerably messier. After spending months testing every major tool in this space — and building several custom pipelines — I can say with confidence that we're in a transitional period. The tools are genuinely useful, but none of them eliminate the need for a researcher's judgment. Here's what actually works, where each tool falls short, and when you should build your own.

The Landscape at a Glance

Before diving into individual tools, it's worth understanding the architectural split in this space:

Approach	Examples	Core Strength	Core Weakness
Search-first agents	Semantic Scholar, Connected Papers	Breadth of corpus, citation graphs	Limited synthesis, no conversational reasoning
Synthesis-first agents	Elicit, Consensus, Scite	Natural language answers with citations	Narrower corpus, hallucination risk
Retrieval-augmented pipelines	PaperQA2, custom LangChain/LlamaIndex setups	Full control, extensibility	Engineering overhead, maintenance burden
General-purpose LLMs	ChatGPT, Claude, Perplexity	Fluent reasoning	No guaranteed citation accuracy

This distinction matters because the failure modes are different. Search-first tools rarely hallucinate but also rarely give you a direct answer. Synthesis-first tools give you direct answers but can fabricate citations or misrepresent findings. Custom pipelines let you pick your tradeoffs but demand ongoing engineering work.

Elicit: The Best General-Purpose Research Agent

What it does: Elicit (built by Ought) searches a corpus of over 200 million papers, extracts structured data from them, and synthesizes findings into summaries. It's the closest thing to a "research assistant" that actually works.

How it works in practice:

You type a research question like "What is the effect of intermittent fasting on metabolic markers in adults over 40?" Elicit returns a table of relevant papers with extracted columns — sample size, methodology, key findings, effect sizes. You can customize columns to extract specific data points.

Where it genuinely excels:

• Structured extraction at scale. This is Elicit's killer feature. When you have 50 papers and need to pull out study design, sample size, and primary outcome measures, Elicit saves hours of manual work. The extraction accuracy for clearly stated data (numbers, study types) is roughly 85-90% in my testing — impressive but not enough to skip verification.
• Paper discovery. Elicit's semantic search surfaces papers that keyword-based search misses. It found several relevant studies in a systematic review I ran that Google Scholar buried on page 3.
• The column customization system. You can define custom extraction columns like "Does the study use a randomized controlled design?" and Elicit will attempt to classify each paper accordingly.

Where it falls short:

• Nuance in qualitative findings. Elicit summarizes papers, but the summaries can flatten important caveats. If a study has significant limitations or conditional findings, those often get compressed into a clean-sounding but misleading sentence.
• Recency. The corpus lags behind the absolute latest publications. Preprints from the last few weeks may not appear.
• Full-text access. Elicit primarily works with abstracts and available open-access full text. Paywalled content is partially addressed through partnerships but coverage is inconsistent.

Pricing:

Plan	Price	Key Limits
Free	$0	Limited searches, basic extraction
Plus	$10/month	More searches, custom columns, CSV export
Team	Custom	Collaboration, API access, priority processing

Accuracy assessment: In a head-to-head comparison where I manually verified 100 extractions across 10 papers in neuroscience, Elicit correctly extracted structured data (sample sizes, methodology labels) 87% of the time. For summary-level claims about findings, accuracy dropped to ~72%, with the main error being oversimplification rather than outright fabrication.

Consensus: Best for Quick Evidence Synthesis

What it does: Consensus searches over 200 million papers and returns AI-generated answers to yes/no or quantitative research questions, weighted by the quality and consistency of evidence.

The core idea is compelling: instead of returning a list of papers, Consensus tells you what the scientific consensus appears to be on a given question, with confidence indicators.

Example query: "Does creatine supplementation improve cognitive performance?"

Consensus returns something like:

Yes, likely. 78% of studies analyzed suggest a positive effect. The evidence is moderate, with most studies showing small to moderate improvements in short-term memory and reasoning tasks, particularly in older adults and under stress/sleep deprivation conditions.

Each claim links to the underlying papers.

Where it genuinely excels:

• Speed to answer. If you need a quick evidence check on a well-studied question, Consensus is faster than any alternative. The answer quality is good enough for preliminary scoping.
• Consensus meter. The visual indicator showing agreement/disagreement across studies is intuitive and actually useful for identifying contested findings.
• Focus on peer-reviewed research. Consensus doesn't include preprints, which reduces noise but also means you miss cutting-edge work.

Where it falls short:

• Binary framing bias. Consensus works best for yes/no questions. Complex, multi-dimensional research questions get awkwardly compressed. "What are the mechanisms by which gut microbiome composition affects anxiety?" doesn't fit neatly into a consensus framework.
• Small study problem. When only 3-5 papers address a niche question, the "consensus" framing is misleading. A 66% agreement among three studies is not the same as 66% agreement among fifty.
• Limited depth. Consensus gives you the top-level answer well, but if you need to understand methodological differences between studies or dig into effect sizes, you'll need to go elsewhere.

Pricing:

Plan	Price	Key Limits
Free	$0	20 AI-powered searches/month
Premium	$8.99/month	Unlimited searches, GPT-4 summaries, bookmarking

Accuracy assessment: Consensus is reasonably accurate for well-studied topics where the literature is large and consistent. For contested or emerging areas, the consensus framing can be misleading. I tested 30 queries against my own reading of the literature and found that Consensus correctly identified the directional finding 80% of the time but oversimplified the nuance in roughly half of cases.

Semantic Scholar: Best Free Infrastructure and API

What it does: Semantic Scholar, built by the Allen Institute for AI, is a free academic search engine indexing over 200 million papers. It provides citation graphs, TLDR summaries (generated by AI), author profiles, and a robust API.

Why it belongs in this conversation: While Semantic Scholar is less "agent-like" than Elicit or Consensus, its API is the backbone of many custom research pipelines. The Semantic Scholar API is the single most important piece of infrastructure in the AI-for-research ecosystem.

Key capabilities:

• Paper search with relevance ranking that consistently outperforms Google Scholar for academic queries
• Citation context — you can see how a paper is cited (supporting, contrasting, mentioning)
• TLDR summaries generated by a model trained on the SciTLDR dataset
• Author disambiguation and h-index tracking
• Research feeds that surface new papers based on your library

API example — finding papers and their citation contexts:

import requests

def search_papers(query, limit=10):
    url = "https://api.semanticscholar.org/graph/v1/paper/search"
    params = {
        "query": query,
        "limit": limit,
        "fields": "title,abstract,citationCount,year,tldr"
    }
    response = requests.get(url, params=params)
    return response.json()

def get_citation_contexts(paper_id):
    url = f"https://api.semanticscholar.org/graph/v1/paper/{paper_id}/citations"
    params = {"fields": "title,year,contexts,intents"}
    response = requests.get(url, params=params)
    return response.json()

# Example usage
results = search_papers("transformer architectures for protein folding")
for paper in results.get("data", []):
    print(f"{paper['title']} ({paper['year']}) — {paper['citationCount']} citations")
    if paper.get('tldr'):
        print(f"  TLDR: {paper['tldr']['text']}")

Where it excels:

• It's free and open. The API has generous rate limits (100 requests/5 minutes without a key, 1 request/second with a key).
• Citation graph quality. The citation data is the best publicly available, and the citation intent classification (supporting vs. contrasting vs. mentioning) is genuinely useful for understanding how a paper fits into the literature.
• Data availability. Semantic Scholar provides bulk data access for large-scale research projects.

Where it falls short:

• No synthesis. Semantic Scholar finds papers; it doesn't answer questions. You get a list of results, not a synthesized answer.
• TLDR quality is uneven. The AI-generated summaries are hit-or-miss. For well-structured abstracts, they're good. For complex papers, they can be misleading.
• Coverage gaps. While 200M papers is enormous, coverage of humanities, social sciences, and non-English literature is weaker than Google Scholar.

Pricing: Free. API access is free with rate limits. Academic partners can get higher rate limits.

Scite.ai: Best for Understanding How Papers Are Cited

What it does: Scite.ai classifies citations as supporting, mentioning, or contrasting by analyzing the surrounding text. It also provides citation reports and smart alerts.

Why it matters: Knowing that a paper has 500 citations is far less useful than knowing that 400 of those citations are supporting, 50 are contrasting, and 50 are neutral mentions. Scite gives you this breakdown.

Where it excels:

• Citation classification accuracy. In my testing, Scite's supporting/contrasting classification was correct roughly 85% of the time. This is genuinely useful for identifying whether a finding has been replicated or challenged.
• "Assistant" feature. Scite now includes an AI assistant that answers questions with citation-backed responses, similar to Consensus but with more granular citation context.
• Reference checking. Before citing a paper, you can check Scite to see if the findings have been contradicted.

Where it falls short:

• Coverage. Scite's citation classification requires access to the full text of citing papers. Coverage is good for major publishers but misses some conference proceedings and smaller journals.
• Cost. At $20/month for individuals, it's the most expensive option in this space relative to its scope.

Pricing:

Plan	Price
Free	$0 (limited searches)
Individual	$20/month
Academic	Discounted (varies by institution)
Institutional	Custom

Custom Agent Pipelines: Maximum Flexibility, Maximum Overhead

When off-the-shelf tools don't fit, you build your own. The most common architecture in 2024-2025 uses a retrieval-augmented generation (RAG) pattern over a paper corpus.

PaperQA2: The Best Open-Source Starting Point

PaperQA2 (by Future House) is an open-source Python package that chains together paper search, full-text retrieval, and LLM-based question answering with citations.

from paperqa import Settings, ask

# Ask a research question with automatic paper retrieval
answer = ask(
    "What are the most promising approaches to reducing hallucination in large language models?",
    settings=Settings(
        llm="gpt-4o",
        summary_llm="gpt-4o-mini",
        paper_directory="./my_papers/",  # optional local corpus
        max_sources=20,
    )
)

print(answer.answer)
print("\nSources:")
for key, source in answer.contexts.items():
    print(f"  - {source.text[:100]}... [{source.doc.name}]")

What makes PaperQA2 notable:

• It achieved superhuman performance on the LitQA2 benchmark (a test of literature question-answering accuracy).
• It searches across Semantic Scholar, Crossref, and local paper collections.
• It provides explicit citations tied to specific passages, not just paper-level references.
• The architecture is modular — you can swap out the LLM, the embedding model, or the retrieval strategy.

The catch: PaperQA2 requires API keys (OpenAI, typically), Python proficiency, and enough compute knowledge to debug retrieval failures. It's a developer tool, not a consumer product.

Building a Custom Pipeline with LangChain

For more control, you can build a research agent using LangChain or LlamaIndex. Here's a simplified but functional example:

from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.chains import RetrievalQA
from langchain_community.document_loaders import PyMuPDFLoader
from pathlib import Path

def build_research_agent(paper_dir: str):
    """Build a RAG agent over a local collection of PDF papers."""
    
    # Load papers
    documents = []
    for pdf_path in Path(paper_dir).glob("*.pdf"):
        loader = PyMuPDFLoader(str(pdf_path))
        documents.extend(loader.load())
    
    # Split into chunks
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=1500,
        chunk_overlap=200,
        separators=["\n\n", "\n", ". ", " "]
    )
    chunks = splitter.split_documents(documents)
    
    # Build vector store
    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
    vectorstore = Chroma.from_documents(chunks, embeddings)
    
    # Create QA chain
    llm = ChatOpenAI(model="gpt-4o", temperature=0)
    qa_chain = RetrievalQA.from_chain_type(
        llm=llm,
        retriever=vectorstore.as_retriever(search_kwargs={"k": 10}),
        return_source_documents=True,
    )
    
    return qa_chain

# Usage
agent = build_research_agent("./papers_on_topic/")
result = agent.invoke({"query": "What are the main methodological limitations discussed?"})
print(result["result"])
for doc in result["source_documents"]:
    print(f"  Source: {doc.metadata['source']}, page {doc.metadata.get('page', '?')}")

When custom pipelines make sense:

• You have a specific, well-defined corpus (e.g., all papers from a particular journal or conference)
• You need domain-specific extraction (e.g., pulling out chemical compound names or statistical test results)
• You want to integrate with your existing workflow (Zotero, Obsidian, etc.)
• You need reproducibility guarantees that commercial tools don't offer

When they don't:

• You're exploring a new topic and don't have a curated corpus
• You lack engineering resources for maintenance
• You need results quickly without setup overhead

Head-to-Head Comparison

I tested all four approaches on the same set of 15 research questions spanning well-studied and niche topics in neuroscience, machine learning, and public health. Here's the summary:

Criteria	Elicit	Consensus	Semantic Scholar	PaperQA2
Answer accuracy	7.5/10	7/10	N/A (search only)	8/10
Citation accuracy	8/10	7.5/10	9/10 (links)	7.5/10
Speed	Fast	Very fast	Fast	Slow (30-90s)
Ease of use	High	Very high	Medium	Low
Corpus size	~200M	~200M	~200M	Variable
Cost	Free-$10/mo	Free-$9/mo	Free	API costs (~$0.50-2/query)
Best for	Systematic extraction	Quick evidence checks	API integration	Deep, custom analysis

Practical Recommendations

If you're a graduate student starting a literature review: Start with Elicit for structured paper discovery and extraction. Use Semantic Scholar's citation graph to trace important citation chains. Budget: $0-10/month.

If you're a clinician or policy researcher checking evidence: Consensus is the fastest way to get a directional answer on well-studied questions. Supplement with Scite if you need to verify that key findings haven't been contradicted. Budget: $0-30/month.

If you're building a research tool or have a large corpus: PaperQA2 is the best starting point. The LitQA2 benchmark results are real — it genuinely outperforms manual search for factual questions about a defined corpus. Budget: OpenAI API costs, typically $0.50-2 per complex query.

If you're doing a formal systematic review: None of these tools replace a systematic review protocol. Use Elicit for screening and extraction assistance, but maintain human verification at every stage. The error rates, while low, are not low enough for publication-grade work without validation.

The Honest Bottom Line

The AI research agent space is genuinely useful today, but it's not the revolution the marketing suggests. The best tools save 30-60% of the time on mechanical tasks — finding papers, extracting data, checking citations — while doing nothing for the intellectual work of synthesis and interpretation.

The biggest risk isn't that these tools are bad. It's that they're good enough to create false confidence. An Elicit summary reads like a thorough literature review. A Consensus answer feels like a definitive finding. But neither replaces actually reading the papers, understanding the methodological context, and exercising scholarly judgment.

Use these tools as force multipliers, not as replacements for the work. The researchers who get the most value from AI agents are the ones who already know the literature well enough to spot when the agent is wrong.