Skip to content

Native to_openeval()/from_openeval() dataset conversion helpers (parity with an in-review openai-python PR) #2901

Description

@adhabnr-ux

Hi — I maintain EvalPort, an open specification for portable LLM evaluation test suites, test cases, and result sets (JSON Schemas + Python/TypeScript SDKs), aimed at letting eval datasets move between frameworks without losing semantic fidelity.

Flagging a concrete precedent before pitching anything: OpenAI's own SDK team is currently reviewing a community PR that adds native to_openeval()/from_openeval() dataset conversion helpers directly into openai-python, in src/openai/types/evals/openeval.py:

openai/openai-python#3619 (fixes openai/openai-python#3549)

To be accurate about where it stands: it is open, not merged — 6 commits in, review requested, real back-and-forth with an OpenAI maintainer in progress. But it's a signal that a major provider SDK team is willing to carry this conversion logic in-tree.

I'd like to propose something similar for python-genai, grounded in your actual types. From google/genai/tests/transformers/test_t_content.py, a turn is types.Content(role=..., parts=[types.Part(text=...)]), with helper constructors types.UserContent / types.ModelContent, and types.Part also carrying function_call, inline_data, etc. A rough sketch:

# e.g. a new google.genai.openeval module

from google.genai import types

def to_openeval(contents: list[types.Content], *, id: str) -> dict:
    """Convert Content/Part turns into an EvalPort test_case (spec/schemas/testcase.json)."""
    ...

def from_openeval(item: dict) -> list[types.Content]:
    """Convert an EvalPort test_case back into Content/Part turns, e.g.
    types.UserContent(parts=[types.Part(text=...)]) for a user turn,
    types.ModelContent(parts=[types.Part(function_call=types.FunctionCall(...))])
    for a tool-call turn."""
    ...

Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

I saw CONTRIBUTING.md currently says the SDK "will accept contributions in the future," so I'm raising this as an early flag rather than assuming a PR is the right vehicle right now. No pressure at all if this isn't a priority — just wanted the precedent and a concrete, type-grounded sketch on record in case it's useful later. Happy to answer questions.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions