Hi — I maintain EvalPort, an open specification for portable LLM evaluation test suites, test cases, and result sets (JSON Schemas + Python/TypeScript SDKs), aimed at letting eval datasets move between frameworks without losing semantic fidelity.
Flagging a concrete precedent before pitching anything: OpenAI's own SDK team is currently reviewing a community PR that adds native to_openeval()/from_openeval() dataset conversion helpers directly into openai-python, in src/openai/types/evals/openeval.py:
openai/openai-python#3619 (fixes openai/openai-python#3549)
To be accurate about where it stands: it is open, not merged — 6 commits in, review requested, real back-and-forth with an OpenAI maintainer in progress. But it's a signal that a major provider SDK team is willing to carry this conversion logic in-tree.
I'd like to propose something similar for python-genai, grounded in your actual types. From google/genai/tests/transformers/test_t_content.py, a turn is types.Content(role=..., parts=[types.Part(text=...)]), with helper constructors types.UserContent / types.ModelContent, and types.Part also carrying function_call, inline_data, etc. A rough sketch:
# e.g. a new google.genai.openeval module
from google.genai import types
def to_openeval(contents: list[types.Content], *, id: str) -> dict:
"""Convert Content/Part turns into an EvalPort test_case (spec/schemas/testcase.json)."""
...
def from_openeval(item: dict) -> list[types.Content]:
"""Convert an EvalPort test_case back into Content/Part turns, e.g.
types.UserContent(parts=[types.Part(text=...)]) for a user turn,
types.ModelContent(parts=[types.Part(function_call=types.FunctionCall(...))])
for a tool-call turn."""
...
Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
I saw CONTRIBUTING.md currently says the SDK "will accept contributions in the future," so I'm raising this as an early flag rather than assuming a PR is the right vehicle right now. No pressure at all if this isn't a priority — just wanted the precedent and a concrete, type-grounded sketch on record in case it's useful later. Happy to answer questions.
Hi — I maintain EvalPort, an open specification for portable LLM evaluation test suites, test cases, and result sets (JSON Schemas + Python/TypeScript SDKs), aimed at letting eval datasets move between frameworks without losing semantic fidelity.
Flagging a concrete precedent before pitching anything: OpenAI's own SDK team is currently reviewing a community PR that adds native
to_openeval()/from_openeval()dataset conversion helpers directly intoopenai-python, insrc/openai/types/evals/openeval.py:openai/openai-python#3619 (fixes openai/openai-python#3549)
To be accurate about where it stands: it is open, not merged — 6 commits in, review requested, real back-and-forth with an OpenAI maintainer in progress. But it's a signal that a major provider SDK team is willing to carry this conversion logic in-tree.
I'd like to propose something similar for
python-genai, grounded in your actual types. Fromgoogle/genai/tests/transformers/test_t_content.py, a turn istypes.Content(role=..., parts=[types.Part(text=...)]), with helper constructorstypes.UserContent/types.ModelContent, andtypes.Partalso carryingfunction_call,inline_data, etc. A rough sketch:Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
I saw CONTRIBUTING.md currently says the SDK "will accept contributions in the future," so I'm raising this as an early flag rather than assuming a PR is the right vehicle right now. No pressure at all if this isn't a priority — just wanted the precedent and a concrete, type-grounded sketch on record in case it's useful later. Happy to answer questions.