Skip to content

fix(generative): send AWS stop_sequences through to the gRPC request - #2131

Open
Anai-Guo wants to merge 1 commit into
weaviate:mainfrom
Anai-Guo:fix-aws-generative-stop-sequences
Open

fix(generative): send AWS stop_sequences through to the gRPC request#2131
Anai-Guo wants to merge 1 commit into
weaviate:mainfrom
Anai-Guo:fix-aws-generative-stop-sequences

Conversation

@Anai-Guo

Copy link
Copy Markdown
Contributor

What

GenerativeConfig.aws_bedrock(...) and aws_sagemaker(...) (the current, non-deprecated AWS generative factories) already accept a stop_sequences argument and store it on the runtime config — but _GenerativeAWS._to_grpc never forwarded it to the gRPC request, so the value was silently dropped and never reached the server.

cfg = GenerativeConfig.aws_bedrock(model="anthropic.claude-3", stop_sequences=["STOP", "\n\n"])
grpc = cfg._to_grpc(_GenerativeConfigRuntimeOptions(return_metadata=True))
# before: grpc.aws.stop_sequences.values == []          <- dropped
# after:  grpc.aws.stop_sequences.values == ['STOP', '\n\n']

Why it's safe to wire unconditionally

The server-side GenerativeAWS message carries stop_sequences in every vendored proto version shipped in this repo (v4216, v5261, v6300), so it can be sent the same way max_tokens already is — no version guard needed. The # TODO - add top_k, top_p & stop_sequences ... comment was stale for stop_sequences; I've narrowed it to the two fields the proto still lacks (top_k, top_p).

Changes

  • _GenerativeAWS._to_grpc: forward stop_sequences=_to_text_array(self.stop_sequences).
  • Narrow the stale TODO to top_k & top_p.
  • Expose stop_sequences on the deprecated aws() factory for parity (it hard-coded stop_sequences=None).
  • Extend the existing AWS _to_grpc test to assert stop_sequences round-trips.

test/collection/test_classes_generative.py passes (20 passed); ruff format --check and ruff check are clean.

🤖 Generated with Claude Code

`GenerativeConfig.aws_bedrock`/`aws_sagemaker` already accept a
`stop_sequences` argument and store it on the runtime config, but
`_GenerativeAWS._to_grpc` never forwarded it, so the value was silently
dropped and never reached the server. The server-side `GenerativeAWS`
proto has carried `stop_sequences` across every vendored version
(v4216/v5261/v6300), so the field can be wired unconditionally the same
way `max_tokens` already is.

Also expose `stop_sequences` on the (deprecated) `aws()` factory for
parity, and narrow the stale TODO to the two fields the proto still
lacks (`top_k`, `top_p`).

@orca-security-eu orca-security-eu Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Infrastructure as Code high 0   medium 0   low 0   info 0 View in Orca
Passed Passed SAST high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Vulnerabilities high 0   medium 0   low 0   info 0 View in Orca

@weaviate-git-bot

Copy link
Copy Markdown

To avoid any confusion in the future about your contribution to Weaviate, we work with a Contributor License Agreement. If you agree, you can simply add a comment to this PR that you agree with the CLA so that we can merge.

beep boop - the Weaviate bot 👋🤖

PS:
Are you already a member of the Weaviate Forum?

target_model: Optional[str] = None,
target_variant: Optional[str] = None,
temperature: Optional[float] = None,
stop_sequences: Optional[List[str]] = None,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remove this field here? We do not want to update deprecated factories

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants