INTEGRATION GUIDE · BETA · SOURCE CHECKED 16 NOV 2026
Migrate research toolkits from the Perspective API (before 31 Dec 2026)
Perspective-shaped requests, independent scores. Change the connection, then validate the moderation decisions.
Google’s Perspective API notice gives 31 December 2026 as the shutdown date. Plan and test your migration before then.
1. Prepare a staging copy
Save the integration version, settings, current thresholds and old scores before changing providers. Create a Decision Models key in the dashboard; store it only on your server. Examples use environment variables, never real keys.
These tools use Perspective as an evaluator, separate from the model being evaluated. Changing the evaluator changes the experiment. Record Decision Models as the scoring provider and retain model/scoring version, dates, dataset hashes, attributes, languages and thresholds. New scores are not comparable as if they were historical Perspective numbers.
2. Change the endpoint and key
HELM
HELM discovery client hard-codes discoveryServiceUrl. Replace that keyword value as shown below and set the Perspective credential used by HELM to your Decision Models key. HELM batch code uses new_batch_http_request(). The replacement has no Google multipart batch transport, so changing the discovery URL alone is insufficient.
- discoveryServiceUrl="https://commentanalyzer.googleapis.com/$discovery/rest?version=v1alpha1",
+ discoveryServiceUrl="https://moderation.decisionmodels.io/$discovery/rest?version=v1alpha1",In your maintained HELM fork, replace the batching block with sequential calls while holding _client_lock. Keep the outer cache wrapper and returned text_to_response map. The following is the replacement for the batch construction/add/execute block, not a complete upstream patch:
import time
with self._client_lock:
for index, text in enumerate(set(request.text_batch)):
if index:
time.sleep(1) # Conservative Free-plan pacing
text_to_response[text] = self._client.comments().analyze(
body=PerspectiveAPIClient.create_request_body(
text[:PerspectiveAPIClient.MAX_TEXT_LENGTH],
request.attributes, request.languages
)
).execute()
return text_to_responseThe adapter paces requests conservatively at one request/s. Add bounded timeouts and rate-limit handling around those individual calls. This transport adaptation needs local integration tests before a full run. Start with a new cache namespace; HELM’s request cache must not mix providers.
lm-evaluation-harness
PR #4448 proposes PERSPECTIVE_API_URL and was open/unmerged when inspected on 16 November 2026. These variables work only in a revision containing that change; they are not a promise about your installed release:
export PERSPECTIVE_API_URL='https://moderation.decisionmodels.io/v1alpha1/comments:analyze'
export PERSPECTIVE_API_KEY="$DM_MODERATION_KEY"
export PERSPECTIVE_API_QPS=1If absent, apply the PR’s endpoint change in a maintained fork of lm_eval/tasks/realtoxicityprompts/metric.py, retaining the existing key query parameter. The URL value is the full analyze endpoint, not the discovery URL.
TrustLLM and SB Intuitions flexeval
TrustLLM discovery client and SB Intuitions flexeval metric each hard-code discoveryServiceUrl. Apply the same one-line discovery URL patch shown for HELM. TrustLLM reads trustllm.config.perspective_key; set that field privately to your Decision Models key. flexeval reads PERSPECTIVE_API_KEY at import time; set it before starting Python. Here “flexeval” means sbintuitions/flexeval, not another project with a similar name.
TrustLLM also reads spanScores without requesting spanAnnotations. Our API only returns spanScores when requested. In get_toxicity_value(), apply this additional response-extraction patch; changing the URL alone will otherwise leave that helper failing:
- return response['attributeScores']['TOXICITY']['spanScores'][0]['score']['value']
+ return response['attributeScores']['TOXICITY']['summaryScore']['value']Attributes and compatibility
HELM defaults request six Perspective attribute names plus SEXUALLY_EXPLICIT and FLIRTATION, both experimental here. SEVERE_TOXICITY is also experimental here. For evaluated-only experiments, explicitly choose TOXICITY, INSULT, THREAT, PROFANITY and IDENTITY_ATTACK and label the changed protocol. Held-out English measurements are published in the documentation. TrustLLM requests TOXICITY; its original helper reads the first span score, and the patch above reads summaryScore instead. The harness PR requests TOXICITY and reads summaryScore. flexeval requests those six Perspective attributes (including experimental SEVERE_TOXICITY here) and reports per-attribute means. Whole-text spans are compatible in shape, not equivalent in meaning to historical span scores.
Built-in and experimental attribute names
Built-in: TOXICITY, IDENTITY_ATTACK, INSULT, PROFANITY, THREAT. Built-in does not mean score parity with Perspective.
Experimental, not evaluated and uncalibrated: SEVERE_TOXICITY, SEXUALLY_EXPLICIT, FLIRTATION, SPAM, ATTACK_ON_AUTHOR, ATTACK_ON_COMMENTER, INCOHERENT, INFLAMMATORY, LIKELY_TO_REJECT, OBSCENE, UNSUBSTANTIAL. Responses identify requested experimental attributes in X-DM-Experimental-Attributes. Unknown attributes return 400. Check the current attribute and language documentation before enabling automatic actions.
Re-tune the thresholds
The harness metric defaults to toxicity_threshold=0.5, with ≥ treated as toxic. The inspected HELM client, TrustLLM helper and flexeval metric return continuous values rather than imposing one common moderation threshold; downstream analyses may add thresholds. Re-tune decision thresholds, and report the new evaluator separately even when retaining a historical cutoff for a clearly labeled sensitivity analysis.
Use the migration tool with representative comments, their saved Perspective scores and optional human labels. Compare flag rates, flagged-set overlap and, where labels exist, precision and recall. Matching a flag rate does not guarantee the same flagged comments or accuracy. Tune each attribute and action separately; do not copy the old numbers into production without review.
3. Test and re-tune
With DM_MODERATION_KEY set privately, this smoke command should return HTTP 200 and a numeric attributeScores.TOXICITY.summaryScore.value between 0 and 1. It checks transport and response shape, not moderation quality.
curl --fail-with-body --max-time 15 \
'https://moderation.decisionmodels.io/v1alpha1/comments:analyze' \
-H "Authorization: Bearer $DM_MODERATION_KEY" \
-H 'Content-Type: application/json' \
--data '{"comment":{"text":"Thank you for explaining."},"requestedAttributes":{"TOXICITY":{}},"doNotStore":true}'Run the curl smoke first. For patched flexeval, this is a one-output metric check; replace the language only after checking support:
PERSPECTIVE_API_KEY="$DM_MODERATION_KEY" python3 - <<'PYTEST'
from flexeval.core.metric.perspective_api import PerspectiveAPI
print(PerspectiveAPI(languages=["en"]).evaluate(["Thank you for explaining."]))
PYTESTFor patched HELM, verify one request through the individual-call adapter before a batch. For TrustLLM, call PerspectiveEval().get_toxicity_value() on one staged string. For the harness, run the PR’s test_realtoxicityprompts.py tests locally before a tiny dataset subset. Compare counts: the harness PR can turn retrieval failures into zero scores, and TrustLLM can skip failed items. Do not interpret missing or failed items as evidence of low toxicity.
In staging, test benign, abusive, borderline, quoted and multilingual comments; compare actual queue or warning actions with human decisions. Also test invalid keys, untested-language warnings, 429 rate limits and a timeout. Keep a manual moderation fallback rather than interpreting a failed request as a safe score. Check limits and errors and plan quotas and request rates.
4. Roll out with a fallback
Use fresh result directories and provider-specific caches. Do not resume a run whose saved scores came from Google. Preserve old raw scores for reproducibility; rollback means restoring the code revision, credential, provider-specific cache and experiment manifest together. A full migration requires validating each toolkit’s orchestration, not just this transport smoke test.
Keep the saved version and settings so you can reverse this change. Before Google’s shutdown, you can restore the Google endpoint together with its Google key and old thresholds if that service remains available to your account. After shutdown, use a tested alternative or manual review. Never send a Decision Models key to the Google host, or a Google key to Decision Models.
Shared gaps: scores come from an independent model; sentence-level span annotations are unavailable (one whole-text span only). Only PLAIN_TEXT is accepted. Valid explicit language hints are accepted and echoed. Untested hints or detected languages carry X-DM-Language-Untested. Tested languages: English (5 attributes) and German (offensive-language proxy only). This is not evidence of language quality. Suggested-score feedback is acknowledged but not stored or used to train the model. See the full API contract.
Source notes
These instructions follow the linked source revisions, inspected on 16 November 2026. Your installed release may differ. Patches and examples are migration recipes, not upstream endorsements or a claim of an end-to-end verified integration.
HELM discovery client · HELM batching/cache · TrustLLM discovery client · SB Intuitions flexeval metric · proposed harness request and failure behavior