dsh-llm-as-a-verifier
DeepSeek Harness tool plugin for fine-grained LLM-based candidate verification, best-of-N selection, and step-progress scoring.
Install
$ dsh plugin --profile web add dsh-llm-as-a-verifierPlugin Overview & Capabilities
AI-assisted organization based on the public repository snapshot. The content must be grounded in source evidence and does not replace compatibility or security verification.
Key Capabilities
- Registers `verify_compare` to score two candidates against supplied criteria and return fine-grained rewards.
- Registers `verify_select` for best-of-N selection using a Probabilistic Pivot Tournament.
- Registers `verify_track` to score checkpoints on an A–T (0–100%) progress scale.
- Supports configurable model endpoint, credentials, timeout, concurrency, and per-tool registration.
- Can also be imported as a TypeScript library through its `Verifier` API.
Useful For
- Compare alternative code implementations or solution plans against evaluation criteria.
- Choose a preferred candidate from multiple generated answers.
- Assess whether a sequence of repair or reasoning steps is making progress.
- Use DeepSeek, OpenAI-compatible, vLLM, or SGLang verification backends that return token-level logprobs.
Who It Fits
- DeepSeek Harness users building agent evaluation workflows.
- Developers comparing generated code or solution candidates.
- Teams running compatible local or hosted LLM verification endpoints.
Documented Limitations
- Requires dsh version 0.1.0-rc.6 or later and Node.js 18 or later.
- Verification requires an OpenAI-compatible service that returns token-level logprobs.
- A configured API key or supported environment-variable credential is required when invoking verification tools.
- Without credentials, tools can register but invocation raises `MissingAPIKeyError`.
- The documented default model behavior and DeepSeek-specific path depend on the configured or inferred endpoint.
DSH Compatibility
Version-specific runtime evidence collected by DSH Plugin. A missing result means we have not tested that combination yet.
Security Signals
Objective signals discovered from package metadata and source inspection. These are not a guarantee that a plugin is safe.
package.json declares 0 runtime, 7 development, 4 peer, and 0 optional dependencies.
package.json declares DSH bundle metadata.
GitHub reports the repository license as NOASSERTION.
A root package.json was captured and can be inspected by the registry.
Public GitHub source metadata is available for this registry snapshot.
Source & Registry Notes
Public provenance, Registry classification, and the latest source check for this entry, kept separate from runtime verification.
- Source repository
- TaurenMountain/dsh-llm-as-a-verifier
- Registry source
- GitHub · dsh-plugin topic
- Registry classification
- Plugin
- Source checked
- 4d69526 · 2026-09-14
This project is independently indexed from public source information. DSH Plugin is not affiliated with DeepSeek or the plugin author. Always check the author repository before installation.
Repository Activity
- GitHub stars
- 5GitHub stars
- Forks
- 0
- Open issues
- 0
- Last commit
- 2026-08-24
- Last release
- No release detected
Related DSH Plugins
Ranked by overlapping capabilities, use cases, plugin type, categories, and DSH profile.
DeepSeek Harness-native vision toolkit for image Q&A, OCR, grounding, UI restoration, pixel comparison, and visual artifacts.
View plugin →Vision routing and pixel-level visual tool suite for DeepSeek Harness agents with built-in free fallbacks.
View plugin →Image generation and iterative editing plugin for DeepSeek Harness supporting cloud providers and local ComfyUI.
View plugin →Browser automation plugin enabling DeepSeek Harness agents to interact with logged-in browser sessions safely via dedicated Agent Windows.
View plugin →