Backend and Model Provider Implementation

This page is for maintainers who need to understand or extend osdk's download capabilities. SDKs and models share networking, source selection, and CAS infrastructure, but use two distinct domain interfaces: SDKs implement Backend, while model repositories implement ModelProvider. Models are not special SDK backends in disguise.

SDK backend contract

Backend is the uniform boundary for every SDK. An implementation supplies a canonical ID, optional aliases, default sources, a probe URL, remote versions, installation logic, executable paths, and executable names. It may also override version resolution, uninstall, post-install behavior, activation environment, and idiomatic version files. The default resolver handles latest, prefix, range, and exact requests and always preserves -o/--opt options in ToolVersion.

Ctx carries directory layout, target platform, merged configuration, the HTTP client, CAS, and progress preference. Archive backends generally produce an InstallPlan and delegate to the shared installation pipeline:

1. obtain a best-first source list using a pin, configured order, or cached probes;
2. download into the shared cache and fail over in source order;
3. verify SHA-256, SHA-512, SRI, or authenticated attestation evidence;
4. safely extract into a scratch directory;
5. ingest content into the BLAKE3 CAS and materialize with hardlink, reflink, or copy;
6. write an artifact receipt and .osdk-complete marker for idempotent and offline reinstall behavior.

The Registry registers built-in backends and aliases and recognizes github:owner/repo, npm:<package>, strict http:https://...{version}..., cargo:<crate-or-https-url>, and go:<module-or-command-path> IDs dynamically. It also loads declarative backends from plugins/*.toml in the user config and data directories. Duplicate IDs or aliases are rejected, so an external definition cannot shadow a built-in backend.

These dynamic namespaces share an option-identity contract for osdk-owned installs. Before resolution or installation, osdk projects the supported public options into a canonical map, rejects unknown public keys, excludes internal __osdk_* lock replay metadata, and computes an order-independent, domain-separated BLAKE3 b3-v2: identity over tool, exact version, platform, scope, canonical material_options, dependencies, and materials. .osdk-install.json schema 1 stores those fields in a nested identity together with install_id, and the fingerprint is part of the physical root. Same-backend/version identities can therefore coexist. Reuse, activation, shim execution, where, uninstall, and reshim require the exact configured identity and never fall back to another fingerprint. .osdk-tool.json, in either old schema 1 or 2 form, is legacy detection only and cannot authorize reuse or execution. Project-managed npm packages remain outside this osdk-owned install identity.

Built-in backend matrix

Backend	Resolution and acquisition	Integrity and installation semantics	Notable behavior or limitation
node (nodejs)	Merged reachable Node indexes from official, npmmirror, TUNA, and USTC	SHASUMS256.txt; shared pipeline verifies each source	Optional arch and corepack; Corepack is a post-install action
npm	npm registry packument/tarball	npm SRI, always required; generates npm/npx launchers	Installed independently of Node, but needs an active Node at runtime
pnpm	Complete pnpm JavaScript distribution	npm SRI; osdk-generated Node launcher	Adds managed Node automatically; store variable depends on major version
yarn	yarn for 1.x, @yarnpkg/cli-dist for 2+	npm SRI; generates Node launchers	Manages Classic and Berry directly instead of delegating to Corepack
go (golang)	go.dev JSON index; mirrors may reuse the official index	Per-file SHA-256; archive pipeline	Exports GOROOT
python (py, cpython)	Built-in PBS release index; merges a tag's SHA256SUMS across sources	Per-release SHA-256; per-source archive fallback	CPython, PyPy, GraalPy, Pyodide, and variants; historical releases can pin tag
java (jdk, openjdk)	Queries Foojay-compatible sources in order, defaulting to Temurin	Per-source checksum detail; JDK/JRE archive	distribution, package-type=jdk|jre; exports JAVA_HOME
maven (mvn)	Built-in single-release record; effective sources ranked by probes	Fixed SHA-512; verified per-source fallback	Current catalog contains one version
gradle	Ranked Gradle version indexes; absolute distribution URLs rebased across sources	Index SHA-256; verified per-source fallback	Supports finished releases and explicitly requested previews from the index
kotlin (kotlinc)	Built-in single GitHub release; effective sources ranked by probes	Fixed SHA-256; verified per-source fallback	Current catalog contains one version
rust (rustup)	rustup channel/version; official, rsproxy, and TUNA	SHA-256 for rustup-init, then isolated rustup runs against each ranked source until the complete command succeeds	Toolchains bypass archive CAS; supports profile, components, and targets; exports isolated RUSTUP_HOME/CARGO_HOME
deno	deno packument plus @deno/<platform>	npm SRI	Platform package; exports DENO_DIR
bun	bun packument plus @oven/bun-<platform>	npm SRI	Platform package; exports BUN_INSTALL_CACHE_DIR
zig	Ranked index.json endpoints; absolute tarball URLs rebased across sources	SHA-256 from the index entry; verified per-source fallback	Platform key uses LLVM CPU tokens; the archive name is read from the index rather than assembled, because the layout changed during 0.14; master is exposed as a prerelease; exports ZIG_GLOBAL_CACHE_DIR
npm:<package>	npm packument; isolated installs use a managed npm subprocess, while project/global use can plan npm or pnpm	A native lock carries transitive integrity; scripts denied by default; .osdk-install.json schema 1 binds installer/build identity in fingerprinted osdk-owned isolated/global roots	Discovers .bin dynamically, adds managed Node, and records scope, installer, optional native-lock identity, and public options in lock schema 4
cargo:<crate-or-https-url>	crates.io-compatible metadata with paired sparse index, or canonical HTTPS Git URL	Registry selection additionally probes the exact .crate download endpoint; exact managed Rust; isolated cargo-binstall/cargo install	Registry exact/latest/prefix or Git latest/tag/branch/full revision; schema 4 records runtime, replay class, and final registry source
go:<module-or-command-path>	Go proxy @latest, version-list, and exact .info metadata, with longest-module-root discovery	Exact managed Go; fresh resolutions supply ranked sources as a native `	-joined GOPROXY; one isolated go install`
github:owner/repo	GitHub API with Atom/public release-page fallback on rate limiting; optional static catalog	Checksums, optional minisign, GitHub artifact attestations; .osdk-install.json schema 1 binds asset/layout/material identity in fingerprinted roots	Selects a host asset; supports archives and bare binaries; regex/template/bin/rename/strip rules handle complex releases

These implementations live under backend/. The npm-backed implementations share packument, version, and SRI handling in npm.rs. Generic source ranking is in source/select.rs. See npm developer tool implementation for the complete dynamic backend project/global/isolated installation, cache, metadata-only lock, legacy lock-schema-2 sidecar compatibility, and shim boundaries. See Cargo developer tool implementation for strict selectors, exact Rust binding, controlled provider fallback, and native publication. See Go developer tool implementation for module-root discovery, proxy routing, build-environment policy, runtime binding, and replay boundaries.

Declarative and GitHub backends

DeclarativeBackend is a constrained schema-1 TOML extension point. It supports static or URL version lists, platform template variables, tar.gz/tar.xz/tar.zst/zip/7z, fixed or remote checksums, strip_root, binary paths, and idiomatic version files. Platform templates expose both {arch}, osdk's short token, and {arch_llvm}, the CPU part of an LLVM target triple, because compiler and toolchain archives are normally published as x86_64/aarch64 rather than x64/arm64; the latter reuses Arch::llvm_token instead of introducing a second naming table. Definitions are limited to 1 MiB, remote lists to 10,000 versions, and URLs, filenames, relative paths, and checksums are strictly validated. It intentionally cannot execute hooks or arbitrary commands; every installation goes through the shared verification and CAS pipeline. When a project lock supplies a generic artifact receipt, the backend consumes the recorded URL, filename, checksum, and subdirectory before consulting its current templates. This gives declarative tools the same metadata-free offline reinstall contract as built-in archive backends. An optional [env] table lets a definition describe the environment its toolchain needs, which is what makes a compiler usable: build systems locate a cross compiler through CC, SYSROOT, and similar variables rather than through PATH. Values are rendered from {install_path}, {version}, and {id} only, and exec_env fails closed if rendering would leave an unresolved placeholder. Names are validated as conventional environment identifiers; PATH and the dynamic-loader variables (LD_PRELOAD, LD_LIBRARY_PATH, DYLD_INSERT_LIBRARIES, DYLD_LIBRARY_PATH) are reserved case-insensitively, and absolute paths, .., and control characters are rejected at parse time, so a data-only definition cannot point a child process outside its installation root. An optional [[archive.overrides]] list exists because a single template cannot express an upstream that renames its assets. LLVM is the case that shaped the design: Linux x86-64 moved from clang+llvm-<version>-x86_64-linux-gnu-ubuntu-18.04 to LLVM-<version>-Linux-X64 in 19.1.0 while Windows kept the earlier spelling, and the embedded distro version cannot be derived from any platform fact, so the rename is per platform and partly unpredictable. Each entry therefore matches on versions (a semver::VersionReq) combined with os, arch, and libc, and replaces whole fields (url, file, kind, strip_root, checksum) rather than fragments, falling back to [archive] for anything it leaves unset. Resolution ranks matches by how many conditions they constrain, so the outcome does not depend on declaration order; a tie is an error rather than an order-dependent pick, because ordering is easy to reshuffle by accident. Arch conditions accept both the {arch} and {arch_llvm} spellings to avoid a second vocabulary, and a non-semver version simply fails to match a requirement instead of aborting an install. Everything checkable without a concrete platform — an empty condition set, an entry that replaces nothing, an invalid requirement, a template that cannot vary by version — is rejected at parse time, so a broken definition fails on load rather than on whichever machine matches it. archive.checksum.attestation exists because some upstreams publish no digest file at all: LLVM ships .sig and, from 19.1.0, .jsonl sigstore bundles, but no .sha256. A bundle's in-toto subject already carries the artifact's SHA-256, so it is simultaneously the signature and the digest source, and verify_github_attestation already implemented the check that gh attestation verify --repo <owner>/<repo> performs, without depending on the gh CLI. The pipeline already treated attestation evidence as an authenticated checksum, so the backend returns Ok(None) from checksum — the digest is genuinely unknown until the bytes exist — and passes a GithubAttestation instead; the archive is still never extracted unverified. The policy is pinned to Required rather than inherited from settings.attestations, whose default is off: when the attestation is the digest source, deferring to a global off would install an archive with no integrity evidence, so attestation_request sets the policy itself and a test asserts it does not follow the global default. repo becomes a GitHubWorkflowRepository certificate-identity policy, so it is parsed and validated as owner/repo rather than interpolated as given. Coverage is not universal and cannot be assumed: attestations exist only for workflow-built artifacts after the upstream adopted them, which for LLVM means 19.1.0 onward and only some assets per release, so switching digest source per version and platform is the [[archive.overrides]] case rather than an edge case.

GithubBackend is a namespaced backend constructed at runtime. It reads up to 1,000 paginated releases, ignores drafts, applies prerelease policy, and scores assets for OS, architecture, and libc. Explicit rules handle non-standard asset names. Online, when signature verification is enabled, an available trusted minisign checksum manifest overrides a preloaded static digest; otherwise the static digest is used before ordinary sidecar/shared checksum discovery. The configured GitHub attestation policy is applied independently. GitHub API, page, Raw, release asset, and attestation URLs all use the same normalized source candidates, while credentials are sent only to the official API host. Its supported asset, platform, catalog-digest, rename, bin, and strip options are validated as public identity inputs and stored in the schema-1 dynamic install manifest. catalog-url is accepted for acquisition but deliberately omitted because the required catalog-sha256 identifies content without persisting the catalog location in the dynamic inventory. HTTP(S) catalog URLs containing userinfo, query parameters, or fragments are rejected. Consequently, a complete marker alone cannot reuse a GitHub install produced by different options or by a legacy inventory. Locked replay additionally matches the persisted artifact receipt's filename, checksum, and subdirectory. The caller must uninstall and reinstall that version after a mismatch. Inventory is published before the completion marker so an interrupted finalization cannot be treated as reusable.

Models are separate and provider-specific

A model reference includes its provider: repository providers use hf:owner/repo@revision or ms:owner/repo@revision, while Civitai uses exact civitai:model-id@model-version-id. ProviderId and ModelRef make the provider part of identity. ModelProvider standardizes only the resolved revision plus file manifest; it does not assume compatible service APIs.

Semantic	Hugging Face	ModelScope	Civitai
Implementation	huggingface.rs	modelscope.rs	civitai.rs
Metadata API	/api/models/{repo}/revision/{revision}?blobs=true	/api/v1/models/{repo}/repo/files?Revision=...&Recursive=true	/api/v1/model-versions/{version-id}
File URL	/{repo}/resolve/{commit}/{path}	/api/v1/models/{repo}/repo?Revision=...&FilePath=...	API-provided downloadUrl, which may redirect to a CDN
Immutable revision	Commit SHA returned by the service	Requested revision plus a BLAKE3 digest of the sorted path/size/SHA-256 manifest	Exact model-version ID
Selection and digest	LFS entries carry SHA-256; a missing regular-blob digest is computed after download	The API must return a valid SHA-256 for every file	Select one Model weight by SafeTensor, primary, then response order; SHA-256 is mandatory and the path is normalized to loras/<filename>
Token	OSDK_HF_TOKEN → HF_TOKEN → HUGGING_FACE_HUB_TOKEN; Bearer	OSDK_MODELSCOPE_TOKEN → MODELSCOPE_API_TOKEN; Bearer plus m_session_id cookie	OSDK_CIVITAI_TOKEN → CIVITAI_API_TOKEN → CIVITAI_TOKEN; Bearer, removed on cross-origin redirects
Default endpoint	https://huggingface.co	Prefer https://modelscope.cn, then https://www.modelscope.ai	Both official front doors: https://civitai.com and https://civitai.red

The three providers have different metadata schemas, download URLs, authentication, and immutable identities; each implementation rejects another provider's ModelRef. Automatic ranking and failover remain inside one provider's endpoint set. Civitai treats .com (the SFW front door) and .red (the full-catalog front door) as credential-bearing official sources, preserves the historical official id, and adds official-red; both canonicalize to .com in the lock, while probing still requires the requested exact version to be available through that front door. The Civitai provider accepts exact IDs already selected by an upper layer such as ogen; it does not search, rank, or match trigger words.

Model resolution, download, and materialization

The CLI entry point for osdk model sync [name] is in commands/runtimes.rs, with the core flow in model/pull.rs:

1. resolve an explicit --endpoint or provider endpoint environment variable; otherwise use that provider's default and custom sources;
2. in auto mode, model/source.rs fetches a real repository manifest and performs a Range request of at most 64 KiB against the smallest non-empty file; ranking is cached by provider, repository, revision, and source configuration;
3. let the provider resolve its remote manifest, then apply --include/--exclude globs. variant and semantic metadata kind/family/derived_from all enter snapshot identity. An omitted Civitai kind consistently becomes lora; other providers are not guessed. Caller-supplied family and derived_from are checked for length, trimming, and control characters;
4. download selected files concurrently up to settings.jobs into a provider/repository/revision-separated cache, with resume support and size/SHA-256 verification;
5. have ModelStore verify again, ingest each file into the shared CAS, finish the snapshot in a hidden temporary directory, rename it to <models>/<logical-name>/snapshots/<snapshot-key>, and update current.json through another temporary-file rename; these renames have no portable replace-atomicity or durability guarantee; then repoint the <models>/<logical-name>/current directory link at the new snapshot. Snapshot directory names are derived from a content hash that covers the file selection, so changing --include produces a different directory and any path written into an external config silently stops matching -- while ComfyUI, llama.cpp and vLLM all take a path and keep it. Windows uses a junction rather than a symlink because a symlink needs Developer Mode or elevation and a junction needs neither; repointing fails loudly if a real directory occupies current, rather than silently deleting user data. A link failure is logged as a warning and does not fail the publish: the snapshot and current.json are already durable, discarding a completed download over one link would be wrong, and model path without --stable still answers from current.json;
6. record provider, repository, requested/resolved revision, endpoint, variant, kind/family/derived_from, and every file's size/SHA-256 in top-level [models] in osdk.lock. The same semantic metadata is stored in .osdk-model.json, and machine JSON reads it back from the manifest. Tokens and short-lived download URLs are never persisted.

Local imports use model/local.rs rather than ModelProvider. The walker uses symlink_metadata and rejects symlinks, Windows reparse points/junctions, and special files before descending. Separators are normalized to /, then absolute paths, empty segments, ./.., colons, and control characters are rejected. Each regular file receives SHA-256; sorted path/size/SHA-256 tuples produce a local-<24 hex> revision; the same ModelStore::publish then copies bytes into CAS and publishes atomically. ProviderId::Local exists only for manifests and machine output: ModelRef::parse, source/provider, and environment paths do not accept it.

A local snapshot manifest never persists the input absolute path. Both lock writing and lock replay reject provider=local, because another machine cannot reproduce an arbitrary local path. Before publishing, the CLI rejects same-name project declarations/locks and an hf-cache view. It may render a ComfyUI view, and a later import re-renders any persisted view membership. --json reuses schema-1 ModelShowOutput without human text on stdout.

model list/show/path/verify/remove operate on the current logical name. Verification checks both the CAS BLAKE3 hash and SHA-256. Removal deletes all snapshots under that logical name, then runs CAS GC with SDK installs and models as roots. Offline sync still requires cached provider metadata and every selected download, after which it can rematerialize a removed snapshot.

The machine protocol is centralized in model_output.rs and is independent of snapshot, lock, and view-state disk schemas. import --json, list/show/path/verify, and read-only view commands emit one schema-1 JSON document, while sync --jsonl uses one emitter for line-delimited events; human mode keeps its existing messages. Every machine stdout write passes through the same serializer. View reconciliation still runs in JSONL mode but suppresses its human report; the top level continues to write errors to stderr and exit non-zero. Absolute paths in the protocol are machine locations and retain native separators, while manifest-relative paths remain /-normalized cross-platform data.

Declarative [models], views, and trust classification

model use writes declarations in the project's osdk.toml under [models.<name>], shaped by ModelDeclaration in config/mod.rs (deny_unknown_fields; like tasks it sits behind the install feature, so the shim's dependency graph does not carry it). An entry has source/include/exclude/variant/kind/family/derived_from/when plus views: consumer -> ModelViewDeclaration{profile, map}. During sync [name], the merged Config.models supplies it: after fetching, the views are written into the lock via locked_views_from_declaration, and model_view::reconcile_declared_views renders them immediately.

On the lock side LockedModel.views is consumer -> LockedModelView{profile, map} with #[serde(default, skip_serializing_if)], so it is omitted when empty; the schema stays 4 and an older binary ignores the unknown field (verified with a real older build). set_model_views gives the field a read/update path that mirrors its write path, avoiding the write-only gap AGENTS.md records for npm. After restoring a snapshot, model sync calls the same reconcile, so the view declarations are rebuilt on another machine by sync alone -- no second model view add.

Trust lives in trust.rs: model configuration belongs to no scope -- no key requires trust, including endpoint. Model bytes are content and osdk never executes them; downloads still verify against pinned digests, so a redirected endpoint cannot turn a content fetch into code execution. Model keys never enter the normalized hash, so editing model entries neither re-prompts nor invalidates a record. affects_tool_dispatch returns false for every models.. key.

Provider environment persistence

osdk model env enable [provider] [--force] writes only sources.<provider>.env and optional env_force to the user-level config; project config cannot override those switches. Activation behavior lives in model/env.rs:

• Hugging Face exports HF_ENDPOINT, HF_HOME, HF_HUB_CACHE, HF_XET_CACHE, and HF_ASSETS_CACHE; osdk offline mode additionally exports the officially supported HF_HUB_OFFLINE=1.
• ModelScope exports MODELSCOPE_ENDPOINT and MODELSCOPE_CACHE; osdk does not invent a MODELSCOPE_OFFLINE variable.
• Civitai has no standard downstream environment adapter; model env civitai fails before writing configuration.
• Existing user variables win by default; --force permits replacement. Shell activation captures original values so disable/deactivate can restore them.
• Custom endpoints default to forward_credentials=false. When osdk manages such an endpoint, it clears provider token variables and uses an isolated anonymous home to prevent local login cookies or tokens from leaking. Download requests carry credentials only for recognized official endpoints or after explicit --forward-credentials. Tokens are never stored in osdk configuration.

Boundaries and caveats

• SDK locks are platform-keyed; model locks are top-level because model files are normally platform-independent. A model variant is a caller-supplied label: it neither infers a quantization format nor changes file selection.
• Online provider identity is present in references, metadata/ranking/download caches, snapshot keys, manifests, and locks. Local imports retain provider=local only in manifests and machine output and never enter source selection or locks. However, the top-level models map and local current.json are keyed by the caller's logical name. Pulling another provider under the same logical name switches that name's current snapshot and replaces its lock entry, although stored snapshots remain provider-distinct.
• A non-LFS Hugging Face blob may lack a server-provided SHA-256. osdk computes and locks one after download, but that is not an independent digest supplied by the service. ModelScope requires a valid SHA-256 in its API manifest; Civitai likewise requires one for the selected weight.
• Online metadata failures may fall back to stale cache. A custom endpoint must implement the selected provider's actual API; hosting compatible files or replacing only the domain is insufficient.
• GitHub asset scoring is heuristic. Use explicit asset rules or a trusted static catalog when names are ambiguous or a release contains several similar artifacts.
• Rust is a delegate backend: isolated rustup owns the toolchain, so it does not get ordinary archive backends' per-file CAS deduplication. Maven, Gradle, and Kotlin currently expose a built-in one-version catalog rather than a complete remote version index.