Hare/yoi
1
1
Fork 0
Code Actions Packages Releases Activity
fed461036a
yoi/work-items/open/20260528-152959-web-search-fetch-tools/item.md

6.9 KiB
Raw Blame History

id slug title status kind priority labels created_at updated_at assignee legacy_ticket
20260528-152959-web-search-fetch-tools web-search-fetch-tools Add WebSearch and WebFetch tools open task P2
tools
web
llm
2026-05-28T15:29:59Z 2026-05-28T15:29:59Z null null

Background

Insomnia currently has strong local filesystem / shell / memory tools, but the agent cannot directly consult current web information except through user-provided excerpts or shell commands. Add first-class WebSearch and WebFetch tools so the model can gather public web information through bounded, observable tool calls.

This should be implemented as normal built-in tools, not as hidden context injection. Tool calls and results must remain visible in history, subject to manifest permission policy, and bounded by output limits.

Requirement

  • Add WebSearch tool.
    • Input includes query string and optional result limit.
    • Output returns structured results: title, URL, snippet/summary, source/search provider metadata where available.
    • Search provider must be configurable. If no provider/API key is configured, the tool should fail with a clear diagnostic instead of falling back to scraping arbitrary search pages.
  • Add WebFetch tool.
    • Input includes URL and optional mode/limits.
    • Output returns normalized text content plus metadata such as final URL, status, content type, title if available, and byte/token truncation indication.
    • HTML should be converted to readable text. Non-text content should be rejected or summarized only when a safe explicit handler exists.
  • Add manifest configuration for web tools.
    • Enable/disable controls.
    • Search provider/API key configuration.
    • Fetch timeout, max response bytes, max output bytes/tokens, redirect limit.
    • Allowed/denied URL schemes and host policy.
  • Integrate with built-in tool registration and manifest permission policy.
    • Web tools are normal tool calls and should go through the existing tool permission mechanism.
    • No implicit network access should happen outside a tool call.
  • Add security and reliability protections.
    • Only http/https by default.
    • Reject local/private/link-local/loopback addresses by default unless explicitly configured.
    • Bound redirects and re-check final URLs.
    • Bound download size and output size.
    • Provide clear errors for timeout, DNS/network failure, unsupported content, blocked host/scheme, and truncation.
  • Prompts/tool descriptions should tell the model when to use WebSearch vs WebFetch and that fetched content may be stale/untrusted.

Brave Search API notes

https://github.com/brave/brave-search-mcp-server was checked as the reference implementation. It is an MCP server around Brave Search APIs, not a general page fetcher.

Useful details to mirror for WebSearch:

  • Use https://api.search.brave.com/res/v1/web/search for the first provider.
  • Authenticate with X-Subscription-Token: <api key>; Insomnia should read the key from a configured environment variable such as BRAVE_SEARCH_API_KEY rather than storing raw secrets in the manifest.
  • Send q for the query. Validate query length up front: Brave's MCP server caps at 400 characters and 50 words.
  • Expose a small initial subset of Brave parameters rather than the full API surface:
    • count / result limit: 1-20.
    • offset: 0-9 if pagination is included.
    • country, search_lang, ui_lang as optional config/defaults, not necessarily per-call in the first version.
    • safesearch: default moderate.
    • freshness: optional (pd, pw, pm, py, or date range) can be added if easy, but is not required for the first cut.
  • Format output conservatively. The MCP server reduces web results to { url, title, description, extra_snippets } and optionally emits FAQ, discussions, news, and videos. Insomnia's first version should return the core web result fields plus provider metadata, and may ignore non-web result buckets unless explicitly requested later.
  • Brave's public docs/reference include an LLM Context endpoint (/res/v1/summarizer/llm_context) that returns extracted snippets/content with token and per-URL limits. This is useful as a future WebContext/enhanced search provider, but should not replace WebFetch in the first implementation because it is provider-specific and not a direct URL fetch tool.
  • Brave MCP defines a nominal free-plan rate limit of 1 request/second and 15,000/month and does not implement robust self-throttling. Insomnia should at least surface HTTP 429/rate-limit errors clearly; local throttling can be a follow-up unless implementation is cheap.

Implementation plan

  • First implementation should be provider-independent normal function tools, not OpenAI/Codex hosted web_search.
  • Add a small typed web config to manifest resolution and pass it into built-in tool registration.
    • Default disabled/no provider should produce clear tool errors, not implicit network access.
    • Prefer environment-variable secret references for API keys; do not require raw keys in manifest files.
  • Implement WebSearch with Brave as the first provider.
    • Keep the first-call schema small: query, optional limit, and optional offset only unless a broader parameter is trivial and well-tested.
    • Use deterministic JSON output with provider metadata and a bounded result list.
  • Implement WebFetch independently from Brave.
    • Use an HTTP client with timeout, redirect limits, response byte limits, and content-type checks.
    • Enforce URL/host/IP policy before fetch and on every redirect.
    • Convert HTML to readable text with a scoped dependency or a small internal extractor; reject unsupported binary content.
  • Register both tools through the existing built-in tool path so existing permission policy and history/logging behavior applies naturally.
  • Do not implement provider-hosted OpenAI/Codex web_search in this ticket; that is a separate model/provider hosted-tool feature.

Acceptance criteria

  • WebSearch and WebFetch are registered built-in tools when enabled/configured.
  • Tool schemas are typed and validated.
  • Manifest docs/config examples describe how to enable/configure web tools.
  • Permission policy can allow/deny/ask these tools like other tools.
  • Tool results are bounded and visible in history; no hidden web context is injected.
  • Unit tests cover input validation, disabled/unconfigured errors, URL policy, redirect/final URL policy, output truncation, and representative HTML-to-text conversion.
  • At least one integration-style test uses a local test HTTP server or mock provider rather than the public internet.
  • cargo fmt --check
  • cargo check -p tools -p manifest -p pod
  • Relevant focused tests for tools/manifest.

Out of scope

  • Browser automation.
  • Authenticated browsing / cookies / sessions.
  • Javascript rendering.
  • File downloads as attachments.
  • Using arbitrary shell commands as the primary web access path.
  • Hidden pre-request browsing or automatic web context injection.