Skip to content

add-new-tool.md

Latest commit

 

History

History
486 lines (342 loc) · 27.5 KB

add-new-tool.md

File metadata and controls

486 lines (342 loc) · 27.5 KB
Raw
Error in user YAML: (<unknown>): did not find expected comment or line break while scanning a block scalar at line 6 column 1
---

## name: add-new-tool
description: Add a new tool from scratch — research auth, validate against a live instance, write files to TENX_PRIVATE_DIR/personal/{tool-name}/. Use when a tool has no recipe yet. Do NOT use this if the tool already exists in tool_connections/ — use setup.md instead.

# Add a New Tool

> **What this file is for:** The tool has no recipe yet anywhere (`tool_connections/` or `TENX_PRIVATE_DIR/personal/`). You are building one from scratch — researching auth, validating against a live instance, and writing the files to `TENX_PRIVATE_DIR/personal/{tool-name}/` for your own use.
>
> **Wrong file?** If the tool already exists in `tool_connections/` or `TENX_PRIVATE_DIR/personal/`, use `setup.md` instead — it will route you to the right recipe and handle patching if something is broken.
>
> **Want to contribute back?** After completing Phase 1, read `contributing.md`.

## Purpose

Turn "I want my agent to access Tool X" into a working, verified connection file that any agent can pick up and use.

**Phase 1 (always):** Research, validate, and write the connection for your own use.
**Phase 2 (optional):** Contribute it back to the repo as a PR — only if the tool is commercial and publicly available.

---

Operating context

Connections here automate daily personal and professional workflows through the coding agent — check messages, read feeds, look up context, draft replies, file tickets, and similar one-off tasks. They are not for mass scraping, bulk harvesting, high-frequency unattended polling, or simulating heavy human usage at scale.

When researching and choosing a connection method:

  • Do not overweight generic bot-detection or account-ban warnings aimed at scrapers, growth bots, or commercial automation. That is not this repo's threat model.
  • Do prioritize:
    1. Feasibility — does the method work for this account/instance? (e.g. Web WeChat login allowed, API reachable)
    2. Privacy — does the connector or its dependencies send data to third parties? Review source before run.
    3. Write safety — preview + explicit user approval before every outbound action (post, send, delete, purchase, etc.)
    4. Proportionate use — on-demand or modest polling while the agent is working; not continuous bulk export
  • Still stop when a platform hard-blocks the auth path. Treat that as a feasibility gate, not a lecture about account safety.
  • Document technical constraints the agent must implement correctly — session TTL, calls that must run inside a browser context, rate limits, instance policy — as facts, not as reasons to abandon the connection without trying.

Dedicated automation accounts (separate from day-to-day personal use) are acceptable when a platform allows it.

Non-negotiable rules

  1. TENX_PRIVATE_DIR/personal/ first, always. All work — new tools, improvements to existing connections, new auth variants, fixes — starts in TENX_PRIVATE_DIR/personal/. Never edit tool_connections/ directly. TENX_PRIVATE_DIR/personal/ lives outside the public repo and is safe for your email, org URLs, tokens, and company-specific details. Nothing leaves TENX_PRIVATE_DIR/personal/ until it is verified, scrubbed, and promoted via staging/ → PR. This applies to improvements just as much as new tools.
  2. Research viability first. Before asking the user for anything, determine what auth methods exist for this tool. Prefer a supported API. If no suitable API exists, treat Agent Browser over an authenticated browser session as the default operational fallback; do not jump straight to a custom Playwright/CDP script.
  3. Ask only what the auth method actually needs. The credential ask must be proportional to the auth method: SSO/browser-session → ask for nothing (just a URL to confirm the instance); API token → ask for the token and where to generate it; username+password → ask for both. Never ask vague questions the user can't answer.
  4. A URL is your best minimal input. If you need to confirm an instance, ask for any URL from that tool (profile page, dashboard, ticket). It reveals the base URL, regional variant, and proves the user has access — without requiring them to know anything about auth.
  5. Run before you write. Every snippet must be code you actually executed and saw succeed against a live instance. No copy-paste from docs. No illustrative output. The reason you haven't run them does not matter — unverified snippets do not belong in a connection file.
  6. Write for the next agent. Strip session-specific IDs, one-time URLs, org-specific data. Document the pattern, not the artifact.
  7. Nothing broken. If an endpoint didn't work, cut it. One working snippet beats five broken ones.
  8. Python SSL: use urlopen(), never roll ssl.CERT_NONE. Managed laptops may run Zscaler, which can intercept HTTPS traffic. from tool_connections.shared_utils.browser import urlopen tries normal TLS verification first, then retries only SSL failures with the Zscaler-compatible context. Do not copy-paste ssl.CERT_NONE blocks — they break on machines without Zscaler and hide real certificate problems.

Safety gate before third-party code

Before cloning third-party repos, running install scripts, installing packages, or executing a local bridge/binary, run the Genesis package safety review:

$GENESIS_DIRECTORY/.genesis/skills/package-safety-review/SKILL.md

If GENESIS_DIRECTORY is unset, use the local Genesis checkout for this machine, for example $HOME/git_repos/the-genesis.

The safety review must classify provenance, source/binary availability, data egress, locality, and write-safety before moving from source review to download, install, or execution. Do not run curl | bash; download and inspect the script first, then ask for explicit approval.

Phase 1: Create and Verify

Step 0: Research viability — stop here if no path exists

Before asking the user for anything:

  1. Research what auth methods exist for this tool (official API docs, OAuth, browser session, etc.)
  2. Pick the best viable method using the priority order below
  3. Determine exactly what that method needs from the user

This repo's goal is zero-friction setup. The user should never have to create an app, register OAuth credentials, or configure anything outside of this repo's own flow. Reject any auth approach that requires that — even if it's technically cleaner.

Connection-method priority order:

Priority Auth method User friction Ask the user for
1 Supported API token / preconfigured OAuth Near-zero — token or authorize click Only what the supported flow needs
2 Agent Browser + existing browser session Near-zero — reuse authenticated browser state Usually nothing
3 Agent Browser + one-time browser login Low — user signs in once, session persists A URL from the tool
4 Custom CDP/Playwright automation Higher maintenance — tool-specific code A URL from the tool
5 Username + password Low — but only for legacy tools Username and password
✗ OAuth requiring user to create their own app High — stop, do not use N/A

Browser fallback rule: when no suitable API is available, use Agent Browser as the default browser interface. It provides compact accessibility snapshots, semantic interaction, existing-session reuse, and action-time confirmation without requiring a connector-specific script.

Escalate from Agent Browser to custom CDP/Playwright only when at least one is true:

  • a stable structured read runs repeatedly and a script materially reduces cost;
  • deterministic batch processing, monitoring, or scheduling is required;
  • Agent Browser cannot expose the needed UI surface reliably;
  • traffic capture is needed to discover a replayable API.

Do not create custom browser code merely to open, inspect, click, draft, or submit through a normal authenticated UI.

On OAuth: OAuth is acceptable only when the repo ships pre-configured client credentials (the user just clicks "Authorize" in their browser — zero app creation). OAuth that requires the user to create a Google Cloud project, register a redirect URI, or configure a consent screen is not acceptable — the friction cost makes it worse than a browser session.

Stop and explain if the only viable path requires the user to create an app or register OAuth credentials. Don't propose it as an option — it violates this repo's zero-friction goal.

SSO-only tools: If the tool uses enterprise SSO and has no API token path, the only option is browser session capture (Priority 2). This is fine — but do three things:

  1. Write a plugin-compliant sso.py in TENX_PRIVATE_DIR/personal/{tool-name}/ (not tool_connections/) with TOOL_NAME, check(env) -> bool, and capture(env) -> dict. Run it directly: python3 "${TENX_PRIVATE_DIR:-$HOME/.10xProductivity}/personal/{tool-name}/sso.py". If you later contribute the recipe upstream (owner-add or contribute workflow), sso.py is copied to tool_connections/ as part of that process — not before.
  2. Document the refresh command in the connection file: python3 "${TENX_PRIVATE_DIR:-$HOME/.10xProductivity}/personal/{tool-name}/sso.py" — the agent cannot self-refresh without the user present.
  3. Document the token TTL (usually ~8h) — so the user knows when to expect re-authentication prompts.

Use Agent Browser for browser-backed operation as well as reconnaissance. Read workflows/discover-ui-surface/agent-browser.md when you need the agent to open a page, inspect visible UI, use a personalized feed, click through a flow, draft content, or perform an approved action. Keep recipes generic: machine-specific CDP ports are runtime-only and must not be written into them.

Prefer replayable APIs when they exist; keep browser operation when the UI is the product surface. Personalized feeds, recommendations, chat, and UI-only workflows may remain browser-backed by design. Document the limitation and the reason. Do not reverse-engineer private endpoints solely to avoid Agent Browser.

Use tool_connections/shared_utils/traffic_sniffer.py — a ready-to-run generic sniffer. It attaches a context-level listener before any page loads (catches service workers and background frames that page.on misses), opens the persistent profile, and records all matching traffic to a JSONL file while the user performs actions manually. Response bodies are off by default (LinkedIn and other heavy SPAs: reading large bodies in the sync handler can stall the driver and drop most later traffic); pass --capture-bodies only when you need response payloads.

Once the tool is set up (connection file written with a sniffer: frontmatter block), use the --tool shortcut — no need to remember paths or filters:

source .venv/bin/activate

# Shortcut — reads profile/url/filter from TENX_PRIVATE_DIR/personal/{tool}/connection-*.md:
python3 tool_connections/shared_utils/traffic_sniffer.py --tool {tool}

# Explicit — full control (for first-time discovery before connection file exists):
# ⚠ --filter is a SUBSTRING match, NOT regex. Pass it multiple times for multiple substrings.
# Wrong:  --filter "api|auth|token"   (treats the whole string as one substring — matches nothing)
# Right:  --filter api.tool.com --filter /auth
# Tip:    omit --filter entirely to capture all traffic, then grep the output.
# ⚠ If the sniffer fails to start (profile locked), kill any lingering browser first:
#   pkill -f {tool}_profile
python3 tool_connections/shared_utils/traffic_sniffer.py \
    --profile ~/.browser_automation/{tool}_profile \
    --url https://app.tool.com \
    --filter /api \
    --output /tmp/{tool}_traffic.jsonl

# Inspect results:
# ⚠ Headers are stored under "request_headers" (not "headers") in the JSONL.
python3 -c "
import json
for e in (json.loads(l) for l in open('/tmp/{tool}_traffic.jsonl')):
    if e['type'] == 'request':
        print(e['method'], e['url'])
        auth = e.get('request_headers', {}).get('authorization', '')
        if auth: print('  auth:', auth[:80])
        if e.get('post_data'): print('  body:', e['post_data'][:200])
"

Capture a full session in one run — perform all typical actions without closing the browser: scroll the main feed, open a post, react, comment, open messaging, send a message, search. Each action captures its own endpoint family. Document these in a ## Typical actions to capture section in the connection file.

  • The user's only job is to perform the target action (post, search, open a file). The sniffer silently records URLs, headers, and request bodies — no DevTools needed.
  • Some platforms issue separate tokens per service. Capture all distinct Authorization header values — each unique one is a separate credential to store in .env.
  • ⚠ Never disable TLS verification (ssl.CERT_NONE, verify=False, ignore_https_errors=True). Production services always have valid certificates. SSL errors mean the base URL is wrong — not the cert.
  • ⚠ If using a persistent browser profile and sso.py fails to launch, a stale SingletonLock file is the likely cause. Add rm -f "$PROFILE_DIR/SingletonLock" at the top of sso.py before launching, and kill any lingering browser processes tied to that profile.

When to stop trying: If browser session auth succeeds (you can log in and see data in the browser) but REST API calls return 401 anyway, the instance has API-level access restrictions that session cookies can't bypass. This is an admin policy, not a fixable bug. Document it as "API access restricted — this tool cannot be automated at this instance" and move on. Do not keep probing different endpoints.

If a viable zero-friction method exists → ask the user only for what that method requires, then proceed to Step 1.

Step 1: Identify the base URL

If the user provided a URL (login page, dashboard, ticket), probe it first:

curl -sI --max-time 10 "https://{the-url}" | head -5

Sites redirect. Confirm the real base URL before researching. Note any site-variant clues (e.g. us5.datadoghq.com → API base is api.us5.datadoghq.com).

Step 2: Research the API

If you can browse it, the API exists. Every web app is a REST/GraphQL client. The browser already has a valid session and makes every call you need to replicate. Official docs are the fastest path — but if they're incomplete or the key endpoints return 403, the browser Network tab is the ground truth.

Search order:

  1. Official docs (docs.tool.com/api or developer.tool.com)
  2. OpenAPI/Swagger spec (/api/swagger.json, /openapi.json)
  3. GitHub code search — working callers are more accurate than docs
  4. Agent Browser operation/reconnaissance — when the UI is the required surface: read workflows/discover-ui-surface/agent-browser.md, inspect compact snapshots, and use semantic interactions.
  5. Browser traffic capture — when docs are missing or incomplete and you need replayable endpoints: run tool_connections/shared_utils/traffic_sniffer.py (see above), ask the user to perform the target action, and read the JSONL output. The URL, headers, and body are everything you need to replay the call directly.

Collect before moving on:

  • Base URL (production)
  • Auth mechanism (API key, Bearer token, session cookie, OAuth2) and header name
  • Token lifetime and refresh method
  • Key endpoints: health/version (no auth), list, get
  • Search/query interface if any
  • Network requirements (VPN?)
  • Env var names to use

Step 3: Store credentials

Add to TENX_PRIVATE_DIR/.env only — do not edit root env.sample (it is a stub) or other shared index files. Document new variables in TENX_PRIVATE_DIR/personal/{tool}/setup.md under **.env entries**.

**sso.py must read credentials from TENX_PRIVATE_DIR/.env, never hardcode them.** Even in TENX_PRIVATE_DIR/personal/, scripts must call load_env() and read SF_USERNAME, SF_PASSWORD, etc. from the parsed env dict — not from module-level string literals. Hardcoded credentials in scripts are a scrubbing risk and make the recipe non-generalizable. If a value is missing from TENX_PRIVATE_DIR/.env, prompt the user at runtime (input() / getpass) rather than baking it in.

Watch for tools with explicit resource-sharing requirements. Some tools (e.g. Notion) require you to explicitly grant the integration access to specific resources (pages, databases) even after auth succeeds. Workspace-level installation ≠ data access. If auth passes but read endpoints return 404 or empty results, look for a resource-level sharing step — usually found in the tool's Settings → Integrations/Apps → edit the integration → content/resource access panel. Document this in the Notes section of the connection file.

# --- Tool Name ---
TOOL_API_TOKEN=your-api-token-here
TOOL_BASE_URL=https://api.tool.com
# Generate at: https://tool.com/settings/api-tokens
# Token lifetime: long-lived / ~8h (refresh with: ...)

Step 4: Validate against the live instance

Do not use dev environments. Validate on the actual production endpoint.

Choose the validation track selected in Step 0:

  • API track: validate auth plus at least two real read endpoints.
  • Agent Browser track: validate at least two real read surfaces and one interaction flow up to—but not through—its final write action. For a write, execute only after explicit approval, then verify the result URL.

4a. Connectivity (no auth)

curl -sI --max-time 10 "$TOOL_BASE_URL/health"    # or /version, /ping, /api/v1/status
  • 200 → proceed
  • SSL error → VPN may be required; document it
  • Timeout → wrong URL

4b. Auth

# ⚠ Avoid bare `source .env` if .env has non-env-var lines (e.g. long SSO cookie values) — it errors silently.
# Use tool-scoped export instead:
export $(grep -v '^#' .env | grep 'TOOL_' | xargs)
# Try the auth pattern from docs
curl -s "$TOOL_BASE_URL/some-read-endpoint" \
  -H "Authorization: Bearer $TOOL_API_TOKEN" | jq .

# If header name is unclear, probe common patterns:
for h in "Authorization: Bearer $TOOL_API_TOKEN" "X-API-Key: $TOOL_API_TOKEN" "api-key: $TOOL_API_TOKEN"; do
  code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 "$TOOL_BASE_URL/some-endpoint" -H "$h")
  echo "$h → HTTP $code"
done

4c. Key read endpoints

Run at least 2 read endpoints and capture real output:

export $(grep -v '^#' .env | grep 'TOOL_' | xargs)
curl -s "$TOOL_BASE_URL/users/me" -H "Authorization: Bearer $TOOL_API_TOKEN" | jq .
# → {"id": "u_123", "name": "Alice", "email": "alice@example.com"}

curl -s "$TOOL_BASE_URL/items?limit=5" -H "Authorization: Bearer $TOOL_API_TOKEN" | jq .
# → [{"id": "p_1", "name": "My Item"}, ...]

Record both successes and permission errors. At least one failure case is required — a 403, a deprecated endpoint, a missing permission, or an explicit "no search API" note. A connection file with only 200s won't pass the community review checklist.

4d. Native search and AI/chat

Always check — this is what makes a connection genuinely useful to an agent.

For every tool, answer these two questions before writing the connection file:

  1. Does it have a search API? (full-text, title-based, filter-based — any kind)
  • Try common patterns: /search, /api/search, ?q=, ?query=
  • Run it. Record what fields it searches, what it returns, and any limitations (e.g. title-only, indexed with delay).
  1. Does it have an AI or chat API? (LLM-backed Q&A, summarization, assistant endpoint)
  • Check official docs for "AI", "assistant", "chat", "copilot" endpoints.
  • If none exist in the public API, say so explicitly — do not leave it ambiguous.

Document the result in the Notes section of the connection file:

  • If search works: show a verified snippet with real output.
  • If AI/chat exists: show the endpoint and a verified call.
  • If neither exists or is paywalled: state it clearly (e.g. "No search API." or "AI chat is enterprise-only, no public endpoint.").

Skipping this step leaves the agent blind to the tool's most useful capabilities.

Step 5: Write the connection files

Location: TENX_PRIVATE_DIR/personal/{tool-name}/ — always. This lives outside the public repo and is never committed. Do not write to tool_connections/, staging/, or anywhere else outside TENX_PRIVATE_DIR/personal/.

Two files are required — both must be present before you can contribute:

  1. connection-{auth-method}.md — the verified connection (format below)
  2. setup.md — setup UX: what to ask the user, .env entries, and the verify snippet (use staging/_example/setup.md as template)

Format for connection-{auth-method}.md (use staging/_example/ as reference, and tool_connections/slack/connection-sso.md as a real-world example of good style):

---
tool: {tool-name}
auth: {api-token|oauth|sso|ad-sso|session-cookie}
author: {github-username}
verified: {YYYY-MM}
env_vars:
  - TOOL_API_TOKEN
  - TOOL_BASE_URL
sniffer:
  profile: ~/.browser_automation/{tool}_profile
  url: https://app.tool.com
  filter: /api
---

# {Tool Name} — {auth method}

{1-2 sentences: what it is, who uses it.}

API docs: {URL}

**Verified:** Production ({base-url}) — {endpoints tested} — {YYYY-MM}. {VPN required / not required.}

---

## Credentials

\`\`\`bash
# Add to .env:
# TOOL_API_TOKEN=your-token-here
# TOOL_BASE_URL=https://api.tool.com
# Generate at: {URL}
\`\`\`

---

## Auth

{Auth flow in 1-2 sentences.}

\`\`\`bash
export $(grep -v '^#' .env | grep 'TOOL_' | xargs)
curl -s "$TOOL_BASE_URL/endpoint" \
  -H "Authorization: Bearer $TOOL_API_TOKEN" | jq .
# → {actual output}
\`\`\`

---

## Verified snippets

\`\`\`bash
export $(grep -v '^#' .env | grep 'TOOL_' | xargs)
BASE="$TOOL_BASE_URL"

# {What this does}
curl -s "$BASE/endpoint" -H "Authorization: Bearer $TOOL_API_TOKEN" | jq .
# → {actual output}
\`\`\`

---

## Agent behavior

**Read actions — run freely, no approval needed:**
- list any read/GET operations here

**Write/interact actions — show preview + target URL, get explicit user approval before executing:**
- list any POST/PUT/DELETE operations here
- Always provide the direct URL so the user can verify or fix manually on error

---

## Typical actions to capture with the sniffer

Run `python3 tool_connections/shared_utils/traffic_sniffer.py --tool {tool-name}` then perform:
- {list the key actions that cover all endpoint families}

---

## Notes

- {Permission requirements}
- {VPN requirement}
- {Known limitations}
- {Which endpoints require browser (headless Playwright) vs plain urllib}

Writing style — the connection file is read by an LLM agent, not a human:

  • Don't explain what the LLM already knows. Skip boilerplate like "Bearer tokens are sent in the Authorization header" or "HTTP 200 means success." Document only what's specific to this tool: its URL patterns, quirks, header names, token format, known failures.
  • Be concise. One sentence beats three. A table beats a paragraph. Cut every word that doesn't add tool-specific information.
  • Inline code over helper functions. The agent will copy and adapt snippets — it doesn't need a library. Write flat, readable code that shows exactly what's happening.
  • Examples teach faster than prose. Where you'd write "use the after: filter for date queries", instead show: "query": "from:@me after:2026-03-24". A concrete example with real values is worth more than a description.
  • ⚠ marks the non-obvious. Use it only for gotchas that would cause silent failure — things the agent couldn't infer from the API docs. e.g. # ⚠ bash truncates long xoxc tokens silently — always load .env in Python.

Snippet rules:

  • Only include commands you actually ran and saw succeed
  • Every snippet has a # → {actual output} comment (truncate long output with # → [{...}, ...])
  • Permission errors are valid: # → 403 Forbidden — requires Admin role
  • Cut anything that didn't work

Step 6: Update verified_connections.md

Once both files are written and at least 2 snippets are verified with real output, add the tool to your active capability index.

Eligibility vs scrubbing — these are different gates at different stages. Credentials, org-specific URLs, and personal data should always go in .env — not hardcoded in scripts. But if they do exist in TENX_PRIVATE_DIR/personal/ files, TENX_PRIVATE_DIR/personal/ is gitignored and they will never be committed. Either way, they are scrubbing concerns handled later in the owner-add or contribute workflow, not eligibility disqualifiers at this stage. Eligibility (Step 2 of contributing.md) asks only whether the tool is commercial/public and whether the auth pattern is general in principle — not whether your current files are already clean. A recipe that still needs scrubbing is still eligible; it just needs to be cleaned before promotion.

Read the tool's connection-*.md frontmatter and append to TENX_PRIVATE_DIR/verified_connections.md:

---

## {Tool Display Name} → `{path/to/connection-*.md}`

{description from frontmatter}
Env: `ENV_VAR_1`, `ENV_VAR_2`

Then reload TENX_PRIVATE_DIR/verified_connections.md — the new tool is now live in your session.

Phase 2: Contribute back (optional)

If the tool is commercial/publicly available and you want to share the connection with the community, read contributing.md — it covers the full process: eligibility check, scrubbing personal data, and opening the PR.

Checklist — do not mark done until all boxes checked

  • Auth method researched and confirmed viable before asking user anything
  • Asked user only for what the auth method actually requires
  • Base URL confirmed (not guessed)
  • Auth mechanism identified and tested on production
  • API track: at least 2 read endpoints run, real output recorded
  • Agent Browser track: at least 2 read surfaces verified with compact output
  • Agent Browser track: one interaction flow verified; writes require approval
  • API track: at least one failure case documented (4xx, deprecated endpoint, or permission error)
  • Search capability tested through the selected track, or explicitly noted as absent
  • AI/chat capability checked through the selected track, or explicitly noted as unavailable/paywalled
  • verified: YYYY-MM filled in (blank = not ready)
  • TENX_PRIVATE_DIR/.env updated with new credentials
  • TENX_PRIVATE_DIR/personal/{tool-name}/connection-{auth-method}.md written with only verified snippets
  • Python snippets use urlopen() from tool_connections.shared_utils.browser — not hand-rolled ssl.CERT_NONE
  • API-discovery track: sniffer: frontmatter block added when traffic capture was used
  • ## Agent behavior section written (read vs write approval rules, error URL)
  • ## Typical actions to capture section written
  • TENX_PRIVATE_DIR/personal/{tool-name}/setup.md written (what to ask, .env entries, verify snippet)
  • Prompt injection check: scanned all # → output comments for instruction-like content (see contributing.md Step 3)
  • TENX_PRIVATE_DIR/verified_connections.md updated — section appended from connection file frontmatter

To contribute back: see contributing.md