Connectors: sync external content into your wiki, with provenance

Six sources, one sync model

A connector is a continuous sync, not a one-time import. Once configured, it runs on its own schedule and keeps the corresponding wiki pages current as the source changes:

• Filesystem — a local or mounted directory of documents.
• Web crawler — a seeded crawl with depth limits, robots.txt respect, and a path-prefix scope.
• Sitemap — an XML sitemap URL list, fetched and walked directly.
• RSS / Atom feeds — feed entries synced as pages, with optional full-content fetch.
• Google Drive — OAuth2-authorized folders, synced by ID.
• GitHub — a repository path, synced by branch and path prefix.
• Confluence — a space, synced page by page via the Confluence API.

Every connector shares the same sync engine underneath: content is fetched, hashed, and compared against what is already there. Unchanged items are skipped, changed items update the existing page, and items removed from the source are tombstoned rather than silently vanishing. A sync that fails partway never leaves the wiki in a worse state than before it started.

Derived pages: real wiki pages, honestly labeled

A page created by a connector is not a special second-class object — it is a normal Markdown page with YAML frontmatter, stored the same way as anything a human wrote. What makes it a derived page is a derived_from field recording the retained source, plus derived_connector and derived_source_url naming which connector produced it and where the content originated.

Readers are never left guessing. Every derived page carries a provenance banner — "Synced from [source] · last synced [date] · via connector [name]" — and a compact badge appears next to derived pages in search results, page lists, and the cluster tree. If a connector is later deleted but its pages are kept, the banner says so plainly: the source is no longer syncing.

Synced content is first-class in retrieval and the knowledge graph

A derived page is indexed exactly like any other page: it participates in hybrid retrieval, is eligible to appear in a context bundle with the same version-pinned citations as hand-written content, and — if its assigned cluster is included under the knowledge graph's inclusion policy — gets entities and relationships extracted from it the same as any other page. Connecting an external source is a way to widen what your agents and search can actually find, not a side channel that content has to "graduate" out of.

A guided admin experience

Setting up a connector is a stepped wizard in the admin panel, not a properties file and a restart: pick a source type, fill in its settings with inline guidance (down to the exact GitHub token scopes or the Confluence API-token walkthrough), authorize where needed (Google Drive is a real OAuth consent redirect), and run a dry-run connection test before saving anything. The test reuses the connector's own fetch path — so "success" means the real sync will work too — and reports what it found in plain language, not a raw API response.

Once saved, the connector's detail page shows its last runs, sync status, and the pages it has produced, with a one-click "Sync Now" alongside its schedule. A wikantik.connectors.enabled kill switch lets an operator pause all syncing instantly if something looks wrong.

Credentials stay encrypted, never echoed

GitHub personal access tokens, Confluence API tokens, and Google Drive OAuth client secrets and refresh tokens are encrypted at rest (AES-256-GCM) in a dedicated credential store — connector configuration rows themselves hold no secret fields. The admin UI shows only whether a credential is set, never its value, and the dry-run test endpoint can use a freshly entered credential without ever persisting or logging it. Filesystem connectors — the one type that reads arbitrary server paths — are deliberately not creatable from the web UI at all.

Sync scheduling, per connector

Each connector has its own sync interval, in hours, editable from its settings — zero or blank means manual-only. A due-tick scheduler checks every connector against its own interval rather than running everything on one fixed clock, so a fast-moving GitHub repo and a slow-changing sitemap can each be scheduled sensibly. Every run — scheduled or manually triggered — is recorded with its timing, counts of created/updated/unchanged/deleted items, and any error text, so a connector's recent history is always visible on its detail page.

Frequently asked questions

What sources can a Wikantik connector sync from?

Six connector types today: a local filesystem, a web crawler, an XML sitemap, RSS/Atom feeds, Google Drive (OAuth2), GitHub, and Confluence. Each is a continuous sync, not a one-time import — content is re-pulled on a schedule and kept current.

Will readers know a page came from an external source?

Yes. Every synced page carries a provenance banner naming the source and connector, plus a small badge in search results, page lists, and the cluster tree. Nothing is silently absorbed into the wiki as if a human wrote it.

Are connector credentials stored securely?

Yes. GitHub tokens, Confluence API tokens, and Google Drive OAuth secrets are encrypted at rest (AES-256-GCM) in a dedicated credential store. The admin UI only ever shows whether a secret is set, never its value, and a dry-run connection test can use a credential without persisting or logging it.

Does synced content get indexed for search and the knowledge graph?

Yes. A derived page is a normal wiki page — it is indexed for hybrid search, eligible for context bundles, and, if its cluster is included under the knowledge graph's inclusion policy, extracted into the knowledge graph like any other page. There is no separate second-class content tier.