Reference

Building sources

A source is a TypeScript package that the collector loads and runs. This page is for developers adding a new source to Omnesis: what qualifies, what a source must do, the data it emits, where its code lives, and how it is tested and reviewed. It links to the SDK types for detail.

What qualifies

One rule decides whether a source is worth building: after a single one-time authentication, it must be able to sync automatically in the background from then on. The collector schedules it forever; no user action is ever part of the ingestion path.

Platforms whose only way out is a manual, user-triggered export — a "download your data" archive the user has to request each time — do not qualify. A stale snapshot that decays until the user remembers to re-export is not a source. The available sources list shows what has cleared this bar.

The contract

The source contract lives in @omnesis/source-sdk; the document shapes you emit (DocumentInput, PersonMention) come from @omnesis/types. At its core a source is a function from a cursor to a page: the collector calls sync(cursor), the source returns what changed upstream since that cursor plus a new cursor, and the gateway stores both. One scheduled tick looks like this:

That loop is why every source ships both a bootstrap test and an incremental test (see Testing). The sections below cover the rest of the contract in the order you build a source.

Entry points

There are three entry points, and every package default-exports exactly one of them:

• defineSource() — a single source (Obsidian, Things).
• defineProvider() — a platform hosting several sources behind one authentication (Google → Gmail, Calendar, Drive, Contacts). Sources share a context object created once per account. The account model is declared once here and inherited by every source under it: most providers accept several accounts, while one reading a single local artifact declares that a host holds at most one. A child source may narrow the provider's discover() result when its own local store can be absent even though a sibling store exists; child discovery requires the provider hook.
• defineStructuredSource() — a source whose output is chiefly typed analytics rows, optionally with summary documents bound to them (Browser History, Screen Time).

The descriptor

The descriptor carries everything consumers need: name, description, icon, unit noun ("notes", "emails"), the auth kind (oauth, qr, api-key, link-widget, or local), and any user-supplied parameters. Push-only and gateway-hosted sources declare local. The portal, CLI, and apps read all of it through the registry — shared code never branches on a source name. If a consumer needs something source-specific, the descriptor grows a field; the name never leaks downstream.

Give every source an icon: an SF Symbol name, a brand color and bgColor, and a renderable image as imageDataUri or url. The portal and Android cannot draw an SF Symbol, so a source without the image shows a generic glyph there. Declare urlPatterns whose first capture group is the item's external ID, so a link to one of the source's items anywhere in the corpus resolves to its document; without them, those links are dropped.

Declare settings with config.object(): string, path, number, boolean, select, secret, and list fields produce both the form and the typed value passed to create(). Defaults must match their field types. Required whitespace is missing input, not a value. Host path checks run on each list element as well as scalar paths. advanced: true omits a setting from the add form without removing it from validation.

Lists accept arrays or text; path lists split text on newlines, other lists on commas or newlines. Set separator: "newline" for values containing commas, such as *.{tmp,bak} glob patterns.

Accounts and discovery

Discovery may return account IDs or AccountDescriptor objects with a stable id, a display label, a typed subject, and optional tenant and aliases. Descriptors refresh existing source rows when collectors announce metadata; discovery that returns bare IDs leaves stored descriptors intact. The account key never changes when its label changes. Tenant and alias declarations do not merge accounts, and there is no user-facing account rename API.

singleInstance: true means one instance of that source type per collector host, not one account across the gateway. Clients choose the target collector and discover its account before deciding whether to join an exact source already registered elsewhere or create a separate source for a different account.

Once the gateway assigns an existing source to a collector, a defined discover() hook is authoritative: the collector will not instantiate that account-qualified configuration unless discovery finds it on the host. A fresh interactive add may still complete its provider setup before the new account becomes discoverable; definitions without discovery derive accounts from their saved configuration.

Syncing

At runtime, create() returns the live instance. The sync engine calls sync(cursor) until hasMore is false, and stores the opaque cursor between runs, so an interrupted sync continues from its last cursor. watchPaths starts a sync when local files change. Use watchDirectoryPaths for paths that are directories but can be absent at startup. The optional watchFileExtensions field limits a recursive directory watch to specified filename suffixes. onPushEvent sends new data without a scheduled sync.

A minimal source fits in one file. This one reads notes that each machine keeps in its own local folder, so every machine that hosts it contributes different notes:

import { config, defineSource, syncPage } from "@omnesis/source-sdk";
import type { SourceInstance } from "@omnesis/source-sdk";
import { FIELDNOTES_ICON } from "./icons.js";
import { readNotesSince } from "./normalizer.js";

interface Cursor extends Record<string, unknown> {
  lastModified: number;
}

export default defineSource({
  id: "fieldnotes",
  name: "Field Notes",
  description: "Notes from the Field Notes desktop app",
  authType: "local",
  unitName: "notes",
  icon: {
    sfSymbol: "note.text",
    color: "#2F81F7",
    bgColor: "#0F1B2E",
    imageDataUri: FIELDNOTES_ICON,
  },
  multiDevice: { mode: "partitioned" },
  contract: {
    apiVersion: 2,
    requires: ["state-envelope", "typed-config"],
    state: {
      version: 1,
      decode: (value) =>
        value !== null && typeof value === "object" &&
        "lastModified" in value && typeof value.lastModified === "number"
          ? value as Cursor : null,
      onUnreadable: "rebootstrap",
      maxBytes: 4096,
    },
  },
  config: config.object({
    dataPath: config.path({
      label: "Data folder",
      scope: "member",
      required: true,
      mustExist: "directory",
    }),
  }),
  create: async ({ config }): Promise<SourceInstance<Cursor>> => ({
    sync: async (cursor) => {
      const since = cursor?.lastModified ?? 0;
      const { documents, newest } = await readNotesSince(config.dataPath, since);
      return syncPage(documents, { lastModified: newest });
    },
  }),
});

Use the injected SourceHost for logging, now(), attachment extraction, optional audio transcription, and scoped analytics reads. Its analytics.query() is read-only and restricted to the source's declared tables; writes belong on a sync page. stateDir is scoped to the provider account and shared by its sibling sources, so use distinct filenames; host.configDir remains available for credential layouts that already use it.

The host interface is not a sandbox for in-process provider code, nor a promise of access to the whole gateway client. Factories and shared provider contexts do not receive a gateway client. A source or provider child declaring execution: "external" needs no factory and is never instantiated by the collector.

A throw from create() — or from the provider's discover() or createContext() — means the source is not added, and the message is what the operator reads as the reason, so word it to name the condition and the fix. Nothing is latched: the collector re-attempts the source on its next connection to the gateway, so a condition the operator resolves needs no re-add. Under a provider, one source's failure costs only that source; its siblings on the same account still come up.

A failure during sync() is thrown as a SyncError from @omnesis/types, whose kind — auth, network, rate-limit, permission, transient or unknown — tells the collector whether the source needs a new sign-in, should wait, or has failed. It may carry retryAfterMs from the platform's retry hint, and a remediation — a one-line summary, the steps, and whether the collector must restart — for a condition only the operator can clear, such as a missing operating-system permission. Clients show the remediation as a fix instead of the raw message. An untyped throw counts as a failure of the whole source.

Some sources read files that another program maintains. If that program stops, the source cannot report an error — it just sees no changes. The freshness field defines a plausible quiet period, so Omnesis can tell a stalled source from a quiet one. When the program can be started unattended, the declaration may also name its macOS bundle identifier; a Mac collector then opens it hidden when it finds it quit, backing off between attempts and never more than a few times a day, and shows the declaration's failedHint once the program has repeatedly failed to stay up.

A completed sync may also return a watermark: a small statement of what the source knows it has covered. A watermark covers the whole configured source, not individual folders, tables, or documents. Omit it when the upstream system gives no stronger boundary; Omnesis records an observation instead, meaning only that a successful sync finished at that time. A watermark is valid only on the final page (hasMore: false); a paginated source must return it with its last page.

Use the strongest honest guarantee: change-cut for a stable upstream change boundary, snapshot for a completed consistent listing, or best-effort-scan for a completed listing that may race concurrent changes.

Local sources

A local source has no credential to renew. It reports an unavailable database, missing operating-system permission, or broken local dependency as a sync error with the relevant remedy; it never sends the user into a re-authentication flow. A local source whose database can live outside its default search location may mark a member-scoped path parameter with provesLocalAvailabilityForAccount: "local". The parameter needs a host check (mustExist, mustContain or check) and the source a discover() hook, or the definition is rejected. A non-empty value that passes that validator proves availability for that exact source on that member; it never authorizes a sibling source or a shared account setting.

Sources that read local files should implement probeReadAccess({ signal }) on the live instance. The collector calls it during device health checks, in the daemon process whose permissions matter. Freshly open the source-owned inputs read-only and close every handle; do not reuse cached handles, read indexed content, copy databases, run sync, or change a cursor. Bounded discovery metadata may be read to select current inputs, but must stay out of reports and logs.

Do not let a cached input roster silently omit newly eligible inputs. Bound discovery and honour cancellation between operations, including cleanup after a late open. Return an object whose status is readable, denied, unavailable, or unsupported; missing inputs and incomplete discovery are unavailable. Optional remediation must not include paths, filenames, or raw errors. Watch paths are not a permission contract: they may include optional files such as database journals.

The SDK's probeFileReadAccess helper closes its descriptor on the JavaScript thread to avoid releasing process-owned SQLite locks during a synchronous statement. It is suitable for eager, same-thread SQLite readers only. A source that holds a transaction or iterator across an await, or reads that same file on another thread, must coordinate its readers and probes instead of using the helper unguarded.

A source maintaining an encrypted local cache or archive also implements probeLocalStores({ signal }). Inspect read-only with the host's storage key; never migrate, repair, or write during a health check. Return the registered keyName, an operator-safe label, and one of encrypted, plaintext, absent, locked, or unverifiable. Missing stores are absent, not evidence of upstream deletion. Reports must contain no local paths, filenames, raw errors, or indexed content.

Deletions

Deletions reach Omnesis through two different channels, and the difference matters. A source that knows an item was deleted names it directly (a tombstone), and Omnesis removes it immediately. A source with no deletion signal instead sends a snapshot — the complete list of what exists upstream — and Omnesis infers the rest from what the list leaves out.

An inference is not an observation. A store that is locked, a permission that lapsed, a database mid-migration: each answers a source short, and the source has no way to tell a short answer from a small one. The snapshot it builds is complete on its own terms and omits items that still exist — a page indistinguishable from a genuine mass deletion, and the two outcomes are not symmetric. So an item a snapshot omits is not deleted at once.

Omnesis records the omission with a deadline: several later snapshots must agree, and a minimum span of time must pass, before the item is removed. A snapshot that names it again cancels the record. A source that stops before enough corroborating snapshots cannot spend the deadline merely by remaining offline. After a gateway restart, a startup grace gives collectors time to revoke stale evidence before deletion resumes. The thresholds are gateway.snapshotAbsence, and a legitimate deletion still lands once that configured deadline has passed.

The delay buys back a bad read that recovers, not one that persists, so a source should still withhold the snapshot for any enumeration it cannot vouch for. An omitted snapshot has no effect at all; a snapshot built from a doubtful read starts a clock on items that are still there. When enumeration is incomplete, include the enumeration's withheldIssue() result (from the SnapshotEnumeration you built) in the finalized page's issues, or return issues: [] after a complete assessment.

Omit issues on incremental ticks that do not assess enumeration: success alone must not clear an earlier warning. Reports accumulate across pages and replace that member's durable warnings only when the run completes; they do not turn a successful partial sync into an error. Runtime guards report invalid snapshots separately: a later valid snapshot clears only the matching document or table diagnostic, leaving unassessed warnings intact.

Withholding is all-or-nothing, which is costly for a source that reads several stores: one address book that will not open suspends deletion detection for the two that did, for as long as it stays broken. Such a source can instead vouch store by store. It names each store it read in full, with what that store holds, and stamps every document it emits with the store it came from. Omnesis then judges only the documents belonging to a store that was named; anything in a store the source could not read is left exactly alone. The two halves have to agree — a store claimed while its documents carry no store name asks for deletions that then never happen.

The wire form is presentClaims: [{ partition: "book-a", ids: ["note-1"] }], paired with partitionKey: "book-a" on those documents; SnapshotEnumeration.claims() builds it. Partition keys must be stable and at most 256 characters. Emit either claims or a whole-source presentExternalIds snapshot, never both, and only on the final page.

Claims are for the cycle that could not read everything. When every store was read in full, send the whole-source snapshot instead: claims never reconcile documents emitted without a partition key or partitions that are no longer listed, so a source that claims on every cycle strands them silently.

Every source that emits a snapshot, of documents or of analytics rows, also has an entry in packages/source-sdk/src/snapshot-emitters.json stating how it guarantees the snapshot is complete; a contract test fails the build without one.

Repeated I/O failures never become evidence of deletion. Preserve known IDs under an unreadable file while reconciling readable siblings; keep an unknown partition gapped.

Several devices

A descriptor declares how one configured source may be hosted across devices with multiDevice: { mode }. Each hosting device is a member. Users see these modes under plain names, described in Sources on several devices.

The mode applies to the source as a whole, including hybrid sources that emit both documents and analytics rows. For a replicated source, a table write's deletedKeys and presentKeys are read from every member whether or not it holds the lease, and applied under the same rule as documents: one replica leads a deletion nobody has reported, and a row another replica still holds is kept as disputed until that replica agrees. Emit a tombstone once rather than repeating it on every sync, and emit tombstones and snapshots without regard to the lease.

For partitioned sources, the gateway derives the stream identity from the authenticated member device and stores it separately from the provider's external ID. Providers therefore keep external IDs stable only within their local upstream and must not embed a device or stream key in them.

Source parameters are shared by every member unless the descriptor marks one with scope: "member". Use member scope only for host-local settings such as a filesystem path: each collector then supplies the value for its own environment. Account identity and other settings that define the logical source remain source-scoped; omitting scope keeps that default.

Adding member-local fields does not stop an existing host that explicitly understands every pinned field from receiving its effective configuration and syncing. Configuration scope changes only when all members advertise the same expanded set; existing local overrides are preserved. Member-local configuration and membership mutations require exact agreement, while missing field declarations or incompatible storage modes still refuse execution.

State and compatibility

Declare contract.apiVersion: 2; a source that omits it is treated as generation 1, which hosts still run. List in contract.requires every host capability the source cannot run without; a host that lacks one refuses that source before creation rather than running it unsafely, and supported siblings may still start. The host supports state-envelope, multi-table-batch, snapshot-sessions, tuple-deletes, scoped-host, and typed-config. Scoped claims require snapshot-sessions. The type also names connection-identity, which no host advertises yet: a source that requires it is refused everywhere.

contract.state declares version, a validating decode(value), and every migrate[n] step from version n to n + 1. The host unwraps state before sync and envelopes each saved cursor. legacyVersion(value) classifies bare cursors saved before the source adopted the envelope; optional minorVersion additions must decode older values. A newer major state version is refused without overwriting it.

Choose onUnreadable: "stop" when discarding a bookmark could lose unreplayable history; the default "rebootstrap" restarts from upstream. maxBytes caps the encoded envelope, not just its payload. Migrations, legacyVersion, a stop policy, or a size ceiling require state-envelope. Every returned cursor, including an absent-store seed, must decode. Test both document and structured sync paths.

outputRevision describes changed output meaning, not cursor shape. It does not trigger reprocessing or a backfill: implement and test any required re-emission separately. Provider contracts contribute their API generation and required capabilities; each child declares its own state. Never migrate one sibling's bookmark using another's decoder.

Authentication sessions

Implement authentication with authenticate(session). show(challenge) presents information and continues; ask(challenge) waits for a typed answer. Challenges carry their own title and instructions: redirect, QR, pasted code, typed fields, widget, or wait notice. Use canShow(kind) before selecting a client interaction; unsupported questions fail immediately rather than waiting for timeout. Clients that declare no capabilities support only redirect and QR challenges. The generic answer endpoint accepts only pending code or fields questions; redirects and widgets complete through their own callbacks.

For API-key providers, where the pasted value is the account's own credential rather than a shared app credential, authenticate(session) receives the fields, verifies them, resolves which account they belong to, and only then stores them under that account, so nothing is persisted until authentication succeeds.

Validate credentials before saving them, including values a client supplied up front in session.supplied. A re-authentication is for session.accountId, not an opportunity to connect a different account. Report failure with a typed AuthFailure — credential-rejected, denied, unavailable, identity-mismatch and the rest — with a remedy the operator can act on, and keep secrets out of notices and logs. The older authFlow and boolean auth hooks remain only as compatibility paths.

credentialState() describes locally stored credential state without a network request; an unreadable credential is not an absent one. A standalone source may expose the same hook on its live instance. Return unknown for unreadable credential storage, and unlinked only when the platform has removed a paired device. A factory unable to open its credentials fails visibly during setup; it cannot publish a live connection-state hook until setup succeeds.

Documents, rows, people

A source emits documents, analytics rows, or both. Hybrid sources do both: a calendar source emits one document per event for search and a typed events table for queries like "how many meetings last month".

Documents

A document is markdown content plus metadata. Each document may carry PersonMention entries (emails, phone numbers, names) that feed the people graph, so a sender in one source and a contact in another resolve to the same person. Roles come from PERSON_ROLES (author, sender, recipient, participant, attendee, mentioned, contact, owner). The source normalises identifiers itself: emails lower-cased, phone numbers in E.164, and platform handles namespaced by source, such as github:<login>. A document that is inherently the user's own marks its author isSelf: true; a source whose account ID is itself a platform identity declares selfIdentity instead, so that identity resolves to you rather than to a duplicate person.

Set metadata.sourceUrl only for a reliable destination represented by the document: normally the exact source item, or the described resource for a reference source. Set metadata.appUrl only for a working item-specific phone deep link. A phone opens appUrl when an installed app handles it and otherwise falls back to sourceUrl, so a deep link that works on only one phone platform is fine. Both are optional; omit them when no reliable destination exists.

Analytics rows

Analytics rows are typed tables in the analytics database (DuckDB), declared in analyticsSchemas and synced through syncStructured(). Beside its name (tableName), displayName, description and typed columns, each table declares:

Columns may be marked volatile when their value changes on every fetch without meaning anything, such as a fetch timestamp. The column name _stream_id is reserved for the host. The full schema type, including the rarely needed sharedDiscriminatorParent for child rows that inherit their owner from a parent table, is in packages/source-sdk/src/structured-source.ts.

A structured page carries its rows in analytics: one TableWrite, or an ordered list of them. Each names its tableName and may carry records to insert or update by primary key, deletedKeys to remove, and presentKeys as that table's snapshot; both kinds of key are records over the table's deleteKey columns. The host writes the list in order, and a table may appear twice. One page fills as many tables as the upstream record spans, so a workout that is also its splits, laps, and best efforts writes all four together under the checkpoint that covers them.

Group writes on one page only when a checkpoint between them would be a lie. Rows that merely arrive at the same moment — an account list fetched before a long walk of each account's history — belong on pages of their own, so a failure later in the walk leaves the earlier rows stored. The older single-column spellings, deletedIds and presentIds, are deprecated, and a page that mixes them with keys is refused.

Where code lives

Each source is its own package under packages/providers/<name>/, with the definition default-exported from src/index.ts:

package.json            name: "@omnesis/provider-fieldnotes"
src/
├── index.ts            defineSource({ ... }) — the default export
├── icons.ts            the icon as a data URI
├── normalizer.ts       raw platform rows → DocumentInput
├── normalizer.test.ts
└── fieldnotes.test.ts  bootstrap + incremental sync

Discovery is automatic: the collector reads the dependencies in its own package.json and imports every @omnesis/provider-* package it finds. To ship a source, create the package and add it there, as a runtime dependency; that is the only registration step. The build still has to know about the package (root tsconfig.json references, the lockfile, and the collector's entry in knip.json), and /source-review lists every other touchpoint. A source marked experimental is offered only by a collector running with OMNESIS_EXPERIMENTAL=1, and one that declares supportedPlatforms only on those platforms. On the next collector start the source appears in + Add source.

Testing

Every source ships tests. These cover the baseline:

• Bootstrap — a first sync against a populated fixture produces the expected documents.
• Incremental — a second sync with the first sync's cursor produces nothing (idempotency); adding one item and syncing again produces only that item.
• Normalization — pure-function tests of the raw-row → DocumentInput mapping against representative fixtures.
• Error handling — auth and platform failures surface with recognisable errors, so the engine can flag the source instead of silently stalling.
• A cycle against unchanged upstream is a no-op — for a source with more than one phase. Each phase can be right on its own while the loop they form is not: two phases that persist the same field from different inputs, or write a value one way and read it back another, disagree forever, and every cycle then redoes work nobody asked for at a cost that never stops. expectUnchangedUpstreamIsNoOp from @omnesis/source-sdk/testing drives your real phases to quiescence and then once more, and fails if the second pass rewrites any meaningful column. Pass its volatileColumns callback the columns your schema declares volatile, derived from the schema rather than restated, so a fetch stamp moving is not a failure.

Tests are vitest files named *.test.ts, placed next to the code they test. The neighbouring providers are the pattern to copy — every sync-based source ships these tests in some form (push-based sources, which have no sync loop, test their push path instead). Tests that need on-disk state use unique temp paths, never a shared fixed path.

The same entry point carries the contract checks. runSourceConformance (and runProviderConformance for a provider) checks a definition against the contract — its declared capabilities, its configuration, and its state, given sample saved states to decode — and formatConformanceReport prints the result. fakeSourceHost and fakeProviderHost stand in for the host in unit tests.

Also exercise installed cursors through the real state decorator, including pending pages, absent stores, downgrades, and unreadable state. Drive create, update, move, and delete through the real collector and an isolated gateway, with one backing store unavailable across several cycles. Hybrid sources must prove that a failure between documents and table writes does not advance the checkpoint. Typed auth tests should cross the subprocess protocol and client endpoint, not just a fake session.

Reviewing a source

This page covers what a source must do. The full contract is spread across the SDK, the gateway, the portal, the mobile apps, and the synthetic universes, and most of the ways a source can be wrong are silent: it builds, its tests pass, it syncs — and it shows a placeholder icon, never contributes people, drops inbound links, or duplicates itself on a second device.

The repository carries the complete checklist as a Claude Code slash command, /source-review, at .claude/commands/source-review.md. Run it from the branch that adds or changes a source, naming the package:

/source-review packages/providers/fieldnotes

It reads the contract, runs the mechanical checks (typecheck, the source's tests, the repo-wide contract tests, universe validation, lint, the privacy scan), then reviews the source section by section: viability and the platform's terms, encapsulation, auth and credentials, the sync runtime, deletions and snapshots, documents and analytics rows, people, URLs, icons, multiple accounts, the multi-device mode, phone-pushed sources, every registration point outside the package, and the tests.

It fixes mechanical omissions on the branch, ranks the rest, and ends with the validation steps only a person can do, such as opening documents on desktop and on a phone, and adding a second account. The file is plain markdown. Without Claude Code, read it as the checklist and walk it by hand before opening the pull request.

Contributing a source

Sources ship with Omnesis itself: a new source reaches users when its package is merged into the repository and released. The @omnesis/* packages named on this page are workspace packages in that repository, not packages published to npm. Before you start on a new platform, open a data source proposal so its viability is settled before the code is written. The contribution guide covers the development setup, the checks a pull request must pass, and the contributor licence agreement.