PERSONAL INTELLIGENCE PIPELINE · ARCHITECTURE & DATA FLOW

Public Intelligence: Collection to Publication

Production architecture, data flow, storage model, schedules, and failure paths. Counts come from project YAML at build time, not live telemetry.

9targets (30 tags)
32 / 38enabled / total channels
30 mindue-channel scan cadence
1 hanalysis queue cadence

01 / ARCHITECTURE

Components and control flow

Multica triggers the control plane; fixed collectors and the Worker API execute the data plane. Model writes pass schema, idempotency, and publication gates.

02 / SEQUENCE

One complete run: inputs, processing, outputs

  1. 1

    Schedule and due query

    Every 30 minutes, collect-due queries channels due from enabled state, interval, and last check time.

    Input
    catalog + channel cursor
    Output
    GET /v1/channels/due
  2. 2

    Fixed-route collection

    Each channel has a fixed collector and template. MCP bindings are pre-verified and configured fallbacks run on failure.

    Enabled collectors
    MCP 20 · RSS 4 · GitHub API 5 · HTTP 2
    Output
    CollectionPage(items, next_cursor)
  3. 3

    Normalize, date evidence, deduplicate

    Normalize URL, body, and time. Dedupe order: channel + external ID, canonical URL, channel + content hash.

    Write
    POST /v1/items/batch
    Cursor
    advances after successful write
  4. 4

    Enrichment and pending queue

    Research resolves full text, final URL, and date evidence. Only recent, complete items without analysis for the current revision enter the queue.

    Read
    GET /v1/items/pending-analysis
    Guard
    content_revision compare-and-swap
  5. 5

    Structured analysis

    The model returns strict JSON for headline, summary, change, impact, scores, topics, next checks, and evidence. Scores are internal ranking signals.

    Validate
    analysis-batch.schema.json
    Write
    POST /v1/analyses/batch
  6. 6

    Deterministic selection and rendering

    Sort by importance, confidence, and time; dedupe by item ID and source URL. Budget: 12 items, up to 3 leads. Midday requires importance ≥ 4.

    Windows
    08:15 / 12:45 / 18:45 / weekly 7d
    Output
    content/posts/intelligence/*.zh.md
  7. 7

    Publication gates and live verification

    Public-source, evidence, secret, front matter, Hugo, diff, and path gates run before push. The live page and artifact fingerprint are verified afterward.

    Status
    draft → validating → ready → published
    Failure
    keep draft; record Run / Issue

03 / STORAGE

Core D1 data model

targets 1 ── N channelschannels 1 ── N itemsitems 1 ── 1 analysesreports N ── N itemspipeline_runsaudit_events
TablePurpose
channelsconfig, cursor, health, failures
itemsitems, hashes, dates, metadata
analysesanalysis, model, prompt version, evidence
reportswindow, edition, Markdown, state, commit, URL

04 / SCHEDULE

Production schedule

JobCadencePurpose
collect-due*/30 mincollect due
analyze-pendinghourly :05enrich/analyze
morning08:30morning
midday13:00high-signal brief
evening19:00evening
weeklySun 20:007-day review

Timezone: Asia/Shanghai. Schedule and report policy remain separate.

05 / FAILURE PATHS

How failures converge

Collector failure

primary → fallback → retry / last_error → health review

Incomplete body/date

discovery_only / rejected → enrichment → max attempts

Schema failure

one repair retry → run failed → exact error in issue

Publish failure

stop → keep draft → preserve Git/D1 evidence → no duplicate run