IR passes and fragment contracts¶

TeXSmith no longer reads HTML and no longer emits LaTeX from Python. The parser, the intermediate representation, the resolver and the writers are tmark's; what remains on the Python side is the part that needs the filesystem, the network and a process — and the part that decides what a construct looks like.

tmark.parse ─▶ IR ─▶ TeXSmith passes ─▶ tmark.resolve ─▶ tmark.write ─▶ Body { text, map, requires }

There are therefore two extension points, and the @reads / @writes decorators of 0.6 map onto them:

All five — readers, writer, latex.template.override, fragment partials and required_partials — were deprecated in 0.7.0 and are gone in 0.8.0: neither the template manifest nor fragment.toml reads them any more, so a package that still declares one converts as if it did not.

IR passes¶

A pass is a pure function (Document, PassContext) -> Document over the generated IR models. It never mutates its input, returns the same object when it has nothing to do, and otherwise rebuilds the tree with document.evolve(...). Failures never raise: a node is replaced by a visible literal and a diagnostic is emitted at its span. An exception escaping a pass is a bug.

Two stages surround the one Rust step in the middle:

pre

before tmark.resolve. A pre pass may add, remove or rewrite blocks — this is where includes are spliced, diagrams converted, DOIs fetched, moustaches expanded.

post

after tmark.resolve. A post pass only slices the block list or computes per-body options, so that the Resolved of the whole document stays valid.

from dataclasses import replace

from tmark.ir import model

from texsmith.core.documents import Document
from texsmith.passes import PassContext, spec


@spec("exam-questions", stage="pre", after=("include",))
def run(document: Document, ctx: PassContext) -> Document:
    ir = document.ir
    if ir is None:
        return document
    blocks = tuple(_number(block, solution=ctx.attribute("solution", False)) for block in ir.blocks)
    if blocks == ir.blocks:
        return document          # nothing to do: hand back the same object
    return document.evolve(ir=replace(ir, blocks=blocks))

Order is declared, never implicit: a PassSpec names the passes it must run after, and build_pipeline performs a stable topological sort that raises PassOrderError on a cycle. Declare a template's passes in its manifest:

[latex.template]
name = "exam"
version = "1.0.0"
entrypoint = "template/template.tex"
engine = "lualatex"

passes = ["my_exam_pkg.questions:run"]

They are resolved with the same import machinery as attribute normalisers and fragment entrypoints, so a typo fails early with an actionable TemplateError, and they apply only while the declaring template renders — exam-style rules would corrupt unrelated documents, so there is no global entry point. [typst.template] reads the same key, and a template that declares both sections lists its passes in both.

Each entry names an attribute of a module ("package.module:attribute"). It may be a PassSpec, used as declared; a callable registered by @spec(...), whose registered spec is reused, so the name, stage and after of the decorator hold; or a plain (Document, PassContext) -> Document callable, wrapped in a pre pass named after the reference. An after naming a bundled pass orders the template's pass among them, and a stage="post" pass runs after tmark.resolve with the bundled post ones.

A template pass reads its options from the PassContext: ctx.attribute("solution", False) walks ctx.contexts — the template overrides first (the document's front matter merged with the CLI's -a key=value, so -a solution=true is True here), then the front matter — and a dotted name reaches into a nested mapping (ctx.attribute("press.title")). The attribute defaults declared in the manifest are resolved by the template renderer and never reach the contexts, so the pass supplies its own default.

Keep passes semantic and backend-neutral: a pass rewrites IR nodes and their attributes, never LaTeX strings. The one exception is deliberate and narrow — the bundled highlight pass turns code blocks into RawBlock{format=latex} holding a Pygments payload, because that work is Python-only and reuses the latex raw path rather than making the writer Pygments-aware.

Custom constructs¶

The container names TMark knows are a closed registry: the callout types (built-in and declared), aside, figure, tabs, tab, multicolumn, landscape and div. A new kind of thing is a declaration under press.declare.admonitions, not a new container name; a wrapper the template styles is div with a class.

::: div {#s1 .sidebar}
Content.
:::

On the paged backends every layout container is rendered by one contract, \begin{tsdiv}{name}[attrs] (#ts-div("name", ..) in Typst), dispatched on the name with the attributes forwarded as keys: #id becomes id, classes become class={a,b}, key=val is forwarded as is (lang and media never are). A template styles a class through the pgfkeys family, or redefines the environment for a name it knows:

\tcbset{/ts/div/class/sidebar/.style={grow to left by=2cm}}
\RenewDocumentEnvironment{tsdiv@multicolumn}{O{}}{\begin{multicols}{3}}{\end{multicols}}

A container name the registry does not know raises container-unknown and renders its content transparently.

A template whose own IR pass reads such a name declares it, next to the pass that claims it, and the warning stops for that name alone:

[latex.template]
passes = ["my_exam_pkg.questions:run"]
containers = ["solution"]

The names are read from the manifest before the documents are parsed — tmark reports an unknown container while it parses, and no pass can take a record back — so solution is a known container for the whole run, on both the LaTeX and the Typst path, and every undeclared name still warns. Bad entries (not a list, not a string, empty) raise TemplateError when the manifest loads.

Contract macros¶

The writers emit a fixed macro or environment per construct, and a fragment must define every macro the writer can name. Naming is regular: macros are \ts<name>, environments ts<name>, lowercase, one word per construct. At most one optional keyval group comes first, then the mandatory arguments, content last.

\tslead{…}
\tsaside[side=left]{…}
\tsprogress[thin]{0.45}{Launch}
\tskeys{Ctrl,Alt,Del}
\tsindex[registry=physics, main]{sort@formatted!sub}
\begin{tscallout}[kind=warning, title={…}, id=w1, collapsed] … \end{tscallout}
\begin{tscode}[lang=py, title={bubble\_sort.py}, linenums=1, hl_lines={2-3}] … \end{tscode}
\tscodeinline[lang=py]{xs.sort()}

Options are pgfkeys keyvals under /ts/<name>/, and every family ignores unknown keys, so an older fragment survives a newer writer. Only keys with a value are serialised; booleans are bare, text is escaped and braced, lists are braced with commas. Typst mirrors each contract with one hyphenated function (#ts-callout, #ts-keys) whose named arguments match the keys.

Which fragment provides which macro is the writer's Requires.fragments: a body that emits \tscallout activates ts-callouts, one that emits \tsindex activates ts-index. Activation is by construction, not by sniffing the rendered LaTeX.

Overriding a construct¶

Three levels, in increasing order of violence. All of them go in the template's .tex, after \VAR{extra_packages} (that is where fragments are \usepackaged).

\VAR{extra_packages}
% (a) restyle through the pgfkeys family
\tcbset{/ts/code/.append style={frame hidden, boxrule=0pt}}

% (b) redefine the macro, same signature
\RenewDocumentCommand{\tscodeinline}{O{}m}{\texttt{#2}}
\RenewDocumentEnvironment{tsdiv@multicolumn}{O{}}{\begin{multicols}{3}}{\end{multicols}}

# (c) replace the fragment outright
press:
  fragments:
    disable: [ts-code]
    append: [./my-code.sty]

A replacement fragment must define every entry of the fragment's provides list; the check runs at load time against tmark.fragments(). See Contract macros for the full table mapping each construct to its macro.

Reference¶

IdAllocator(floor: int = 1)

110
111

def __init__(self, floor: int = 1) -> None:
    self.floor = max(1, floor)

next() -> int

117
118
119
120
121

def next(self) -> int:
    """One fresh id."""
    value = self.floor
    self.floor += 1
    return value

observe(root: Any) -> None

113
114
115

def observe(self, root: Any) -> None:
    """Raise the floor above the ids of ``root`` (a document, node or record)."""
    self.floor = max(self.floor, highest_id(root) + 1)

reserve(count: int) -> int

123
124
125
126
127

def reserve(self, count: int) -> int:
    """Reserve ``count`` consecutive ids; returns the first one."""
    first = self.floor
    self.floor += max(0, count)
    return first

PassContext(
    files: FileTable = FileTable(),
    ids: IdAllocator = IdAllocator(),
    diagnostics: DiagnosticSink = DiagnosticSink(),
    loader: Any = MemoryLoader(),
    output_dir: Path = Path.cwd(),
    include_paths: tuple[Path, ...] = (),
    root_dir: Path | None = None,
    request: ConversionRequest | None = None,
    contexts: tuple[Mapping[str, Any], ...] = (),
    emitter: DiagnosticEmitter = NullEmitter(),
    template: SlotTemplate = SlotTemplate(),
    backend: str = "latex",
    code: Mapping[str, Any] = dict(),
    bibliography: list[Path] = list(),
    pygments_styles: dict[str, str] = dict(),
    doi_fetcher: Any = None,
    copy_assets: bool = True,
    convert_assets: bool = False,
    hash_assets: bool = False,
    values: dict[str, Any] = dict(),
)

attribute(name: str, default: Any = None) -> Any

222
223
224
225
226
227
228
229
230
231
232
233
234
235
236

def attribute(self, name: str, default: Any = None) -> Any:
    """The template attribute ``name`` as the conversion resolved it, else ``default``.

    The lookup walks :attr:`contexts` in order — template overrides (front
    matter merged with the CLI's ``-a key=value``), then the front matter,
    then the mustache defaults — and a dotted ``name`` reaches into a nested
    mapping (``ctx.attribute("press.title")``). A template pass reads its
    options here: ``ctx.attribute("solution", False)`` is ``True`` under
    ``-a solution=true`` and falls back to the pass's own default when
    nothing overrides it, since manifest defaults never reach the contexts.
    """
    from texsmith.passes.var import MISSING, lookup

    value = lookup(name.split("."), self.contexts)
    return default if value is MISSING else value

PassSpec(
    name: str,
    run: Pass,
    after: tuple[str, ...] = (),
    needs_io: bool = False,
    stage: Stage = "pre",
)

SlotTemplate(
    default_slot: str = "mainmatter",
    requests: Mapping[str, str] = dict(),
    levels: Mapping[str, int] = dict(),
    strip_heading: frozenset[str] = frozenset(),
    base_level: int = 0,
)

build_pipeline(
    names: Iterable[str] = DEFAULT_PIPELINE,
    *,
    extra: Iterable[PassSpec] = (),
) -> tuple[PassSpec, ...]

284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320

def build_pipeline(
    names: Iterable[str] = DEFAULT_PIPELINE,
    *,
    extra: Iterable[PassSpec] = (),
) -> tuple[PassSpec, ...]:
    """The passes to run, ordered.

    ``names`` gives the base order (registered passes, looked up by name);
    ``extra`` adds specs placed by their ``after`` constraints. The sort is
    stable: without constraints the listed order stands, and every ``pre``
    pass precedes every ``post`` pass.
    """
    import texsmith.passes.builtins

    ordered: list[PassSpec] = []
    for name in names:
        try:
            ordered.append(REGISTRY[name])
        except KeyError:
            raise PassOrderError(f"unknown pass '{name}'") from None
    ordered.extend(extra)
    by_name = {item.name: item for item in ordered}
    if len(by_name) != len(ordered):
        seen: set[str] = set()
        for item in ordered:
            if item.name in seen:
                raise PassOrderError(f"pass '{item.name}' listed twice")
            seen.add(item.name)

    for item in ordered:
        for dependency in item.after:
            if dependency not in by_name:
                raise PassOrderError(f"pass '{item.name}' runs after unknown pass '{dependency}'")

    pre = [item for item in ordered if item.stage == "pre"]
    post = [item for item in ordered if item.stage == "post"]
    return (*_sort_stage(pre, by_name), *_sort_stage(post, by_name))

highest_id(root: Any) -> int

130
131
132

def highest_id(root: Any) -> int:
    """The largest node id under ``root`` (a document, node, record or tuple), ``0`` when none."""
    return max(_iter_ids(root), default=0)

register(pass_spec: PassSpec) -> PassSpec

244
245
246
247

def register(pass_spec: PassSpec) -> PassSpec:
    """Register a pass by name (a template or plugin adds its own here)."""
    REGISTRY[pass_spec.name] = pass_spec
    return pass_spec

run_pipeline(
    document: Document,
    ctx: PassContext,
    pipeline: Iterable[PassSpec] = (),
    *,
    resolve: Callable[[Document, PassContext], Document]
    | None = None,
) -> Document

358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378

def run_pipeline(
    document: Document,
    ctx: PassContext,
    pipeline: Iterable[PassSpec] = (),
    *,
    resolve: Callable[[Document, PassContext], Document] | None = None,
) -> Document:
    """Run ``pipeline`` on ``document``: the ``pre`` passes, ``resolve``, the ``post`` passes."""
    passes = tuple(pipeline) if pipeline else build_pipeline()
    current = document
    for item in passes:
        if item.stage != "pre":
            continue
        current = item.run(current, ctx)
    if resolve is not None:
        current = resolve(current, ctx)
    for item in passes:
        if item.stage != "post":
            continue
        current = item.run(current, ctx)
    return current

spec(
    name: str,
    *,
    after: Iterable[str] = (),
    needs_io: bool = False,
    stage: Stage = "pre",
) -> Callable[[Pass], Pass]

250
251
252
253
254
255
256
257
258
259
260
261
262
263

def spec(
    name: str,
    *,
    after: Iterable[str] = (),
    needs_io: bool = False,
    stage: Stage = "pre",
) -> Callable[[Pass], Pass]:
    """Decorator registering ``run`` under ``name``."""

    def decorate(run: Pass) -> Pass:
        register(PassSpec(name=name, run=run, after=tuple(after), needs_io=needs_io, stage=stage))
        return run

    return decorate

Body(
    text: str,
    map: list[tuple[int, int, int]] = list(),
    requires: Requires = Requires(),
)

Requires(
    packages: list[str] = list(),
    fragments: list[str] = list(),
    shell_escape: bool = False,
    assets: list[dict[str, Any]] = list(),
    bibliography: bool = False,
    citations: list[str] = list(),
    acronyms: list[str] = list(),
    index: list[str] = list(),
    counters: list[str] = list(),
)

merge(other: Requires) -> None

83
84
85
86
87
88
89
90
91
92
93

def merge(self, other: Requires) -> None:
    """Union ``other`` into this record, keeping first-seen order of the lists."""
    _extend(self.packages, other.packages)
    _extend(self.fragments, other.fragments)
    self.shell_escape = self.shell_escape or other.shell_escape
    self.assets.extend(item for item in other.assets if item not in self.assets)
    self.bibliography = self.bibliography or other.bibliography
    _extend(self.citations, other.citations)
    _extend(self.acronyms, other.acronyms)
    _extend(self.index, other.index)
    _extend(self.counters, other.counters)

build_writer_options(
    *,
    backend: str,
    language: str | None = None,
    code_options: Mapping[str, Any] | None = None,
    legacy_accents: bool = False,
    media: str | None = None,
    base_level: int = 1,
    numbered: bool = True,
    refs: Mapping[str, str] | None = None,
    numbering: Mapping[str, str] | None = None,
    source_map: bool = False,
) -> dict[str, Any]

134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174

def build_writer_options(
    *,
    backend: str,
    language: str | None = None,
    code_options: Mapping[str, Any] | None = None,
    legacy_accents: bool = False,
    media: str | None = None,
    base_level: int = 1,
    numbered: bool = True,
    refs: Mapping[str, str] | None = None,
    numbering: Mapping[str, str] | None = None,
    source_map: bool = False,
) -> dict[str, Any]:
    """The ``options`` dict of ``tmark.write``, one key per ``WriterOptions`` field.

    ``code_options`` is the merged ``code`` section of the template context
    (``_resolve_code_options``): ``engine``, ``inline.plain``, ``inline.breaks``.
    """
    code = dict(code_options or {})
    engine = str(code.get("engine") or "pygments").strip().lower()
    if engine not in _CODE_ENGINES:
        engine = "pygments"
    inline = normalise_inline_options(code.get("inline"))
    options: dict[str, Any] = {
        "media": media or ("web" if backend == "html" else "print"),
        "code": {
            "engine": engine,
            "inline_plain": bool(inline["plain"]),
            "inline_breaks": str(inline["breaks"]),
        },
        "latex": {"legacy_accents": bool(legacy_accents)},
        "headings": {"base_level": int(base_level), "numbered": bool(numbered)},
        "source_map": bool(source_map),
    }
    if language:
        options["lang"] = language
    if refs:
        options["refs"] = {key: str(value) for key, value in refs.items()}
    if numbering:
        options["numbering"] = {key: str(value) for key, value in numbering.items()}
    return options

write_body(
    ir_document: Document,
    backend: str,
    options: Mapping[str, Any],
    *,
    blocks: tuple[Block, ...] | None = None,
    loader: Any = None,
    resolved: Any = None,
    resolve_options: ResolveOptions | None = None,
    sink: DiagnosticSink | None = None,
) -> Body

177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208

def write_body(
    ir_document: model.Document,
    backend: str,
    options: Mapping[str, Any],
    *,
    blocks: tuple[model.Block, ...] | None = None,
    loader: Any = None,
    resolved: Any = None,
    resolve_options: ResolveOptions | None = None,
    sink: DiagnosticSink | None = None,
) -> Body:
    """Write ``ir_document`` (or the slice ``blocks`` of it) with ``backend``.

    ``resolved`` is the ``tmark.resolve`` result (or its ``handle``) shared by
    every body of the document; without it tmark resolves the slice itself
    through ``loader`` with ``resolve_options``.
    """
    target = ir_document if blocks is None else replace(ir_document, blocks=blocks)
    payload = codec.encode_document(target)
    payload["tmark"] = tmark.version()
    handle = resolved.get("handle", resolved) if isinstance(resolved, Mapping) else resolved
    result = tmark.write(
        payload,
        backend,
        dict(options),
        loader,
        handle,
        resolve_options.to_json() if resolve_options is not None else None,
    )
    if sink is not None:
        sink.extend_from_tmark(result.get("diagnostics") or ())
    return Body.from_json(result)