a whirlwind tour of rustdoc
I’m generally a fan of rustdoc. I love that there’s effectively a one-touch access to creating
good-looking documentation for my libraries. It’s not without its own issues, but the fact that it’s
distributed alongside rust, and produces pretty good output, is enough to make me happy about it.
But before we get too far, what even is rustdoc? Put simply, it’s the thing that runs
under-the-hood whenever you run cargo doc. It takes a crate as input, and generates a pile of HTML
pages (and their supporting assets) as output. You can preview what the output looks like by
checking out rust’s standard library documentation. By default, it includes the theme
shown there, the search box that searches anything within that crate, and docs for all a crate’s
dependencies, whose items are also included in that search index.
(For a bit of history: rustdoc is pretty old. The commit that added it dates to December 2011! That makes it almost as old as rustc itself (started in July 2009 and successfully self-hosted in April 2011), well before Rust 1.0 landed in May 2015.)
where oh where has my little crate gone
That’s all well and good, but what happens if you want to tweak something about it? Say, there’s a formatting issue you want to fix, or it’s not outputting docs for something you would expect it to pick up. Where is rustdoc?
It actually doesn’t have a repo of its own. It’s situated alongside the source for the rust
compiler, and is built alongside it whenever you build the compiler as a whole. This
means that if you want to tweak it and test out your change, you need to submit yourself to get
used to the compiler’s build system. This can be an issue if you haven’t dealt with rustbuild
before, or if you just want to test a small change.
show me the way to docsville
So here’s a bit of a crash course in rustbuild, the build system for the compiler. A better
introduction is available in the main repo’s README, but here are the basics:
All interaction with the build system is either done through git (if you need to deal with
submodules) or through a wrapper script called x.py. This provides a means to download known-good
versions of rustc and cargo to compile the actual build system, written fully in
rust. To get to just the parts we care about, there are two commands worth knowing: ./x.py build
and ./x.py doc. The first just compiles the bits we ask for (or everything, if you don’t ask for
something specific), and the second compiles everything and then renders the full suite of
documentation: standard library docs, the book, the reference, anything you see at
https://doc.rust-lang.org. It’s useful if your rustdoc change can be seen by looking at the
standard library docs instead of having to render a separate crate’s docs for your test.
follow the rusty-brick road
If you just run one of those commands by themselves, you’ll start to notice something annoying: they
compile a lot of things. This is the nature of building the compiler: it goes through a
“bootstrap” sequence, where it starts with a known version (in this case, the most recent beta),
builds the compiler, then uses the result to build the compiler again, just to prove that
everything still works. (There’s an optional third step involving building a third time to prove
its reproducible. This is turned off by default.)
If you were just working on the compiler or standard library, then you could get around this with
the --stage and --keep-stage flags, but this doesn’t quite work when testing rustdoc on the
standard library documentation. At least, as of this writing. So for now, if we want to test
rustdoc the easy way, we’re stuck recompiling the whole stage to re-render the standard library
docs. I’ll mention an alternate method later on.
To test changes to rustdoc by rendering the standard library documentation, make your changes to the
code, then call ./x.py doc --stage 1 to build rustdoc and render the library documentation. This
will compile just rustdoc on stage 0, then all of stage 1, then render the set of documentation. The
rendering will be available in the build directory in your rust clone, specifically
build/$TRIPLE/doc, where $TRIPLE is the “build triple” of the system you’re working on. In my
case, it’s x86_64-unknown-linux-gnu, since I do my Rust development on a 64-bit Arch Linux
system.
If you don’t want to wait for the entire stage to build every time, or if you need to test something
that doesn’t have a good representative case in the standard library, then you can take your custom
rustc and rustdoc and use them to render documentation for a separate crate. To do this, run ./x.py
build src/libtest --stage 1 as your build step. This builds just enough that you get useable
binaries for the next steps.
(UPDATE: Thanks to PR 43482, rustdoc is now outside the regular compilation flow! This means a lot
of the above gripes and instructions are invalid. To build standard library documentation, you can
just run ./x.py doc --stage 1 src/libstd, and it will output the docs in build/$TRIPLE/doc,
where $TRIPLE is the build triple of the system you’re working on. In my case, it’s
x86-unknown-linux-gnu. This will compile stage 1 artifacts, but only the first time. After that,
only rustdoc itself will be rebuilt if that’s all you changed. To build a rustdoc you can use to
compile other crates, run ./x.py build --stage 1 src/tools/rustdoc src/libstd. Then set up a
linked toolchain like below.)
If you’re using rustup to manage your regular rust installs, it has a fantastic feature for people
who work on the compiler. You can “link” a local directory to create a custom toolchain. To do this,
run the command rustup toolchain link local /path/to/build/$TRIPLE/stage1. (Replace the path in
the last argument with the actual path to the build directory in your Rust clone. The
build/$TRIPLE/stage1 is the actual structure that needs to be at the end of the path, much like
the standard library docs were in build/$TRIPLE/doc.)
Once you’ve set this up, you can use your “local” rustc and rustdoc just like you would use the
“stable” or “nightly” toolchains! The “local” in that rustup command earlier set the name for a new
toolchain. Therefore, you can generate documentation with cargo +local doc. If you’ve rebuilt your
rustdoc but haven’t touched the other crate, you may need to run cargo clean beforehand so it
actually rebuilds everything.
(If you don’t have rustup, don’t fret! It’s still possible to use your custom toolchain for a
one-off documentation run. It’s just a little more complicated. Cargo recognizes two environment
variables, RUSTC and RUSTDOC, which point to their respective binaries. By setting these two
environment variables and calling cargo doc, you can make cargo use your custom toolchain even
without rustup!)
the winding streets of the verdant rustdoc city
(caution: the only part of rustdoc i’ve extensively worked with is the part that generates the HTML;
i make no claims of accuracy for any other component of rustdoc >_>)
Rustdoc uses the compiler-internal libraries, so it can read a crate the same way the compiler would. However, to guard itself against changes to the compiler, it takes the “raw” syntax tree and converts it into its own simplified version before working on it. This “doctree” version of the crate is then “cleaned” to strip the AST to only those elements that make sense for documentation (for example, items inside function bodies are never visible outside that function, so they’re removed when crawling the AST).
With this “clean” version in hand, it then generates the search index, writes out the static files
that go with the output, and walks the crate, writing each item in turn to the appropriate folder
and file. The formatting is actually implemented as a bunch of fmt::Display impls, so the template
is literally a big write! call with the page template as the format string. This way,
modules render links to all their items, traits render a listing of their methods and implementors,
enums list their variants, and so on.
To render documentation comments, the Rust compiler first converts /// and //! comments into
#[doc] attributes, then rustdoc collects all of these together and hands them off to a Markdown
renderer to convert to HTML. The resulting HTML is then used wherever the documentation for an item
needs to be.
At the moment, rustdoc defaults to using a renderer called Hoedown, but we’re currently testing an
alternative implementation called pulldown-cmark that conforms to the CommonMark
specification and allows for easier extension due to being written in Rust. As of Rust 1.18, you can
test the pulldown version by passing --enable-commonmark to rustdoc, like with
RUSTDOCFLAGS="--enable-commonmark" cargo doc.
If you’re asking rustdoc to run documentation tests, it runs most of the steps above, but stops
short of emitting the final HTML. Instead, it collects all the documentation comments from the crate
and hands them to the Markdown renderer to see what it picks up as a code block. From these code
blocks, it reads through them to find whether it needs to remove a trailing # from hidden lines,
or add the required extern crate line and fn main() wrapper, or even needs to process it at all
due to a no_run or ignore flag on the block.
For those that need to run, it takes the code after processing and hands it off to the compiler to build the test. For those that need to run, it then executes the test and checks the result to evaluate the final test result.
pay no attention to those maintainers behind the curtain
Rustdoc is an official Rust project, and falls under the resposibility of the Dev tools team. In
addition, there are three “rustdoc tool peers” who could be considered the primary maintainers of
rustdoc: @steveklabnik, @GuillaumeGomez, and myself (@QuietMisdreavus). Issues against rustdoc can
be posted on the main rust-lang/rust issue tracker, and short-form discussion about
it can be had in the #rust-dev-tools IRC channel on the Mozilla IRC network.