https://ourbigbook.com source code + a compatible local CLI static wiki generator and markup language to write complex structured wikis/books/blogs with reference implementation in JavaScript.
This repository (The OurBigBook Project) contains two main things:
- OurBigBook CLI: a static Wiki generator that can be invoked from the command line with the
ourbigbook
executable. The OurBigBook CLI calls the OurBigBook Library to convert each input file.The markup language used is called OurBigBook Markup (file extension:.bigb
), and it is compatible with https://ourbigbook.com, TODO support import/export to/from https://ourbigbook.com to local filesystem. - OurBigBook Web: the source code of https://ourbigbook.com, which is a regular database-backed dynamic website. This is basically separate package that depends on the OurBigBook Library:Everything else in this repository outside of that section/directory is about the OurBigBook Library and the OurBigBook CLI, not the dynamic website.
- information about OurBigBook Web is present at: Section 7. "OurBigBook Web (OurBigBook.com, Dynamic website)"
- the source code of the OurBigBook Web is present under
web/
Key links:
- https://cirosantilli.com: showcase demo document with interesting content. Primary inspiration for OurBigBook development.
- https://cirosantilli.com/oxford-nanopore-river-bacteria: a self-contained tutorial style part of the above. Note how internal links integrate seamlessly into the more global topic of biology, e.g. when talking about DNA we link to the global topic https://cirosantilli.com/dna.
- https://github.com/cirosantilli/cirosantilli.github.io and https://github.com/cirosantilli/cirosantilli.github.io/blob/dev/oxford-nanopore-river-bacteria.bigb: source of the above showcase documents
- Section 3. "Design goals": feature overview
- https://github.com/cirosantilli/ourbigbook: OurBigBook source code
- https://github.com/cirosantilli/ourbigbook/blob/master/README.bigb: source for this document
- https://docs.ourbigbook.com: rendered version of this document
- https://docs.ourbigbook.com/editor: live in-browser demo
- https://cirosantilli.com/ourbigbook-template: good template to get started, see Section 2. "Quick start"
- cross references to any header (including e.g. h2, h3, etc. in other files), images, etc. with amazing error checking and reporting: never break internal links without knoing again, and quickly find out what broke when you do. E.g.:animal.bigb
= Animal <Bats> are <flying animals>.
= Flying animal == Bat
Animal.bigb
would render something like:<a href="flying-animal.html#bat">Bats</a> are <a href="flying-animal.html">flying animals</a>.
- nonexistent id:
<Weird animal not documented>
- duplicate IDs:
= Animal == Dog == Cat == Dog
- nonexistent id:
- KaTeX server side mathematics, works on browsers with JavaScript disabled:
I like $\sqrt{2}$, but I adore this \x[equation-quadratic-equation]: $$ x^2 + 2x + 1 $$ {title=Quadratic equation}
- multi-file features out of the box so you don't need a separate wrapper like Jekyll to make a multi-page website:
- cross file references
- single-source multi-format output based on includes and build options:
- by default, one HTML per source with includes rendered as links between pages, e.g.:README.bigb
= My website == h2 \Include[not-readme]
= Not readme == Not readme h2
index.html
andnot-readme.html
- with the
-S
,--split-headers
option, you can output each header of an input file into a separate output file. The previous filesystem would produce:Each of those pages automatically gets a table of contentsindex.html
: which contains the fullREADME.bigb
outputsplit.html
: split version of the above containing only the= My website
header and noth2
h2.html
: only contains theh2
headernot-readme.html
contains the full output ofnot-readme.bigb
not-readme-split.html
: only contains the= Not readme
headernot-readme-h2.html
: only contains the= Not readme h2
header
--embed-includes
single file output from multiple input files. Includes are parsed smartly, not just source copy pasted, e.g. included headers are shifted fromh1
toh2
correctly.On the previous sample filesystem, it would produce a single output fileindex.html
which would contain a header structure like:= My website == h2 === Not readme ==== Not readme h2
- supports both local serverless rendering to HTML files for local viewing, and server oriented rendering such as GitHub pages, e.g. cross references automatically get
.html
extension and or not. E.g.:- locally, a link
\x[not-readme]
would render as<a href="not-readme.html">
andnot-readme.bigb
producesnot-readme.html
- when publishing,
\x[not-readme]
would render as<a href="not-readme">
andnot-readme.bigb
also producesnot-readme.html
, which the server converts to justhttp://my-website.com/not-readme
- locally, a link
- cross file configuration files to factor out common page parts like headers, footers and other metadata, e.g.:
main.liquid.html
: Liquid template used for all pages, see example at: Section 2.1. "Play with the template"main.scss
: CSS stylesheet generated from SASS input, see example at: Section 2.1. "Play with the template"ourbigbook.tex
: global LaTeX math definitions, e.g.:\newcommand{\abs}[1]{\left|#1\right|}
$\abs{x}$
ourbigbook.json
: per repository configuration options
- table of contents that crosses input files via includes. E.g. in:README.bigb
= My website == h2 \Include[not-readme]
= Not readme == Not readme h2
index.html
also contains the headers fornot-readme.bigb
producing:This means that you can split large splitDefault input files if rendering starts to slow you down, and things will still render exactly the same.- My website
- h2
- Not readme
- Not readme h2
- Not readme
- h2
- My website
- check that local files and images linked to actually exist:
\a
check
argument. E.g.:\a[i-don-exist.txt]
- associate headers to files with the
\H
file
argument e.g.:Here's an example of a nice image: \x[file/path/to/my/image.png]. = path/to/my/image.png {file} This image was taken when I was on vacation!
- advanced header/ID related features:
- ID-based header levels:
= Furry animal I like \x[furry-animal]{p}, especially my cat, here is his photo: \x[image-my-cat]. == Cat \Image[My_cat.jpg] {title=My cat}
- scopes either with directories or with within a single file:
See the important conclusion of my experiment: \x[report-of-my-experiment/conclusion] = Report of my experiment {scope} == Introduction == Middle == Conclusion
- cross reference title inflection for capitalization and pluralization, e.g.;
= Dog == Snoopy {c} \x[dog]{c}{p} are fun. But the \x[dog] I like the most is \x[snoopy]!
\x[dog]{c}{p}
asDogs
: capitalized because of{c}
and pluralized because of{p}
\x[dog]
asdogs
: auto lowercased because its header= Dog
does not have{c}
\x[snoopy]
asSnoopy
: title capitalization kept to upper case due to{c}
on the header== Snoopy
- synonyms, e.g.:
= User interface = UI {c} {synonym} {title2} \x[user-interface]{c} is too long, I just say \x[ui].
<a href="#user-interface">User interface</a> is too long, I just say <a href="user-interface">UI</a>
ui.html
user-interface.html
, so it serves as a way to have backward compatibility on page renames.And thetitle2
makes it appears on the main title under parenthesis, something like:<h1>User interface (UI)</h1>
- header disambiguation, e.g.:
My favorite fruits are \x[apple-fruit]{p}! My favorite least favorite brand is is \x[apple-company]! \x[apple] computers are too expensive. == Apple {disambiguate=fruit} == Apple {c} {disambiguate=company} = Apple {c} {synonym}
\x[apple-fruit]{p}
:<a href="apple-fruit">apples</a>
\x[apple-company]
:<a href="apple-company">Apple</a>
\x[apple]
: also<a href="apple-company">Apple</a>
because of the synonym== Apple\n{disambiguate=fruit}
:<h2 id="apple-fruit">Apple (fruit)</h2>
== Apple\n{disambiguate=company}
:<h2 id="apple-company">Apple (company)</h2>
- tags are regular headers:
\H
child
argument,\x
child
argument= Animal == Dog {tag=domestic} {tag=cute} == Cat {tag=domestic} {tag=cute} == Bat {tag=flying} = Flying = Cute = Domestic
- unlimited header levels, levels higher than 6 are rendered in HTML as an appropriately styled
div
s with an ID:= h1 == h2 === h3 ==== h4 ===== h5 ====== h6 ======= h7 ======== h8
- generate lists of incoming links between internal headers: it shows every internal link coming into the current page
- ID-based header levels:
- is written in JavaScript and therefore runs natively on the browser to allow live previews as shown at: https://docs.ourbigbook.com/editor
- helps you with the publishing:
ourbigbook --publish
publishes in a single command to the configured target (default GitHub pages): Section 5.2.16. "-p, --publish
"- OurBigBook tries to deal with media such as images and video intelligently for you, e.g.: Section 4.1.8.3. "Where to store images". E.g. you can keep media in a separate media repository,
my-media-repository
, and then by configuring onourbigbook.json
:"media-providers": { "github": { "default-for": ["image", "video"], "path": "media", "remote": "yourname/myproject-media" } }
\Image[My_image_basename.jpg]
\Image[https://raw.githubusercontent.com/cirosantilli/myproject--media/master/My_image_basename.jpg]
inotifywait
watch and automatically rebuild with-w
,--watch
:ourbigbook --watch input-file.bigb
- automatic code formatting:
--format-source
Learn the syntax basics in 5 minutes: https://docs.ourbigbook.com/editor.
Play with a OurBigBook template locally:
That template can be seen rendered live at: http://cirosantilli.com/ourbigbook-generate-multifile/ Other templates are documented at:
git clone https://github.com/cirosantilli/ourbigbook-template
cd ourbigbook-template
npm install
npx ourbigbook .
firefox index.html
--generate
.To publish to GitHub Pages on your repository you can just fork the repository https://github.com/cirosantilli/ourbigbook-template to your own https://github.com/johndoe/ourbigbook-template and then:
and it should now be visible at: https://johndoe.github.io/ourbigbook-template
git remote set-url origin git@github.com:johndoe/ourbigbook-template.git
npx ourbigbook --publish
Then, every time you make a change you can publish the new version with:
or equivalently with the
git add .
git commit --message 'hacked stuff'
ourbigbook --publish .
-P, --publish-commit <commit-message>
shortcut:
ourbigbook --publish-commit 'hacked stuff'
If you want to publish to your root page https://johndoe.github.io instead of https://johndoe.github.io/ourbigbook-template you need to rename the
master
branch to dev
as mentioned at publish to GitHub pages root page:
git remote set-url origin git@github.com:johndoe/johndoe.github.io.git
# Rename master to dev, and delete the old master.
git checkout -b dev
git push origin dev:dev
git branch -D master
git push --delete origin master
npx ourbigbook --publish
The following files of the template control the global style of the output, and you are free to edit them:
main.liquid.html
: global HTML template in Liquid format. Available variables are documented at--template
, and it is being selected in that repository with :"template": "main.liquid.html"
ourbigbook.json
configuration file.- Sass is just much more convenient to write than raw CSS.That file gets included into the global HTML template inside
main.liquid.html
at:<link rel="stylesheet" href="{{ root_relpath }}main.css">
When you run:
it converts all files in the current directory separately, e.g.:
npx ourbigbook .
README.bigb
toindex.html
, sinceREADME
is a magic name that we want to show on the root URLnot-readme.bigb
tonot-readme.html
, as this one is a regular name unlikeREADME
main.scss
tomain.css
If one of the input files starts getting too large, usually the toplevel
Note however that when those individual files have a cross file reference to something defined in
to parse all files and extract all necessary IDs to the ID database. That would be optimized slightly wit the
to only extract the IDs but not render, which speeds things up considerably
README.bigb
in which you dump everything by default like Ciro does, you can speed up development and just compile files individually with either:
npx ourbigbook README.bigb
npx ourbigbook not-readme.bigb
not-readme.bigb
, e.g. via \x[h2-in-not-the-readme]
, then you must first do a first pass once with:
npx ourbigbook .
--no-render
command line option:
npx ourbigbook --no-render .
When dealing with large files, you might also be interested in the following amazing options:
To produce a single standalone output file that contains everything in a directory run:
You can now just give
npx ourbigbook --embed-resources --embed-includes README.bigb
xdg-open index.html
index.html
to any reader and they should be able to view it offline without installing anything. The flags are:--embed-includes
: without this,\Include[not-readme]
shows as a link to the filenot-readme.html
which comes fromnot-readme.bigb
With the flag,not-readme.bigb
output gets embedded into the outputindex.html
directly--embed-resources
: by default, we link to CSS and JavaScript that lives insidenode_modules
. With this flag, that CSS and JavaScript is copied inline into the document instead. One day we will try to handle images that way as well
Install the NPM package globally and use it from the command line for a quick conversion:
or to a file:
You almost never want to do this except when developing OurBigBook, as it won't be clear what version of
npm install -g ourbigbook
printf 'ab\ncd\n' | ourbigbook --body-only
printf 'ab\ncd\n' | ourbigbook > tmp.html
ourbigbook
the document should be compiled with. Just be a good infant and use OurBigBook with the template that contains a package.json
via npx
, OK?Furthermore, the default install of Chromium on Ubuntu 21.04 uses Snap and blocks access to dotfiles. For example, in a sane NVM install, our global CSS would live under One workaround is to use
/home/ciro/.nvm/versions/node/v14.17.0/lib/node_modules/ourbigbook/dist/ourbigbook.css
, which gets blocked because of the .nvm
part:- https://forum.snapcraft.io/t/dot-files/7062
- https://bugs.launchpad.net/snapd/+bug/1607067
- https://superuser.com/questions/1546550/chromium-81-wont-display-dotfiles-anymore
- https://askubuntu.com/questions/1184357/why-cant-chromium-suddenly-access-any-partition-except-for-home
- https://askubuntu.com/questions/1214346/as-a-user-is-there-any-way-to-change-the-confinement-of-a-snap-package
--embed-resources
, but this of course generates larger outputs.To run master globally from source for development see: Section 11.1. "Run OurBigBook master". This one actually works despite the dotfile thing since your development path is normally outside of dotfiles.
Try out the JavaScript API with lib_hello.js:
npm install ourbigbook
./lib_hello.js
OurBigBook is designed entirely to allow writing complex professional HTML and PDF scientific books, blogs, articles and encyclopedias.
OurBigBook aims to be the ultimate LaTeX "killer", allowing books to be finally published as either HTML or PDF painlessly (LaTeX being only a backend to PDF generation).
It aims to be more powerful and saner and than Markdown and Asciidoctor.
Originally, OurBigBook was is meant to be both saner and more powerful than Markdown and Asciidoctor.
But alas, as Ciro started implementing and using it, he started to bring some Markdown insanity he missed back in.
And so this "degraded" slightly into a language slightly saner than Asciidoctor but with an amazing Node.js implementation that makes it better for book writing and website publishing.
Notably, we hope that our escaping will be a bit saner backslash escapes everything instead of Asciidoctor's "different escapes for every case" approach: https://github.com/asciidoctor/asciidoctor/issues/901
But hopefully, having starting from a saner point will still produce a saner end result, e.g. there are sane constructs for every insane one.
It is intended that this will be an acceptable downside as OurBigBook will be used primarily large complex content such as books rather than forum posts, and will therefore primarily written either:
- in text editors locally, where users have more features than in random browser textareas
- in a dedicated website that will revolutionize education, and therefore have a good JavaScript editing interface: https://github.com/cirosantilli/write-free-science-books-to-get-famous-website
For example, originally OurBigBook had exactly five magic characters, with similar functions as in LaTeX:and double blank newlines for paragraphs if you are pedantic, but this later degenerated into many more with insane macro shortcuts.
\
backslash to start a macro, like LaTeX{
and}
: left and right square brackets to delimit optional macro arguments[
and]
: left and right curly braces bracket to start an optional arguments
We would like to have only square brackets for both optional and mandatory to have even less magic characters, but that would make the language difficult to parse for computer and humans. LaTeX was right for once!
This produces a very regular syntax that is easy to learn, including doing:
- arbitrary nesting of elements
- adding arbitrary properties to elements
This sanity also makes the end tail learning curve of the endless edge cases found in Markdown and Asciidoctor disappear.
The language is designed to be philosophically isomorphic to HTML to:
- further reduce the learning curve
- ensure that most of HTML constructs can be reached, including arbitrary nesting
More precisely:
- macro names map to tag names, e.g.:
\\a
to<a
- one of the arguments of macros, maps to the content of the HTML element, and the others map to attributes.E.g., in a link:
\a[http://example.com][Link text\]
http://example.com
href
of<a
, and the second macro argument:Link text
<a>Link text<>
.
The high sanity of OurBigBook, also makes creating new macro extensions extremely easy and intuitive.
All built-in language features use the exact same API as new extensions, which ensures that the extension API is sane forever.
Markdown is clearly missing many key features such as block attributes and cross references, and has no standardized extension mechanism.
The "more powerful than Asciidoctor" part is only partially true, since Asciidoctor is very featureful can do basically anything through extensions.
The difference is mostly that OurBigBook is completely and entirely focused on making amazing scientific books, and so will have key features for that application out-of-the box, notably:and we feel that some of those features have required specialized code that could not be easily implemented as a standalone macro.
- amazing header/ToC/ID features including proper error reports: never have a internal broken link or duplicate ID again
- server side pre-rendered maths with KaTeX: all divs and spans are ready, browser only applies CSS, no JavaScript gets executed
-p, --publish
: we take care of website publishing for you out-of-the-box, no need to integrate into an external project like Jekyll-S
,--split-headers
:- https://github.com/asciidoctor/asciidoctor/issues/626 feature request
- https://github.com/owenh000/asciidoctor-multipage third party plugin that does it
Another advantage over Asciidoctor is that the reference implementation of OurBigBook is in JavaScript, and can therefore be used on browser live preview out of the box. Asciidoctor does Transpile to JS with Opal, but who wants to deal with that layer of complexity?
Static wiki generators: this is perhaps the best way of classifying this project :-)
- https://github.com/gollum/gollum: already has a local server editor! But no WYSIWYG nor live preview. Git integration by default, so when you save on the UI already generates a Git commit. We could achieve that with: https://github.com/isomorphic-git/isomorphic-git, would be really nice. Does not appear to have built-in static generation:Does not appear to check that any links are correct.
- https://github.com/wcchin/markypydia
- https://obsidian.md/ closed source, Markdown with cross file reference + a SaaS. Appears to require payment for any publishing. 28k followers 2021: https://twitter.com/obsdmd. Founders are likely Canadians of Asian descent from Waterloo University: https://www.linkedin.com/in/lishid/ | https://www.linkedin.com/in/ericaxu/ also working in parallel on https://dynalist.io/ 2020 review at: https://www.youtube.com/watch?v=aK2fOQRNSxc Has offline editor with side-by-side preview. Compares with Roam and Notion, but can't find any public publishing on those, seem to be enterprise only things.
Static book generators:
- https://github.com/rstudio/bookdown, https://bookdown.org/. Very similar feature set to what we want!!! Transpiles to markdown, and then goes through Pandoc: https://bookdown.org/yihui/bookdown/pandoc.html, thus will never run on browser without huge translation layers. But does have an obscene amount of output formats however.
- Hugo. Pretty good, similar feature set to ours. But Go based, so hard on browser, and adds adhoc features on top of markdown once again
- https://en.wikipedia.org/wiki/Personal_wiki
- https://github.com/hplgit/doconce
- https://www.gwern.net/About#source is pretty interesting, uses https://github.com/jaspervdj/Hakyll/ + some custom stuff.
- https://github.com/JerrySievert/bookmarkdown
- https://www.gitbook.com/
- https://github.com/rust-lang/mdBook. Impressive integrated search feature. Like Gitbook but implemented in Rust.
- https://github.com/facebook/docusaurus React + markdown based, written in TypeScript. So how can it be build fast? Gotta benchmark.
- vimdoc: http://vimdoc.sourceforge.net/ They do have perfectly working Internal cross file references, see any page e.g. http://vimdoc.sourceforge.net/htmldoc/pattern.html.
Less related but of interest, similar philosophy to what Ciro wants, but no explicitly reusable system:
Ciro Santilli developed OurBigBook to perfectly satisfy his writing style, which is basically "create one humongous document where you document everything you know about a subject so everyone can understand it, and just keep adding to it".
https://cirosantilli.com is the first major document that he has created in OurBigBook.
He decided to finally create this new system after having repeatedly facing limitations of Asciidoctor which were ignored/wontfixed upstream, because Ciro's writing style is not as common/targeted by Asciidoctor.
Following large documents Ciro worked extensively on:made the limitations of Asciidoctor clear to Ciro, and were major motivation in this work.
The key limitations have repeatedly annoyed Ciro were:
- cannot go over header level 6, addressed at: unlimited header levels
- the need for
-S
,--split-headers
to avoid one too large HTML output that will never get indexed properly by search engines, and takes a few seconds to load on any browser, which is unacceptable user experience
OurBigBook Markup is the markup language used in the OurBigBook project.
It works both on the OurBigBook Web dynamic website, and on OurBigBook CLI static websites from the command line.
This section documents all OurBigBook macros.
Macros are magic commands that do cool stuff, e.g.
\Image
to create an image.The most common macros also have insane macro shortcuts to keep the syntax shorter.
The general macro syntax is described at Section 4.2. "OurBigBook Markup syntax".
Insane autolink (link text is the same as the link):
The website http://example.com is cool. See also:
\Q[http://example.com/2]
which renders as:
Exact parsing rules described at: Section 4.1.1.5. "Insane link parsing rules".The website http://example.com is cool. See also:http://example.com/2
Equivalent sane version:
The website \a[http://example.com] is cool.
\Q[\a[http://example.com/2]]
which renders as:
The website http://example.com is cool.http://example.com/2
Insane link with custom text:
The website http://example.com[example.com] is cool.
which renders as:
Equivalent sane version:The website example.com is cool.
The website \a[http://example.com][example.com] is cool.
which renders as:
If the custom text is empty, an autolink is generated. This is often useful if you want your link to be followed by punctuation:The website example.com is cool.
The website is really cool: http://example.com[].
which renders as:
This could also be achieved with the sane syntax of course, but this pattern saves a tiny bit of typing.The website is really cool: http://example.com.
Link with multiple paragraphs inside it:
\a[http://example.com][Multiple
paragraphs]
which renders as:
If
{check=0}
is given, this disables the local file existence check that is done by default for relative links.If and the following are:
check
is not given explicitly, OurBigBook does the check by default if the link is not a URL with protocol. For example, the following are not checked:http://cirosantilli.com
https://cirosantilli.com
file:///etc/fstab
ftp://cirosantilli.com
index.js
../index.js
path/to/index.js
/path/to/index.js
. Since it stats with a/
, this path has to exist relative to project toplevel directory, not relative to the current.bigb
source.//example.com/path/to/index.js
. This gets treated the same as/example.com/path/to/index.js
.
For example, it is often the case in computer programming tutorials that we want to refer to source files in the current directory, so you could have in your
and then if
README.bigb
:
See this awesome source file: \a[index.js]
index.js
does not exist in the project, compilation leads to an error.This check is important because as you start documenting several source files, it is almost inevitable getting wrong paths due to renames or typos without this type of error checking
The most common use case for disabling checks is as follows.
https://github.com/cirosantilli/cirosantilli.github.io/blob/da296c3c933704484d5cd1a42754f60ec00f672b/README.bigb was rendered at https://cirosantilli.com, and contains links to https://cirosantilli.com/markdown-style-guide, whose source lives in a separate non-OurBigBook repository: https://github.com/cirosantilli/markdown-style-guide/
Therefore, if we did from
we do not want OurBigBook do check that the file
README.bigb
:
See this awesome style guide: \a[markdown-style-guide]
markdown-style-guide
exists in the local filesystem.For this reason, we have instead to write:
See this awesome style guide: \a[markdown-style-guide]{check=0}
The general lesson is clear: in a server, paths could be re-routed to anything, including content that lies outside of a OurBigBook project.
The link target, e.g. in:
\a[http://example.com]
href
equals http://example.com
.Important behaviours associated with this property for local links:
- they are checked for existence in the local filesystem:
\a
check
argument - they are modified as per Section 4.1.1.4.1. "Relative link" to account for scopes with
-S
,--split-headers
Analogous to the
\x
ref
argument, e.g.:
Trump said this and that.https://en.wikipedia.org/wiki/Donald_Trump_Access_Hollywood_tape#Trump's_responses{ref}https://web.archive.org/web/20161007210105/https://www.donaldjtrump.com/press-releases/statement-from-donald-j.-trump{ref} Then he said that and this.https://en.wikipedia.org/wiki/Donald_Trump_Access_Hollywood_tape#Trump's_responses{ref}https://web.archive.org/web/20161007210105/https://www.donaldjtrump.com/press-releases/statement-from-donald-j.-trump{ref}
which renders as:
If given, this boolean argument forces a given link to be a relative link or not.
Otherwise, the link is automatically guessed based on the address given as explained at Section 4.1.1.4.1. "Relative link".
A relative link is a link that points to a resource that will be present a final URL relative to the input
.bigb
file.For example, it is often the case in computer programming tutorials that we want to refer to source files in the current directory.
So from our
README.bigb
, we could want to write something like:
Have a look at this amazing source file: \a[index.js].
which renders as:
and hereHave a look at this amazing source file: index.js.
\a[ourbigbook]
is a relative link.A non-relative link is something like:
This is great website: https://cirosantilli.com
which renders as:
which points to an absolute URL.This is great website: https://cirosantilli.com
A link being relative has the following effects
- the correct relative path to the file is used when using nested scopes with
-S
,--split-headers
. For example, if we have:= h1 == h2 {scope} === h3 \a[index.js]
h3
will be rendered toh2/h3.html
.Therefore, if we didn't do anything about it, the link toindex.js
would render ashref="index.js"
and thus point toh2/index.js
instead of the correctindex.js
.Instead, OurBigBook automatically converts it to the correcthref="../index.js"
OurBigBook considers a link relative by default if:
- it is not a URL with protocol
- it does not start with a slash
/
Therefore, the following are not relative links by default:and the following are:
http://cirosantilli.com
https://cirosantilli.com
file:///etc/fstab
ftp://cirosantilli.com
/path/to/index.js
//example.com/path/to/index.js
index.js
../index.js
path/to/index.js
Implemented at: https://github.com/cirosantilli/ourbigbook/issues/87.
A URL with protocol is a URL that matches the regular expression
^[a-zA-Z]+://
. The following are examples of URLs with protocol:http://cirosantilli.com
https://cirosantilli.com
file:///etc/fstab
ftp://cirosantilli.com
The following aren't:
index.js
../index.js
path/to/index.js
/path/to/index.js
//example.com/path/to/index.js
. This one is a bit tricky. Web browsers would consider this as a protocol-relative URL, which technically implies a protocol, although that protocol would be different depending how you are viewing the file, e.g. locally throughfile://
vs on a with websitehttps://
.For simplicity's sake, we just consider it as a URL without protocol.
Insane start at any of the recognized protocols are the ones shown at: Section 4.3.3. "Known URL protocols".absolutely anywhere if not escaped, e.g.:
renders something like:
To prevent expansion, you have to escape the protocol with a backslash
Empty domains like:
don't becomes links however. But this one does:
http://
https://
ahttp://example.com
a <a href="http://example.com">
\\
, e.g.:
\http://example.com
http://
http://a
Insane links end when either the end of the document or one of the following characters is found:
- space
- newline
\n
- open or close square bracket
[
or]
- open or close curly braces
{
or}
As a consequence, to have an insane link followed immediately by a punctuation like a period you should use an empty argument as in:
Check out this website: http://example.com[].
which renders as:
otherwise the punctuation will go in it. Another common use case is:Check out this website: http://example.com.
As mentioned on the tutorial (http://example.com[see this link]).
which renders as:
As mentioned on the tutorial (see this link).
If you want your link to include one of the terminating characters, e.g.
]
, all characters can be escaped with a backslash, e.g.:
Hello http://example.com/\]a\}b\\c\ d world.
which renders as:
Hello http://example.com/]a}b\c d world.
Note that the
http://example.com
inside \a[http://example.com]
only works because we do some post-processing magic that prevents its expansion, otherwise the link would expand twice:
\P[http://example.com]
\a[http://example.com]
which renders as:
This magic can be observed with
--help-macros
by seeing that the href
argument of the a
macro has the property:
"elide_link_only": true,
Some \b[bold] text.
which renders as:
Some bold text.
There is basically one application for this: poetry, which would be too ugly with code block due to fixed width font:
Paragraph 1 Line 1\br
Paragraph 1 Line 2\br
Paragraph 2 Line 1\br
Paragraph 2 Line 2\br
which renders as:
Paragraph 1 Line 1
Paragraph 1 Line 2Paragraph 2 Line 1
Paragraph 2 Line 2
Inline code (code that should appear in the middle of a paragraph rather than on its own line) is done with a single backtick (
`
) insane macro shortcut:
My inline `x = 'hello\n'` is awesome.
which renders as:
and block code (code that should appear on their own line) is done with two or more backticks (My inlinex = 'hello\n'
is awesome.
``
):
``
f() {
return 'hello\n';
}
``
which renders as:
f() { return 'hello\n'; }
The sane version of inline code is a lower case
c
:
My inline \c[[x = 'hello\n']] is awesome.
which renders as:
and the sane version of block math is with an upper caseMy inlinex = 'hello\n'
is awesome.
C
:
\C[[
f() {
return 'hello\n';
}
]]
which renders as:
f() { return 'hello\n'; }
The capital vs lower case theme is also used in other elements, see: block vs inline macros.
If the content of the sane code block has many characters that you would need to escape, you will often want to use literal arguments, which work just like the do for any other argument. For example:
\C[[[
A paragraph.
\C[[
And now, some long, long code, with lots
of chars that you would need to escape:
\ [ ] { }
]]
A paragraph.
]]]
which renders as:
Note that the initial newline is skipped automatically in code blocks, just as for any other element, due to: argument leading newline removal, so you don't have to worry about it.A paragraph. \C[[ And now, some long, long code, with lots of chars that you would need to escape: \ [ ] { } ]] A paragraph.
The distinction between inline
\c
and block \C
code blocks is needed because in HTML, pre
cannot go inside P
.We could have chosen to do some magic to differentiate between them, e.g. checking if the block is the only element in a paragraph, but we decided not to do that to keep the language saner.
And now a code block outside of
\OurBigBookExample
to test how it looks directly under the \Toplevel
implicit macro:Hello
Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello
HelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHello
Hello
And nos a very long inline code:
Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello
Block code can have a title, e.g. see this one: Code 1. "My nice code block":
``
ab
cd
``
{id=code-my-nice-code}
{title=My nice code block}
which renders as:
Code 1. My nice code block.ab cd
Shows both the OurBigBook code and its rendered output, e.g.:
\OurBigBookExample[[
Some `ineline` code.
]]
which renders as:
Some `ineline` code.
which renders as:Someineline
code.
Its input should be thought of as a literal code string, and it then injects the rendered output in the document.
This macro is used extensively in the OurBigBook documentation.
The
Comment
and comment
macros are regular macros that does not produce any output. Capitalization is explained at: Section 4.3.2. "Block vs inline macros".You will therefore mostly want to use it with a literal argument, which will, as for any other macro, ignore any macros inside of it.
Before comment.
\Comment[[
Inside comment.
]]
After comment.
which renders as:
Before comment.After comment.
And an inline one:
My inline \comment[[inside comment]] is awesome.
\comment[[inside comment]] inline at the start.
which renders as:
My inline is awesome.inline at the start.
Insane with
Insane headers end at the first newline found. They cannot therefore contain raw newline tokens.
=
(equal sign space):
= My h1
== My h2
=== My h3
Equivalent sane:
\H[1][My h1]
\H[2][My h2]
\H[3][My h3]
Custom ID for cross references on insane headers:
= My h1
{id=h1}
== My h2
{id=h2}
=== My h3
{id=h3}
Sane equivalent:
\H[1][My h1]{id=h1}
\H[2][My h2]{id=h2}
\H[3][My h3]{id=h3}
There is no limit to how many levels we can have, for either sane or insane headers!
HTML is randomly limited to
h6
, so OurBigBook just renders higher levels as an h6
with a data-level
attribute to indicate the actual level for possible CSS styling:
<h6 data-level="7">My title</h6>
The recommended style is to use insane headers up to
h6
, and then move to sane one for higher levels though, otherwise it becomes very hard to count the =
signs.To avoid this, we considered making the insane syntax be instead:
but it just didn't feel as good, and is a bit harder to type than just smashing
= 1 My h1
= 2 My h2
= 3 My h3
=
n times for lower levels, which is the most common use case. So we just copied markdown.The very first header of a document can be of any level, although we highly recommend your document to start with a
\H[1]
, and to contain exactly just one \H[1]
, as this has implications such as:\H[1]
is used for the document title: HTML document title\H[1]
does not show on the table of contents
After the initial header however, you must not skip a header level, e.g. the following would give an error because it skips level 3:
= my 1
== my 1
==== my 4
If the document has only a single header of the highest level, e.g. like the following has only a single
then this has some magical effects.
h2
:
== My 2
=== My 3 1
=== My 3 2
Header IDs won't show for the toplevel level. For example, the headers would render like:
rather than:
This is because in this case, we guess that the
My 2
1. My 3 1
2. My 3 2
1. My 2
1.2. My 3 1
1.2. My 3 2
h2
is the toplevel.TODO: we kind of wanted this to be the ID of the toplevel header instead of the first header, but this would require an extra postprocessing pass (to determine if the first header is toplevel or not), which might affect performance, so we are not doing it right now.
When the OurBigBook input comes from a file (and not e.g. stdin), the default ID of the first header in the document is derived from the basename of the OurBigBook input source file rather than from its title.
This is specially relevant when including other files.
For example, in file named
the ID of the header is
my-file.bigb
which contains:
= Awesome ourbigbook file
]]
my-file
rather than awesome-ourbigbook-file
. See also: automatic ID from title.If the file is an index file other than the toplevel index file, then the basename of the parent directory is used instead, e.g. the toplevel ID of a file:
would be:
rather than:
my-subdir/README.bigb
#my-subdir
#README.bigb
For the toplevel index file however, the ID is just taken from the header itself as usual. This is done because you often can't general control the directory name of a project.
For example, a GitHub pages root directory must be named as
<username>.github.io
. And users may need to rename directories to avoid naming conflicts.As a consequence of this, the toplevel index file cannot be included in other files.
If given, makes the header capitalized by default on cross file references.
More details at: Section 4.1.20.2. "Cross reference title inflection".
This
multiple
argument marks given IDs as being children of the current page.The effect is the same as adding the
\x
child
argument argument to an under the header. Notably, such marked target IDs will show up on the tagged autogenerated header metadata section.Example:
renders exactly as:
= Animal
== Mammal
=== Bat
=== Cat
== Wasp
== Flying animal
{child=bat}
{child=wasp}
\x[bat]
\x[wasp]
= Animal
== Mammal
=== Bat
=== Cat
== Wasp
== Flying animal
\x[bat]{child}
\x[wasp]{child}
The header
child
syntax is generally preferred because at some point while editing the content of the header, you might accidentally remove mentions to e.g. \x[bat]{child}
, and then the relationship would be lost.The
\H
tag
argument does the same as the \x
child
argument but in the opposite direction.Sometimes the short version of a name is ambiguous, and you need to add some extra text to make both its title and ID unique.
For example, the word "Python" could either refer to:
- the programming language: https://en.wikipedia.org/wiki/Python_(programming_language)
- the genus of snakes: https://en.wikipedia.org/wiki/Python_(genus)
The
disambiguate
name argument, which is automatically defined for all macros, helps you deal more neatly with such problems.Have a look at this example:
My favorite snakes are \x[python-genus]{p}!
My favorite programming language is \x[python-programming-language]!
\x[python-genus]{full}
\x[python-programming-language]{full}
= Python
{disambiguate=genus}
{parent=h-disambiguate-argument}
= Python
{c}
{disambiguate=programming language}
{parent=h-disambiguate-argument}
{title2=.py}
{wiki}
which renders as:
from which we observe howMy favorite snakes are pythons!My favorite programming language is Python!
disambiguate
:- gets added to the ID after conversion following the same rules as automatic ID from title
- shows up on the header between parenthesis, much like Wikipedia, as well as in
full
cross references - does not show up on non-
full
references. This makes it much more likely that you will be able to reuse the title automatically on a cross reference withoutcontent
: we wouldn't want to say "My favorite programming language is Python (programming language)" all the time, would we? - gets added to the default
\H
wiki
argument inside parenthesis, following Wikipedia convention, therefore increasing the likelihood that you will be able to go with the default Wikipedia value
Besides disambiguating headers, the
disambiguate
argument has a second related application: disambiguating IDs of images. For example:
\x[image-the-title-of-my-disambiguate-image]{full=0}
\x[image-the-title-of-my-disambiguate-image-2]{full=0}
\x[image-the-title-of-my-disambiguate-image]{full}
\x[image-the-title-of-my-disambiguate-image-2]{full}
\Image[Tank_man_standing_in_front_of_some_tanks.jpg]
{title=The title of my disambiguate image}
\Image[Tank_man_standing_in_front_of_some_tanks.jpg]
{title=The title of my disambiguate image}
{disambiguate=2}
which renders as:
Note that unlike for headers,![]()
Figure 1. The title of my disambiguate image. ![]()
Figure 2. The title of my disambiguate image.
disambiguate
does not appear on the title of images at all. It serves only to create an unique ID that can be later referred to. Headers are actually the only case where disambiguate
shows up on the visible rendered output. We intend on making this application obsolete however with:This use case is even more useful when
title-from-src
is enable by default for the media-providers
entry, so you don't have to repeat titles several times over and over.If given, the current section contains metadata about file or other resource with the given URL.
If empty, the URl of the file is extracted directly from the header. Otherwise, the given URL is used.
for example:
renders the same as:
so note how:
The file \x[file/path/to/myfile.c] is very useful.
= path/to/myfile.c
{file}
An explanation of what this file is about.
The file \x[file/path/to/myfile.c] is very useful.
= path/to/myfile.c
{file}
{id=file/path/to/myfile.c}
An explanation of what this file is about.
\a[path/to/myfile.c][View file]
- Also, a
file/
prefix is automatically added to the ID. This is needed with-S
,--split-headers
to avoid a collision between:path/to/myfile.c
: the actual filefile/path/to/myfile.c
: the metadata about that file. Note that locally the.html
extension is added as infile/path/to/myfile.c.html
which avoids the collision. But on a server deployment, the.html
is not present, and there would be a conflict if we didn't add thatfile/
prefix.
- a link to the is added automatically, since users won't be able to click it from the header, as clicking on the header will just link to the header itself
- a preview is added. The type of preview is chosen as follows:
In some cases however, especially when dealing with external URLs, we might want to have a more human readable title with a
which renders something like:
file
specified as in:
The video \x[tank-man-by-cnn-1989] is very useful.
= Tank Man by CNN (1989)
{file=https://www.youtube.com/watch?v=YeFzeNAHEhU}
An explanation of what this video is about.
The video \x[tank-man-by-cnn-1989] is very useful.
= Tank Man by CNN (1989)
\a[https://www.youtube.com/watch?v=YeFzeNAHEhU][View file]
An explanation of what this video is about.
\Video[https://www.youtube.com/watch?v=YeFzeNAHEhU]
An explanation of what this video is about.

An explanation of what this video is about.
This boolean argument determines whether renderings of a header will have section numbers or not. This affects all of:This option can be set by default for all files with:
- headers themselves
- table of contents links
- cross references with the
\x
full
argument
By default, headers are numbered as in a book, e.g.:
renders something like:
= h1
== h2
=== h3
==== h4
= h1
Table of contents
* 1. h2
* 1.1. h3
* 1.1.1. h4
== 1. h2
=== 1.1. h3
==== 1.1.1. h4
However, for documents with a very large number of sections, or deeply nested headers those numbers start to be more noise than anything else, especially in the table of contents and you are better off just referring to IDs. E.g. imagine:
1.3.1.4.5.1345.3.2.1. Some deep level
When documents reach this type of scope, you can disable numbering with the
numbered
option.This option can be set on any header, and it is inherited by all descendants.
The option only affects descendants.
E.g., if in the above example turn numbering off at
then it renders something like:
h2
:
= h1
== h2
{numbered=0}
=== h3
==== h4
= h1
Table of contents
* 1. h2
* h3
* h4
== 1. h2
=== h3
==== h4
The more common usage pattern to disable it on toplevel and enable it only for specific "tutorial-like sections". An example can be seen at:which is something like:
then it renders something like:
Note how in this case the number for
- https://cirosantilli.com/: huge toplevel wiki, for which we don't want numbers
- https://cirosantilli.com/x86-paging: a specific tutorial, for which we want numbers
= Huge toplevel wiki
{numbered=0}
== h2
=== A specific tutorial
{numbered}
{scope}
==== h4
===== h5
= Huge toplevel wiki
Table of contents
* h2
* A specific tutorial
* 1. h4
* 1.1. h5
== h2
=== A specific tutorial
==== 1. h4
===== 1.1. h5
h4
is just 1.
rather than 1.1.1.
. We only show numberings relative to the first non-numbered header, because the 1.1.
wouldn't be very meaningful otherwise.In addition to the basic way of specifying header levels with an explicit level number as mentioned at Section 4.1.7. "Header (
\H
)", OurBigBook also supports a more indirect ID-based mechanism with the parent
argument of the \H
element.We hightly recommend using
parent
for all but the most trivial documents.For example, the following fixed level syntax:
is equivalent to the following ID-based version:
= My h1
== My h2 1
== My h2 2
=== My h3 2 1
= My h1
= My h2 1
{parent=my-h1}
= My h2 2
{parent=my-h1}
= My h3 2 1
{parent=my-h2-h}
The main advantages of this syntax are felt when you have a huge document with very large header depths. In that case:
- it becomes easy to get levels wrong with so many large level numbers to deal with. It is much harder to get an ID wrong.
- when you want to move headers around to improve organization, things are quite painful without a refactoring tool (which we intend to provide in the browser editor with preview), as you need to fix up the levels of every single header.If you are using the ID-based syntax however, you only have to move the chunk of headers, and change the
parent
argument of a single top-level header being moved.
Note that when the
because the second header has level
parent=
argument is given, the header level must be 1
, otherwise OurBigBook assumes that something is weird and gives an error. E.g. the following gives an error:
= My h1
== My h2
{parent=my-h1}
2
instead of the required = My h2
.When scopes are involved, the rules are the same as those of internal reference resolution, including the leading
/
to break out of the scope in case of conflicts.Like the
which is equivalent to:
\H
child
argument, parent
also performs ID target from title on the argument, allowing you to use the original spaces and capitalization in the target as in:
= Flying animal
= Bat
{parent=Flying animal}
= Flying animal
= Bat
{parent=flying-animal}
See also: Section 4.1.7.4.6.2. "Header explicit levels vs nesting design choice" for further rationale.
When mixing both
\H
parent
argument and scopes, things get a bit complicated, because when writing or parsing, we have to first determine the parent header before resolving scopes.As a result, the follow simple rules are used:
- start from the last header of the highest level
- check if the
{parent=XXX}
is a suffix of its ID - if not, proceed to the next smaller level, and so on, until a suffix is found
Following those rules for example, a file
will lead to the following header tree with
tmp.bigb
:
= h1
{scope}
= h1 1
{parent=h1}
{scope}
= h1 1 1
{parent=h1-1}
= h1 1 2
{parent=h1-1}
= h1 1 3
{parent=h1/h1-1}
= h1 2
{parent=h1}
{scope}
= h1 2 1
{parent=h1-2}
{scope}
= h1 2 1 1
{parent=h1-2/h1-2-1}
--log headers
:
= h1 tmp
== h2 1 tmp/h1-1
=== h3 1.1 tmp/h1-1/h1-1-1
=== h3 1.2 tmp/h1-1/h1-1-2
=== h3 1.3 tmp/h1-1/h1-1-3
== h2 2 tmp/h1-2
=== h3 2.1 tmp/h1-2/h1-2-1
==== h4 2.1.1 tmp/h1-2/h1-2-1/h1-2-1-1
Arguably, the language would be even saner if we did:
rather than having explicit levels as in
\H[My h1][
Paragraph.
\H[My h2][]
]
\H[1][My h1]
and so on.But we chose not to do it like most markups available because it leads to too many nesting levels, and hard to determine where you are without tooling.
Ciro later "invented" (?)
\H
parent
argument, which he feels reaches the perfect balance between the advantages of those two options.In some use cases, the sections under a section describe inseparable parts of something.
For example, when documenting an experiment you executed, you will generally want an "Introduction", then a "Materials" section, and then a "Results" section for every experiment.
On their own, those sections don't make much sense: they are always referred to in the context of the given experiment.
The problem is then how to get unique IDs for those sections.
One solution, would be to manually add the experiment ID as prefix to every subsection, as in:
= Experiments
See: \x[full-and-unique-experiment-name/materials]
== Introduction
== Full and unique experiment name
=== Introduction
{id=full-and-unique-experiment-name/introduction}
See our awesome results: \x[full-and-unique-experiment-name/results]
For a more general introduction to all experiments, see: \x[introduction].
=== Materials
{id=full-and-unique-experiment-name/materials}
=== Results
{id=full-and-unique-experiment-name/results}
but this would be very tedious.
To keep those IDs shorter, OurBigBook provides the
scope
boolean argument property of headers, which works analogously to C++ namespaces with the header IDs.Using
scope
, the previous example could be written more succinctly as:
= Experiments
See: \x[full-and-unique-experiment-name/materials]
== Introduction
== Full and unique experiment name
{scope}
=== Introduction
See our awesome results: \x[results]
For a more general introduction to all experiments, see: \x[/introduction].
=== Materials
=== Results
Note how:
- full IDs are automatically prefixed by the parent scopes prefixed and joined with a slash
/
- we can refer to other IDs withing the current scope without duplicating the scope. E.g.
\x[results]
in the example already refers to the IDfull-and-unique-experiment-name/materials
- to refer to an ID outside of the scope and avoid name conflicts with IDs inside of the current scope, we start a reference with a slash
/
So in the example above,\x[/introduction]
refers to the IDintroduction
, and notfull-and-unique-experiment-name/introduction
.
When nested scopes are involved, cross references resolution peels off the scopes one by one trying to find the closes match, e.g. the following works as expected:
Here OurBigBook:
= h1
{scope}
== h2
{scope}
=== h3
{scope}
\x[h2]
- first tries to loop for an
h1/h2/h3/h2
, sinceh1/h2/h3
is the current scope, but that ID does not exist - so it removes the
h3
from the current scope, and looks forh1/h2/h2
, which is still not found - then it removes the
h2
, leading toh1/h2
, and that one is found, and therefore is taken
Putting files in subdirectories of the build has the same effect as adding a scope to their top level header.
Notably, all headers inside that directory get the directory prepended to their IDs.
The toplevel directory is determined as described at: the toplevel index file.
For fun and profit.
Let's break this local link: ourbigbook.
When the toplevel header is given the
scope
property OurBigBook automatically uses the file path for the scope and heaves fragments untouched.For example, suppose that file
full-and-unique-experiment-name
contains:
= Full and unique experiment name
{scope}
== Introduction
== Materials
In this case, multi-file output will generate a file called instead of
full-and-unique-experiment-name.html
, and the URL of the subsections will be just:full-and-unique-experiment-name.html#introduction
full-and-unique-experiment-name.html#materials
full-and-unique-experiment-name.html#full-and-unique-experiment-name/introduction
full-and-unique-experiment-name.html#full-and-unique-experiment-name/materials
Some quick interactive cross file link tests:
When using
-S
, --split-headers
, cross references always point to non-split pages as mentioned at cross reference targets in split headers.If the
splitDefault
boolean argument is given however:- the split header becomes the default, e.g.
index.html
is now the split one, andnosplit.html
is the non-split one - the header it is given for, and all of its descendant headers will use the split header as the default internal cross target, unless the header is already rendered in the current page. This does not propagate across includes however.
For example, consider
and
Then the following links would be generated:
README.bigb
:
= Toplevel
{splitDefault}
\x[h2][toplevel to h2]
\x[notreadme][toplevel to notreadme]
\Include[notreadme]
== h2
notreadme.bigb
:
= Notreadme
\x[h2][notreadme to h2]
\x[notreadme][notreadme to notreadme h2]
== Notreadme h2
index.html
: split version ofREADME.bigb
, i.e. does not containh2
toplevel to h2
:h2.html
. Links to the split version ofh2
, sinceh2
is also affected by thesplitDefault
of its parent, and therefore links to it use the split version by defaulttoplevel to notreadme
:notreadme.html
. Links to non-split version ofnotreadme.html
since that header is notsplitDefault
, becausesplitDefault
does not propagate across includes
nosplit.html
non-split version ofREADME.bigb
, i.e. containsh2
toplevel to h2
:#h2
, because even thoughh2
issplitDefault
, that header is already present in the current page, so it would be pointless to reload the split onetoplevel to notreadme
:notreadme.html
h2.html
split version ofh2
fromREADME.bigb
notreadme.html
: non-split version ofnotreadme.bigb
notreadme to h2
:h2.html
, becauseh2
issplitDefault
notreadme to notreadme h2
:#notreadme-h2
notreadme-split.html
: split version ofnotreadme.bigb
notreadme to h2
:h2.html
, becauseh2
issplitDefault
notreadme to notreadme h2
:notreadme.html#notreadme-h2
, becausenotreadme-h2
is notsplitDefault
The major application of this is that Ciro likes to work with a huge
README.bigb
containing thousands of random small topics. This is the case for example for: https://cirosantilli.comAnd splitting those into separate source files would be quite laborious, as it would require duplicating IDs on the filename, and setting up includes.
And even worse: this README corresponds to the main index page of the website, which will make what a large number of users will see be that slowness.
Therefore, once this README reaches a certain size, you can add the
splitDefault
attribute to it, to make things smoother for readers.And if you have a smaller, more self-contained, and highly valuable tutorial such as https://cirosantilli.com/x86-paging, you can just split that into a separate
.bigb
source file.This way, any links into the smaller tutorial will show the entire page as generally desired.
And any links from the tutorial, back to the main massive README will link back to split versions, leading to fast loads.
This feature was implemented at: https://github.com/cirosantilli/ourbigbook/issues/131
If given, add a custom suffix to the output filename of the header when using
-S
, --split-headers
.If the given suffix is empty, it defaults to
-split
.For example, given:
a
However, if we instead wrote:
it would not be placed under:
and if we set a custom one as:
it would go instead to:
= my h1
== my h2
--split-headers
conversion would normally place my h2
into a file called:
my-h2.html
== my h2
{splitSuffix}
my-h2-split.html
== my h2
{splitSuffix=asdf}
my-h2-asdf.html
This option is useful if the root of your website is written in OurBigBook, and you want to both:
- have a section that talks about some other project
- host the documentation of that project inside the project source tree
For example, https://cirosantilli.com with source at https://github.com/cirosantilli/cirosantilli.github.io has a quick section about OurBigBook: https://cirosantilli.com#ourbigbook.
Therefore, without a custom suffix, the split header version of that header would go to https://docs.ourbigbook.com, which would collide with this documentation, that is present in a separate repository: https://github.com/cirosantilli/ourbigbook.
Therefore a
splitSuffix
property is used, making the split header version fall under /ourbigbook-split
, and leaving the nicer /ourbigbook
for the more important project toplevel.If given on the the toplevel headers, which normally gets a suffix by default to differentiate from the non-split version, it replaces the default
-split
suffix with a custom one.For example if you had
then it would render to:
but if you used instead:
then it would instead be:
notindex.bigb
as:
= Not index
notindex-split.bigb
= Not index
{splitSuffix=asdf}
notindex-asdf.bigb
This option is similar to
\H
title2
argument but it additionally:- creates a new ID that you can refer to, and renders it with the alternate chosen title
- the rendered ID on cross references is the same as what it is a synonym for
- the synonym header is not rendered at all, including in the table of contents
- when using
-S
,--split-headers
, a redirect output file is generated from the synonym to the main ID
Example:
renders something like:
Furthermore, if
which contains a redirection from
= Parent
== GNU Debugger
{c}
= GDB
{c}
{synonym}
I like to say \x[gdb] because it is shorter than \x[gnu-debugger].
= GNU Debugger
I like to say \a[#gnu-debugger][GDB] because it is shorter than \x[#gnu-debugger][GNU Debugger].
-S
, --split-headers
is used, another file is generated:
gdb.html
gdb.html
to gnu-debugger.html
.Implemented at: https://github.com/cirosantilli/ourbigbook/issues/114
Contains the main content of the header. The insane syntax:
is equivalent to the sane:
and in both cases
= My title
\H[1][My title]
My title
is the title argument.The title argument is also notably used for automatic ID from title.
If a non-toplevel macro has the Note how those rules leave non-ASCII Unicode characters untouched, except for:as capitalization and determining if something "is a letter or not" in those cases can be tricky.
title
property is present but no explicit id
, an ID is created automatically from the title
, by applying the following transformations:- do a
id
output format conversion on the title to remove for example any HTML tags that would be present in the conversion output - convert all characters to lowercase. This uses JavaScript case conversion. Note that this does convert non-ASCII characters to lowercase, e.g.
É
toé
. - if
id
normalize
latin
istrue
(the default) do Latin normalization. This converts e.g.é
toe
. - if
id
normalize
punctuation
istrue
(the default) do Punctuation normalization. This converts e.g.+
toplus
. - convert consecutive sequences of all non
a-z0-9
ASCII characters to a single hyphen-
. Note that this leaves non-ASCII characters untouched. - strip leading or trailing hyphens
- capitalization changes wher applicable, e.g.
É
toé
For toplevel headers, see: the ID of the first header is derived from the filename.
So for example, the following automatic IDs would be generated: Table 1. "Examples of automatically generated IDs".
Table 1. Examples of automatically generated IDs.
title | id | latin normalization | punctuation normalization | comments |
---|---|---|---|---|
My favorite title | my-favorite-title | |||
Ciro's markdown is awesome | ciro-s-markdown-is-awesome | ' is an ASCII character, but it is not in a-z0-9 , therefore it gets converted to a hyphen - |
||
É你 | e你 | true | The Latin acute accented e , É , is converted to its lower case form é as per the JavaScript case conversion.The Chinese character 你 is left untouched as Chinese characters have no case, and no ASCII analogue. |
|
É你 | é你 | false | Same as the previous, but é is not converted to e since Latin normalization is turned off. |
|
C++ is great | c-plus-plus-is-great | true | This is the effect of Punctuation normalization. | |
I love dogs. | i-love-dogs | love is extracted from the italic tags <i>love</i> with id output format conversion.
|
For the toplevel header, its ID is derived from the basename of the OurBigBook file without extension instead of from the
title
argument.This conversion type is similar to Automatic ID from title, but it is used in certain cases where we are targeting IDs rather than setting them, notably:
Unlike
which renders something like:
Note how we added the synonym to the title only when it is not just a simple flexion variant, since
\H
title2
argument, the synonym does not show up by default next to the title. This is because we sometimes want that, and sometimes not. To make the title appear, you can simply add an empty title2
argument to the synonym header as in:
= GNU Debugger
{c}
= GDB
{c}
{synonym}
{title2}
= Quantum computing
= Quantum computer
{synonym}
= GNU Debugger (GDB)
= Quantum computing
Quantum computing (Quantum computer)
would be kind of useless would be kind of useless.Same as
is equivalent in every way to:
\x
child
argument but in the opposite direction, e.g.:
== Mammal
=== Bat
{tag=flying-animal}
=== Cat
== Flying animal
== Mammal
=== Bat
=== Cat
== Flying animal
{child=bat}
Naming rationale:So
parent
as the opposite of child is already taken to be then "main parent" via the "\H
parent
argument"- we could have renamed the
\H
child
argument totags
as in "this header tags that one", but it would be a bit confusingtags
vstag
child
vs tag
it is for now.You generally want to use
tag
instead of the \H
child
argument because otherwise some very large header categories are going to contain Huge lists of children, which is not very nice when editing.It is possible to enforce the
\H
child
argument or the \H
tag
argument in a given project with the lint
h-tag
option.The
title2
argument can be given to any element that has the title
argument.Its usage is a bit like the
description=
argument of images, allowing you to add some extra content to the header without affecting its ID.Unlike
description=
however, title2
shows up on all full
references, including appearances in the table of contents, which make it more searchable.Its primary use cases are:
- give acronyms, or other short names names of fuller titles such as mathematical/programming notationOne primary reason to not use the acronyms as the main section name is to avoid possible ID ambiguities with other acronyms.
- give the header in different languages
For example, given the OurBigBook input:
the rendered output looks like:
= Toplevel
The Toc follows:
== North Atlantic Treaty Organization
{c}
{title2=NATO}
\x[north-atlantic-treaty-organization]
\x[north-atlantic-treaty-organization]{full}
= Toplevel
The ToC follows:
* North Atlantic Treaty Organization (NATO)
== North Atlantic Treaty Organization (NATO)
North Atlantic Treaty Organization
Section 1. "North Atlantic Treaty Organization (NATO)"
Related alternatives to
title2
include:\H
disambiguate
argument when you do want to affect the ID to remove ambiguities\H
synonym
argument
Parenthesis are added automatically around all rendered
title2
.The
title2
argument has a special meaning when applied to a header with the \H
synonym
argument, see \H
title2
argument of a synonym header.If given, show a link to the Wikipedia article that corresponds to the header.
If a value is not given, automatically link to the Wiki page that matches the header exactly with spaces converted to underscores.
Here is an example with an explicit wiki argument:
==== Tiananmen Square
{wiki=Tiananmen_Square}
which looks like:
or equivalently with the value deduced from the title:
= Tiananmen Square
{wiki}
which looks like:
You can only link to subsections of wiki pages with explicit links as in:
= History of Tiananmen Square
{wiki=Tiananmen_Square#History}
which looks like:
OurBigBook adds some header metadata to the toplevel header at the bottom of each page. this section describes this metadata.
Although the table of contents has a macro to specify its placement, it is also automatically placed at the bottom of the page, and could be considered a header metadata section.
Lists other sections that link to the current section.
E.g. in:
the page since those pages link to the
= tmp
== tmp 1
=== tmp 1 1
=== tmp 1 2
\x[tmp-1]
== tmp 2
\x[tmp-1]
tmp-1.html
would contain a list of incoming links as:tmp-1-2
tmp-2
tmp-1
ID.Lists sections that are secondary children of the current section, i.e. tagged under the current section.
The main header tree hierarchy descendants already show under the table of contents instead.
E.g. in:
the tagged sections for:
= tmp
== Mammal
== Flying
== Animal
=== Bat
{tag=mammal}
{tag=flying}
=== Bee
{tag=flying}
=== Dog
{tag=mammal}
- Mammal will contain Bat and Dog
- Flying will contain Bat and Bee
Shows a list of ancestors of the page. E.g. in:
the ancestor lists would be for:so we see that this basically provides a type of breadcrumb navigation.
= Asia
== China
=== Beijing
==== Tiananmen Square
=== Hong Kong
- Hong Kong: China, Asia
- Tiananmen Square: Beijing, China, Asia
- Beijing: China, Asia
- China: Asia
A block image with capital 'i'
For further discussion on the effects of ID see: Section 4.1.8.2. "Image ID".
Image
showcasing most of the image properties Figure 3. "The title of my image".
Have a look at this amazing image: \x[image-my-test-image].
\Image[Tank_man_standing_in_front_of_some_tanks.jpg]
{title=The title of my image}
{id=image-my-test-image}
{width=600}
{height=200}
{source=https://en.wikipedia.org/wiki/File:Tianasquare.jpg}
{description=The description of my image.}
which renders as:
This exemplifies the following parameters:Have a look at this amazing image: Figure 3. "The title of my image".![]()
Figure 3. The title of my image. Source. The description of my image.
title
: analogous to the\H
title
argument. Shows up preeminently, and sets a default ID if one is not given. It is recommended that you don't add a period.
to it, as that would show in cross referencesdescription
: similar totitle
, but allow for further explanations without them appearing in cross references to the imagesource
: a standardized way to credit an image by linking to a URL that contains further image metadata
And this is how you make an inline image inline one with lower case
i
:
My inline \image[Tank_man_standing_in_front_of_some_tanks.jpg][test image] is awesome.
which renders as:
Inline images can't have captions.My inlineis awesome.
And now for an image outside of
\OurBigBookExample
to test how it looks directly under the \Toplevel
implicit macro: Figure 4.By default, we fix image heights to
height=315
, and let the width
be calculated proportionally once the image loads. We therefore ignore the actual image size. This is done to:- prevent reflows as the page loads images and can determine their actual sizes, especially is the user opens the page at a given ID in the middle of the page
- create a more uniform media experience by default, unless a custom image size is actually needed e.g. if the image needs to be larger
Here is an image without a description but with an ID so we can link to it: Figure 5.
Have a look at this amazing image: \x[image-my-test-image-2].
\Image[Tank_man_standing_in_front_of_some_tanks.jpg]
{id=image-my-test-image-2}
which renders as:
This works becauseHave a look at this amazing image: Figure 5.![]()
Figure 5.
full
is the default cross reference style for Image
, otherwise the link text would be empty since there is no title
, and OurBigBook would raise an error.OurBigBook can optionally deduce the title from the basename of the
src
argument if the titleFromSrc
boolean argument is given, or if title-from-src
is set as the default media provider for the media type:
Have a look at this amazing image: \x[image-tank-man-standing-in-front-of-some-tanks].
\Image[Tank_man_standing_in_front_of_some_tanks.jpg]
{titleFromSrc}
which renders as:
Have a look at this amazing image: Figure 6. "Tank man standing in front of some tanks.".![]()
Figure 6. Tank man standing in front of some tanks.
If the image has neither ID nor title nor description nor
source
, then it does not get a caption at all:
\Image[Tank_man_standing_in_front_of_some_tanks.jpg]
which renders as:
If the image does not have an ID nor title, then it gets an automatically generated ID, just like every other OurBigBook output HTML element, and it is possible for readers to link to that ID on the rendered version, e.g. as:
Note that the
#_123
123
is not linked to the Figure <number>.
, but just a sequential ID that runs over all elements.This type of ID is of course not stable across document revisions however, since if an image is added before that one, the link will break. So give an ID or title for anything that you expect users to link to.
Also, it is not possible to link to such images with an cross reference, like any other OurBigBook element with autogenerated temporary IDs.
Another issue to consider is that in paged output formats like PDF, the image could float away from the text that refers to the image, so you basically always want to refer to image by ID, and not just by saying "the following image".
We can also see that such an image does not increment the Figure count:
\Image[Tank_man_standing_in_front_of_some_tanks.jpg]{id=image-my-test-image-count-before}
\Image[Tank_man_standing_in_front_of_some_tanks.jpg]
\Image[Tank_man_standing_in_front_of_some_tanks.jpg]{id=image-my-test-image-count-after}
which renders as:
If the image has any visible metadata such as
source
or description
however, then the caption does show and the Figure count gets incremented:
\Image[Tank_man_standing_in_front_of_some_tanks.jpg]{source=https://en.wikipedia.org/wiki/File:Tianasquare.jpg}
\Image[Tank_man_standing_in_front_of_some_tanks.jpg]{description=This is the description of my image.}
which renders as:
If you are making a limited repository that will not have a ton of images, then you can get away with simply git tracking your images in the main repository.
However, if you are making a huge tutorial, which can have a huge undefined number of images (i.e. any scientific book), then you likely don't want to git track your images in the git repository.
A generally better alternative is to store images in a separate media repository, and especially store images in a separate media repository and track it as a git submodule.
In this approach, you create a separate GitHub repository in addition to the main one containing the text to contain only media such as images.
This approach is more suitable than store images inside the repository itself if you are going to have a lot of images.
When using this approach, you could of course just point directly to the final image URL, e.g. as in:
\Image[https://raw.githubusercontent.com/cirosantilli/media/master/Chrysanthemum_Xi_Jinping_with_black_red_liusi_added_by_Ciro_Santilli.jpg]
which renders as:
but OurBigBook allows you use configurations that allow you to enter just the image basename:![]()
Chrysanthemum_Xi_Jinping_with_black_red_liusi_added_by_Ciro_Santilli.jpg
which we will cover next.In order to get this to work, the recommended repository setup is:The directory and repository names are not mandatory, but if you place media in
./main-repo/.git
: main repository at https://github.com/username/main-repo./main-repo/data/media/.git/
: media repository at https://github.com/username/main-repo-media, and wheredata/
is gitignored.
data/media
and name its repository by adding the *-media
suffix, then ourbigbook
will handle everything for you without any further configuration in media-providers
.This particular documentation repository does have a different setup as can be seen from its ourbigbook.json. Then, when everything is setup correctly, we can refer to images simply as:
\Image[Chrysanthemum_Xi_Jinping_with_black_red_liusi_added_by_Ciro_Santilli.jpg]{provider=github}
which renders as:
In this example, we also needed to set![]()
{provider=github}
explicitly since it was not set as the default image provider in our ourbigbook.json
. In most projects however, all of your images will be in the default repository, so this won't be needed.provider
must not be given when a full URL is given because we automatically detect providers from URLs, e.g.:
\Image[https://raw.githubusercontent.com/cirosantilli/media/master/Chrysanthemum_Xi_Jinping_with_black_red_liusi_added_by_Ciro_Santilli.jpg]{provider=github}
TODO implement:
ourbigbook
will even automatically add and push used images in the my-tutorial-media
repository for you during publishing!You should then use the following rules inside This way, even though the repositories are not fully in sync, anyone who clones the latest version of the
my-tutorial-media
:- give every file a very descriptive and unique name as a full English sentence
- never ever delete any files, nor change their content, unless it is an improvement in format that does change the information contained of the image TODO link to nice Wikimedia Commons guideline page
*-media
directory will be able to view any version of the main repository.Then, if one day the media repository ever blows up GitHub's limit, you can just migrate the images to another image server that allows arbitrary basenames, e.g. AWS, and just configure your project to use that new media base URL with the
media-providers
option.The reason why images should be kept in a separate repository is that images are hundreds or thousands of times larger than hand written text.
Therefore, images could easily fill up the maximum repository size you are allowed: https://webapps.stackexchange.com/questions/45254/file-size-and-storage-limits-on-github#84746 and then what will you do when GitHub comes asking you to reduce the repository size?
Git LFS is one approach to deal with this, but we feel that it adds too much development overhead.
This is likely the sanest approach possible, as it clearly specifies which media version matches which repository version through the submodule link.
Furthermore, it is possible to make the submodule clone completely optional by setting things up as follows. For your OurBigBook project
yourname/myproject
create a yourname/myproject-media
with the media, and track it as a submodule under yourname/myproject/media
.Then, add to
media-providers
:
"media-providers": {
"github": {
"default-for": ["image", "video"],
"path": "media",
"remote": "yourname/myproject-media"
}
}
Now, as mentioned at
media-providers
, everything will work beautifully:ourbigbook .
local conversion will use images frommedia/
if it exists, e.g.:\Image[myimage.jpg]
media/myimage.jpg
. So after cloning the submodule, you will be able to see the images on the rendered pages without an internet connection.But if the submodule is not cloned, not problem, renders will detect that and automatically use GitHub images.Then, when you do:ourbigbook --publish
\Image[myimage.jpg]
uses the GitHub URL- automatically push
media/
to GitHub in case there were any updates - also, that directory is automatically gitignore, so it won't be pushed as part of the main render and thus duplicate things
Wikimedia Commons is another great possibility to upload your images to:
\Image[https://upload.wikimedia.org/wikipedia/commons/thumb/5/5b/Gel_electrophoresis_insert_comb.jpg/450px-Gel_electrophoresis_insert_comb.jpg]
{source=https://commons.wikimedia.org/wiki/File:Gel_electrophoresis_insert_comb.jpg}
which renders as:
![]()
Figure 11. . Source.
OurBigBook likes Wikimedia Commons so much that we automatically parse the image URL and if it is from Wikimedia Commons, automatically deduce the
source
for you. So the above image renders the same without the source
argument:
\Image[https://upload.wikimedia.org/wikipedia/commons/5/5b/Gel_electrophoresis_insert_comb.jpg]
which renders as:
![]()
Figure 12. . Source.
And like for non-Wikimedia images, you can automatically generate a
title
from the src
by setting the titleFromSrc
boolean argument or if title-from-src
is set as the default media provider for the media type:
\Image[https://upload.wikimedia.org/wikipedia/commons/5/5b/Gel_electrophoresis_insert_comb.jpg]
{titleFromSrc}
which renders as:
![]()
Figure 13. Gel electrophoresis insert comb. Source.
If you really absolutely want to turn off the
source
, you can explicitly set:
\Image[https://upload.wikimedia.org/wikipedia/commons/5/5b/Gel_electrophoresis_insert_comb.jpg]
{source=}
which renders as:
but you don't want to do that for the most commonly Wikimedia Commons used license of CC BY+, do you? :-)![]()
Upsides of using Wikimedia Commons for your images:Downsides:
- makes it easier for other writers to find and reuse your images
- automatically generates resized versions of the uploaded images into several common dimensions so you can pick the smallest one that fits your desired image height to reduce bandwidth usage
- if you have so many images that they would blow even the size of a separate media repository, this will still work
- forces you to use the Creative Commons license
- requires the content to be educational in nature
- uploading a bunch of images to Wikimedia Commons does feel a bit more laborious than it should because you have to write down so much repeated metadata for them
We do this by default because OurBigBook is meant to allow producing huge single page documents like Ciro likes it, and in this way:
- images that the user is looking at will load first
- we save a lot of bandwidth for the user who only wants to browse one section
TODO: maybe create a mechanism to disable this for the entire build with
ourbigbook.json
.For the love of God, there is no standardized for SVG to set its background color without a rectangle? https://stackoverflow.com/questions/11293026/default-background-color-of-svg-root-element
viewport-fill
was just left in limbo?And as a result, many many many SVG online images that you might want to reuse just rely on white pages and don't add that background rectangle.
Therefore for now we just force white background on our default CSS, which is what most SVGs will work with. Otherwise, you can lose the entire image to our default black background.
Then if someone ever has an SVG that needs another background color, we can add an image attribute to set that color as a local style.
TODO implement: mechanism where you enter a textual description of the image inside the code body, and it then converts to an image, adds to the
-media
repo and pushes all automatically. Start with dot.Analogous to the
\a
check
argument when checking if the image src
argument exists or not.The
description
argument similar to the image title
argument argument, but allows allowing longer explanations without them appearing in cross references to the image.For example, consider:
See this image: \x[image-description-argument-test-1].
\Image[Tank_man_standing_in_front_of_some_tanks.jpg]
{title=Tank man standing in front of some tanks}
{id=image-description-argument-test-1}
{description=Note how the tanks are green.}
{source=https://en.wikipedia.org/wiki/File:Tianasquare.jpg}
which renders as:
In this example, the referenceSee this image: Figure 15. "Tank man standing in front of some tanks".![]()
Figure 15. Tank man standing in front of some tanks. Source. Note how the tanks are green.
\x[image-description-argument-test-1]
expands just to
Tank man standing in front of some tanksand does not include the description, which only shows on the image.
The description can be as large as you like. If it gets really large however, you might want to consider moving the image to its own header to keep things slightly saner. This will be especially true after we eventually do: https://github.com/cirosantilli/ourbigbook/issues/180.
If the description contains any element that would take its own separate line, like multiple paragraphs or a list, we automatically add a line grouping the description with the corresponding image to make that clearer, otherwise it can be hard to know which title corresponds to a far away image. Example with multiple paragraphs:
Stuff before the image.
\Image[Tank_man_standing_in_front_of_some_tanks.jpg]
{title=Tank man standing in front of some tanks}
{id=image-description-argument-test-2}
{source=https://en.wikipedia.org/wiki/File:Tianasquare.jpg}
{description=Note how the tanks are green.
But the shirt is white.}
Stuff after the image description.
which renders as:
Stuff before the image.![]()
Figure 16. Tank man standing in front of some tanks. Source. Note how the tanks are green.But the shirt is white.Stuff after the image description.
We recommend adding a period or other punctuation to the end of every description.
The address of the image, e.g. in:
the
\Image[image.png]
src
is image.png
.Analogous to the
\a
href
argument.Analogous to the
\H
title
argument.The
\Include
macro allows including an external OurBigBook headers under the current header.It exists to allow optional single page HTML output while still retaining the ability to:
- split up large input files into multiple files to make renders faster during document development
- suggest an optional custom output split with one HTML output per OurBigBook input, in order to avoid extremely large HTML pages which could be slow to load
\Include
takes one mandatory argument: the ID of the section to be included, much like cross references.There is however one restriction: only the toplevel headers can be pointed to. This restriction allows us to easily find the included file in the filesystem, and dispenses the need to do a first
./ourbigbook
run to generate the ID database. This works because the ID of the first header is derived from the filename.Headers of the included document are automatically shifted to match the level of the child of the level where they are being included.
If
--embed-includes
is given, the external document is rendered embedded into the current document directly, essentially as if the source had been copy pasted (except for small corrections such as the header offsets).Otherwise, the following effects happen:
- The headers of the included tree appear in the table of contents of the document as links to the corresponding external files.This is implemented simply by reading a previously generated database file much like cross file reference internals, which avoids the slowdown of parsing all included files every time.As a result, you have to do an initial parse of all files in the project to extract their headers however, just as you would need to do when linking to those headers.
- the include itself renders as a link to the included document
--embed-includes
Here is an example of inclusion of the files
The above is the recommended and slightly insaner version of:
The insaner version is a bit insaner because the
not-readme.bigb
and not-readme-2.bigb
:
\Include[not-readme]
\Include[not-readme-2]
\Include[not-readme-with-scope]
\Include[not-readme]
\Include[not-readme-2]
\Include[not-readme-with-scope]
\Include
magically discards the following newline node that follows it if it just a plaintext node containing exactly a newline. With a double newline, the newline would already have been previously taken out on the lexing stage as part of a paragraph.Section 4.1.9.3. "
\Include
example" shows what those actually render like.When you are in a subdirectory, include resolution just is simply relative to the subdirectory. E.g. we could do:
subdir/index.bigb
= Subdir
\Include[notindex]
\Include[subdir2/notindex]
subdir/notindex.bigb
= Notindex
subdir/subdir2/notindex.bigb
= Notindex
It is not currently possible to include from ancestor directories: https://github.com/cirosantilli/ourbigbook/issues/214.
This option is analogous to
\H
parent
argument, but for includes.For example, consider you have:
and now you want to split
= Animal
== Dog
== Cat
== Bat
Cat
to cat.bigb
.If you wrote:
Cat would be a child of Dog, since that is the previous header, which is not what we want.
= Animal
== Dog
\Include[cat]
== Bat
Instead, we want to write:
and now Cat will be a child of Animal as desired.
= Animal
== Dog
\Include[cat]{parent=animal}
== Bat
Implemented at: https://github.com/cirosantilli/ourbigbook/issues/127
This shows what includes render as.
Some \i[italic] text.
which renders as:
Some italic text.
The
JsCanvasDemo
macro allows you to create interactive HTML/JavaScript canvas demos easily.These demos:so you can stuff as many of them as you want on a page, and they won't cause the reader's CPU to fry an egg.
- only start running when the user scrolls over them for the first time
- stop automatically when they leave the viewport
\JsCanvasDemo[[
new class extends OurbigbookCanvasDemo {
init() {
super.init('hello');
this.pixel_size_input = this.addInputAfterEnable(
'Pixel size',
{
'min': 1,
'type': 'number',
'value': 1,
}
);
}
draw() {
var pixel_size = parseInt(this.pixel_size_input.value);
for (var x = 0; x < this.width; x += pixel_size) {
for (var y = 0; y < this.height; y += pixel_size) {
var b = ((1.0 + Math.sin(this.time * Math.PI / 16)) / 2.0);
this.ctx.fillStyle =
'rgba(' +
(x / this.width) * 255 + ',' +
(y / this.height) * 255 + ',' +
b * 255 +
',255)'
;
this.ctx.fillRect(x, y, pixel_size, pixel_size);
}
}
}
}
]]
which renders as:
new class extends OurbigbookCanvasDemo { init() { super.init('hello'); this.pixel_size_input = this.addInputAfterEnable( 'Pixel size', { 'min': 1, 'type': 'number', 'value': 1, } ); } draw() { var pixel_size = parseInt(this.pixel_size_input.value); for (var x = 0; x < this.width; x += pixel_size) { for (var y = 0; y < this.height; y += pixel_size) { var b = ((1.0 + Math.sin(this.time * Math.PI / 16)) / 2.0); this.ctx.fillStyle = 'rgba(' + (x / this.width) * 255 + ',' + (y / this.height) * 255 + ',' + b * 255 + ',255)' ; this.ctx.fillRect(x, y, pixel_size, pixel_size); } } } }
And another one showing off some WebGL:
new class extends OurbigbookCanvasDemo {
init() {
super.init('webgl', {context_type: 'webgl'});
this.ctx.viewport(0, 0, this.ctx.drawingBufferWidth, this.ctx.drawingBufferHeight);
this.ctx.clearColor(0.0, 0.0, 0.0, 1.0);
this.vertexShaderSource = `
#version 100
precision highp float;
attribute float position;
void main() {
gl_Position = vec4(position, 0.0, 0.0, 1.0);
gl_PointSize = 64.0;
}
`;
this.fragmentShaderSource = `
#version 100
precision mediump float;
void main() {
gl_FragColor = vec4(0.18, 0.0, 0.34, 1.0);
}
`;
this.vertexShader = this.ctx.createShader(this.ctx.VERTEX_SHADER);
this.ctx.shaderSource(this.vertexShader, this.vertexShaderSource);
this.ctx.compileShader(this.vertexShader);
this.fragmentShader = this.ctx.createShader(this.ctx.FRAGMENT_SHADER);
this.ctx.shaderSource(this.fragmentShader, this.fragmentShaderSource);
this.ctx.compileShader(this.fragmentShader);
this.program = this.ctx.createProgram();
this.ctx.attachShader(this.program, this.vertexShader);
this.ctx.attachShader(this.program, this.fragmentShader);
this.ctx.linkProgram(this.program);
this.ctx.detachShader(this.program, this.vertexShader);
this.ctx.detachShader(this.program, this.fragmentShader);
this.ctx.deleteShader(this.vertexShader);
this.ctx.deleteShader(this.fragmentShader);
if (!this.ctx.getProgramParameter(this.program, this.ctx.LINK_STATUS)) {
console.log('error ' + this.ctx.getProgramInfoLog(this.program));
return;
}
this.ctx.enableVertexAttribArray(0);
var buffer = this.ctx.createBuffer();
this.ctx.bindBuffer(this.ctx.ARRAY_BUFFER, buffer);
this.ctx.vertexAttribPointer(0, 1, this.ctx.FLOAT, false, 0, 0);
this.ctx.useProgram(this.program);
}
draw() {
this.ctx.clear(this.ctx.COLOR_BUFFER_BIT);
this.ctx.bufferData(this.ctx.ARRAY_BUFFER, new Float32Array([Math.sin(this.time / 60.0)]), this.ctx.STATIC_DRAW);
this.ctx.drawArrays(this.ctx.POINTS, 0, 1);
}
}
Equivalent fully sane with explicit container:
\Ul[
\L[a]
\L[b]
\L[c]
]
which renders as:
- a
- b
- c
The explicit container is required if you want to pass extra arguments properties to the
ul
list macro, e.g. a title and an ID: Ul 1:
\Ul
{id=list-my-id}
[
\L[a]
\L[b]
\L[c]
]
which renders as:
This is the case because without the explicit container in an implicitUl 1.
- a
- b
- c
ul
list, the arguments would stick to the last list item instead of the list itself.
It is also required if you want ordered lists:
\Ol[
\L[first]
\L[second]
\L[third]
]
which renders as:
- first
- second
- third
Insane nested list with two space indentation:
* a
* a1
* a2
* a2
* b
* c
which renders as:
The indentation must always be exactly equal to two spaces, anything else leads to errors or unintended output.
- a
- a1
- a2
- a2
- b
- c
Equivalent saner nested lists with implicit containers:
\L[
a
\L[a1]
\L[a2]
\L[a2]
]
\L[b]
\L[c]
which renders as:
- a
- a1
- a2
- a2
- b
- c
Insane list item with a paragraph inside of it:
* a
* I have
Multiple paragraphs.
* And
* also
* a
* list
* c
which renders as:
- a
I haveMultiple paragraphs.
- And
- also
- a
- list
- c
Equivalent sane version:
\L[a]
\L[
I have
Multiple paragraphs.
\L[And]
\L[also]
\L[a]
\L[list]
]
\L[c]
which renders as:
- a
I haveMultiple paragraphs.
- And
- also
- a
- list
- c
Insane lists may be escaped with a backslash as usual:
\* paragraph starting with an asterisk.
which renders as:
* paragraph starting with an asterisk.
You can also start insane lists immediately at the start of a positional or named argument, e.g.:
\P[* a
* b
* c
]
which renders as:
- a
- b
- c
And now a list outside of
\OurBigBookExample
to test how it looks directly under the \Toplevel
implicit macro:- a
- b
- c
Via KaTeX server side, oh yes!
Inline math is done with the dollar sign (
$
) insane macro shortcut:
My inline $\sqrt{1 + 1}$ is awesome.
which renders as:
and block math is done with two or more dollar signs (My inline is awesome.
$$
):
$$
\sqrt{1 + 1} \\
\sqrt{1 + 1}
$$
which renders as:
The sane version of inline math is a lower case
m
:
My inline \m[[\sqrt{1 + 1}]] is awesome.
which renders as:
and the sane version of block math is with an upper caseMy inline is awesome.
M
:
\M[[
\sqrt{1 + 1} \\
\sqrt{1 + 1}
]]
which renders as:
The capital vs lower case theme is also used in other elements, see: block vs inline macros.
In the sane syntax, as with any other argument, you have to either escape any closing square brackets
]
with a backslash \
:
My inline \m[1 - \[1 + 1\] = -1] is awesome.
which renders as:
or with the equivalent double open and close:My inline is awesome.
My inline \m[[1 - [1 + 1] = -1]] is awesome.
Equation IDs and titles and linking to equations works identically to images, see that section for full details. Here is one equation reference example that links to the following insane syntax equation: Equation 4. "My first insane equation":
$$
\sqrt{1 + 1}
$$
{title=My first insane equation}
which renders as:
and the sane equivalent Equation 5. "My first sane equation":Equation 4. My first insane equation
\M{title=My first sane equation}[[
\sqrt{1 + 1}
]]
which renders as:
Equation 5. My first sane equation
Here is a raw one just to test the formatting outside of a
ourbigbook_comment
:
First here is an invisible block (with
{show=0}
) defining with a \newcommand
definition after this paragraph:
$$
\newcommand{\foo}[0]{bar}
$${show=0}
which renders as:
We make it invisible because this block only contains KaTeX definitions, and should not render to anything.Analogously with
\def
, definition:
$$
\gdef\foogdef{bar}
$${show=0}
which renders as:
and the second block using it:
$$
\foogdef
$$
which renders as:
If your project has multiple
.bigb
input files, you can share Mathematics definitions across all files by adding them to the ourbigbook.tex
file on the toplevel directory.For example, if
then from any
ourbigbook.tex
contains:
\newcommand{\foo}[0]{bar}
.bigb
file we in the project can use:
$$
\foo
$$
OK, this is too common, so we opted for some insanity here: double newline is a paragraph!
Paragraph 1.
Paragraph 2.
which renders as:
Paragraph 1.Paragraph 2.
Equivalently however, you can use an explicit
\P
macros as well, which is required for example to add properties to a paragraph, e.g.:
\P{id=paragraph-1}[Paragraph 1]
\P{id=paragraph-2}[Paragraph 2]
which renders as:
Paragraphs are created automatically inside macro argument whenever a double newline appears.Note that OurBigBook paragraphs render in HTML asdiv
withclass="p"
and not asp
. This means that you can add basically anything inside them, e.g. a list:which renders as a single paragraph.My favorite list is: \Ul[ \li[aa] \li[bb] ] because it is simple.
One major advantage of this, is that when writing documentation, you often want to keep lists or code blocks inside a given paragraph, so that it is easy to reference the entire paragraph with an ID. Think for example of paragraphs in the C++ standard.Dumps its contents directly into the rendered output.This construct is not XSS safe, see: Section 10.2. "unsafe-xss (--unsafe-xss
)".Here for example we define a paragraph in raw HTML:\passthrough[[ <p>Hello <b>raw</b> HTML!</p> ]]
which renders as:Hello raw HTML!
And for an inline passthrough:Hello \passthrough[[<b>raw</b>]] world!
which renders as:Hello raw world!Withq
:And so he said: \Q[ Something very smart And with multiple paragraphs. ] and it was great.
which renders as:And so he said:Something very smartAnd with multiple paragraphs.and it was great.The insane syntax marks:For example:
- headers with
||
(pipe, pipe space) at the start of a line- regular cells with
|
(pipe, space) at the start of a line- separates rows with double newline
|| Header 1 || Header 2 | 1 1 | 1 2 | 2 1 | 2 2
which renders as:Empty cells are allowed without the trailing space however:
Header 1 Header 2 1 1 1 2 2 1 2 2 | 1 1 | | 1 3 | 2 1 | | 2 3
which renders as:
1 1 1 3 2 1 2 3 Equivalent fully explicit version:\Table[ \Tr[ \Th[Header 1] \Th[Header 2] ] \Tr[ \Td[1 1] \Td[1 2] ] \Tr[ \Td[2 1] \Td[2 2] ] ]
which renders as:Any white space indentation inside an explicit
Header 1 Header 2 1 1 1 2 2 1 2 2 \Tr
can make the code more readable, and is automatically removed from final output due toremove_whitespace_children
which is set for\Table
.To pass further arguments to an implicit table such astitle
orid
, you need to use an explicittable
macro as in: Table 2. "My table title".\Table {title=My table title} {id=table-my-table} [ || Header 1 || Header 2 | 1 1 | 1 2 | 2 1 | 2 2 ]
which renders as:We would like to remove that explicit toplevel requirement as per: https://github.com/cirosantilli/ourbigbook/issues/186 The rules of when the caption shows up or not similar to those of images as mentioned at Section 4.1.8.2.1. "Image caption".Table 2. My table title.
Header 1 Header 2 1 1 1 2 2 1 2 2 Multiple source lines, including paragraphs, can be added to a single cell with insane syntax by indenting the cell with exactly two spaces just as for lists, e.g.:|| h1 || h2 || h3 h3 2 | 11 | 12 12 2 | 13 | 21 | 22 | 23
which renders as:Arbitrarily complex nested constructs may be used, e.g. a table inside a list inside table:
h1 h2 h3h3 211 1212 213 21 22 23 | 00 | 01 * l1 * l2 | 20 | 21 | 30 | 31 | 10 | 11
which renders as:
00 01
- l1
l2
20 21 30 31 10 11 And now a table outside of\OurBigBookExample
to test how it looks directly under the\Toplevel
implicit macro:Table 3. My table title.
Header 1 Header 2 1 1 1 2 2 1 2 2 And a fully insane one:
Header 1 Header 2 1 1 1 2 2 1 2 2 JavaScript interactive on-click table sorting is enabled by default, try it out by clicking on the header row:|| String col || Integer col || Float col | ab | 2 | 10.1 | a | 10 | 10.2 | c | 2 | 3.4
which renders as:Powered by: https://github.com/tristen/tablesort
String col Integer col Float col ab 2 10.1 a 10 10.2 c 2 3.4 OurBigBook automatically adds a ToC at the end of the first non-toplevel header of every document.For example, on a standard document with a single toplevel header:the ToC is rendered like:= Animal Animals are cute! == Dog == Cat
= Animal Animals are cute! Table of Contents * Dog * Cat == Dog == Cat
You may customize ToC placement with\Toc
macro, but just don't do it, it will just go against common convention and confuse readers.This is especially important when considering-S
,--split-headers
, where you almost always want an automatically generated ToC for every split header, otherwise you would need to add them all manually.Only one ToC is rendered per document. Any ToC besides the first one is ignored. In particular, this means that includes with--embed-includes
work seamlessly and render a single table of contents, even if multiple\Toc
macros are present in the included pages.The ID of the ToC is always fixed to#toc
. If you try to use that for another element, you will get the following error:error: tmp.bigb:3:1: reserved ID "toc"
The ToC ignores the toplevel header if you have one.For when you want a quick outline of the header tree on the terminal, also consider the--log headers
option.To the left of table of content entries you can click on an open/close icon to toggle the visibility of different levels of the table of contents.The main use case covered by the expansion algorithm is as follows:
- the page starts with all nodes open to facilitate Ctrl + F queries
- if you click on a node in that sate, you close all its children, to get a summarized overview of the contents
- if you click one of those children, it opens only its own children, so you can interactively continue exploring the tree
The exact behaviour is:
- the clicked node is open:
- state 1 all children are closed. Action: open all children recursively, which puts us in state 2
- state 2: not all children are closed. Action close all children, which puts us in state 1. This gives a good overview of the children, without any children of children getting in the way.
- state 3: the clicked node is closed (not showing any children). Action: open it to show all direct children, but not further descendants (i.e. close those children). This puts us in state 1.
Note that those rules make it impossible to close a node by clicking on it, the only way to close a node os to click on its parent, the state transitions are:but we feel that it is worth it to do things like this to cover the main use case described above without having to add two buttons per entry.
- 3 -> 1
- 1 -> 2
- 2 -> 1
Clicking on the link from a header up to the table of contents also automatically opens up the node for you in case it had been previously closed manually.Very analogous to images, only differences will be documented here.In the case of videos, where to store images becomes even more critical since videos are even larger than images, such that the following storage approaches are impractical off the bat:As a result, then Wikimedia Commons is one of the best options much like for images:\Video[https://upload.wikimedia.org/wikipedia/commons/8/85/Vacuum_pump_filter_cut_and_place_in_eppendorf.webm] {id=sample-video-in-wikimedia-commons} {title=Nice sample video stored in Wikimedia Commons} {start=5}
which renders as:We also handle more complex transcoded video URLs just fine:Video 2. Nice sample video stored in Wikimedia Commons. Source. \Video[https://upload.wikimedia.org/wikipedia/commons/transcoded/1/19/Scientific_Industries_Inc_Vortex-Genie_2_running.ogv/Scientific_Industries_Inc_Vortex-Genie_2_running.ogv.480p.vp9.webm] {id=sample-video-in-wikimedia-commons-transcoded} {title=Nice sample video stored in Wikimedia Commons transcoded}
which renders as:Commons is better than YouTube if your content is on-topic there because:Video 3. Nice sample video stored in Wikimedia Commons transcoded. Source.
- they have no ads
- it allows download of the videos: https://www.quora.com/Can-I-download-Creative-Commons-licensed-YouTube-videos-to-edit-them-and-use-them.
- it makes it easier for other users to find and re-use your videos
If your video does not fit the above Wikimedia Commons requirements, YouTube could be a good bet. OurBigBook automatically detects YouTube URLs for you, so the following should just work:\Video[https://youtube.com/watch?v=YeFzeNAHEhU&t=38] {id=sample-video-from-youtube-implicit-youtube} {title=Nice sample video embedded from YouTube implicit from `youtube.com` URL}
which renders as:TheVideo 4. Nice sample video embedded from YouTube implicit from youtube.com
URL. Source.youtu.be
domain hack URLs also work;\Video[https://youtu.be/YeFzeNAHEhU?t=38] {id=sample-video-from-youtube-implicit-youtu-be} {title=Nice sample video embedded from YouTube implicit from `youtu.be` URL}
which renders as:Alternatively, you can reach the same result in a more explicit and minimal way by settingVideo 5. Nice sample video embedded from YouTube implicit from youtu.be
URL. Source.{provider=youtube}
and thestart
arguments:\Video[YeFzeNAHEhU]{provider=youtube} {id=sample-video-from-youtube-explicit} {title=Nice sample video embedded from YouTube with explicit `youtube` argument} {start=38}
which renders as:When theVideo 6. Nice sample video embedded from YouTube with explicit youtube
argument. Source.youtube
provider is selected, the Video address should only to contain the YouTube video ID, which shows in the YouTube URL for the video as:Remember that you can also enable thehttps://www.youtube.com/watch?v=<video-id>
youtube
provider by default on yourourbigbook.json
with:"media-provider" { "youtube": {"default-for": "video"} }
But you can also use raw video files from any location that can serve them of course, e.g. here is one stored in this repository: Video 7. "Nice sample video stored in this repository".\Video[Tank_man_side_hopping_in_front_of_some_tanks.mp4] {id=sample-video-in-repository} {title=Nice sample video stored in this repository} {source=https://www.youtube.com/watch?v=YeFzeNAHEhU} {start=3}
which renders as:Video 7. Nice sample video stored in this repository. Source. Unlike image lazy loading, we don't support video lazy loading yet because:
- non-
youtube
videos use thevideo
tag which has noloading
property yetyoutube
videos are embedded withiframe
andiframe
has noloading
property yetBoth of this cases could be worked around with JavaScript:
non-youtube
: setsrc
from JavaScript as shown for images: https://stackoverflow.com/questions/2321907/how-do-you-make-images-load-lazily-only-when-they-are-in-the-viewport/57389607#57389607.But this breaks page semantics however, we don't know how to work around thatyoutube
videos: same as above for theiframe
, but this should be less problematic since YouTube videos are not viewable without JavaScript anyways, and who cares aboutiframe
semantics?The time to start playing the video at in seconds. Works for bothyoutube
and non-YouTube videos.Every macro in OurBigBook can have an optionalid
and many also have a reservedtitle
property.When a macro in the document has atitle
argument but noid
argument given, get an auto-generated ID from the title: automatic ID from title.Usually, the most convenient way to write cross references is with the insane syntax with delimited angled braces:<Cross references> are awesome.
which renders as:More details at: insane cross reference.Cross references are awesome.The sane equivalent to this is:\x[cross-reference]{c}{p} are awesome section.
which renders as:Note how that is more verbose, especially because here we use both theCross references are awesome section.\x
c
argument and\x
p
argument to capitalize and pluraize as desired.Another sane equivalent would be to add an explicit link body as in:\x[cross-reference][Cross references] are awesome.
which renders as:Cross references are awesome.When you use an insane cross reference (<>
) such as in:<Cross references> are awesome.
which renders as:it gets expanded exactly to the sane equivalent:Cross references are awesome.so we see that the\x[Cross references]{magic} are alwasome
\x
magic
argument gets added. It is that argument that for example adds the missing-
, and removes the pluralization to find the correct IDcross-reference
. For more details, see the documentation of the\x
magic
argument.Like other insane constructs, insane cross references are exactly equivalent to the sane version, so you can just add other arguments after the construct, e.g.:<Cross references>{full} are awesome.
which renders as:which gets converted to exact the same as the sane:Section 4.1.20. "Cross reference (\x
macro)" are awesome.\x[cross-reference]{full} are awesome.
which renders as:Section 4.1.20. "Cross reference (\x
macro)" are awesome.In most cases it is generally more convenient to simply use the\x
magic
argument through insane cross references instead of thec
andp
arguments as described on the rest of this section, see also: Section 4.1.20.3. "Inflection vs magic".A common usage pattern is that we want to use header titles in non-full cross references as the definition of a concept without repeating the title, for example:== Dog Cute animal. \x[cats][Cats] are its natural enemies. == Cats This is the natural enemy of a \x[dog][dog]. \x[dog][Dogs] are cute, but they are still the enemy. One example of a cat is \x[felix-the-cat]. === Felix the Cat Felix is not really a \x[cats][cat], just a carton character.
However, word inflection makes it much harder to avoid retyping the definition again.For example, in the previous example, without any further intelligent behaviour we would be forced to re-type\x[dog][dog]
instead of the desired\x[dog]
.OurBigBook can take care of some inflection cases for you.For capitalization, both headers and cross reference macros have thec
boolean argument which stands for "capitalized":Capitalization is handled by a JavaScript case conversion.
for headers,c
means that the header title has fixed capitalization as given in the title, i.e.
- if the title has a capital first character, it will always show as a capital, as is the case for most proper noun
- if it is lower case, it will also always remain lower case, as is the case for some rare proper nouns, notably the name of certain companies
This means that for such headers,c
in thex
has no effect. Maybe we should give an error in that case. But lazy now, send PR. for cross reference macros,c
means that the first letter of the title should be capitalized.Using this option is required when you are starting a sentence with a non-proper noun.For pluralization, cross reference macros have thep
boolean argument which stands for "pluralize":If your desired pluralization is any more complex than modifying the last word of the title, you must do it manually however.
- if given and true, this automatically pluralizes the last word of the target title by using the https://github.com/blakeembrey/pluralize library
- if given and false, automatically singularize
- if not given, don't change the number of elements
With those rules in mind, the previous OurBigBook example can be written with less repetition as:== Dog Cute animal. \x[cats]{c} are its natural enemies. == Cats This is the natural enemy of a \x[dog]. \x[dog]{p} are cute, but they are still the enemy. One example of a cat is \x[Felix the Cat]. === Felix the Cat {c} Felix is not really a \x[cats][cat], just a carton character.
If plural and capitalization don't handle your common desired inflections, you can also just create custom ones with the\H
synonym
argument.Now for a live example for quick and dirty interactive testing.\x[inflection-example-not-proper]{full}
which renders as:The\x
magic
argument was introduced later, and basically became a better alternative to cross reference title inflection in all but the following cases:
\H
disambiguate
argument: disambiguate prevents the determination of plural inflection, e.g. in:there is currently no way to make it output= Python {disambiguate=animal} I like <python animal>.
Pythons
in the plural without resorting to either\x
p
argument or an explicit content, because if you wrote:it would just lead to Id not found, as we would try the plural vs singular onI like <pythons animal>.
animal
only.Maybe one day we can implement an even insaner system that understands that parenthesis should skipped for the inflection as in:https://github.com/cirosantilli/ourbigbook/issues/244I like <pythons (animal)>.
- plural headers. We only attempt to singularize arguments for now, not pluralize them. So if you had:
you would instead need to write:My <dog> is nice. == Dogs
or:My <dog>{p=0} is nice.
My <dog>[dog] is nice.
If you use\x
within atitle
, which most commonly happens for image titles, that can generate complex dependencies between IDs, which would either be harder to implement, or lead to infinite recursion.To prevent such problems, OurBigBook emits an error if you use an\x
without content in thetitle
of one of the following elements:
any header. For example, the following gives an error:= h1 {id=myh1} == \x[myh1]
This could be solved by either adding a content to the reference:or by adding an explicit ID to the header:= h1 {id=myh1} == \x[myh1][mycontent]
= h1 {id=myh1} == \x[myh1] {id=myh2}
non-header (e.g. an image) that links to the title of another non-headerFor non-headers, things are a bit more relaxed, and we can link to headers, e.g.:This is allowed because OurBigBook calculates IDs in two stages: first for all headers, and only later non non-headers.= h1 \Image[myimg.jpg] {title=my \x[h1]}
What you cannot do is link to another image e.g.:and there the workaround are much the same as for headers: either explicitly set the cross reference content:\Image[myimg.jpg] {id=myimage1} {title=My image 1} \Image[myimg.jpg] {title=my \x[h1]}
or explicitly set an ID:\Image[myimg.jpg] {id=myimage1} {title=My image 1} \Image[myimg.jpg] {title=my \x[h1][My image 1]}
\Image[myimg.jpg] {id=myimage1} {title=My image 1} \Image[myimg.jpg] {id=myimage2} {title=my \x[h1]}
While it is technically possible relax the above limitations and give an error only in case of loops, it would require a bit of extra work which we don't want to put in right now: https://github.com/cirosantilli/ourbigbook/issues/95.Furthermore, the above rules do not exclude infinite rendering loops, but OurBigBook detects such loops and gives a nice error message, this has been fixed at: https://github.com/cirosantilli/ourbigbook/issues/34For example this would contain an infinite loop:\Image[myimg.jpg] {id=myimage1} {title=\x[myimage2]} \Image[myimg.jpg] {id=myimage2} {title=\x[myimage1]}
This infinite recursion is fundamentally not technically solved: the user has to manually break the loop by providing anx
content explicitly, e.g. in either:or:\Image[myimg.jpg] {id=myimage1} {title=\x[myimage2][my content 2]} \Image[myimg.jpg] {id=myimage2} {title=\x[myimage1]}
\Image[myimg.jpg] {id=myimage1} {title=\x[myimage2]} \Image[myimg.jpg] {id=myimage2} {title=\x[myimage1][my content 1]}
A closely related limitation is the simplistic approach to\x
id
output format.Reference to a non-first header of another file:\x[h2-in-not-the-readme]
which renders as:To make toplevel links cleaner,if target header is the very first element of the other page, then the link does not get a fragment, e.g.:\x[not-readme]
rendered as:and not:<a href="not-readme"
while<a href="not-readme#not-readme"
\x[h2-in-not-the-readme]
is rendered with the fragment:<a href="not-readme#h2-in-not-the-readme"
Reference to the first header of another file that is a second inclusion:\x[included-by-not-readme]
which renders as:Reference to another header of another file, withfull
:\x[h2-in-not-the-readme]{full}.
which renders as:Note that whenfull
is used with references in another file in multi page mode, the number is not rendered as explained at: Section 4.1.20.6.3.1. "\x
full
argument in cross file references".Reference to an image in another file:\x[image-not-readme-xi]{full}.
which renders as:Reference to an image in another file:\x[image-figure-in-not-the-readme-without-explicit-id]{full}.
which renders as:Remember that the ID of the toplevel header is automatically derived from its file name, that's why we have to use:\x[not-readme]
which renders as:instead of:\x[not-the-readme]
Reference to a subdirectory:\x[subdir] \x[subdir/h2] \x[subdir/notindex] \x[subdir/notindex-h2]
which renders as:Implemented at: https://github.com/cirosantilli/ourbigbook/issues/116Reference to an internal header of another file: h2 in not the README. By default, That header ID gets prefixed by the ID of the top header.When using--embed-includes
mode, the cross file references end up pointing to an ID inside the current HTML element, e.g.:rather than:<a href="#not-readme">
This is why IDs must be unique for elements across all pages.<a href="not-readme.html/#not-readme">
When running in Node.js, OurBigBook dumps the IDs of all processed files to aout/db.sqlite3
file in theout
directory, and then reads from that file when IDs are needed.When converting under a directory that containsourbigbook.json
,out/db.sqlite3
is placed inside the same directory as theourbigbook.json
file.If there is noourbigbook.json
in parent directories, thenout/db.sqlite3
is placed in the current working directory.These follows the principles described at: the current working directory does not matter when there is aourbigbook.json
.db.sqlite3
is not created or used when handling input from stdin.When running in the browser, the same JavaScript API will send queries to the server instead of a local SQLite database.To inspect the ID database to debug it, you can use:sqlite3 out/db.sqlite3 .dump
It is often useful to dump a single table, e.g. to dump theids
table:and one particularly important query is to dump a list of all known IDs:sqlite3 out/db.sqlite3 '.dump ids'
sqlite3 out/db.sqlite3 'select id from ids'
You can forceourbigbook
to not use the ID database with the--no-db
command line optionIf the targettitle
argument contains a link from either another cross references or a regular external hyperlink, OurBigBook automatically prevents that link from rendering as a link when no explicit body is given.This is done because nested links are illegal in HTML, and the result would be confusing.This use case is most common when dealing with media such as images. For example in:the= afds \x[image-aa-zxcv-lolol-bb] == qwer \Image[Tank_man_standing_in_front_of_some_tanks.jpg] {title=aa \x[zxcv][zxcv] \a[http://example.com][lolol] bb} == zxcv
\x[image-aa-zxcv-lolol-bb]
renders something like:and not:<a href="#image-aa-zxcv-lolol-bb">aa zxcv lolol bb</a>
<a href="#image-aa-zxcv-lolol-bb">aa <a href="zxcv">zxcv</a> <a href="http://example.com">lolol</a> bb</a>
Live example:This is a nice image: \x[image-aa-zxcv-lolol-bb]. \Image[Tank_man_standing_in_front_of_some_tanks.jpg] {title=aa \x[cross-reference-title-link-removal][zxcv] \a[http://example.com][lolol] bb}
which renders as:This is a nice image: Figure 17. "aa zxcv lolol bb".![]()
Figure 17. aa zxcv lolol bb. Capitalizes the first letter of the target title.For more details, see: Section 4.1.20.2. "Cross reference title inflection".Setting thechild
boolean argument on a cross reference to a header as in:makes that header show up on the list of extra parents of the child.\x[my-header]{child}
This allows a section to have multiple parents, e.g. to include it into multiple categories. For example:would render something like:= Animal == Mammal === Bat === Cat == Flying animal These animals fly: * \x[bat]{child} These animals don't fly: * \x[cat]
so note how "Bat" has a list of tags including "Flying animal", but Cat does not, due to the= Animal == Mammal === Bat (Parent section: Mammal) (Tags: Flying animal) === Cat (Parent section: Mammal) == Flying animal (Parent section: Animal) These animals fly: * \x[bat] These animals don't fly: * \x[cat]
child
.This property does not affect how the\Toc
is rendered. We could insert elements sections there multiple times, but it has the downside that browser Ctrl + F searches would hit the same thing multiple times on the table of contents, which might make finding things harder.== My title{id=my-id} Read this \x[my-id][amazing section].
If the second argument, thecontent
, is not present, it expand to the header title, e.g.:is the same as:== My title{id=my-id} Read this \x[my-id].
== My title{id=my-id} Read this \x[my-id][My title].
A live demo can be seen at:\x
child
argument demo.Generally, a better alternative to this argument is to use\H
child
argument.The term refers to sections that have a parent/child relationship via either of the:rather than via the usual header hierarchy.Secondary children show up for example on the tagged metadata section, but not on the table of contents, which is what the header hierarchy already shows.Secondary children are normally basically used as "tags": a header such asBat
can be a direct child ofMammal
, and a secondary child ofFlying animal
, or vice versa. BothMammal
andFlying animal
are then basically ancestors. But we have to chose one main ancestor as "the parent", and other secondary ancestors will be seen as tags.This option first does ID target from title conversion on the argument, so you can e.g. keep any spaces or use capitalization in the title as in:TODO the fact that this transformation is done currently makes it impossible to use "non-standard IDs" that contain spaces or uppercase letters. If someone ever wants that, we could maybe add a separate argument that does not do the expansion e.g.:= Animal == Flying animal {child=Big bat} == Big bat
but definitely the most important use case is having easier to type and read source with the standard IDs.= Animal == Flying animal {childId=Big bat} == Big bat {id=Big bat}
Oh, and cows are also mammals.Bats can fly.But cats can't.Theparent
argument is exactly like the\x
child
argument, but it reverses the direction of the parent/child relation.To also show the section auto-generated number as in "Section X.Y My title" we add the optional{full}
boolean argument to the cross reference, for example:\x[x-full-argument]{full}.
which renders as:{full}
is not needed for cross references to most macros besides headers, which usefull
by default as seen by thedefault_x_style_full
macro property in--help-macros
. This is for example the case for images. You can force this to be disabled with{full=0}
:Compare \x[image-my-test-image]{full=0} vs \x[image-my-test-image]{full=1}.
which renders as:For example in the following cross file reference:\x[h2-in-not-the-readme]{full}.
which renders as:we get just something like:instead of:Section "h2 in not the readme"
This is because the number "Section 1.2" might already have been used in the current page, leading to confusion.Section 1.2 "h2 in not the readme"
This argument makes writing many internal links more convenient, and it was notably introduced because it serves as the sane version of insane cross references.If given e.g. as in:the link treated magically as follows:= Internal reference \x[Internal references]{magic}
content capitalization and pluralization are detected from the string, and implicitly set the\x
c
argument and\x
p
argument. In the example:In this simple example, the content therefore will be exactly
{c}
capitalization is set becauseInternal references
starts with an upper case characterI
{p}
pluralization is set becauseInternal references
ends in a plural wordInternal references
as in the source. But note that this does not necessarily need to be the case, e.g. if we had done:then the content would be:\x[Internal Reference]{magic}
without capitalInternal reference
R
, i.e. everything except capitalization and pluralization is ignored. This forgiving way of doing things means that writers don't need to remember the exact ideal capitalization of everything, which is very hard to remember.It also means that any more complex elements will be automatically rendered as usual, e.g. if we had:then the output would still contain the= \i[Internal] reference \x[internal reference]{magic}
<i>
italic tag.If we had a scope as in\x[my scope/Internal references]
, then each scope part is checked separately. E.g. in this case we would have upper caseInternal references
, even thoughmy scope
is lowercase, and so{c}
would be set.- the ID is calculated as follows:
- if the ID starts with
file/
, use it as is, to work better with the\H
file
argument- otherwise:
automatic ID from title conversion is performed, with you exception: forwards slashs/
are kept, in order to make scopes work.In our case, there aren't any slashes/
, so it just givesinternal-references
. But if instead we had e.g.:\x[my scope/internal reference]{magic}
, then we would reachmy-scope/internal-reference
and notmy-scope-internal-reference
.- if there is a match to an existing ID use it.
internal-references
in the plural does not match, so go to the next step- if the above failed, try singularizing the last word as in the
\x
p
argument withp=0
before doing automatic ID from title conversion. This givesinternal-reference
, which does exist, and so we use that.There may be some cases where you might still want to use cross reference title inflection however, see: Section 4.1.20.3. "Inflection vs magic".Pluralizes or singularizes the last word of the target title.For more details, see: Section 4.1.20.2. "Cross reference title inflection".Theref
argument of\x
marks the link as reference, e.g.:renders something like:Trump said this and that.\x[donald-trump-access-hollywood-tape]{ref} = Donald Trump Access Hollywood tape
Trump said this and that.<a href="donald-trump-access-hollywood-tape">*</a>
This could currently be replicated withoutref
by just using:but later on we might add more precise reference fields like the page of a book or date fetched as Wikipedia supports.Trump said this and that.\x[donald-trump-access-hollywood-tape][*]
Implemented at: https://github.com/cirosantilli/ourbigbook/issues/137Certain commonly used macros have insane macro shortcuts that do not start with backslash (\
).Originally, Ciro wanted to avoid those, but they just feel too good to avoid.Every insane syntax does however have an equivalent sane syntax.The style recommendation is: use the insane version which is shorter, unless you have a specific reason to use the sane version.Insane in our context does not mean worse. It just mean "harder for the computer to understand". But it is more important that humans can understand in the first place! It is find to make the computer work a bit more for us when we are able to.
- Section 4.1.14. "Paragraph (
\P
)":\n\n
(double newline)- Section 4.1.1. "Link (
\a
)":a http://example.com b
(space followed byhttp://
)- Section 4.1.20. "Cross reference (
\x
macro)":<Cross references>
(angle brackets), see: Section 4.1.20.1. "Insane cross reference (<>
)"- Section 4.1.13. "Mathematics (
$$
,$
,\M
,\m
)":$
, described at: insane code and math shortcuts- Section 4.1.4. "Code block (
``
,`
,\C
,\c
)":`
, described at: insane code and math shortcuts- Section 4.1.12. "List (
*
,\L
,\Ul
,\Ol
)":*
and indentation- Section 4.1.17. "Table (
||
,|
,\Table
,\Tr
,\Th
and\Td
)":||
,|
and indentationThe insane code and math shortcuts work very analogously and are therefore described together in this section.The insane inline code syntax:a `b c` d
which renders as:and is equivalent to the sane:ab c
da \c[[b c]] d
The insane block code:a `` b c `` d
which renders as:and is equivalent to the sane:ab c
da \C[[ b c ]] d
Insane arguments always work by abbreviating:This means that you can add further arguments as usual.
- the macro name
- one or more of its positional arguments, which are fixed as either literal or non-literal for a given insane construct
For example, an insane code block with an id can be written as:because that is the same as:a `b c`{id=ef} g
a \c[b c]{id=ef} g
which renders as:Extra arguments must come after the insane opening, e.g. the following does not work:a {id=ef}`b c` g
This restriction things easy to parse for humans and machines alike.Literal backticks and dollar signs can be produced witha backslash escape as in:a \` \$ b
which renders as:a ` $ bIt is not possible to escape backticks (`
) inside an insane inline code, or dollar signs ($
) in insane math.The design reason for that is because multiple backticks produce block code.The upside is that then you don't have to escape anything else, e.g. backslashes (\
) are rendered literally.The only way to do it is to use the sane syntax instead:a \c[[b ` c]] d a \m[[\sqrt{\$4}]] d
which renders as:ab ` c
da dWithin block code and math, you can just add more separators:``` code with two backticks `` nice ```
which renders as:code with two backticks `` nice
OurBigBook Markup macro identifiers can consist of the following letters:Since underscores
a-z
lowercaseA-Z
uppercase0-9
_
or hyphens=
are not allowed, camel case macro names are recommended, e.g. for\OurBigBookExample
we use the name:OurBigBookExample
Every argument in OurBigBook is either positional or named.For example, in a header definition with an ID:which is equivalent to the sane version:= My asdf {id=asdf qwer} {scope}
we have:\H[1][My asdf] {id=asdf qwer} {scope}
- two positional argument:
[1]
and[My asdf]
. Those are surrounded by square brackets[]
and have no name two named arguments:{id=asdf qwer}
and{scope}
.The first one has nameid
, followed by the separator=
, followed by the valueasdf qwer
.The separator=
always is optional. If not given, it is equivalent to an empty value, e.g.:is the same as:{id=}
{id}
You can determine if a macro is positional or named by using--help-macros
. Its output contains something like:and so we see that"h": { "name": "h", "positional_args": [ { "name": "level" }, { "name": "content" } ], "named_args": { "id": { "name": "id" } "scope": { "name": "scope" } },
level
andcontent
are positional arguments, andid
andscope
are named arguments.Generally, positional arguments are few (otherwise it would be hard to know which is which is which), and are almost always used for a given element so that they save us from typing the name too many times.The order of positional arguments must of course be fixed, but named arguments can go anywhere. We can even mix positional and named arguments however we want, although this is not advised for clarity.The following are therefore all equivalent:\H[1][My asdf]{id=asdf qwer}{scope} \H[1][My asdf]{scope}{id=asdf qwer} \H{id=asdf qwer}{scope}[1][My asdf] \H{scope}[1]{id=asdf qwer}[My asdf]
Just like named arguments, positional arguments are never mandatory.Most positional arguments will default to an empty string if not given.However, some positional arguments can have special effects if not given.For example, an anchor with the first positional argument present (the URL), but not the second positional argument (the link text) as in:\a[http://example.com]
which renders as:has the special effect of generating automatic links as in:\a[http://example.com][http://example.com]
This can be contrasted with named arguments, for which there is always a default value, notably for boolean arguments.See also: Section 4.1.1. "Link (\a
)".Some positional arguments are required, and if not given OurBigBook reports an error and does not render the node.This is for example thelevel
of a header.These arguments marked with themandatory: true
--help-macros
argument property.Name arguments marked in--help-macros
asboolean: true
must either:
- take no value and no
=
sign, in which case the value is implicitly set to1
- take value exactly
0
or1
- not be given, in which case a custom per-macro default is used. That value is the
default
from--help-macros
, or0
if such default is not givenFor example, the\x
full
argument of cross references is correctly written as:\x[boolean-argument]{full}
which renders as:without the=
sign, or equivalently:\x[boolean-argument]{full=1}
which renders as:Thefull=0
version is useful in the case of reference targets that unlike headers expand the title on the cross reference by default, e.g. images:\x[boolean-argument]{full=1}
which renders as:The name "boolean argument" is given by analogy to the "boolean attribute" concept in HTML5.The JavaScript interface sees arguments as follows:where args is a dict such that:function macro_name(args)
- optional arguments have the key/value pairs explicitly given on the call
mandatory arguments have a key documented by the API, and the value on the call.For example, the link API names its argumentshref
andtext
.Arguments that are opened with more than one square brackets[
or curly braces{
are literal arguments.In literal arguments, OurBigBook is not parsed, and the entire argument is considered as text until a corresponding close with the same number of characters.Therefore, you cannot have nested content, but it makes it extremely convenient to write code blocks or mathematics.For example, a multiline code block with double open and double close square brackets inside can be enclosed in triple square brackets:A literal argument looks like this in OurBigBook: \C[[ \C[ A multiline code block. ] ]] And another paragraph.
which renders as:A literal argument looks like this in OurBigBook:\C[ A multiline code block. ]
And another paragraph.The same works for inline code:The program \c[[puts("]");]] is very complex.
which renders as:The programputs("]");
is very complex.Within literal blocks, only one thing can be escaped with backslashes are:
- leading open square bracket
[
- trailing close square bracket
]
The rule is that:
- if the first character of a literal argument is a sequence of backslashes (
\
), and it is followed by another argument open character (e.g.[
, remove the first\
and treat the other characters as regular text- if the last character of a literal argument is a
\
, ignore it and treat the following closing character (e.g.]
) as regular textSee the following open input/output pairs:and close examples:\c[[\ b]] <code>\ b</code> \c[[\a b]] <code>\a b</code> \c[[\[ b]] <code>[ b</code> \c[[\\[ b]] <code>\[ b</code> \c[[\\\[ b]] <code>\\[ b</code>
\c[[a \]] <code>a \</code> \c[[a \]]] <code>a ]</code> \c[[a \\]]] <code>a \]</code>
If the very first or very last character of an argument is a newline, then that character is ignored if it would be part of a regular plaintext node.For example:generates something like:\C[[ a b ]]
instead of:<pre><code>a b</code></pre>
This is extremely convenient to improve the readability of code blocks and similar constructs.<pre><code> a b </code></pre>
The newline is however considered if it would be part of some insane macro shortcut. For example, we can start an insane list inside a quotations as in:\Q[ * a * b ]
which renders as:where the insane list requires a leading newline
- a
- b
\n*
to work. That newline is not ignored, even though it comes immediately after the\Q[
opening.The macro name and the first argument, and any two consecutive arguments, can be optionally separated by exactly one newline character, e.g.:is equivalent to:\H [2] {scope} [Design goals]
which is also equivalent to:\H[2]{scope}[Design goals]
This allows to greatly improve the readability of long argument lists by having them one per line.\H[2]{scope} [Design goals]
There is one exception to this however: inside an insane header, any newline is interpreted as the end of the insane header. This is why the following works as expected:and the== My header 2 `some code` {id=asdf}
id
gets assigned to the header rather than the trailing code element.If the document ends one newline, it is ignored.If it is two or more, then that generates an error.Every character that cannot be a macro identifier can be escaped with a backslash\
. If you try to escape a macro identifier it of course treats the thing as a macro instead and fails, e.g. in\a
it would try to use a macro called\a
, not escape the charactera
.For some characters, escaping or not does not make any difference because they don't have any meaning to OurBigBook Markup, e.g. currently%
is always the exact same as\%
.But in non-literal macro arguments, you have to use a backslash to escape the following if you want them to not have any magical meaning:
\
: backslashes start macros\[
and\]
: open and close positional macro arguments\{
and\}
: open and close optional macro arguments- escapes for macros with insane shortcut:
<
(open angle brackets, less than sign): insane macro shortcut for insane cross references$
(dollar sign): insane macro shortcut for mathematics`
(backtick): insane macro shortcut for code blocksFurthermore, only at:you must also escape the following macros with insane shortcut:
- at the start of the document
- after a newline
- at the start of a new argument
The escape rules for literal arguments are described at: Section 4.2.3.4. "Literal arguments ([[...]]
and{{key=...}}
)".This is good for short arguments of regular text, but for longer blocks like code blocks or mathematics, you may want to use literal argumentsIn HTML, certain elements such as<ul>
cannot have anytext
nodes in them, and any whitespace is ignored, see https://stackoverflow.com/questions/2161337/can-we-use-any-other-tag-inside-ul-along-with-li/60885802#60885802.A similar concept applies to OurBigBook, e.g.:does not parse as:\Ul[ \L[aa] \L[bb] ]
but rather as:\Ul[\L[aa]<NEWLINE>\L[bb]<NEWLINE>]
because the\Ul[\L[aa]\L[bb]]
content
argument oful
is marked withremove_whitespace_children
and automatically removes any whitespace children (such as a newline) as a result.This also applies to consecutive sequences ofauto_parent
macro property macros, e.g.:also does not include the newline between the list items.\L[aa] \L[bb]
The definition of whitespace is the same as the ASCII whitespace definition of HTML5:\r\n\f\t
.By default, arguments can be given only once.However, arguments with themultiple
property set totrue
can be given multiple times, and each time the argument is given, the new value is appended to a list containing all the values.An example is the\H
child
argument.Internally, multiple is implemented by creating a new level in the abstract syntax tree, and storing each argument separately under a newly generated dummy nodes as in:AstNode: H AstArgument: child AstNode: Comment AstArgument: content AstNode: plaintext AstNode: x AstNode: Comment AstArgument: content AstNode: plaintext AstNode: x
These are shared concepts that are used across other sections.The children are always added as arguments of thecontent
argument of the implicit parent.If present, theauto_parent
macro property determines which auto-parent gets added to those macros.Every OurBigBook macro is either block or inline:Some macros have both a block and an inline version, and like any other macro, those are differentiated by capitalization:Certain common URL protocols are treated as "known" by OurBigBook, and when found they have special effects in some parts of the conversion.The currently known protocols are:
http://
https://
Effects of known protocols include:
- insane link parsing rules: mark the start of insane links
- store images in a separate media repository: mark an image
src
to ignoreprovider
Some parts of OurBigBook use "JavaScript case conversion".This means that the conversion is done as if by thetoLowerCase
/toUpperCase
functions.The most important fact about those functions is that they do convert non-ASCII Unicode capitalization, e.g. betweenÉ
andé
:These conversions are also specified in the Unicode standard.Convert a.bigb
file to HTML and output the HTML to a file with the same basename without extension, e.g.:ourbigbook hello.bigb firefox hello.html
Files namedREADME.bigb
are automatically converted toindex.html
so that they will show on both GitHub READMEs and at the website's base address:ourbigbook README.bigb firefox hello.html
Convert all.bigb
files in a directory to HTML files next to each corresponding.bigb
file, e.g.somefile.bigb
tosomefile.html
:The HTML output files are placed right next to each correspondingourbigbook .
.bigb
.The output file can be selected explicitly with:--outfile <outfie>
.Output to stdout instead of saving it to a file:ourbigbook --stdout README.bigb
In order to resolve cross file references, this actually does two passes:
- first an ID extraction pass, which parses all inputs and dumps their IDs to the ID database
- then a second render pass, which uses the IDs in the ID database
Convert a.bigb
file from stdin to HTML and output the contents of<body>
to stdout:printf 'ab\ncd\n' | ourbigbook --body-only
The following basenames are considered "index files":
README.bigb
index.bigb
Those basenames have the following magic properties:
- the default output file name for an index file in HTML output is either:
index.html
when in the project toplevel directory. E.g.README.bigb
renders toindex.html
. Note that GitHub and many other static website hosts then automatically hide theindex.html
part from the URL, so that yourREADME.bigb
hosted athttp://example.com
will be accessible simply underhttp://example.com
and nothttp://example.com/index.html
the name of the subdirectory in which it is located when not in the project toplevel directory. E.g.mysubdir/index.bigb
outputs tomysubdir.html
Previously, we had placed the output inmysubdir/index.html
, but this is not as nice as it makes GitHub pages produce URLs with a trailing slash asmysubdir/
, which is ugly, see also: https://stackoverflow.com/questions/5948659/when-should-i-use-a-trailing-slash-in-my-url- the default toplevel header ID of an index files is derived from the parent directory basename rather than from the source file basename
This directory is determined by first checking the presence of aourbigbook.json
file.If aourbigbook.json
is found, then the project toplevel directory is the directory that contains that file.
- otherwise, if the input path is a descendant of the current working directory, then the current working directory is used, see also: the current working directory does not matter when there is a
ourbigbook.json
- otherwise, if the input path is a directory, it is used
- otherwise, the directory containing the input file is used
For example, consider the file following file structure relative to the current working directory:path/to/notindex.bigb
In this case:
- if there is no
ourbigbook.json
file:
- if we run
ourbigbook .
: the toplevel directory is the current directory.
, and sonotindex.bigb
has IDpath/to/notindex
- if we run
ourbigbook path
: same- if we run
ourbigbook path/to
: same- if we run
ourbigbook path/to/notindex.bigb
: same- if there is a
path/ourbigbook.json
file:
- if we run
ourbigbook .
: the toplevel directory is the current directory.
because theourbigbook.json
is below the entry point and is not seen, and sonotindex.bigb
has IDpath/to/notindex
- if we run
ourbigbook path
: the toplevel directory is the directory with theourbigbook.json
,path
, and sonotindex.bigb
has IDto/notindex
- if we run
ourbigbook path/to
: same- if we run
ourbigbook path/to/notindex.bigb
: sameThis is the index file present in the project toplevel directory.Being the toplevel index file has the following implications compared to other index files:
- its ID is derived from the header itself, not the directory, see also: the ID of the first header is derived from the filename
When the file or directory being converted has an ancestor directory with aourbigbook.json
file, then your current working directory does not have any effect on OurBigBook output. For example if we have:then all of the following conversions produce the same output:/project/ourbigbook.json /project/README.bigb /project/subdir/README.bigb
- directory conversion:
cd /project && ourbigbook .
cd / && ourbigbook project
cd project/subdir && ourbigbook ..
- file conversion:
cd /project && ourbigbook README.bigb
cd / && ourbigbook project/README.bigb
cd project/subdir && ourbigbook ../README.bigb
When there isn't aourbigbook.json
, everything happens as though there were an emptyourbigbook.json
file in the current working directory. So for example:
- outputs that would be placed relative to inputs are still placed in that place, e.g.
README.bigb -> index.html
always stay together- outputs that would be placed next to the
ourbigbook.json
are put in the current working directory, e.g. theout
directoryInternally, the general philosophy is that the JavaScript API in index.js works exclusively with paths relative to the project toplevel directory. It is then up to callers such as ourbigbook to ensure that filesystem specifics handle the relative paths correctly.This is the most important option of the software.It produces a copy of the HTML of https://cirosantilli.com/china-dictatorship to stdout.The data is stored inside an NPM package, making it hard to censor that information, see also: https://cirosantilli.com/china-dictatorship#mirrorsUsage:ourbigbook --china > china.html firefox china.html
The--dry-run
option is a good way to debug--publish
option, as it builds the publish output files without doing any git commands that would be annoying to revert. So after doing:you can just go and inspect the generated HTML to see what would get pushed at:./ourbigbook --dry-run --publish .
see also: thecd out/publish/out/publish/
out
directory.Similar to--dry-run
, but it runs all git commands except forgit push
, which gives a clearer idea of what--publish
would actually do including the git operations, but without publishing anything:./ourbigbook --dry-run --publish .
Makes includes render the included content in the same output file as the include is located, instead of the default behaviour of creating links.In addition to this:
cross file references are disabled, and the cross file ID database does not get updated.It should be possible to work around this, but we are starting with the simplest implementation that forbids it.The problem those cause is that the IDs of included headers show as duplicate IDs of those in the ID database.This should be OK to start with because the more common use case with--html-sinle-page
is that of including all headers in a single document.Otherwise,include
only adds the headers of the other file to the table of contents of the current one, but not the body of the other file. The ToC entries then point to the headers of the included external files.You may want to use this option together with--embed-resources
to produce fully self-contained individual HTML files for your project.Embed as many external resources such as images and CSS as possible into the HTML output files, rather than linking to external resources.The use case for this option is to produce a single HTML file for an entire build that is fully self contained, and can therefore be given to consumers and viewed offline, much like a PDF.Examples of embeddings done:
CSS and JavaScript are copy pasted in place into the HTML.The default built-in CSS and JavaScript files used by OurBigBook (e.g. the KaTeX CSS used for mathematics) are currently all automatically downloaded as NPM package dependencies to ourbigbookWithout--embed-resources
, those CSS and JavaScript use their main cloud CDN URLs, and therefore require Internet connection to view the generated documents.The embedded version of the document can be viewed offline however.There is however a known bug: KaTeX fonts are not currently embedded, so math won't work properly. The situation is similar as for images, but a bit harder because we also need to fetch the blobs from the CSS, which is likely doable from Webpack:Examples of embedding that could be implemented in the future:
images are downloaded if needed and embedded asdata:
URLs.Doing this however has a downside: it would slow the page loading down. The root problem is that HTML was not designed to contain assets, and notably it doesn't have byte position indices that can tell it to skip blobs while parsing, and how to refer to them later on when they show up on the screen. This is kind of why EPUB exists: https://github.com/cirosantilli/ourbigbook/issues/158Images that are managed by the project itself and already locally present, such as those inside the project itself or due tomedia-providers
usually don't require download.For images linked directly from the web, we maintain a local download cache, and skip downloads if the image is already in the cache.To re-download due to image updates, use either:
--asset-cache-update
: download all images such that the local disk timestamp is older than the HTTP modification date withIf-Modified-Since
--asset-cache-update-force
: forcefully redownload all assetsKeep in mind that certain things can never be embedded, e.g.:
- YouTube videos, since YouTube does not offer any download API
Parse and overwrite the local .bigb OurBigBook Markup input source files with the recommended code format. E.g.:overwritesourbigbook README.bigb
README.bigb
with the recommended formatting, and:does that for every single file in the current directory.ourbigbook .
This option uses thebigb
output format.In order to reach a final stable state, you might need to run the conversion twice. This is not ideal but we don't have the patience to fix it. The reason is that links in image titles may expand twice. This is the usual type of two level recursion that has caused much more serious problems, see e.g.\x
withintitle
restrictions. E.g. starting with:the first conversion leads to uppercasing inside the image title:<image my big dog> \Image[image.png]{title=My <big dog>} = Big dog
and the second one to uppercasing the reference to the image title:<image my big dog> \Image[image.png]{title=My <big Dog>} = Big Dog
<image my big Dog> \Image[image.png]{title=My <big Dog>} = Big Dog
Generate one of the template repositories locally:
ourbigbook --generate default
: a good starter template that illustrates many key OurBigBook featuresourbigbook --generate min
: a minimal template that is still saneourbigbook --generate subdir
: a template in which OurBigBook source is located a subdirectorydocs/
:This template illustrates that everything works exactly as if OurBigBook source were in the git repository toplevel.This is a convenient setup for programming projects that want to use OurBigBook for their documentation without polluting their toplevel.End users almost never want this, because it means that to have a sane setup you need to:so maybe we should just get rid of that option and just ensure that we can provide an up-to-date working template for the latest relase.
- install OurBigBook globally with
npm install -g ourbigbook
- generate the template
- then install OurBigBook locally again with
npm install
For now we are keeping this as it is useful to automate the updating of templates during the release procedure.You can get an overview of all macros in JSON format with:ourbigbook --help-macros
Give multiple times to enable a list of certain types of logs to stderr help debugging, e.g.:Note that this follows commander.js' insane variadic argumentso syntax, and thus the./ourbigbook --log ast tokens -- README.bigb
--
is required above. If you want to omit it for a single value you have to add the=
sign as in:./ourbigbook --log=ast README.bigb
Values not documented in other sections:
ast
: the full final parsed abstract syntax tree as JSONast-simple
: a simplified view of the abstract syntax tree with one AstNode or AstArgument per line and showing only the most important fieldsast-pp-simple
: view snapshots of the various abstract syntax tree post process stages, more info at: conversion process overviewast-inside
: print the AST from inside theourbigbook.convert
call before it returns.This is useful to debug the program ifourbigbook.convert
blows up on the next stages before returning.db
: show database transactions done by OurBigBook, to help debug stuff like cross file referencesmem
: show process memory usage as per Node.js'process.memoryUsage()
after each--log perf
step: https://stackoverflow.com/questions/12023359/what-do-the-return-values-of-node-js-process-memoryusage-stand-for. Implies--log perf
.To use this options, you must run OurBigBook with the--expose-gc
command line option, e.g. with:node --expose-gc $(which ourbigbook) myfile.bigb
parse
: parsing stepstokenize
: tokenization stepstokens
: final parsed token streamtokens-inside
: likeast-inside
but for tokens.Also adds token index to the output, which makes debugging the parser way easier.This nifty little option outputs to stderr what the header graph looks like!It is a bit like a table of contents in your terminal, for when you need to have a look at the outline of the document to decide where to place a new header, but are not in the mood to open a browser or use the browser editor with preview.Sample output excerpt for this document:= h1 ourbigbook == h2 1 quick-start == h2 2 design-goals === h3 2.1 saner === h3 2.2 more-powerful == h2 3 paragraphs == h2 4 links
This option can also serve as a debug tool for header tree related features (confession: that was its original motivation!).TODOprint performance statistics to stderr, for examplecould output:./ourbigbook --log=perf README.bigb
which shows how long different parts of the conversion process took to help identify bottlenecks.perf start: 181.33060800284147 perf tokenize_pre: 181.4424349963665 perf tokenize_post: 318.333980999887 perf parse_start: 319.1866770014167 perf post_process_start: 353.5477180033922 perf post_process_end: 514.1527540013194 perf render_pre: 514.1708239987493 perf render_post: 562.834307000041 perf end: 564.0349840000272 perf convert_input_end 566.1234430000186 perf convert_path_pre_sqlite 566.1564619988203 perf convert_path_pre_sqlite_transaction 566.2528780028224 perf convert_path_post_sqlite_transaction 582.256645001471 perf convert_path_end 582.3469280004501
This option can also be useful to mark phases of the conversion to identify from which phase other logs are coming from, e.g. if we wanted to know which part of the conversion is making a ton of database requests we could run:and we would see the database requests made at each conversion phase.ourbigbook --log db perf -- README.bigb
Note that--log perf
currently does not take sub-converts into account, e.g. include and\OurBigBookExample
both call the toplevel conversion functionconvert
, and therefore go through all the conversion intervals, but we do not take those it account, and just dump them all into the same toplevel interval that they happen in, currently betweenpost_process_start
andpost_process_end
.Don't use the ID database during this run. This implies that the on-disk database is not read, and also not written to. Instead, a temporary clean in-memory database is used.If not given, cross references render with the.html
extension as in:<a href=not-readme.html#h2-in-not-the-readme>
This way, those links will work when rendering locally to.html
files which is the default behaviour of:ourbigbook .
If given however, the links render without the.html
as in:which is what is needed for servers such as GitHub Pages, which automatically remove the<a href=not-readme#h2-in-not-the-readme>
.html
extension from paths.This option is automatically implied when publishing to targets that remove the.html
extension such as GitHub pages.Only extract IDs to fill the ID database, don't render. This saves time if you only want to render a single file which has references to other files without getting any errors.OurBigBook stores the timestamp of the last successful:for each file.For ID extraction, we always skip the extraction if the filesystem timestamp of a source file is older than the last successful extraction.For render:
- we also skip rendering by default when you invoke ourbigbook on a directory, e.g.
ourbigbook .
, as this greatly speeds up the interactive error fixing turnaround time- we always re-render fully when you specify a single file, e.g.
ourbigbook path/to/README.bigb
However, note that skipping renders, unlike for ID extraction, can lead to some outdated pages.This option disables the timestamp skip for rendering, so ensure that you will get a fully clean updated render.E.g. consider if you had two files:file1.bigb= File 1 == File 1 1
file2.bigb= File 2 == File 2 1 \x[file-1-1]
We then do the initial conversion:we see output like:ourbigbook .
indicating full conversion without skips.extract_ids file1.bigb extract_ids file1.bigb finished in 45.61287499964237 ms extract_ids file2.bigb extract_ids file2.bigb finished in 15.163879998028278 ms render file1.bigb render file1.bigb finished in 23.21016100049019 ms render file2.bigb render file2.bigb finished in 25.92908499762416 ms
But then if we just modify fil1.bigb as:the following conversion with= File 1 == File 1 1 hacked {id=file-1-1}
ourbigbook .
would look like:and because we skippedextract_ids file1.bigb extract_ids file1.bigb finished in 45.61287499964237 ms extract_ids file2.bigb extract_ids file2.bigb skipped by timestamp render file1.bigb render file1.bigb finished in 41.026930000633 ms render file2.bigb render file2.bigb skipped by timestamp
file2.bigb
render, it will still have the outdated "File 1 1" instead of "File 1 1 hacked".We could in principle solve this problem by figuring out exactly which files need to be changed when a given ID changes, and we already have to solve a similar problem due to query bundling. Also, this will need to be done sonner or later for the OurBigBook Web. But lazy now: https://github.com/cirosantilli/ourbigbook/issues/207, this is hard stuff.Set a custom output directory for the conversionIf not given, the project toplevel directory is used.Suppose we have an input file./test.bigb
. Then:places its output at:ourbigbook --outdir my_outdir test.bigb
my_outdir/test.html
The same would happen if we instead did a full directory conversion as in:The output would also be placed inourbigbook --outdir my_outdir .
my_outdir/test.html
.This option also relocates theout
directory to the target destination, e.g.:would generate:ourbigbook --outdir my_outdir test.bigb
This means that the source tree remains completely clean, and every output and temporary cache is put strictly under the selectedmy_outdir/out
--outdir
.Save the output to a given file instead of outputting to stdout:./ourbigbook --outfile not-readme.html not-readme.bigb
The generated output is slightly different than that of:because with./ourbigbook not-readme.bigb > not-readme.html
--outfile
we know where the output is going, and so we can generate relative includes to default CSS/JavaScript files.Default:html
output format.The default output format. Web pages!!!Outputs as OurBigBook Markup, i.e. the same format as the input itself!While using-O bigb
is not a common use case, the existence of this format has the following applications:
- automatic source code formatting e.g. with
--format-source
. The recommended format, including several edge cases, can be seen in the test file test_bigb_output.bigb, which should be left unchanged by abigb
conversion.- manipulating source code on OurBigBook Web to allow editing either individual sections separatelly, or multiple sections at once
- this could be adapted to allows us to migrate updates with breaking changes to the source code more easily. Alternatively on OurBigBook Web, we might just start storing the AST instead of source, and just rendering the source whenever users want to edit it.
Can be tested interactively with:ourbigbook --no-db -O bigb --stdout --log=ast-simple test_bigb_output.bigb
One important property of thebigb
conversion is that is must not alter the AST, and therefore neither the final output, in any way.One good test is:ourbigbook README.bigb && mv out/html/index.html out/html/old.html && ourbigbook --format-source README.bigb && ourbigbook README.bigb && diff -u out/html/old.html out/html/index.html
This was tracked at: https://github.com/cirosantilli/ourbigbook/issues/83This output format is mostly a joke.It is an intermediate step in automatic ID from title, that unlike HTML output does not have any tags.So for example, converting:with the\i[asdf]
id
output format produces simply:instead of the HTML output:asdf
<i>asdf</i>
This conversion type is useful in situations that users don't expect conversion to produce any HTML tags. For example, you could create a header:and then following the automatic ID from title algorithm, that header would have the more commonly desired ID= My \i[asdf]
my-asdf
, and notmy-<i>asdf</i>
.Besides being more intuitive, this also guarantees greater format portability, in case we ever decide to support other output formats!We decided to expose this format from the CLI just for fun, as it posed no extra work at all as it is treated internally exactly like any other conversion format.And it also was a good start generalizing OurBigBook to multiple outputs, as this is a simple format.The conversion is very simplistic, it basically just pastes thecontent
of most macros. Important exceptions to that include:
\x
: see\x
id
output formatSo for example if we had:then the image ID would be= Title with more {id=title} \Image[my.png]{title=My \i[nice] \x[title]
image-my-nice-title
:
image-
automatically added prefixmy
: literal textnice
: extract the content of the italictitle
: note how we use thehref
title
instead of what is actually rendered,Title with more
, because\x
is treated somewhat specially for performance reasonsStuff that is primarily non-textual like images is just completely removed. We could put effort in outputting their title correctly, but meh, not worth the effort.\x
useshref
if the content is not given explicitly.Previously, if\x
didn't have a content, we were actually rendering the\x
to calculate the ID. But then we noticed that doing so would require another parse pass, so we just went for this simpler approach. This is closely linked to\x
withintitle
restrictions.For example in:note that the ID of the image is= Animal \x[image-i-like-dog] \Image[dog.jpg] {title=I like \x[dog]} == Dog hacked {id=dog}
image-i-like-dog
, i.e.\x[dog]
is not expanded toDog hacked
, thehref
is used directly.If you wantedimage-i-like-dog-hacked
instead, you would need to explicitly give it as in:= Animal \x[image-i-like-dog-hacked] \Image[dog.jpg] {title=I like \x[dog][dog hacked]} == Dog hacked {id=dog}
For similar reasons as the above,{p}
inflection with the\x
p
argument is not considered either, e.g. you would have:and not:= Animal \x[image-i-like-dog] \Image[dog.jpg] {title=I like \x[dog]{p}} == Dog
\x[image-i-like-dogs]
This can however be worked around with the\x
magic
argument as in;= Animal \x[image-i-like-dogs] \Image[dog.jpg] {title=I like <dogs>} == Dog
One day, one day. Maybe.OurBigBook tooling is so amazing that we also take care of the HTML publishing for you!Once publish target is properly setup, all you have to do is run:and your changes will be published to the default target specified ingit add README.bigb git commit -m 'more content!' ourbigbook --publish
ourbigbook.json
.If not specified, the default target is publish to GitHub Pages.Only changes committed to Git are pushed.Files thatourbigbook
knows how to process get processed and only their outputs are added to the published repo, those file types are:Every other Git-tracked file is pushed as is.
.bigb
files are converted to.html
.scss
files are converted to.css
When--publish
is given, stdin input is not accepted, and so the current directory is built by default, i.e. the following two are equivalent:./ourbigbook --publish ./ourbigbook --publish .
Publishing only happens if the build has no errors.GitHub pages is the default OurBigBook publish target.Since that procedure is so important, it is documented directly at: play with the template.If you want to publish your root user page, which appears at/
(e.g. https://github.com/cirosantilli/cirosantilli.github.io for the usercirosantilli
), GitHub annoyingly forces you to use themaster
branch for the HTML output:This means that you must place your.bigb
input files in a branch other thanmaster
to clear upmaster
for the generated HTML.ourbigbook
automatically detects if your repository is a root repository or not by parsinggit remote
output, but you must setup the branches correctly yourself.So on a new repository, you must first checkout to a different branch as in:or to move an existing repository to a non-master branch:git init git checkout -b dev
git checkout -b dev git push origin dev:dev git branch -D master git push --delete origin master
You then will also want to set your default repository branch todev
in the settings for that repository: https://help.github.com/en/github/administering-a-repository/setting-the-default-branchLike-p, --publish
, but also automatically:
git add -u
to automatically add change to any files that have been previously git trackedgit commit -m <commit-message>
to create a new commit with those changesThis allows you to publish your changes live in a single command such as:ourbigbook --publish-commit 'my amazing change' .
With great power comes great responsibility of course, but who cares!Split each header into its own separate HTML output file.This option allows you to keep all headers in a single file, which is much more convenient than working with a billion separate source files, and let them grow naturally as new information is added, but still be able to get a small output page on the rendered website that contains just the content of the given header. Such split pages load faster on the browser, and might get better Google PageRank.For example given an input file calledhello.bigb
and containing:a conversion command:= h1 h1 content. A link to another section: \x[h1-1]. == h1 1 h1-1 content. == h1 1 1 h1-1-1 content. == h1 1 2 h1-1-2 content.
would produce the following output files:ourbigbook --split-headers hello.bigb
hello.html
: contains the entire rendered document as usual.Remember that this is calledhello.html
instead ofh1.html
because the toplevel header ID is automatically derived from its filename.Each header contains a on-hover link to the single-file split version of the header.hello-split.html
: contains only the contents directly under= h1
, but not under any of the subheaders, e.g.:The
h1 content.
appears in this rendered outputh1-1-1
does not appear in this rendered output-split
suffix can be customized with the\H
splitSuffix
argument option. The-split
suffix is appended in order to differentiate the output path fromhello.html
h1-1.html
,h1-1-1.html
,h1-1-2.html
: contain only the contents direcly under their headers, analogously tohello-split.html
, but now we don't need to worry about the input filename and collisiont, and just directly use the ID of each header--split-headers
is implied by-p, --publish
: the published website will automatically get the split pages. There is no way to turn it off currently. A pull request would be accepted, especially if it offers aourbigbook.json
way to do it. Maybe it would be nice to have a more generalized way of setting any CLI option equivalent from theourbigbook.json
, and an optioncli
vscli-publish
so thatcli-publish
is publish only. Just lazy for now/not enough pressing use case met.By default, all cross references point to the non-split version of headers, including those found in split headers.The rationale for this is that it gives readers the most context around the header by simply scrolling.For example, considering the example document at-S
,--split-headers
, cross references such as\x[h1-1]
would point:The same applies to cross file references when there are multiple input files.
- from the non-split
hello.html
to the section in the current non-split file#h1-1
- from split
hello-split.html
to the same section in non-split file withhello.html#h1-1
In order to make the split version be the default for some headers, you can use the\H
splitDefault
argument.When converting a file, output output to stdout in addition to outputting to a file:The regular output file is also saved.convert --stdout input.bigb
Cannot be used when converting a directory.Select a custom Liquid template file for the output.The recommended file name for this file is:which is already ignored by default from the published output.main.liquid.html
This repository has a samplemain.liquid.html
at: main.liquid.html.If not given, the default template at one point was:This will get out of sync sooner or later with the code, but this should still serve as a good base example for this documentation.<!doctype html> <html lang=en> <head> <meta charset=utf-8> <title>{{ title }}</title> <style>{{ style }}</style> </head> <body class="ourbigbook"> {{ body }} </body> </html>
Defined variables:
body
: the rendered bodygit_sha
: SHA of the latest git commit of the source code if in a git repositoryhtml_ext
:.html
for local renders, empty for server renders.So e.g. to link to an IDmyid
you can use:<a href="{{ root_relpath }}myid{{ html_ext }}">
This will ideally be replaced with a more generic link to arbitrary ID mechnism at some point: https://github.com/cirosantilli/ourbigbook/issues/135input_path
: path to the OurBigBook Markup source file relative to the project toplevel directory that generated this output, e.g.path/to/myfile.bigb
root_page
: relative path to the toplevel page, e.g. eitherindex.html
,../index.html
locally or./
,../
on server oriented rendereingroot_replath
: relative path from the rendered output to the toplevel directory.This allows for toplevel resources like CSS to be found seamlessly form inside subdirectories, specially when rendering locally.For example, for the toplevel CSSmain.css
which is generated from main.scss, we can use:<link rel="stylesheet" type="text/css" href="{{ root_relpath }}main.css">
Then, when a file is locally, for example under a subdirectorymysubdir/myfile.html
, OurBigBook will set:giving the desired:root_relpath=../
<link rel="stylesheet" type="text/css" href="../main.css">
And if the output path were instead justmyohterfile.html
,root_replath
expands to an empty string, giving again the correct:<link rel="stylesheet" type="text/css" href="main.css">
This will ideally be replaced with a more generic link to arbitrary ID mechnism at some point: https://github.com/cirosantilli/ourbigbook/issues/135style
: default OurBigBook stylesheetstitle
We pick Liquid because it is server-side safe: if we ever some day offer a compilation service, Liquid is designed to prevent arbitrary code execution and infinite loops in templates.This option enables actions that would allow arbitrary code execution, so you should only pass it if you trust the repository author. Enabled functionality includes:Don't quitourbigbook
immediately.Instead, watch the selected file or directory for changes, and rebuild individual files when changes are detected.Watch every.bigb
file in an entire directory:When a directory is given as the input path, this automatically first does an ID extraction pass on all files to support cross file references.ourbigbook --watch .
Now you can just edit any OurBigBook file such hasREADME.bigb
, save the file in your editor, and refresh the webpage and your change should be visible, no need to run aourbigbook
command explicitly every time.Exit by entering Ctrl + C on the terminal.Watch a single file:When a single file is watched, the reference database is not automatically updated. If it is not already up-to-date, you should first update it with:ourbigbook --watch README.bigb
otherwise you will just get a bunch of undefined ID errors every time the input file is saved.ourbigbook .
TODO: integrate Live Preview: https://asciidoctor.org/docs/editing-asciidoc-with-live-preview/ to also dispense the browser refresh.Sync local directory to OurBigBook Web instead of doing anything else.The source code is uploaded, and conversion to HTML happens on the server, no conversion is done locally.This option is not amazing right now. It was introduced mostly to allow uploading the reference demo content from https://cirosantilli.com to https://ourbigbook.com/cirosantilli, and it is not expected that it will be a major use case for end users for a long time.Some important known limitations:
- file renaming does not work, it will think that you are creating a new file and blows up duplicates
- everything file is rebuilt every time, even if it was not modified. To avoid this, we would need to use or reimplement parts of Git, this is the kind of stuff that Git does very well.
- if there's an error in a later file, the database is still modified by the previous files, i.e. there is no atomicity. A way to improve that would be to upload all files to the server in one go, and let the server convert everything in one transaction. However, this would lead to a very long server action, which would block any other incoming request (I tested, everything is single threaded)
Set a custom URL for-W
,--web
from the command line. If not given, the canonical https://ourbigbook.com is used. This option is used e.g. for testing locally e.g. with:ourbigbook --web --web-url http://localhost:3000
Set the username for-W
,--web
from the command line, e.g.:If not given, you will be prompted for it from the command line.ourbigbook --web --web-url http://localhost:3000 --web-username donald-trump
OurBigBook configuration file that affects the behaviour of ourbigbook for all files in the directory.ourbigbook.json
not used for input from stdin, since we are mostly doing quick tests in that case.Whileourbigbook.json
is optional, it is used to determine the toplevel directory of a OurBigBook project, which has some effects such as those mentioned at the toplevel index file.Therefore, it is recommended that you always have aourbigbook.json
in your project's toplevel directory, even if it is going to be an empty JSON containing just:{}
For example, if you convert a file in a subdirectory such as:thenourbigbook subdir/notindex.bigb
ourbigbook
walks up the filesystem tree looking forourbigbook.json
, e.g.:and so on.
- is there a
./subdir/ourbigbook.json
?- otherwise, is there a
./ourbigbook.json
?- otherwise, is there a
../ourbigbook.json
?- otherwise, is there a
../../ourbigbook.json
?If we reach the root path/
and noourbigbook.json
is found, then we understand that there is noourbigbook.json
file present.List of paths relative to the project toplevel directory that OurBigBook CLI will ignore.Useful if your project has a large directory that does not contain OurBigBook sources, and you don't want OurBigBook to mess with it.Only ignores recursive conversions, e.g. given:doing:"ignore": [ "web" ]
skips that directory, butourbigbook .
converts it because it was explicitly requested.ourbigbook web/myfile.bigb
TODO: also ignore during-w
,--watch
.Dictionary of options that control automatic ID from title generation.Iftrue
, does Latin normalization on the title.Default:true
.ASCII normalization is a custom OurBigBook defined normalization that converts many characters that look like latin characters into latin characters.For now, we are using thedeburr
method of Lodash: https://lodash.com/docs/4.17.15#deburr, which only affects latin-like characters.One notable effect is that it converts variants of ASCII letters to ASCII letters. E.g.é
toe
removing the accent.This operation is kind of a superset of Unicode normalization acting only on Latin-like characters. where unicode basically only removes things like diactricts.OurBigBook normalization also does other natural transformations that Unicode does not do, e.g.æ
toae
.Iftrue
, does Punctuation normalization on the title.Default:true
.Some selected punctuation marks are automatically converted into their dominant corresponding pronunciations. These are:
%
:percent
&
:and
+
:plus
@
:at
Dashes are added around the signs if needed, e.g.:
C++
:c-plus-plus
Q&A
:q-and-a
Folding@home
:folding-at-home
Dictionary of lint options to enable. OurBigBook tries to be strict about forcing specific styles by default, e.g. forbids triple newline paragraph. But sometimes we just can't bear it :-)Possible values:
parent
: forces headers to use\H
parent
argument to specify their levelnumber
: forces headers to not use\H
parent
argument to specify their level, i.e. to use a number or a number of=
You should basically always set either one of those on any serious project. Forgetting aparent=
in a project that usesparent=
everywhere else is a common cause of build bugs, and can be hard to debug without this type of linting enabled.Possible values:
child
: forbids headers from using the\H
tag
argument. They should instead use the\H
child
argument.tag
: forbids headers from using the\H
child
argument. They should instead use the\H
tag
argument.This dictionnary stores options related to headers.Sets the default\H
numbered
argument argument of the toplevel headers of each source file.Note that since the option is inherited by descendants, this can also affect the rendering of ancestors.https://github.com/cirosantilli/ourbigbook/issues/188 contains a proposal to instead inherit this property across includes.This options is exactly analogous to thenumbered
option, but it affects the\H
splitDefault
argument instead of the\H
numbered
argument.If given, the toplevel output of each input source is always non-split, and a split version is not generated at all.This overrides the\H
splitDefault
argument for the toplevel header.E.g.:ourbigbook.json{ "h": { "splitDefault": true, "splitDefaultNoToplevel": true, } }
my-first-header.bigb= My first header == My second header
When converted with:would lead only to two output files:ourbigbook --split-headers my-first-header.bigb
- my-first-header: not split
- my-second-header: split
WithoutsplitDefaultNoToplevel
we would instead have:
- my-first-header: split
- my-first-header-nosplit: not split
- my-second-header: split
The initial use case for this was in OurBigBook Web. If we didn't do this, then there would be two versions of every article at the toplevel of a file: split and nosplit.This would be confusing for users, who would e.g. see two new articles on the article index every time they create a new one.It would also mean that metadata such as comments would be visible in two separate locations.So instead of filtering the duplicate articles on every index, we just don't generate them in the first place.The general format ofmedia-providers
looks like:"media-providers": { "github": { "default-for": ["image"], // "all" to default for both image, video and anything else "path": "data/media/", // data is gitignored, but should not be nuked like out/ "remote": "cirosantilli/ourbigbook-media" }, "local": { "default-for": ["video"], "path": "media/", }, "youtube": {} }
Properties that are valid for every provider:
default-for
: use this provider as the default for the given types of listed macros.The first character of the macros are case insensitive and must be given as lower case. Therefore e.g.:
image
applies to bothimage
andImage
- giving
Image
is an error because that starts with an upper case charactertitle-from-src
(bool
): extract thetitle
argument from thesrc
by default for media such as images and videos as if thetitleFromSrc
macro argument had been given, see also: Section 4.1.8.2. "Image ID"Direct children of media-providers and subproperties that are valid only for them specifically:
local
: tracked in the current Git repository as mentioned at Section 4.1.8.3.1. "Store images inside the repository itself"
path
: location of the cloned local repository relative to the root the main repositorygithub
: tracked in a separate Git repository as mentioned at Section 4.1.8.3.2. "Store images in a separate media repository"
path
: analogous topath
forlocal
: a local location for this GitHub provider, where the repository can optionally be cloned.TODO implement the amazing behaviour described below:If this directory exists then local renders automatically use it instead of linking to the GitHub resources, therefore allowing you to develop locally without the internet and Still see images.Then, during deployment, the GitHub version is used on the render regardless. Furthermore, we also automaticallygit push
this repository during deployment to ensure that any asset changes will be available.This path is ignored from OurBigBook conversion as if added toignore
, and is not added to the final output, because you are already going to have a copy of it.The sanest approach is generally to track this directory as a Git submodule as mentioned at: store images in a separate media repository and track it as a git submoduleremote
:<github-username>/<repo-name>
youtube
: YouTube videosDefault:false
For example:would be placed under:ourbigbook hello.bigb
instead ofout/html/hello.html
./hello.html
.Advantages:Disadvantages:
- the source tree becomes cleaner, especially when using
-S
,--split-headers
which can produce hundreds of output files from a single input file- if you want to track several
.html
source files in-tree, you don't need to add an exception to each of of them on the.gitignore
as:*.html !/main.liquid.html
- you have to type more to open each output file on the terminal
Generally we recommend starting out without this option, and enabling it as the project grows and you feel like the.html
floating around in-tree are starting to annoy you too much.This option is always forced tofalse
when--outdir <outdir>
is given.Implemented at: https://github.com/cirosantilli/ourbigbook/issues/163Path of a script that gets executed before runningourbigbook --publish
.The script arguments are:
the publish output directory.That directory is guaranteed to exist whenprepublish
is called.Forgit
-based publish targets, all files are almost ready in there, just waiting for agit add .
that followsprepublish
.This means that you can use this script to place or remove files from the final publish output.If theprepublish
script returns with a non-zero exit value, the publish is aborted.Generate custom redirects.For example:produces a file in the output called"redirects": [ ["cirodown", "ourbigbook"] ],
cirodown.html
that redirects toourbigbook.html
.Absolute URLs are also accepted, e.g.:produces a file in the output called"redirects": [ ["ourbigbook", "https://docs.ourbigbook.com"] ],
ourbigbook.html
that redirects tohttps://docs.ourbigbook.com
.When dealing with regular headers, you generally don't want to use this option and instead use the\H
synonym
argument, which already creates the redirection for you.This JSON option can be useful however for dealing with things that are outside of your OurBigBook project.For example, at one point, this project renamed the repository https://github.com/cirosantilli/cirodown to https://github.com/cirosantilli/ourbigbook.Unfortunately, GitHub Pages does not generate redirects like github.com itself.So in this case, we've added to theourbigbook.json
of the toplevel user repository https://github.com/cirosantilli/cirosantilli.github.io the lines:which produces a file in the output called"redirects": [ ["cirodown", "ourbigbook"] ],
cirodown.html
that redirects toourbigbook.html
.In this case,cirodown
andourbigbook
don't have to be any regular IDs present in the database, those strings are just used directly.TODO ideally we should check for conflicts with regular output from split headers IDs or their synonyms. But lazy.The main entry point for the JavaScript API is theourbigbook.convert
function.An example can be seen under lib_hello.js.Note that while doing a simple conversion is easy, things get harder if you want to take multifile features in consideration, notably cross file reference internals.This is because these features require interacting with the ID database, and we don't do that from the defaultourbigbook.convert
API because different deployments will have very different implementations, notably:
- local Node.js run uses SQLite, an implementation can be seen in the ourbigbook file class
SqliteDbProvider
- the in-browser version that runs in the browser editor of the OurBigBook Web makes API calls to the server
OurBigBook Web is the project that powers https://OurBigBook.com. Some more motivation is available at: https://cirosantilli.com/ourbigbook-com.OurBigBook Web is a regular databased backed dynamic website. This is unlike the static websites generated by OurBigBook CLI. Static websites are simpler and cheaper to run, but they are harder to setup for non-programmers, and they cannot have multiuser features such as likes, comments, and "view versions of this article by other users", which is are core functionality of the OurBigBook Project.The source for OurBigBook Web, source code is located under web/. OurBigBook Web can be seen as a separate Node.js package which uses the OurBigBook Library as a dependency.OurBigBook Web was originally forked from the following starter boilerplate: https://github.com/cirosantilli/node-express-sequelize-realworld-example-app. We are trying to keep tech synced as much as possible between both projects, since the boilerplate is useful as a tech demo to quickly try out new technologies in a more minimal setup.It is highly recommended that you use the exact same Node.js and NPM versions as given under package.jsonengines.js
entry. The best way to do that is likely to use NVM as explained at: https://stackoverflow.com/questions/16898001/how-to-install-a-specific-version-of-node-on-ubuntu/47376491#47376491First time setup:where:cd ourbigbook && npm run link && npm run build-assets && cd web/ && npm install && ./bin/generate-demo-data.js --users 2 --articles-per-user 10 # Or short version: #./bin/generate-demo-data.js -u 2 -a 10
We also provide a shortcut for that setup as:
npm run build-assets
needs to be re-run if any assets (e.g. CSS or Js file mentioned at overview of files in this repository) on the./ourbigbook/
toplevel are modified. No need to re-run it for changes underweb/
.- web/bin/generate-demo-data.js also creates the database and is not optional. If you want to start with an empty database instead of the demo one, you can run instead web/bin/sync-db.js:
./bin/sync-db
npm run web-setup ./bin/generate-demo-data.js --users 2 --articles-per-user 10
After this initial setup, run the development server:npm run dev
And the website is now running at http://localhost:3000. If you created the demo data, you can login with:
- email:
user0@mail.com
,user1@mail.com
, etc. password:asdf
Custom demo user passwords can be set by exporting theOURBIGBOOK_DEMO_USER_PASSWORD
variable, e.g.:this is useful for production.OURBIGBOOK_DEMO_USER_PASSWORD=qwer ./bin/generate-demo-data.js -u 2 -a 10
To run on a different port use:PORT=3001 npm run dev
We also offer shortcuts on toplevel for thenpm install
andnpm run dev
commands so you can skip thecd web
for those:npm install npm run dev
Whenever you save any changes to the backend server, we listen to this and automatically restart the server, so after a few seconds or less, you can refresh the web page to obtain the backend update.For frontend, changes are automatically recompiled by the webpack development server, so you can basically just refresh pages and they will be updated straightaway.You can generate demo data for OurBigBook Web with web/bin/generate-demo-data.js, e.g.:cd web ./bin/generate-demo-data --users 2 --articles-per-user 10
Every time this is run, it tries to update existing entities such as users and articles first, and only creates them if they don't exist. This allows us to update all demo data on a live website that also has users without deleting any user data.Note however that if you ever increase the ammount of demo users, you might overwrite real user data. E.g. if you first do:and then some time later:./bin/generate-demo-data --users 2 --articles-per-user 10
it is possible that some real user will have taken up the username that we use for the third user, which did not exist previously, and then hacks their articles away. So never ever do that! Just stick to the default values in production../bin/generate-demo-data --users 4 --articles-per-user 10
As a safeguard, to be able to run this in production you have to also pass the--force-production
flag;./bin/generate-demo-data --users 2 --articles-per-user 10 --force-production
To first fully clear the database, including any real user data, before doing anything else, use--clear
, e.g.:./bin/generate-demo-data --users 4 --articles-per-user 10 --clear
To clear the database and start with an empty database use--empty
:./bin/generate-demo-data --empty
Shortcut:DEBUG='sequelize:sql:*' npm run dev
npm run devs
First run the first time setup from local development server.Then, when running for the first time, or whenever frontend changes are made, you need to create optimized frontend assets with:before you finally start the server each time with:npm run build-dev
npm start
This setup runs the Next.js server in production mode locally. Running this setup locally might help debug some front-end deployment issues.Building like this notably runs full typescript type checking, which is a good way to find bugs early.But otherwise you will just normally use the local run as identical to deployment as possible setup instead for development, as that makes iterations quicker are you don't have to re-run the slownpm run build-dev
command after every frontend change.build-dev
is needed instead ofbuild
because it usesNODE_ENV_OVERRIDE
which is needed because Next.js forcesNODE_ENV=production
and wontfixed changing it: https://github.com/vercel/next.js/issues/4022#issuecomment-374010365, and that would lead to the PostgreSQL database being used, instead of the SQLite one we want.build
runsnpm run build-assets
on toplevel which repacks ourbigbook itself and is a bit slow. To speed things up during the development loop, you can also use:instead, which builds only the stuff undernpm run build-dev-nodeps
web/
.TypeScript type checking an also be run in isolation as mentioned at Section 7.13. "OurBigBook Web TypeScript type checking" with:npm run tsc
Here we use PostgreSQL instead of SQLite with the prebuilt static frontend.For when you really need to debug some deployment stuff locally.Setup:createdb ourbigbook psql -c "CREATE ROLE ourbigbook_user with login password 'a'" psql -c 'GRANT ALL PRIVILEGES ON DATABASE ourbigbook TO ourbigbook_user' echo "SECRET=$(tr -dc A-Za-z0-9 </dev/urandom | head -c 256)" >> .env
Run after every modificationand then visit the running website at: http://localhost:3000/npm run build-prod npm run start-prod
To optionally nuke the database and create the demo data:or alternatively to start from a clean database:npm run seed-prod
psql -c "DROP DATABASE ourbigbook" createdb ourbigbook psql -c 'GRANT ALL PRIVILEGES ON DATABASE ourbigbook TO ourbigbook_user'
You can inspect the database interactively with:and then running SQL commands.psql ourbigbook
If you have determined that a bug is PostgreSQL specific, and it is easier to debug it interactively, first create the database as mentioned at local run as identical to deployment as possible and then:or shortcut for the run:OURBIGBOOK_POSTGRES=true ./bin/generate-demo-data.js OURBIGBOOK_POSTGRES=true npm run dev
npm run dev-pg
Note that doingsync-db
also requiresNODE_ENV=production
as in:because we have to shell out to the ugly migration CLI, and that only understandsNODE_ENV=production OURBIGBOOK_POSTGRES=true ./bin/sync-db.js
NODE_ENV
.Got it running perfectly at as of April 2021 https://ourbigbook.com with the following steps.Initial setup for a Heroku project calledourbigbook
:sudo snap install --classic heroku heroku login heroku git:remote -a ourbigbook git remote rename heroku prod # Automatically sets DATABASE_URL. heroku addons:create -a ourbigbook heroku-postgresql:hobby-dev # We need this to be able to require("ourbigbook") heroku config:set -a ourbigbook SECRET="$(tr -dc A-Za-z0-9 </dev/urandom | head -c 256)" # Password of users generated with ./web/bin/generate-demo-data heroku config:set -a ourbigbook OURBIGBOOK_DEMO_USER_PASSWORD="$(tr -dc A-Za-z0-9 </dev/urandom | head -c 20)" # You can get it later to login with the demo users from the Heroku web interface
Additionally, you also need to setup the PostgreSQL test database for both OurBigBook CLI and OurBigBook Web as documented at: test system and local run as identical to deployment as possible, because we will be running tests on PostgreSQL before pushing:createdb ourbigbook createdb ourbigbook_cli psql -c "CREATE ROLE ourbigbook_user with login password 'a'" psql -c 'GRANT ALL PRIVILEGES ON DATABASE ourbigbook TO ourbigbook_user' psql -c 'GRANT ALL PRIVILEGES ON DATABASE ourbigbook_cli TO ourbigbook_user'
Then deploy with:cd web npm run deploy-prod
Get an interactive shell on the production server:From there you could then for example update the demo data with:heroku run -a ourbigbook bash
This should in theory not affect any real user data, only the demo articles and users, so it might be safe. In theory!cd web bin/generate-demo-data.js --force-production
Alternatively, we could do this at once with;heroku run -a ourbigbook web/bin/generate-demo-data.js --force-production
Drop into a PostgreSQL shell on production:Of course, any writes could mean loss of user data!heroku psql -a ourbigbook
Run a query directly from your terminal:heroku psql -a ourbigbook -c 'SELECT username,email FROM "User" ORDER BY "createdAt" DESC LIMIT 50'
If some spurious bugs crashes the server, you might want to restart it with:heroku restart -a ourbigbook
Download a dump of the database as per https://devcenter.heroku.com/articles/heroku-postgres-import-export:Restore it to a local database:heroku pg:backups:capture -a ourbigbook heroku pg:backups:download -a ourbigbook
The local password ispg_restore --verbose --clean --no-acl --no-owner -h localhost -U ourbigbook_user -d ourbigbook latest.dump
a
(single letter). TODO automate login.The 10k rows of the free plan are easy to reach, this procedure can be used to upgrade:The domainourbigbook.com
was leased from: https://porkbun.com/. Unfortunatly, HTTPS on Heroku with a custom domain requires using a paying tier, so we upgraded from the free tier to the cheapest paid tier, "Hobby Project", to start with: https://stackoverflow.com/questions/52185560/heroku-set-ssl-certificates-on-free-planOn the Porkbun web UI, we added a DNS record of type :whereALIAS ourbigbook.com <heroku-id>.herokudns.com
heroku-id
was obtained from:and we removed all otherheroku domains:add ourbigbook.com heroku domains
ALIAS
/CNAME
records from Porkbun.