LLMs only produce markdown [1], usually Math is wrapped with KaTeX [2]. The rendering happens in the Web UI. If some math failed to render, you can copy the code and paste to latex-sandbox [3] and fix yourself.
[1] https://docs.github.com/en/get-started/writing-on-github/wor...
When you say consistently render what do you mean? Like without any sort of formatting issue? If thats the case I would lower your expectations a bit almost always as with any bit of copy editing regardless of using ai or not theres going to be issues to review. I have a claude skill using tectonic I believe and it works pretty well, and then I wrote hooks to make sure if certain things are going to be written it does it in a certain way.
It seems that GitHub gist can renders some of \LaTeX, but not perfect.
https://gist.github.com/ontouchstart/bcffb186a753c5b75522fc8...
honestly the magic phrase ive found is "use aligned LaTeX delimiters and assume the renderer supports MathJax." that one line cuts 80% of the inconsistency.
also: put it in your CLAUDE.md, not the system prompt. Claude reads that per-session and it sticks way better than hoping your prompt survives context compression. learned that the hard way after losing formmating mid-session one too many times.
Some things are just too hard for them to do reliably