Looking into this further, I think we may not be sacrificing any functionality with this change. If I'm reading the documentation correctly for the various wint_t functions in wctype.c (for example, iswupper), the unicode paths are only hit if setlocale was previously called (i.e. setlocale(LC_ALL, "en_US.utf8")). We don't export setlocale for wasm builds, so it actually seems very reasonable to treat these code paths as unreachable and just handle the ASCII cases as I've done in this PR.

These re-implementations will need to be tested further before I think this is ok to merge, but I believe the previous unicode concerns I raised are actually a non-issue.

A quick GH search shows that no parsers use setlocale at all, besides tree-sitter-haskell and tree-sitter-synquid. For both of these parsers, the calls to setlocale are gated behind #ifdef TREE_SITTER_DEBUG, so they're not actually used outside of development.

Oh, I actually thought those functions would use UTF-8 character categories by default. I think it would be better to keep the unicode support compiled in, and look into setting the locale somehow. I think those functions are used in some scanners where UTF8 support is intended.

Oh, I actually thought those functions would use UTF-8 character categories by default. I think it would be better to keep the unicode support compiled in, and look into setting the locale somehow. I think those functions are used in some scanners where UTF8 support is intended.

Gotcha. In that case, the recent additions to wctype.c should probably be moved to wctype.h as static inlines so that the wasi implementations are used. This will grow the stdlib size by about 4k to ~18,984. Using our own implementations for the functions in string.c brings the size back down to 15559. As long as those minimal impls are sufficient, that seems like a reasonable tradeoff.

Since we don't export setlocale, the default C locale should always be in use by wasm parsers. Given this, maybe we can use a pared down version of wasi's casemap table and its surrounding functionality to get some more size savings since we don't need to support other locales. Just out of curiosity, I tried adding setlocale to the export list (stdlibsymbols.txt) and run into some odd duplicate symbol errors:

Failed to compile the Tree-sitter Wasm stdlib:
wasm-ld: error: duplicate symbol: malloc
>>> defined in /tmp/stdlib-3eba72.o
>>> defined in /home/lillis/.cache/tree-sitter/wasi-sdk/bin/../share/wasi-sysroot/lib/wasm32-wasi/libc.a(dlmalloc.o)

wasm-ld: error: duplicate symbol: free
>>> defined in /tmp/stdlib-3eba72.o
>>> defined in /home/lillis/.cache/tree-sitter/wasi-sdk/bin/../share/wasi-sysroot/lib/wasm32-wasi/libc.a(dlmalloc.o)

wasm-ld: error: duplicate symbol: calloc
>>> defined in /tmp/stdlib-3eba72.o
>>> defined in /home/lillis/.cache/tree-sitter/wasi-sdk/bin/../share/wasi-sysroot/lib/wasm32-wasi/libc.a(dlmalloc.o)

wasm-ld: error: duplicate symbol: realloc
>>> defined in /tmp/stdlib-3eba72.o
>>> defined in /home/lillis/.cache/tree-sitter/wasi-sdk/bin/../share/wasi-sysroot/lib/wasm32-wasi/libc.a(dlmalloc.o)
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I also tested a rust wasm32-unknown-unknown build via the repro from #5158 and that use case is still supported with these changes.