Introducing concurrency mode in JsonStream. - #373

Introducing concurrency mode in JsonStream.

Performance gains are substantial while not adding too much complexity to the current source code. Although I strongly recommend using this new threaded version, the previous non-threaded version is still supported. The JsonStream API remains the same whether you are using the threaded version or not. The only difference for the end-user is the need to add -pthread flag to the compiler in order to enable the threaded version. In the case where the user would not specify the -pthread flag, the non-threaded version will be used. Note that threads are enabled by default with Visual C++ (MSVC).

Here is a chart comparing the speed of the different alternatives to parse a multiline JSON:

How it works

Context: The parsing in simdjson is divided into 2 stages. First, in stage 1, we parse the document and find all the structural indexes ({, }, ], [, ,, ", ...) and validate UTF8. Then, in stage 2, we go through the document again and build the tape using structural indexes found during stage 1. Although stage 1 finds the structural indexes, it has no knowledge of the structure of the document nor does it know whether it parsed a valid document, multiple documents, or even if the document is complete.

We found a pretty cool algorithm that allows us to quickly identify the position of the last JSON document in a given batch. Knowing exactly where the end of the batch is, we no longer need for stage 2 to finish in order to load a new batch. We already know where to start the next batch. Therefore, we can run stage 1 on the next batch concurrently while the main thread is going through stage 2. Now, running stage 1 in a different thread can, in best cases, remove almost entirely it's cost and replaces it by the overhead of a thread, which is orders of magnitude cheaper. Ain't that awesome!

Future Development

This pull request also acts as a proof of concept for future development where the standard parsing (single file) could be threaded for substantial gains. This technique could be implemented by modifying the stage 2 so that it could parse in streaming mode, while stage 1 would run concurrently. What I mean here by "streaming mode" is that instead of stage 2 parsing a file all at once, it could parse it piece by piece, similar to the JsonStream implementation.
Therefore, running stage 1 concurrently would reduce its cost substantially, which generally represents ~50% of the total work.

TL/DR: We could potentially make simdjson as much as ~50% faster in some cases. Bigger files would get bigger gains.

#188 #274