Worked on some more #Gentoo global #jobserver goodies today.
Firstly, Portage jobserver support patch: https://github.com/gentoo/portage/pull/1528. It's not yet perfect, but it seems to work. If enabled via FEATURES=jobserver-token, it causes emerge to acquire a job token for every job it starts, therefore:
1. emerge won't start more jobs than the jobserver specifies. For example, `steve -j12` will cap emerge at 12 jobs, even if you specify --jobs=16. However, lower cap for emerge will be respected, so you can leave a bunch of job tokens free for other processes to use.
2. emerge will share the job pool with other jobserver-enabled software such as GNU make or Ninja. When emerge is using up all 12 job tokens, all builders inside it will run serial. When it's running 6 jobs, all builders will be able to start up to 6 extra jobs, etc. Ideally, no more than 12 active jobs will be running at a time over the system.
3. Jobserver-enabled software will now correctly respect jobserver job count. When running with externally-provided jobserver, these tools assumed that the jobserver already acquired one token for them, and therefore ran one extra job without a token; say, 13 jobs for a single emerge process. Now the token is actually acquired, so the accounts will match.
Secondly, I've packaged dev-python/pytest-jobserver with a bunch of fixes to make it behave correctly and support Gentoo jobserver. We're not integrating it with the eclasses yet, but you can install it and force-enabled via EPYTEST_FLAGS="-p jobserver". With it, pytest-xdist will acquire job tokens while running tests, and therefore parallel #PyTest jobs will also be counted towards total job count.
Again, it's not a perfect solution, but it works reasonably. The plugin still starts -n jobs as specified by the arguments, but it acquired job tokens prior to executing every test, therefore delaying actual testing until tokens are available. It doesn't seem to cause noticeable overhead either.