Support #323

set c/c++ compiler invocation to "-j2" to default?

Added by Florian Lier over 9 years ago. Updated about 7 years ago.

Status:ClosedStart date:2014-11-15
Priority:NormalDue date:
Assignee:-% Done:

50%

Category:RecipeSpent time:-
Target version:Toolkit 0.1

Description

We currently have 2 "executers" as the default in Jenkins so "make -j2" should be perfectly doable as most machines today have at least 4 threads available.

Shall we do this?

IMHO this would speed up the deployment process.

History

#1 Updated by Florian Lier over 9 years ago

Does feedback mean it is implemented?

#2 Updated by Jan Moringen over 9 years ago

Florian Lier wrote:

Does feedback mean it is implemented?

No.

If we decide to do this, it can mostly be achieved on the template level, I think. Most of the work will be designing good defaults, allowing the defaults to be overwritten and not making the solution CMake/make-specific.

@Johannes: I have never seen parallelization wihtin Jenkins jobs. Do you know whether this is a supported use-case?

#3 Updated by Johannes Wienke over 9 years ago

This somehow contradicts the idea of jenkins executors.

#4 Updated by Robert Haschke about 9 years ago

For me, it's not clear, why jenkins executors should be single-threaded.
If there is only a single job to execute, it will greatly benefit from parallelization. Otherwise, i.e. having multiple parallel jobs that in turn are parallelized, will not harm much, wouldn't it?

Hence, I vote for job-internal parallelization too.

#5 Updated by Johannes Wienke about 9 years ago

Using jenkins parallelization + job parallel builds will easily flood the available working memory on the computer.

#6 Updated by Florian Lier about 9 years ago

I have the feeling at some point this has been implemented in the famula branch, is that correct? I would, again, vote for
a -D j=int option with a comment "expert" mode.

Jenkins comes with #2 executers by default and most machines these days support at least 4 threats. I believe parallel C/C++ builds are more efficient than more executers. For example if a c/c++ job which has a lot of downstream dependencies (that are correctly blocked in this case) takes forever, parallel executers deliver no speed up.

So, if you know what you are doing, and you have 2 executers and let's say 8 threats available -j >2 won't hurt much.

#7 Updated by Robert Haschke about 9 years ago

Yes, in the famula branch I added an appropriate variable make.threads for autotools and freestyle projects. However, I wasn't able to influence cmake builds in this manner.

Even having many executors AND many build threads typically doesn't harm - as long as enough memory is available.

#8 Updated by Florian Lier over 8 years ago

Did this feature make it into the master?

#9 Updated by Jan Moringen over 8 years ago

  • Assignee deleted (Jan Moringen)

Florian Lier wrote:

Did this feature make it into the master?

I don't know but this can be done entirely on the template level.

#10 Updated by Robert Haschke over 8 years ago

As far as I could see (when I was searching for an appropriate option),
this cannot be handled on the template level.
There is no make command specified in the templates for cmake:

grep -r [^c]make .

./toolkit/T600-freestyle.template: "make.command": "
./toolkit/T600-freestyle.template:make -j${make.threads|1}
./toolkit/T700-autotools.template:make -j${make.threads|1}
./ci/T600-freestyle.template: "make.command": "
./ci/T600-freestyle.template:make -j${make.threads|1}
./ci-deploy/T600-freestyle.template: "make.command": "
./ci-deploy/T600-freestyle.template:make -j${make.threads|1}
./ci-deploy/T700-autotools.template:make -j${make.threads|1}
./_common/T800-runnable.template: "make.command": "
./_common/T800-runnable.template:make -j${make.threads|1}

The corresponding jenkins commands are generated by the binary...

#11 Updated by Jan Moringen over 8 years ago

  • Status changed from Feedback to In Progress
  • % Done changed from 0 to 50

build-generator master allows this. Not sure what good defaults are, though.

#12 Updated by Jan Moringen over 8 years ago

  • Category changed from build-generator to Recipe

#13 Updated by Florian Lier about 7 years ago

  • Status changed from In Progress to Closed

Also available in: Atom PDF