FAQ
Frequently asked questions:
Q: Why are there so many hyperparameters?
A: ParMOO is not a solver it is a framework for building and deploying customized solvers. It has been our experience that a single solver cannot obtain the best performance on every problem. Through ParMOO, we are giving you a modeling language and library of useful techniques for building a great solver for your problem and deploying it in parallel environments.
Q: There are too many options, where do I start?
A: If you are not an optimization expert or if you’re just not sure where to start, that’s OK. Start from one of our tutorials and modify it to match your problem. See how that works for your problem. Then start experimenting with other options, referring to this FAQ for some general guidance. Alternatively, check out some of our existing pre-built solvers in the parmoo_solver_farm and see if you can modify one to fit your needs!
Q: I am running ParMOO with a single simulation function with the following settings:
search_budget = B_s
,number of acquisition functions = q
,max_iters (input to solve method) = k
. How many times will ParMOO run my simulation while solving this problem?
A: In this example:
total_budget = B_s + q * k
Q: Going off the previous example, how should I choose the values of
B_s
,q
, andk
?
A: The answer is problem dependent, but these are our recommendations:
For most of our solver settings ParMOO performs pure “exploitation” after the initial search, so you can think of
B_s
as the “exploration” parameter. (There are several notable exceptions, such as if you use theEI_RandomConstraint
acquisition function, which we generally do not recommend due to its computational expense). If your problem is very smooth or doesn’t have many local minima, then you can get away with a small value ofB_s
. If your problem is nonsmooth or highly nonconvex, you will do better with a larger value ofB_s
. At a minimum,B_s
must be at leastn+1
(recall:n
is the number of design variables). All of our existing surrogate models will raise an error ifB_s < n+1
. If you are still unsure, a good starting place that works well on a variety of challenging problems isB_s = min(2000, 0.5*total_budget)
, wheretotal_budget
is the total budget that you intend to use for this problem.The value of
q
determines how many simulations ParMOO will evaluate per batch. If you want to achieve NP-way parallelism (using thelibE_MOOP
class), then you should useq=NP
. If you are running serially (using the baseMOOP
class), then you should use a small number of acquisition functions in order to get more iterations. In many situations, just 1 acquisition function will work best when running serially. In order to amortize the cost of re-fitting the surrogate model, we typically useq=2
,3
, or4
when running serially.The value of
k
should be as large as you can afford, given your simulation costs. Note that theGaussRBF
surrogate becomes expensive to fit fortotal_budget > 4000
, although we have used it for up tototal_budget = 10,000
. If you want more practical advice, see the next question.Q: Given the advice from the previous question, how do I know a good
total_budget
for my problem?
A: It depends on a lot of factors. In general, if your problem is small (
n < 8
design variables ando < 3
objectives) you can probably get away with a total budget in the hundreds. If your problem is large (n > 8
design variables oro > 3
objectives) you will probably need a total budget in the thousands or even tens of thousands. In general, as you increase the number of design variables or objectives the problem expense increases exponentially if you want to maintain the same accuracy. This is part of the curse of dimensionality.Q: I have a lot of design variables but I can’t afford that large of a budget, what can I do?
A: The key issue is that global optimization is expensive. At a fundamental level, no blackbox solver can guarantee global convergence without densely sampling the design space, which is exponentially expensive when
n
(number of design variables) is large. So what can you do? You can switch to using local modeling methods, whose costs generally only grow linearly in the dimension. You will not get any convergence guarantees to global solutions, but in many cases, you will still be able to approximately solve your problem. Check out our High-dimensional multiobjective optimization tutorial to learn more.Q: How can I determine whether my problem was solved by ParMOO?
Short Answer: you can’t. Here’s why – in most situations (unless your objectives are not truly conflicting) there are infinitely many solutions to a multiobjective optimization problem with continuous design variables. We cannot find all such solutions with a finite budget. What we can do is give you as many approximate solutions as possible for the budget allocated, so that you can make informed decisions about the inherent tradeoffs, and possibly run a single-objective solver to refine your favorite solution in the future.
Longer Answers:
For practical purposes: you could solve the problem with ParMOO on as large of a budget as you can afford with checkpointing turned on. Then plot the results using one of the methods from our
viz
library and see how you are doing. If you are unsatisfied with the results, re-load from the last checkpoint and solve with a few added iterations. Then plot your results again and see if the performance has improved.For small problems: you could solve the problem with ParMOO on as large of a budget as you can afford. Then plot the convergence over time, according to one of the common multiobjective performance indicators, such as hypervolume. If you are seeing diminishing to no improvements in late iterations, then you may have solved the problem. Note that hypervolume is exponentially expensive to compute when you have a large number of objectives. Therefore, we do not have a hypervolume metric calculator available in ParMOO at this time, but we will add it in the future.
Q: Why are the iteration costs (time spent generating a batch) so high?
A: The majority of ParMOO’s overhead comes from fitting the surrogate models and solving the scalarized surrogate problems. If you followed the quickstart, then the default method for surrogate modeling was to fit a Gaussian process. For numerical stability reasons, we fit our Gaussian processes via a symmetric-eigensolve, which is not inexpensive. Then you may have to evaluate the Gaussian process thousands of times while solving the surrogate problem. All of this expense adds up, especially if you are using a large total budget, since the cost of fitting such Gaussian processes grows cubically with the number of data points. One solution is to switch to using a
LocalGaussRBF
surrogate, which does not use the entire database when fitting surrogates, and therefore is more scalable for handling large budgets. See our tutorial on local methods for an example.Q: Surrogate models, acquisition functions, search techniques, and optimization solvers – how do I know which ones to pick?
A: Generally, we recommend sticking with
LatinHypercube
search andRandomConstraint
acquisition functions, unless you have a good reason for changing. These options work best for many of our test problems, and they are demonstrated in our tutorials. For the surrogate model and optimization solver, start out withLocalGPS
optimizer andGaussRBF
surrogate model, as in the quickstart. Then:
If you are willing to code the derivative for your objective and constraint functions (not the simulations), then you can follow the advanced_example and switch to using the
LBFGSB
optimizer.If you have a lot of design variables, then you might do better with a local solver, by switching your surrogate to the
LocalGaussRBF
surrogate. If you are using theLBFGSB
optimizer, then you will also need to switch to theTR_LBFGSB
optimizer.If you’re a professional optimizer or researcher and you want to try your own methods, then you can do so by writing your own implementation for one of our
Abstract Base Classes
. If you try a novel method and it works and you’re ready to publish it, consider sharing your novel solver on the parmoo_solver_farm!
We would like to acknowledge the following users, whose helpful discussions with us inspired this FAQ:
Sarah Salem (Bundeswehr University Munich)
Nicholas Antoniou (independent researcher)