In general, there are three options regarding the existence of a solution:[7]: chpt.4
If such a point x* exists, it is referred to as an optimal point or solution; the set of all optimal points is called the optimal set; and the problem is called solvable.
If is unbounded below over , or the infimum is not attained, then the optimization problem is said to be unbounded.
Otherwise, if is the empty set, then the problem is said to be infeasible.
Standard form
A convex optimization problem is in standard form if it is written as
The inequality constraint functions , , are convex functions;
The equality constraint functions , , are affine transformations, that is, of the form: , where is a vector and is a scalar.
The feasible set of the optimization problem consists of all points satisfying the inequality and the equality constraints. This set is convex because is convex, the sublevel sets of convex functions are convex, affine sets are convex, and the intersection of convex sets is convex.[7]: chpt.2
Many optimization problems can be equivalently formulated in this standard form. For example, the problem of maximizing a concave function can be re-formulated equivalently as the problem of minimizing the convex function . The problem of maximizing a concave function over a convex set is commonly called a convex optimization problem.[8]
Epigraph form (standard form with linear objective)
In the standard form it is possible to assume, without loss of generality, that the objective function f is a linear function. This is because any program with a general objective can be transformed into a program with a linear objective by adding a single variable t and a single constraint, as follows:[9]: 1.4
Conic form
Every convex program can be presented in a conic form, which means minimizing a linear objective over the intersection of an affine plane and a convex cone:[9]: 5.1
where K is a closed pointed convex cone, L is a linear subspace of Rn, and b is a vector in Rn. A linear program in standard form in the special case in which K is the nonnegative orthant of Rn.
Eliminating linear equality constraints
It is possible to convert a convex program in standard form, to a convex program with no equality constraints.[7]: 132 Denote the equality constraints hi(x)=0 as Ax=b, where A has n columns. If Ax=b is infeasible, then of course the original problem is infeasible. Otherwise, it has some solution x0 , and the set of all solutions can be presented as: Fz+x0, where z is in Rk, k=n-rank(A), and F is an n-by-k matrix. Substituting x = Fz+x0 in the original problem gives:
where the variables are z. Note that there are rank(A) fewer variables. This means that, in principle, one can restrict attention to convex optimization problems without equality constraints. In practice, however, it is often preferred to retain the equality constraints, since they might make some algorithms more efficient, and also make the problem easier to understand and analyze.
Special cases
The following problem classes are all convex optimization problems, or can be reduced to convex optimization problems via simple transformations:[7]: chpt.4 [10]
The convex programs easiest to solve are the unconstrained problems, or the problems with only equality constraints. As the equality constraints are all linear, they can be eliminated with linear algebra and integrated into the objective, thus converting an equality-constrained problem into an unconstrained one.
In the class of unconstrained (or equality-constrained) problems, the simplest ones are those in which the objective is quadratic. For these problems, the KKT conditions (which are necessary for optimality) are all linear, so they can be solved analytically.[7]: chpt.11
For unconstrained (or equality-constrained) problems with a general convex objective that is twice-differentiable, Newton's method can be used. It can be seen as reducing a general unconstrained convex problem, to a sequence of quadratic problems.[7]: chpt.11 Newton's method can be combined with line search for an appropriate step size, and it can be mathematically proven to converge quickly.
The more challenging problems are those with inequality constraints. A common way to solve them is to reduce them to unconstrained problems by adding a barrier function, enforcing the inequality constraints, to the objective function. Such methods are called interior point methods.[7]: chpt.11 They have to be initialized by finding a feasible interior point using by so-called phase I methods, which either find a feasible point or show that none exist. Phase I methods generally consist of reducing the search in question to a simpler convex optimization problem.[7]: chpt.11
Convex optimization problems can also be solved by the following contemporary methods:[12]
Subgradient methods can be implemented simply and so are widely used.[15] Dual subgradient methods are subgradient methods applied to a dual problem. The drift-plus-penalty method is similar to the dual subgradient method, but takes a time average of the primal variables.[citation needed]
Consider a convex minimization problem given in standard form by a cost function and inequality constraints for . Then the domain is:
The Lagrangian function for the problem is
For each point in that minimizes over , there exist real numbers called Lagrange multipliers, that satisfy these conditions simultaneously:
minimizes over all
with at least one
(complementary slackness).
If there exists a "strictly feasible point", that is, a point satisfying
then the statement above can be strengthened to require that .
Conversely, if some in satisfies (1)–(3) for scalars with then is certain to minimize over .
Software
There is a large software ecosystem for convex optimization. This ecosystem has two main categories: solvers on the one hand and modeling tools (or interfaces) on the other hand.
Solvers implement the algorithms themselves and are usually written in C. They require users to specify optimization problems in very specific formats which may not be natural from a modeling perspective. Modeling tools are separate pieces of software that let the user specify an optimization in higher-level syntax. They manage all transformations to and from the user's high-level model and the solver's input/output format.
The table below shows a mix of modeling tools (such as CVXPY and Convex.jl) and solvers (such as CVXOPT and MOSEK). This table is by no means exhaustive.
Interfaces with CPLEX, GUROBI, MOSEK, SDPT3, SEDUMI, CSDP, SDPA, PENNON solvers; also supports integer and nonlinear optimization, and some nonconvex optimization. Can perform robust optimization with uncertainty in LP/SOCP/SDP constraints.
Can do robust optimization on linear programming (with MOSEK to solve second-order cone programming) and mixed integer linear programming. Modeling package for LP + SDP and robust versions.
^For methods for convex minimization, see the volumes by Hiriart-Urruty and Lemaréchal (bundle) and the textbooks by Ruszczyński, Bertsekas, and
Boyd and Vandenberghe (interior point).
^Nesterov, Yurii; Arkadii, Nemirovskii (1995). Interior-Point Polynomial Algorithms in Convex Programming. Society for Industrial and Applied Mathematics. ISBN978-0898715156.
^Peng, Jiming; Roos, Cornelis; Terlaky, Tamás (2002). "Self-regular functions and new search directions for linear and semidefinite optimization". Mathematical Programming. 93 (1): 129–171. doi:10.1007/s101070200296. ISSN0025-5610. S2CID28882966.
^Ben Haim Y. and Elishakoff I., Convex Models of Uncertainty in Applied Mechanics, Elsevier Science Publishers, Amsterdam, 1990
^Ahmad Bazzi, Dirk TM Slock, and Lisa Meilhac. "Online angle of arrival estimation in the presence of mutual coupling." 2016 IEEE Statistical Signal Processing Workshop (SSP). IEEE, 2016.
Hiriart-Urruty, Jean-Baptiste, and Lemaréchal, Claude. (2004). Fundamentals of Convex analysis. Berlin: Springer.
Hiriart-Urruty, Jean-Baptiste; Lemaréchal, Claude (1993). Convex analysis and minimization algorithms, Volume I: Fundamentals. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences]. Vol. 305. Berlin: Springer-Verlag. pp. xviii+417. ISBN978-3-540-56850-6. MR1261420.
Hiriart-Urruty, Jean-Baptiste; Lemaréchal, Claude (1993). Convex analysis and minimization algorithms, Volume II: Advanced theory and bundle methods. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences]. Vol. 306. Berlin: Springer-Verlag. pp. xviii+346. ISBN978-3-540-56852-0. MR1295240.
Lemaréchal, Claude (2001). "Lagrangian relaxation". In Michael Jünger and Denis Naddef (ed.). Computational combinatorial optimization: Papers from the Spring School held in Schloß Dagstuhl, May 15–19, 2000. Lecture Notes in Computer Science. Vol. 2241. Berlin: Springer-Verlag. pp. 112–156. doi:10.1007/3-540-45586-8_4. ISBN978-3-540-42877-0. MR1900016. S2CID9048698.
Nesterov, Yurii; Nemirovskii, Arkadii (1994). Interior Point Polynomial Methods in Convex Programming. SIAM.