Lecture 3

Encoding matters

Encoding Matters: When measuring running time, we measure the number of state transitions in a Turing machine. Given that a Turing machine can process only one symbol at a time, the encoding of the input matters.

We can find problems where changing the encoding changes the time-complexity bound drastically. Two relevant examples are Weak NP-Hard Decision Problem and Strong NP-Hard Decision Problem.

Knapsack’s dependence on encoding

Consider the Knapsack problem with running time $\Theta (n\cdot W)$. How does $n\cdot W$ relate to the input? What even is the input?

Let

$$x=((v_1,w_1),(v_2,w_2),\dots ,(v_n,w_n),W)$$

be a string. The input length is

$$N=|x|.$$

Here, $N\ne n$ if $W$ changes the length of the input string.

Binary encoding

Encode the integer values $v_i$, $w_i$, and $W$ in binary. The dynamic programming algorithm of Knapsack then runs over

$$\begin{array}{rl}& \mathrm{for}i\leftarrow 1\text{to}n:\\ & \mathrm{for}j\leftarrow 1\text{to}{2}^{\mathrm{bitlen}(W)}:\\ & \text{run some }\Theta (1).\end{array}$$

Here, $n$ is purely structural. It is the count of $(v_i,w_i)$ tuples encoded in the input string; it is not encoded explicitly. Thus, it is encoding-independent.

However, $W=2^{\mathrm{bitlen}(W)}$ is part of the encoded string because it is encoded in binary. We still need to explore all $2^{\mathrm{bitlen}(W)}$ state combinations. We cannot simply skip elements of the state space.

Unary encoding

Encode the integer values in unary. Unary means that the length of the encoding is constrained linearly by the integer’s magnitude:

$$|\mathrm{enc}(v)|=O(v).$$

The dynamic programming algorithm of Knapsack then runs over

$$\begin{array}{rl}& \mathrm{for}i\leftarrow 1\text{to}n:\\ & \mathrm{for}j\leftarrow 1\text{to}W:\\ & \text{run some }\Theta (1).\end{array}$$

Again, $n$ is purely structural; it is not part of the input string. The inner loop now iterates over $W$‘s magnitude. We can imagine iterating over a unary-encoded string space equivalent to appending an element to a linear list:

$$a,aa,aaa,\dots$$

This has running time $O(W)$.

Why not encode with a larger base?

Logarithms of different bases are equivalent in complexity theory:

$$\mathrm{bitlen}(W)\approx {\log }_{10}W=\frac{{\log }_{2}W}{{\log }_{2}10}=\frac{1}{{\log }_{2}10}{\log }_{2}W\in \Theta ({\log }_{2}W).$$

By definition, if a problem $p$ is NP-hard, every problem in NP can be reduced to $p$. Assume there exists at least one weakly NP-hard problem. Why is not every problem weakly NP-hard?

Complexity is always defined with respect to a fixed encoding. We cannot compare complexity relations for different encodings without fixing how their instances are represented.

Why not translate to an easier encoding?

Suppose the binary input contains $W=2^{\ell }$. To convert all $\ell$ bits into unary, the Turing machine must physically produce $2^{\ell }$ symbols on its tape to encode $W$.

Producing those symbols already costs

$$\Omega (2^{\ell })$$

(lower bound; exponential) running time. Therefore, we cannot simply skip the complexity caused by the encoding.

Reduction from Subset Sum to Knapsack

The reduction is a Karp reduction from Subset Sum to Knapsack:

$$\mathrm{SubsetSum}{\le }_p\mathrm{Knapsack}.$$

The two problems

Subset Sum: The input is a multiset $P$ of non-negative integers and a target integer $k$. The output question is

$$\exists S\subseteq P:\sum _{z\in S}z=k.$$

Knapsack: The input is a multiset $Q$ of integer pairs $(w,p)$, where $w$ is the weight and $p$ is the profit; an integer $W$ for the maximum weight; and an integer $P$ for the minimum profit. The output question is

$$\exists S\subseteq Q:\sum _{(w,p)\in S}w\le W\wedge \sum _{(w,p)\in S}p\ge P.$$

Construction

Let $(P,k)$ be an instance of Subset Sum. Construct an instance $(Q,W,P)$ of Knapsack. Set $W\leftarrow k$ and $P\leftarrow k$. The Knapsack condition becomes

$$\exists S\subseteq Q:\sum _{(w,p)\in S}w\le k\wedge \sum _{(w,p)\in S}p\ge k.$$

For every $z\in P$, append $(z,z)$ to $Q$:

$$Q\leftarrow Q\cup \{(z,z)\}.$$

The constructed Knapsack instance is equivalent to the Subset Sum instance because each selected item contributes the same amount to the weight and the profit:

$$\begin{array}{rl}\exists S\subseteq Q:& \sum _{(z,z)\in S}z\le k\wedge \sum _{(z,z)\in S}z\ge k\\ & ⟺\exists S\subseteq P:\sum _{z\in S}z=k.\end{array}$$

Given that Subset Sum is weakly NP-hard, Knapsack is weakly NP-hard.

Dynamic programming for Knapsack

The algorithm is dynamic programming for Knapsack. Sort the items arbitrarily and label them by an integer $i$.

Let $\mathrm{DP}[i,w,p]$ denote whether profit $p$ is achievable using the first $i$ items and a weight bound $w$.

Base case

Given that all profits are non-negative, profit $p=0$ is always achievable. Set

$$\mathrm{DP}[i,w,0]\leftarrow 1$$

for all $i\in \{1,\dots ,n\}$ and all $w\in \{0,\dots ,W\}$.

Progression

In the progression, there are two possible actions to choose from: take and skip.

If item $i$ is skipped, it is achievable only if all previous items $1,\dots ,i-1$ achieve the condition with the capacity constraints $w$ and $p$. This is obtained from $\mathrm{DP}[i-1,w,p]$. Therefore,

$$\mathrm{DP}[i,w,p]\leftarrow \mathrm{DP}[i-1,w,p].$$

If item $i$ is taken, the state is achievable using items $1,\dots ,i$ and constraints $w$ and $p$ if one of two cases is true. First, it is achievable even without $i$, which is the same scenario as skipping it. Second, it is achievable without $i$ and with $i$‘s weight $w_i$ and profit $p_i$. Item $i$ can be taken only when $w_i\le w$ and $p_i\le p$.

Together, we arrive at the formula

$$\mathrm{DP}[i,w,p]\leftarrow \max \left\{\mathrm{DP}[i-1,w,p],\mathrm{DP}[i-1,w-w_i,p-p_i]\right\}.$$

Islands of Intractability

Within NP-complete problems, there are tractable subproblems.

Example: $\text{3-SAT}\subseteq \text{SAT}$, where 3-SAT is efficiently solvable and SAT is NP-complete.

Horn and Krom formulas

A Horn formula is a conjunction of Horn clauses. A Horn clause is a disjunctive clause with at most one positive literal. For example,

$$\neg p\vee \neg q\vee c$$

is a Horn clause.

A unit clause is a clause that consists of only one literal. Unit propagation uses unit clauses to force values on literals and simplify the formula.

For example, consider

$$\begin{array}{rl}F=& (\neg a\vee \neg b\vee c)\wedge (\neg b\vee \neg c\vee f)\wedge (\neg f\vee b)\\ & \wedge (\neg e\vee \neg c\vee a)\wedge f\wedge (\neg d\vee e)\wedge (\neg b\vee \neg c).\end{array}$$

The recorded unit-propagation steps are

$$f=\top ,b=\top ,c=\perp ,a=\perp .$$

The handwritten notes also record $e=\top$. This last assignment does not follow from the displayed clauses alone: the clause $(\neg d\vee e)$ would force $e$ only if $d=\top$ were already known. It should therefore be checked against the original lecture example.

A Krom formula is a formula in which every clause has at most two literals. For example,

$$(a\vee b)\wedge \neg b\wedge (c\vee a).$$

Solving Krom formulas is tractable because every disjunction of two literals can be represented as two implications:

\begin{array}{r@{\;}c@{\;}l@{\quad}c@{\quad}l@{\;}c@{\;}l@{\quad}c@{\quad}l@{\;}c@{\;}l} a & \lor & \neg b & \iff & (b & \Rightarrow & a) & \land & (\neg a & \Rightarrow & \neg b), \\ a & \lor & b & \iff & (\neg a & \Rightarrow & b) & \land & (\neg b & \Rightarrow & a), \\ \neg a & \lor & \neg b & \iff & (a & \Rightarrow & \neg b) & \land & (b & \Rightarrow & \neg a). \end{array}

Unit clauses can be represented in the same way:

$$a⟺(\neg a\Rightarrow a),\neg a⟺(a\Rightarrow \neg a).$$

Consequently, we can represent Krom formulas as an implication graph. For each clause, add the two equivalent implication edges.

For example,

$$(x_0\vee x_2)\wedge (x_0\vee \neg x_3)$$

becomes

$$(\neg x_2\Rightarrow x_0)\wedge (\neg x_0\Rightarrow x_2)\wedge (x_3\Rightarrow x_0)\wedge (\neg x_0\Rightarrow \neg x_3).$$

The graph has six singleton strongly connected components. There is no circularity, so the formula is satisfiable.

As a contrasting example,

$$x_0\wedge \neg x_0\wedge (x_0\vee x_1)$$

becomes

$$(x_0\Rightarrow \neg x_0)\wedge (\neg x_0\Rightarrow x_0)\wedge (\neg x_1\Rightarrow x_0)\wedge (\neg x_0\Rightarrow x_1).$$

Here, the strongly connected components include

$$\{x_0,\neg x_0\},\{x_1\},\{\neg x_1\}.$$

The component containing both $x_0$ and $\neg x_0$ is non-singleton. A variable and its negation belonging to the same strongly connected component makes the Krom formula unsatisfiable.

Tree-like formulas

The incidence graph of a formula $F$ is a bipartite graph whose vertex set consists of all variables and clauses in the formula. A variable is incident to a clause if and only if its literal occurs in that clause.

Formulas whose incidence graphs are acyclic can be solved in polynomial time.

Consider the formula

$$F=\underset{=c_1}{\underbrace{(x_1\vee \neg x_2\vee \neg x_3)}}\wedge \underset{=c_2}{\underbrace{(x_2\vee \neg x_4)}}\wedge \underset{=c_3}{\underbrace{(x_2\vee \neg x_5\vee x_6)}}\wedge \underset{=c_4}{\underbrace{x_5}}.$$

Its incidence graph is a tree. Squares represent variables and circles represent clauses.

Removing satisfied leaves

The tree can be simplified from its leaves upwards. The leaf clause $c_4$ contains only $x_5$, so it forces

$$x_5=\top .$$

The first stage highlights the leaf clause and its parent:

Remove the satisfied leaf pair $c_4,x_5$. The remaining tree is:

The variable $x_6$ is a leaf in $c_3$. Choosing $x_6=\top$ satisfies $c_3$ without branching through $x_2$, which has more children. Remove $c_3$ and $x_6$:

The leaf variable $x_4$ occurs in $c_2=(x_2\vee \neg x_4)$. Choosing $x_4=\perp$ satisfies $c_2$, so remove $c_2$ and $x_4$. The final clause is

$$c_1=(x_1\vee \neg x_2\vee \neg x_3).$$

It can be made true by choosing a satisfying literal: $x_1=\top$, $x_2=\perp$, or $x_3=\perp$.

Conflict-driven clause learning

Conflict-driven clause learning (CDCL) follows the cycle

$$\text{guess}⟶\text{propagate consequences}⟶\text{conflict}⟶\text{learn why}.$$

Consider the clauses

$$\begin{array}{rlrlrl}{c}_1& =(\neg a\vee \neg b\vee c),& c_2& =(\neg a\vee d),& c_3& =(\neg c\vee \neg d\vee e),\\ c_4& =(\neg h\vee \neg e\vee f),& c_5& =(\neg e\vee g),& c_6& =(\neg f\vee \neg g).\end{array}$$

Assume the decisions

$$h=\top ,b=\top ,g=\top ,a=\top .$$

Unit propagation proceeds as follows:

$$\begin{array}{rlrl}{c}_1& =(\neg a\vee \neg b\vee c)=(\perp \vee \perp \vee c)=(c)& & ⟹c=\top ,\\ c_2& =(\neg a\vee d)=(\perp \vee d)=(d)& & ⟹d=\top ,\\ c_3& =(\neg c\vee \neg d\vee e)=(\perp \vee \perp \vee e)=(e)& & ⟹e=\top ,\\ c_4& =(\neg h\vee \neg e\vee f)=(\perp \vee \perp \vee f)=(f)& & ⟹f=\top ,\\ c_5& =(\neg e\vee g)=(\perp \vee g)=(g)& & ⟹g=\top ,\\ c_6& =(\neg f\vee \neg g)=(\perp \vee \perp )=(\perp ).& & \end{array}$$

The last clause is false, so the decisions have produced a conflict. The implication graph records the propagation history as it is built.

After $c_1$, $c_2$, and $c_3$

The decisions $a=\top$ and $b=\top$ force $c=\top$ through $c_1$ and $d=\top$ through $c_2$. Together, $c$ and $d$ force $e=\top$ through $c_3$.

Adding $c_4$ and $c_5$

The decision $h=\top$ and the propagated value $e=\top$ force $f=\top$ through $c_4$. The same value $e=\top$ reduces $c_5$ to the unit clause $g$, giving $g=\top$.

The conflict and the learned clause

Finally, $f=\top$ and $g=\top$ reduce $c_6$ to the empty clause. The graph now exposes the complete causal chain.

Tracing the conflict backwards reaches the decisions $a=\top$, $b=\top$, and $h=\top$. CDCL learns that these three decisions cannot hold simultaneously:

$$c_7=\neg (a\wedge b\wedge h)\equiv \neg a\vee \neg b\vee \neg h.$$

The solver adds $c_7$ and retries after backtracking. The learned clause prevents the same conflicting decision combination from being explored again.