My Blog

MILP Example with JuMP and HiGHS

2023-02-16T00:00:00+00:00

In my previous post</a>, I mentioned that the problem (Advent of Code 2018 day 23</a>) can be reformulated as a mixed-integer linear program (MILP). In this post, we'll walk through a solution using JuMP.jl</a> and HiGHS.jl</a>. The formulation is based on this</a> Reddit comment.</p>

Input parsing is the same as last time. We set up the JuMP problem by defining variables x, y, and z as integers and the vector variable botsinrange</code> as binary. This time, the value of botsinrange[i]</code> is 1 if the point (x, y, z) is in range of bot i, otherwise 0.</p>

using</span> JuMP</span></span>
using</span> HiGHS</span></span>
</span>
struct</span> Nanobot</span></span>
    x</span>::</span>Int</span></span>
    y</span>::</span>Int</span></span>
    z</span>::</span>Int</span></span>
    radius</span>::</span>Int</span></span>
end</span></span>
function</span> processinput</span>(</span>filename</span>)</span></span>
    puzin </span>=</span> [</span>[</span>parse</span>(</span>Int</span>,</span> m</span>.</span>match</span>)</span> for</span> m </span>in</span> eachmatch</span>(</span>r"</span>(</span>\-</span>?</span>\d</span>+)</span>"</span>,</span> line</span>)</span>]</span></span>
             for</span> line </span>in</span> readlines</span>(</span>filename</span>)</span>]</span></span>
    bots </span>=</span> [</span>Nanobot</span>(</span>l</span>...</span>)</span> for</span> l </span>in</span> puzin</span>]</span></span>
    return</span> bots</span></span>
end</span></span>
bots </span>=</span> processinput</span>(</span>"</span>input201823.txt</span>"</span>)</span></span>
</span>
model </span>=</span> Model</span>(</span>HiGHS</span>.</span>Optimizer</span>)</span></span>
set_silent</span>(</span>model</span>)</span></span>
@variable</span>(</span>model</span>,</span> x</span>,</span> Int</span>)</span></span>
@variable</span>(</span>model</span>,</span> y</span>,</span> Int</span>)</span></span>
@variable</span>(</span>model</span>,</span> z</span>,</span> Int</span>)</span></span>
@variable</span>(</span>model</span>,</span> botsinrange</span>[</span>1</span>:</span>length</span>(</span>bots</span>)</span>]</span>,</span> Bin</span>)</span></span></code></pre>
Next, define constraints of the form</p>
abs</span>(</span>b</span>.</span>x </span>-</span> x</span>)</span> +</span> abs</span>(</span>b</span>.</span>y </span>-</span> y</span>)</span> +</span> abs</span>(</span>b</span>.</span>z </span>-</span> z</span>)</span> <=</span> b</span>.</span>radius </span>+</span> (</span>1</span> -</span> botsinrange</span>[</span>i</span>]</span>)</span>*</span>slack</span></span></code></pre>
where i is the index of bot b. For the moment, assume slack is infinity. Then if we think bot i is out of range (botsinrange[i] = 0</code>), the constraint holds (right hand side is Inf). Otherwise, if we think bot i is in range (botsinrange[i] = 1</code>), the constraint holds if and only if abs(b.x - x) + abs(b.y - y) + abs(b.z - z) <= b.radius</code>, i.e. the bot has to actually</em> be in range. We will maximize the sum of botsinrange</code>, so the optimizer will try to set botsinrange[i] = 1</code> for as many i as possible, and in those cases, it must also ensure the corresponding constraint holds.</p>
So far so good, but there are two minor complications. First, the abs function is non-linear, so we can't have it in the constraint definition. However, note that if -x < c and x < c, then abs(x) < c. If x + y < c and -x + y < c and x - y < c and -x - y < c, then abs(x) + abs(y) < c. A similar argument holds for 3 variables, so we just need to create several linear constraints that imply the constraint defined above.</p>
Second, we can't actually use Inf for the slack for numerical stability reasons; we'll have to use a large finite number. The slack can't be too low because we need the constraint to be relaxed if botsinrange[i] = 0</code>. Consider the worst case, where the left hand side of the constraint is as large as possible. This may occur when (x, y, z) is at the "opposite" corner of 3d space from a bot i.  The largest coordinate or radius in our puzzle input is</p>
maxcoord </span>=</span> maximum</span>(</span>b </span>-></span> max</span>(</span>abs</span>(</span>b</span>.</span>x</span>)</span>,</span> abs</span>(</span>b</span>.</span>y</span>)</span>,</span> abs</span>(</span>b</span>.</span>z</span>)</span>,</span> b</span>.</span>radius</span>)</span>,</span> bots</span>)</span></span></code></pre>
In the worst case, we are comparing a point like (maxcoord, maxcoord, maxcoord) to (-maxcoord, -maxcoord, -maxcoord), so the distance between those points is 2*3*maxcoord. We have additional distance for the bots' sensor radius, so we may use slack = 2*4*maxcoord</code>.</p>
Putting it all together, our constraints are</p>
slack </span>=</span> 2</span>*</span>4</span>*</span>maximum</span>(</span>b </span>-></span> max</span>(</span>abs</span>(</span>b</span>.</span>x</span>)</span>,</span> abs</span>(</span>b</span>.</span>y</span>)</span>,</span> abs</span>(</span>b</span>.</span>z</span>)</span>,</span> b</span>.</span>radius</span>)</span>,</span> bots</span>)</span></span>
for</span> (</span>i</span>,</span>b</span>)</span> in</span> enumerate</span>(</span>bots</span>)</span></span>
    for</span> absmult </span>in</span> Iterators</span>.</span>product</span>(</span>(</span>-</span>1</span>,</span>1</span>)</span>,</span>(</span>-</span>1</span>,</span>1</span>)</span>,</span>(</span>-</span>1</span>,</span>1</span>)</span>)</span></span>
        @constraint</span>(</span>model</span>,</span> absmult</span>[</span>1</span>]</span>*</span>(</span>b</span>.</span>x </span>-</span> x</span>)</span> +</span> absmult</span>[</span>2</span>]</span>*</span>(</span>b</span>.</span>y </span>-</span> y</span>)</span> +</span></span>
                            absmult</span>[</span>3</span>]</span>*</span>(</span>b</span>.</span>z </span>-</span> z</span>)</span> <=</span> b</span>.</span>radius </span>+</span> (</span>1</span>-</span>botsinrange</span>[</span>i</span>]</span>)</span>*</span>slack</span>)</span></span>
    end</span></span>
end</span></span></code></pre>
Finally, we can optimize the model.</p>
@objective</span>(</span>model</span>,</span> Max</span>,</span> sum</span>(</span>botsinrange</span>)</span>)</span></span>
optimize!</span>(</span>model</span>)</span></span>
println</span>(</span>"""</span>x=</span>$</span>(</span>round</span>(</span>Int</span>,</span> value</span>(</span>x</span>)</span>)</span>)</span>, y=</span>$</span>(</span>round</span>(</span>Int</span>,</span> value</span>(</span>y</span>)</span>)</span>)</span>, z=</span>$</span>(</span>round</span>(</span>Int</span>,</span> value</span>(</span>z</span>)</span>)</span>)</span></span>
ans=</span>$</span>(</span>round</span>(</span>Int</span>,</span> value</span>(</span>x</span>)</span>)</span> +</span> round</span>(</span>Int</span>,</span> value</span>(</span>y</span>)</span>)</span> +</span> round</span>(</span>Int</span>,</span> value</span>(</span>z</span>)</span>)</span>)</span>"""</span>)</span></span></code></pre>
For my input, we find the correct answer, but recall the problem statement asked if there are multiple points with maximum nanobot coverage, then we should choose the point closest to the origin. So, we can add a constraint that the objective function must be at least as high as we just found1</a></sup></p>
function</span> coordisinrange</span>(</span>x</span>,</span> y</span>,</span> z</span>,</span> bot</span>::</span>Nanobot</span>)</span></span>
    return</span> abs</span>(</span>bot</span>.</span>x </span>-</span> x</span>)</span> +</span> abs</span>(</span>bot</span>.</span>y </span>-</span> y</span>)</span> +</span> abs</span>(</span>bot</span>.</span>z </span>-</span> z</span>)</span> <=</span> bot</span>.</span>radius</span></span>
end</span></span>
maxbots </span>=</span> count</span>(</span>b </span>-></span> coordisinrange</span>(</span>round</span>(</span>Int</span>,</span> value</span>(</span>x</span>)</span>)</span>,</span></span>
         round</span>(</span>Int</span>,</span> value</span>(</span>y</span>)</span>)</span>,</span> round</span>(</span>Int</span>,</span> value</span>(</span>z</span>)</span>)</span>,</span> b</span>)</span>,</span> bots</span>)</span></span>
@constraint</span>(</span>model</span>,</span> maxbots </span><=</span> sum</span>(</span>botsinrange</span>)</span>)</span></span></code></pre>
and add a penalty term to our objective function to minimize abs(x) + abs(y) + abs(z).2</a></sup> We have the same non-linear problem with abs as before, so we have to use a similar trick. The objective function becomes sum(botsinrange) - 0.00000001 * (xt + yt + zt)</code> where the penalty is set by trial and error until the optimizer finds a solution. The coordinates are 8 digit numbers, so the penalty should be similar in scale to 1 unit of the objective—i.e. having one more bot in range.</p>
@variable</span>(</span>model</span>,</span> xt</span>,</span> Int</span>)</span></span>
@variable</span>(</span>model</span>,</span> yt</span>,</span> Int</span>)</span></span>
@variable</span>(</span>model</span>,</span> zt</span>,</span> Int</span>)</span></span>
@constraint</span>(</span>model</span>,</span> x </span><=</span> xt</span>)</span>;</span> @constraint</span>(</span>model</span>,</span> -</span>x </span><=</span> xt</span>)</span>;</span></span>
@constraint</span>(</span>model</span>,</span> y </span><=</span> yt</span>)</span>;</span> @constraint</span>(</span>model</span>,</span> -</span>y </span><=</span> yt</span>)</span>;</span></span>
@constraint</span>(</span>model</span>,</span> z </span><=</span> zt</span>)</span>;</span> @constraint</span>(</span>model</span>,</span> -</span>z </span><=</span> zt</span>)</span>;</span></span>
</span>
@constraint</span>(</span>model</span>,</span> maxbots </span><=</span> sum</span>(</span>botsinrange</span>)</span>)</span></span>
@objective</span>(</span>model</span>,</span> Max</span>,</span> sum</span>(</span>botsinrange</span>)</span> -</span> 0.00000001</span> *</span> (</span>xt </span>+</span> yt </span>+</span> zt</span>)</span>)</span></span>
optimize!</span>(</span>model</span>)</span></span>
println</span>(</span>"""</span>x=</span>$</span>(</span>round</span>(</span>Int</span>,</span> value</span>(</span>x</span>)</span>)</span>)</span>, y=</span>$</span>(</span>round</span>(</span>Int</span>,</span> value</span>(</span>y</span>)</span>)</span>)</span>, z=</span>$</span>(</span>round</span>(</span>Int</span>,</span> value</span>(</span>z</span>)</span>)</span>)</span></span>
ans=</span>$</span>(</span>round</span>(</span>Int</span>,</span> value</span>(</span>x</span>)</span>)</span> +</span> round</span>(</span>Int</span>,</span> value</span>(</span>y</span>)</span>)</span> +</span> round</span>(</span>Int</span>,</span> value</span>(</span>z</span>)</span>)</span>)</span>"""</span>)</span></span></code></pre>
The solution is the same answer as before—which we know is correct thanks to Z3—but I'm curious if any other input needs the tie-breaking rule.</p>
Discussion</a>
</h2>
The MILP solution feels a bit hacky compared to Z3, but it is faster at around 0.3-1.5 seconds without compilation, compared to 50+ seconds for Z3. It took trial and error to get the right slack, and even if the slack is high enough, the solver may provide an incorrect solution.3</a></sup> For my input, slack of 1e11 results in an incorrect solution, while 1e10, 1e12-1e14 are ok.</p>
Handling the L1 norm (abs) is annoying as well. I tried to use MathOptInterface.NormOneCone</code>,</p>
for</span> (</span>i</span>,</span>b</span>)</span> in</span> enumerate</span>(</span>bots</span>)</span></span>
    @constraint</span>(</span>model2</span>,</span></span>
     [</span>b</span>.</span>radius </span>+</span> (</span>1</span>-</span>botsinrange</span>[</span>i</span>]</span>)</span>*</span>Int</span>(</span>1e10</span>)</span>;</span> x </span>-</span> [</span>b</span>.</span>x</span>,</span> b</span>.</span>y</span>,</span> b</span>.</span>z</span>]</span>]</span> in</span> MOI</span>.</span>NormOneCone</span>(</span>4</span>)</span>)</span></span>
end</span></span></code></pre>
but there is a performance issue. With constraints from only 280 bots (the problem has 1000), the solver takes about 4 seconds, but with 290 bots it hangs (or takes longer than I care to wait).</p>
In summary, MILP solvers can solve this problem more quickly than Z3 with a minor reformulation. There may be a better formulation that is less finicky. If you know of one, let me know.</p>
Full Julia script</a>
</h2>
using</span> JuMP</span></span>
using</span> HiGHS</span></span>
</span>
struct</span> Nanobot</span></span>
    x</span>::</span>Int</span></span>
    y</span>::</span>Int</span></span>
    z</span>::</span>Int</span></span>
    radius</span>::</span>Int</span></span>
end</span></span>
</span>
function</span> processinput</span>(</span>filename</span>)</span></span>
    puzin </span>=</span> [</span>[</span>parse</span>(</span>Int</span>,</span> m</span>.</span>match</span>)</span> for</span> m </span>in</span> eachmatch</span>(</span>r"</span>(</span>\-</span>?</span>\d</span>+)</span>"</span>,</span> line</span>)</span>]</span></span>
             for</span> line </span>in</span> readlines</span>(</span>filename</span>)</span>]</span></span>
    bots </span>=</span> [</span>Nanobot</span>(</span>l</span>...</span>)</span> for</span> l </span>in</span> puzin</span>]</span></span>
    return</span> bots</span></span>
end</span></span>
</span>
function</span> coordisinrange</span>(</span>x</span>,</span> y</span>,</span> z</span>,</span> bot</span>::</span>Nanobot</span>)</span></span>
    return</span> abs</span>(</span>bot</span>.</span>x </span>-</span> x</span>)</span> +</span> abs</span>(</span>bot</span>.</span>y </span>-</span> y</span>)</span> +</span> abs</span>(</span>bot</span>.</span>z </span>-</span> z</span>)</span> <=</span> bot</span>.</span>radius</span></span>
end</span></span>
</span>
function</span> main</span>(</span>filename</span>)</span></span>
    bots </span>=</span> processinput</span>(</span>filename</span>)</span></span>
    model </span>=</span> Model</span>(</span>HiGHS</span>.</span>Optimizer</span>)</span></span>
    slack </span>=</span> 2</span>*</span>4</span>*</span>maximum</span>(</span>b </span>-></span> max</span>(</span>abs</span>(</span>b</span>.</span>x</span>)</span>,</span> abs</span>(</span>b</span>.</span>y</span>)</span>,</span> abs</span>(</span>b</span>.</span>z</span>)</span>,</span> b</span>.</span>radius</span>)</span>,</span> bots</span>)</span></span>
    set_silent</span>(</span>model</span>)</span></span>
    @variable</span>(</span>model</span>,</span> x</span>,</span> Int</span>)</span></span>
    @variable</span>(</span>model</span>,</span> y</span>,</span> Int</span>)</span></span>
    @variable</span>(</span>model</span>,</span> z</span>,</span> Int</span>)</span></span>
    @variable</span>(</span>model</span>,</span> botsinrange</span>[</span>1</span>:</span>length</span>(</span>bots</span>)</span>]</span>,</span> Bin</span>)</span></span>
    for</span> (</span>i</span>,</span>b</span>)</span> in</span> enumerate</span>(</span>bots</span>)</span></span>
        for</span> absmult </span>in</span> Iterators</span>.</span>product</span>(</span>(</span>-</span>1</span>,</span>1</span>)</span>,</span>(</span>-</span>1</span>,</span>1</span>)</span>,</span>(</span>-</span>1</span>,</span>1</span>)</span>)</span></span>
            @constraint</span>(</span>model</span>,</span> absmult</span>[</span>1</span>]</span>*</span>(</span>b</span>.</span>x </span>-</span> x</span>)</span> +</span> absmult</span>[</span>2</span>]</span>*</span>(</span>b</span>.</span>y </span>-</span> y</span>)</span> +</span></span>
                                absmult</span>[</span>3</span>]</span>*</span>(</span>b</span>.</span>z </span>-</span> z</span>)</span> <=</span> b</span>.</span>radius </span>+</span> (</span>1</span>-</span>botsinrange</span>[</span>i</span>]</span>)</span>*</span>slack</span>)</span></span>
        end</span></span>
    end</span></span>
    @objective</span>(</span>model</span>,</span> Max</span>,</span> sum</span>(</span>botsinrange</span>)</span>)</span></span>
    optimize!</span>(</span>model</span>)</span></span>
    println</span>(</span>"</span>Step 1 solution:</span>"</span>)</span></span>
    println</span>(</span>"""</span>x=</span>$</span>(</span>round</span>(</span>Int</span>,</span> value</span>(</span>x</span>)</span>)</span>)</span>, y=</span>$</span>(</span>round</span>(</span>Int</span>,</span> value</span>(</span>y</span>)</span>)</span>)</span>, z=</span>$</span>(</span>round</span>(</span>Int</span>,</span> value</span>(</span>z</span>)</span>)</span>)</span>,</span></span>
     ans=</span>$</span>(</span>round</span>(</span>Int</span>,</span> value</span>(</span>x</span>)</span>)</span> +</span> round</span>(</span>Int</span>,</span> value</span>(</span>y</span>)</span>)</span> +</span> round</span>(</span>Int</span>,</span> value</span>(</span>z</span>)</span>)</span>)</span>"""</span>)</span></span>
</span>
    #</span> Step 2, with lower bound constraint for number of bots in range.</span></span>
    maxbots </span>=</span> count</span>(</span>b </span>-></span> coordisinrange</span>(</span>round</span>(</span>Int</span>,</span> value</span>(</span>x</span>)</span>)</span>,</span></span>
         round</span>(</span>Int</span>,</span> value</span>(</span>y</span>)</span>)</span>,</span> round</span>(</span>Int</span>,</span> value</span>(</span>z</span>)</span>)</span>,</span> b</span>)</span>,</span> bots</span>)</span></span>
    @constraint</span>(</span>model</span>,</span> maxbots </span><=</span> sum</span>(</span>botsinrange</span>)</span>)</span></span>
</span>
    @variable</span>(</span>model</span>,</span> xt</span>,</span> Int</span>)</span></span>
    @variable</span>(</span>model</span>,</span> yt</span>,</span> Int</span>)</span></span>
    @variable</span>(</span>model</span>,</span> zt</span>,</span> Int</span>)</span></span>
    @constraint</span>(</span>model</span>,</span> x </span><=</span> xt</span>)</span>;</span> @constraint</span>(</span>model</span>,</span> -</span>x </span><=</span> xt</span>)</span>;</span></span>
    @constraint</span>(</span>model</span>,</span> y </span><=</span> yt</span>)</span>;</span> @constraint</span>(</span>model</span>,</span> -</span>y </span><=</span> yt</span>)</span>;</span></span>
    @constraint</span>(</span>model</span>,</span> z </span><=</span> zt</span>)</span>;</span> @constraint</span>(</span>model</span>,</span> -</span>z </span><=</span> zt</span>)</span>;</span></span>
</span>
    @objective</span>(</span>model</span>,</span> Max</span>,</span> sum</span>(</span>botsinrange</span>)</span> -</span> 0.00000001</span> *</span> (</span>xt </span>+</span> yt </span>+</span> zt</span>)</span>)</span></span>
    optimize!</span>(</span>model</span>)</span></span>
    println</span>(</span>"</span>Step 2 solution:</span>"</span>)</span></span>
    println</span>(</span>"""</span>x=</span>$</span>(</span>round</span>(</span>Int</span>,</span> value</span>(</span>x</span>)</span>)</span>)</span>, y=</span>$</span>(</span>round</span>(</span>Int</span>,</span> value</span>(</span>y</span>)</span>)</span>)</span>, z=</span>$</span>(</span>round</span>(</span>Int</span>,</span> value</span>(</span>z</span>)</span>)</span>)</span></span>
     ans=</span>$</span>(</span>round</span>(</span>Int</span>,</span> value</span>(</span>x</span>)</span>)</span> +</span> round</span>(</span>Int</span>,</span> value</span>(</span>y</span>)</span>)</span> +</span> round</span>(</span>Int</span>,</span> value</span>(</span>z</span>)</span>)</span>)</span>"""</span>)</span></span>
</span>
    return</span> nothing</span></span>
end</span></span>
</span>
@time</span> main</span>(</span>"</span>input201823.txt</span>"</span>)</span></span></code></pre>
Output (second run to avoid compilation time):</p>
Step 1 solution:</span></span>
x=44166763, y=43480550, z=38585775,</span></span>
ans=126233088</span></span>
Step 2 solution:</span></span>
x=44166763, y=43480550, z=38585775</span></span>
ans=126233088</span></span>
  1.688667 seconds (590.15 k allocations: 32.233 MiB, 0.86% gc time)</span></span></code></pre>
^{1</sup>
You may be wondering why I didn't use round(Int, objective_value(model))</code>. I found that for some values of slack</code>, the objective_value</code> is not the value of the objective at the solution (x, y, z). For example, the solution for step one results in an objective_value</code> of 969 when the true number of bots in range of (x, y, z) is 972. 969 is still larger than the objective function evaluated at any other point, though. There may be a setting for HiGHS—that I'm unaware of—to resolve this issue.</p>
</div>
^{2</sup>
We could do this in the first step as well, but I'm more confident in finding the global maxbots</code> without the penalty in the first step, then trying to minimize the L1 norm of the solution.</p>
</div>
^{3</sup>
If the slack is too low</em>, then the problem is incorrectly specified. In the extreme case where slack is 0, the problem is to find a point within every bot's range, which isn't feasible. However, there shouldn't be an issue—mathematically—with a slack that is too large.</p>
</div>}}}


Z3.jl Optimization Example
2023-02-14T00:00:00+00:00
Z3</a> is a satisfiability modulo
theories (SMT) solver callable from Julia with Z3.jl</a>.
Documentation is somewhat lacking, so I'd like to share a worked example
of optimizing an integer programming problem with Z3.jl.</p>
We will solve Advent of Code 2018 day 23</a>, so if you'd like to avoid spoilers,
stop reading.
The problem provides a list of nanobots defined by a center point in 3d space and
radius, measured by Manhattan distance.
For part 2 of the problem, we must find the (integer) point that lies within range of the largest number of nanobots.
If there are multiple such points, we must find the one closest to the origin (0, 0, 0).</p>
First, we'll define a struct to hold the nanobot data and process the input
file.1</a></sup></p>
using</span> Z3</span></span>
</span>
struct</span> Nanobot</span></span>
    x</span>::</span>Int</span></span>
    y</span>::</span>Int</span></span>
    z</span>::</span>Int</span></span>
    radius</span>::</span>Int</span></span>
end</span></span>
</span>
function</span> processinput</span>(</span>filename</span>)</span></span>
    puzin </span>=</span> [</span>[</span>parse</span>(</span>Int</span>,</span> m</span>.</span>match</span>)</span> for</span> m </span>in</span> eachmatch</span>(</span>r"</span>(</span>\-</span>?</span>\d</span>+)</span>"</span>,</span> line</span>)</span>]</span></span>
             for</span> line </span>in</span> readlines</span>(</span>filename</span>)</span>]</span></span>
    bots </span>=</span> [</span>Nanobot</span>(</span>l</span>...</span>)</span> for</span> l </span>in</span> puzin</span>]</span></span>
    return</span> bots</span></span>
end</span></span>
</span>
bots </span>=</span> processinput</span>(</span>"</span>input201823.txt</span>"</span>)</span></span></code></pre>
Let's think about how we may define the problem as an integer optimization
that Z3 can understand.
Consider a potential solution point (x, y, z)</code>.
The point is outside the radius of nanobot bot</code> if and only if
abs(bot.x - x) + abs(bot.y - y) + abs(bot.z - z) > bot.radius</code>.
We want to minimize</em> the number bots for which that statement is true.
We can collect the statements in an array, and sum the boolean values (1 = true).
The resulting sum (call it noutofrange</code>) will be our objective to minimize.</p>
Finally, conditional on minimizing noutofrange</code>, we want to choose the
point closest to the origin, or minimize abs(x) + abs(y) + abs(z)</code> (call this value cost</code>).
Z3 uses by default a lexicographic priority of objectives.
It solves first for the objective that is declared first, so this should be easy.</p>
Define the Z3 context and necessary variables.
We'll see why we need z3zero</code> and z3one</code> in a moment.</p>
ctx </span>=</span> Context</span>(</span>)</span></span>
x </span>=</span> int_const</span>(</span>ctx</span>,</span> "</span>x</span>"</span>)</span></span>
y </span>=</span> int_const</span>(</span>ctx</span>,</span> "</span>y</span>"</span>)</span></span>
z </span>=</span> int_const</span>(</span>ctx</span>,</span> "</span>z</span>"</span>)</span></span>
z3zero </span>=</span> int_val</span>(</span>ctx</span>,</span> 0</span>)</span></span>
z3one </span>=</span> int_val</span>(</span>ctx</span>,</span> 1</span>)</span></span>
noutofrange </span>=</span> int_const</span>(</span>ctx</span>,</span> "</span>noutofrange</span>"</span>)</span></span>
cost </span>=</span> int_const</span>(</span>ctx</span>,</span> "</span>cost</span>"</span>)</span></span></code></pre>
The array of boolean values (converted to integer 1 or 0) measuring whether the point is within range of each bot
may now be defined as</p>
botsoutofrange </span>=</span> [</span>ite</span>(</span>abs</span>(</span>b</span>.</span>x </span>-</span> x</span>)</span> +</span> abs</span>(</span>b</span>.</span>y </span>-</span> y</span>)</span> +</span> abs</span>(</span>b</span>.</span>z </span>-</span> z</span>)</span> ></span> b</span>.</span>radius</span>,</span></span>
                      z3one</span>,</span> z3zero</span>)</span> for</span> b </span>in</span> bots</span>]</span></span></code></pre>
ite(a, b, c)</code> (if then else) is a Z3 method that means</p>
if a</span></span>
    return b</span></span>
else</span></span>
    return c</span></span>
end</span></span></code></pre>
We need b</code> and c</code> to be Z3 expressions, hence the use of our previously defined
z3one</code> and z3zero</code>.</p>
Now, we initialize the optimization problem with the "constraints" that
define our objective variables,</p>
opt </span>=</span> Optimize</span>(</span>ctx</span>)</span></span>
add</span>(</span>opt</span>,</span> noutofrange </span>==</span> sum</span>(</span>botsoutofrange</span>)</span>)</span></span>
add</span>(</span>opt</span>,</span> cost </span>==</span> abs</span>(</span>x</span>)</span> +</span> abs</span>(</span>y</span>)</span> +</span> abs</span>(</span>z</span>)</span>)</span></span></code></pre>
and finally optimize each of the objectives in order.</p>
minimize</span>(</span>opt</span>,</span> noutofrange</span>)</span></span>
minimize</span>(</span>opt</span>,</span> cost</span>)</span></span>
res </span>=</span> check</span>(</span>opt</span>)</span></span></code></pre>
Confirm that the problem is satisfiable and print the optimal point.</p>
@assert</span> res </span>==</span> Z3</span>.</span>sat</span></span>
m </span>=</span> get_model</span>(</span>opt</span>)</span></span>
for</span> (</span>k</span>,</span> v</span>)</span> in</span> consts</span>(</span>m</span>)</span></span>
    println</span>(</span>"</span>$k</span> = </span>$v</span>"</span>)</span></span>
end</span></span></code></pre>
For my input, this prints</p>
noutofrange = 28</span></span>
cost = 126233088</span></span>
z = 38585775</span></span>
y = 43480550</span></span>
x = 44166763</span></span></code></pre>
It takes about a minute to solve on my PC. That may seem slow, but the human
time required to code a faster algorithm is a lot longer than a minute!^{2</a></sup>
Z3 is one of those tools that isn't often applicable, but when it is, it feels like magic.</p>}
Full Julia script</a>
</h2>
using</span> Z3</span></span>
</span>
struct</span> Nanobot</span></span>
    x</span>::</span>Int</span></span>
    y</span>::</span>Int</span></span>
    z</span>::</span>Int</span></span>
    radius</span>::</span>Int</span></span>
end</span></span>
</span>
function</span> processinput</span>(</span>filename</span>)</span></span>
    puzin </span>=</span> [</span>[</span>parse</span>(</span>Int</span>,</span> m</span>.</span>match</span>)</span> for</span> m </span>in</span> eachmatch</span>(</span>r"</span>(</span>\-</span>?</span>\d</span>+)</span>"</span>,</span> line</span>)</span>]</span></span>
             for</span> line </span>in</span> readlines</span>(</span>filename</span>)</span>]</span></span>
    bots </span>=</span> [</span>Nanobot</span>(</span>l</span>...</span>)</span> for</span> l </span>in</span> puzin</span>]</span></span>
    return</span> bots</span></span>
end</span></span>
</span>
function</span> main</span>(</span>filename</span>)</span></span>
    bots </span>=</span> processinput</span>(</span>filename</span>)</span></span>
    ctx </span>=</span> Context</span>(</span>)</span></span>
    x </span>=</span> int_const</span>(</span>ctx</span>,</span> "</span>x</span>"</span>)</span></span>
    y </span>=</span> int_const</span>(</span>ctx</span>,</span> "</span>y</span>"</span>)</span></span>
    z </span>=</span> int_const</span>(</span>ctx</span>,</span> "</span>z</span>"</span>)</span></span>
    z3zero </span>=</span> int_val</span>(</span>ctx</span>,</span> 0</span>)</span></span>
    z3one </span>=</span> int_val</span>(</span>ctx</span>,</span> 1</span>)</span></span>
    noutofrange </span>=</span> int_const</span>(</span>ctx</span>,</span> "</span>noutofrange</span>"</span>)</span></span>
    cost </span>=</span> int_const</span>(</span>ctx</span>,</span> "</span>cost</span>"</span>)</span></span>
    botsoutofrange </span>=</span> [</span>ite</span>(</span>abs</span>(</span>b</span>.</span>x </span>-</span> x</span>)</span> +</span> abs</span>(</span>b</span>.</span>y </span>-</span> y</span>)</span> +</span> abs</span>(</span>b</span>.</span>z </span>-</span> z</span>)</span> ></span> b</span>.</span>radius</span>,</span> z3one</span>,</span> z3zero</span>)</span></span>
                      for</span> b </span>in</span> bots</span>]</span></span>
    opt </span>=</span> Optimize</span>(</span>ctx</span>)</span></span>
    add</span>(</span>opt</span>,</span> noutofrange </span>==</span> sum</span>(</span>botsoutofrange</span>)</span>)</span></span>
    add</span>(</span>opt</span>,</span> cost </span>==</span> abs</span>(</span>x</span>)</span> +</span> abs</span>(</span>y</span>)</span> +</span> abs</span>(</span>z</span>)</span>)</span></span>
    minimize</span>(</span>opt</span>,</span> noutofrange</span>)</span></span>
    minimize</span>(</span>opt</span>,</span> cost</span>)</span></span>
    res </span>=</span> check</span>(</span>opt</span>)</span></span>
    @assert</span> res </span>==</span> Z3</span>.</span>sat</span></span>
    m </span>=</span> get_model</span>(</span>opt</span>)</span></span>
    for</span> (</span>k</span>,</span> v</span>)</span> in</span> consts</span>(</span>m</span>)</span></span>
        println</span>(</span>"</span>$k</span> = </span>$v</span>"</span>)</span></span>
    end</span></span>
    return</span> nothing</span></span>
end</span></span>
</span>
@time</span> main</span>(</span>"</span>input201823.txt</span>"</span>)</span></span></code></pre>
Output:</p>
noutofrange = 28</span></span>
cost = 126233088</span></span>
z = 38585775</span></span>
y = 43480550</span></span>
x = 44166763</span></span>
 52.030344 seconds (71.38 k allocations: 3.053 MiB, 0.07% compilation time)</span></span></code></pre>Comparison with Python</a>
</h2>
The script below is the same problem solved with the Python package z3-solver</code>.
I had to manually define abs</code> to work with Z3 in Python, but Python didn't require ite</code> or
definition of the 0 and 1 Z3 expressions like Julia.
I also had occasional segfaults in the Julia version, although
it wasn't reproducible when executing the file in a fresh REPL.
It may be related to this issue: https://github.com/ahumenberger/Z3.jl/issues/12</a>.</p>
import</span> z3</span></span>
import</span> re</span></span>
import</span> time</span></span>
</span>
with</span> open</span>(</span>'</span>input/input201823.txt</span>'</span>,</span> '</span>r</span>'</span>)</span> as</span> f</span>:</span></span>
    actualinput</span> =</span> f</span>.</span>read</span>(</span>)</span></span>
</span>
bots</span> =</span> [</span>list</span>(</span>map</span>(</span>int</span>,</span> re</span>.</span>findall</span>(</span>'</span>(\-?\d+)</span>'</span>,</span> row</span>)</span>)</span>)</span> for</span> row</span> in</span> actualinput</span>.</span>splitlines</span>(</span>)</span>]</span></span>
</span>
#</span> https://stackoverflow.com/questions/22547988/how-to-calculate-absolute-value-in-z3-or-z3py</span></span>
def</span> z3abs</span>(</span>x</span>)</span>:</span></span>
    return</span> z3</span>.</span>If</span>(</span>x</span> >=</span> 0</span>,</span>x</span>,</span>-</span>x</span>)</span></span>
</span>
x</span> =</span> z3</span>.</span>Int</span>(</span>'</span>x</span>'</span>)</span></span>
y</span> =</span> z3</span>.</span>Int</span>(</span>'</span>y</span>'</span>)</span></span>
z</span> =</span> z3</span>.</span>Int</span>(</span>'</span>z</span>'</span>)</span></span>
noutofrange</span> =</span> z3</span>.</span>Int</span>(</span>'</span>noutofrange</span>'</span>)</span></span>
cost</span> =</span> z3</span>.</span>Int</span>(</span>'</span>cost</span>'</span>)</span></span>
</span>
botsoutofrange</span> =</span> [</span>z3abs</span>(</span>b</span>[</span>0</span>]</span>-</span>x</span>)</span> +</span> z3abs</span>(</span>b</span>[</span>1</span>]</span>-</span>y</span>)</span> +</span> z3abs</span>(</span>b</span>[</span>2</span>]</span>-</span>z</span>)</span> ></span> b</span>[</span>3</span>]</span> for</span> b</span> in</span> bots</span>]</span></span>
</span>
opt</span> =</span> z3</span>.</span>Optimize</span>(</span>)</span></span>
opt</span>.</span>add</span>(</span>noutofrange</span> ==</span> z3</span>.</span>Sum</span>(</span>botsoutofrange</span>)</span>)</span></span>
opt</span>.</span>add</span>(</span>cost</span> ==</span> z3abs</span>(</span>x</span>)</span> +</span> z3abs</span>(</span>y</span>)</span> +</span> z3abs</span>(</span>z</span>)</span>)</span></span>
#</span> Z3 uses by default a lexicographic priority of objectives. It solves first for the objective that is declared first.</span></span>
lowestoutofrange</span> =</span> opt</span>.</span>minimize</span>(</span>noutofrange</span>)</span></span>
closesttoorigin</span> =</span> opt</span>.</span>minimize</span>(</span>cost</span>)</span></span>
print</span>(</span>"</span>Checking optimization...</span>"</span>)</span></span>
start_time</span> =</span> time</span>.</span>time</span>(</span>)</span></span>
opt</span>.</span>check</span>(</span>)</span></span>
print</span>(</span>f</span>"</span>Found solution in </span>{</span>time</span>.</span>time</span>(</span>)</span> -</span> start_time</span>:.2f</span>}</span>s</span>"</span>)</span></span>
print</span>(</span>opt</span>.</span>model</span>(</span>)</span>)</span></span></code></pre>
^{1</sup>
You can use the following example input if you don't want to log in to Advent of Code.

pos=<10,12,12>, r=2</code>

pos=<12,14,12>, r=2</code>

pos=<16,12,12>, r=4</code>

pos=<14,14,14>, r=6</code>

pos=<50,50,50>, r=200</code>

pos=<10,10,10>, r=5</code></p>
</div>}^2</sup>The problem can be reformulated as a mixed-integer linear program, which is
solvable nearly instantly by an optimizer like HiGHS</a>, but that still
requires some human effort to think through and implement the reformulation.</p>
</div>


Scraping Discord pins
2023-02-01T00:00:00+00:00

Discord doesn't like</a> automation of user accounts, which is the method discussed in this post. Use at your own risk.</p>
</blockquote>
As more online communities move to Discord, it's becoming increasingly
difficult to find coherent information within those communities.
Often, important findings and frequently asked questions are scattered in
pinned messages throughout channels in the server.
Discord search is not great, and navigating through pins is tiresome, so I
wrote a hacky script to scrape all pinned messages in a server to
a single Markdown file.</p>
The script grabs the user's authentication token from the browser to
call Discord's API. A bot would be more convenient and maintainable,
but bots need approval by server admins, which is yet another step of friction.
Instead, we may run this script directly in Chrome's JavaScript console, for example, which is
good enough for my purposes. If you don't know what it means to use the JS console,
you shouldn't be copying and pasting code you found on the internet. Please do some
searching to learn before coming back here; this</a> is
a good resource from the Chrome team.</em></p>
The only manual input for the script is the guild ID (guilds are also called servers).
To get the ID, open Discord, go to Settings > Advanced and enable developer mode. Then, right-click on the server title and select "Copy ID" to get the guild ID.</p>
The script grabs the user authentication token from the app's storage.
The code here is copied from Stack Overflow; refer to the linked question if it stops working.
Maybe someone has found a new method.
Next, we use the API endpoint /guilds/{guild.id}/channels</code></a> to get a list of
channels in the server and sort by display position.
Then, we loop through the channels to get pinned message content with
/channels/{channel.id}/pins</code></a>.
Images are attachments to messages, so they require some special handling.
Instead of downloading the images, we create hotlinks to the Discord CDN—which, by the way, requires no authentication.
This may or may not work for your use case; one could manually download the images instead.</p>
Finally, we stuff the channel content and image links into a single <pre></code> tag, which we
then display in a new window.
The Markdown structure is an H1 header of the channel name followed by all pinned message contents.
The content of the pop-up should be copiable as Markdown.
Formatting errors aside, one may render the Markdown however one chooses for a single easily searchable and browseable help file.</p>
//</span> To get the server ID, open Discord, go to Settings > Advanced and enable developer mode. Then, right-click on the server title and select "Copy ID" to get the guild ID.</span></span>
let</span> GUILDID</span> =</span> "</span>1111</span>"</span>;</span> //</span> CHANGE ME</span></span>
</span>
//</span> See https://stackoverflow.com/a/69868564</span></span>
let</span> TOKEN</span> =</span> (</span>webpackChunkdiscord_app</span>.</span>push</span>(</span>[</span>[</span>'</span>'</span>]</span>,</span>{</span>}</span>,</span>e</span>=></span>{</span>m</span>=</span>[</span>]</span>;</span>for</span>(</span>let</span> c</span> in</span> e</span>.</span>c</span>)</span>m</span>.</span>push</span>(</span>e</span>.</span>c</span>[</span>c</span>]</span>)</span>}</span>]</span>)</span>,</span>m</span>)</span>.</span>find</span>(</span>m</span>=></span>m</span>?.</span>exports</span>?.</span>default</span>?.</span>getToken</span>!==</span>void</span> 0</span>)</span>.</span>exports</span>.</span>default</span>.</span>getToken</span>(</span>)</span>;</span></span>
</span>
let</span> textchannels</span> =</span> await</span> fetch</span>(</span>"</span>https://discord.com/api/v9/guilds/</span>"</span> +</span> GUILDID</span> +</span> "</span>/channels</span>"</span>,</span> {</span></span>
  "</span>headers</span>"</span>:</span> {</span></span>
    "</span>accept</span>"</span>:</span> "</span>*/*</span>"</span>,</span></span>
    "</span>authorization</span>"</span>:</span> TOKEN</span>,</span></span>
 }</span>,</span></span>
  "</span>body</span>"</span>:</span> null</span>,</span></span>
  "</span>method</span>"</span>:</span> "</span>GET</span>"</span>,</span></span>
  "</span>mode</span>"</span>:</span> "</span>cors</span>"</span>,</span></span>
  "</span>credentials</span>"</span>:</span> "</span>include</span>"</span></span>
}</span>)</span></span>
  .</span>then</span>(</span>response</span> =></span> response</span>.</span>json</span>(</span>)</span>)</span></span>
  .</span>then</span>(</span>channels</span> =></span> {</span></span>
        let</span> result</span> =</span> [</span>]</span>;</span></span>
        for</span> (</span>let</span> i</span> =</span> 0</span>;</span> i</span> <</span> channels</span>.</span>length</span>;</span> i</span>++</span>)</span> {</span></span>
            if</span> (</span>channels</span>[</span>i</span>]</span>.</span>type</span> ===</span> 0</span>)</span>{</span></span>
            result</span>.</span>push</span>(</span>channels</span>[</span>i</span>]</span>)</span>;</span></span>
            }</span></span>
        }</span></span>
        return</span> result</span>;</span></span>
    }</span></span>
)</span>;</span></span>
</span>
//</span> https://stackoverflow.com/a/69026789</span></span>
textchannels</span>.</span>sort</span>(</span>(</span>a</span>,</span> b</span>)</span> =></span> {</span></span>
    var</span> ret</span>;</span></span>
    if</span> (</span>a</span>.</span>position</span> <</span> b</span>.</span>position</span>)</span> {</span></span>
        ret</span> =</span> -</span>1</span>;</span></span>
    }</span> else</span> if</span> (</span>a</span>.</span>position</span> ></span> b</span>.</span>position</span>)</span> {</span></span>
        ret</span> =</span> 1</span>;</span></span>
    }</span> else</span> {</span></span>
        ret</span> =</span> 0</span>;</span></span>
    }</span></span>
    return</span> ret</span></span>
}</span>)</span>;</span></span>
</span>
let</span> pincontents</span> =</span> [</span>'</span><pre></span>'</span>]</span>;</span></span>
</span>
for</span> (</span>let</span> i</span> =</span> 0</span>;</span> i</span> <</span> textchannels</span>.</span>length</span>;</span> i</span>++</span>)</span>{</span></span>
    console</span>.</span>log</span>(</span>"</span>Getting pins for channel = </span>"</span> +</span> JSON</span>.</span>stringify</span>(</span>textchannels</span>[</span>i</span>]</span>)</span>)</span>;</span></span>
    pincontents</span>.</span>push</span>(</span>"</span>[//]: # (COMMENT: channel = </span>"</span> +</span> JSON</span>.</span>stringify</span>(</span>textchannels</span>[</span>i</span>]</span>)</span> +</span> '</span>)</span>'</span>)</span>;</span></span>
    pincontents</span>.</span>push</span>(</span>"</span># </span>"</span> +</span> textchannels</span>[</span>i</span>]</span>[</span>'</span>name</span>'</span>]</span>)</span>;</span></span>
    //</span> https://stackoverflow.com/a/51939030</span></span>
    //</span> Need to avoid 429</span></span>
    await</span> new</span> Promise</span>(</span>resolve</span> =></span> setTimeout</span>(</span>resolve</span>,</span> 2000</span>)</span>)</span>;</span></span>
    await</span> fetch</span>(</span>"</span>https://discord.com/api/v9/channels/</span>"</span> +</span> textchannels</span>[</span>i</span>]</span>[</span>'</span>id</span>'</span>]</span> +</span> "</span>/pins</span>"</span>,</span> {</span></span>
        "</span>headers</span>"</span>:</span> {</span></span>
            "</span>accept</span>"</span>:</span> "</span>*/*</span>"</span>,</span></span>
            "</span>authorization</span>"</span>:</span> TOKEN</span>,</span></span>
        }</span>,</span></span>
        "</span>body</span>"</span>:</span> null</span>,</span></span>
        "</span>method</span>"</span>:</span> "</span>GET</span>"</span>,</span></span>
        "</span>mode</span>"</span>:</span> "</span>cors</span>"</span>,</span></span>
        "</span>credentials</span>"</span>:</span> "</span>include</span>"</span></span>
        }</span>)</span></span>
        .</span>then</span>(</span>response</span> =></span> response</span>.</span>json</span>(</span>)</span>)</span></span>
        .</span>then</span>(</span>pins</span> =></span> {</span></span>
                for</span> (</span>let</span> i</span> =</span> 0</span>;</span> i</span> <</span> pins</span>.</span>length</span>;</span> i</span>++</span>)</span> {</span></span>
                    pincontents</span>.</span>push</span>(</span>pins</span>[</span>i</span>]</span>[</span>'</span>content</span>'</span>]</span>)</span>;</span></span>
                    for</span> (</span>let</span> j</span> =</span> 0</span>;</span> j</span> <</span> pins</span>[</span>i</span>]</span>[</span>'</span>attachments</span>'</span>]</span>.</span>length</span>;</span> j</span>++</span>)</span> {</span></span>
                        pincontents</span>.</span>push</span>(</span>'</span>\n</span>![Image](</span>'</span> +</span> pins</span>[</span>i</span>]</span>[</span>'</span>attachments</span>'</span>]</span>[</span>j</span>]</span>[</span>'</span>url</span>'</span>]</span> +</span> '</span>)</span>\n</span>'</span>)</span>;</span></span>
                    }</span></span>
                    </span></span>
                }</span></span>
            }</span></span>
        )</span>;</span></span>
}</span></span>
pincontents</span>.</span>push</span>(</span>'</span></pre></span>'</span>)</span>;</span></span>
</span>
console</span>.</span>log</span>(</span>pincontents</span>.</span>join</span>(</span>"</span>\n</span>\n</span>"</span>)</span>)</span>;</span></span>
</span>
//</span> May need to allow pop-ups. If it fails, just run these lines again.</span></span>
var</span> win</span> =</span> window</span>.</span>open</span>(</span>"</span>"</span>,</span> "</span>Pins</span>"</span>,</span> "</span>toolbar=no,location=no,directories=no,status=no,menubar=no,scrollbars=yes,resizable=yes,width=780,height=200,top=</span>"</span>+</span>(</span>screen</span>.</span>height</span>-</span>400</span>)</span>+</span>"</span>,left=</span>"</span>+</span>(</span>screen</span>.</span>width</span>-</span>840</span>)</span>)</span>;</span></span>
win</span>.</span>document</span>.</span>body</span>.</span>innerHTML</span> =</span> pincontents</span>.</span>join</span>(</span>"</span>\n</span>\n</span>"</span>)</span>;</span></span></code></pre>
Sample output (abbreviated):</p>
</span>
[</span>//</span>]</span>:</span> </span>#</span> </span>(</span>COMMENT: channel = {..."name":"displayedfirst","position":0,...}</span>)</span></span>
</span>
#</span> displayedfirst</span></span>
</span>
asdf</span></span>
</span>
[</span>//</span>]</span>:</span> </span>#</span> </span>(</span>COMMENT: channel = {..."name":"general","position":1...}</span>)</span></span>
</span>
#</span> general</span></span>
</span>
longer message </span></span>
😀 </span></span>
maybe</span></span>
</span>
</span>
![</span>Image</span>]</span>(</span>https://cdn.discordapp.com/attachments/....png</span>)</span></span>
</span>
</span>
![</span>Image</span>]</span>(</span>https://cdn.discordapp.com/attachments/....png</span>)</span></span>
</span>
</span>
Another pin with bold </span>*</span>asdf</span>*</span> _</span>bold</span>_</span></span>
1.</span> 3</span></span>
2.</span> 6</span></span>
</span>
test pinned</span></span>
</span>
[</span>//</span>]</span>:</span> </span>#</span> </span>(</span>COMMENT: channel = {..."name":"otherchannel","position":2,...</span>)</span></span>
</span>
#</span> otherchannel</span></span>
</span>
other chan pin</span></span></code></pre>
One nagging issue is 429 (Too Many Requests)
responses when querying each channel. The script simply waits 2 seconds between
requests to avoid them. For large servers, this may be annoyingly long, but
retry logic just complicates the script.</p>


Solving Peg Solitaire with Julia
2022-06-18T00:00:00+00:00

Peg solitaire</a> is a singleplayer board game with the objective to remove all game pieces (pegs or marbles) except one from the board by "jumping" them with another peg.
The 15-hole triangular variant is commonly found in Cracker Barrel</a> restaurants in the US.
We will solve this variant with Julia.</p>
The triangular board is laid out as a hexagonal grid, so jumps can occur by moving northwest, northeast, east, southeast, southwest, or west.
To simplify our code, we will represent the board as a BitMatrix, a 2d array of 1s and 0s, and restrict moves to valid indices.</p>
INDICES </span>=</span> Vector</span>{Tuple{Int, Int}}</span>(</span>)</span></span>
board </span>=</span> falses</span>(</span>5</span>,</span> 9</span>)</span></span>
for</span> col </span>in</span> axes</span>(</span>board</span>,</span> 2</span>)</span></span>
    for</span> row </span>in</span> axes</span>(</span>board</span>,</span> 1</span>)</span></span>
        if</span> (</span>(</span>row </span>%</span> 2</span> ==</span> col </span>%</span> 2</span>)</span> &&</span> (</span>col </span>>=</span> row</span>)</span> &&</span></span>
                (</span>col </span><=</span> size</span>(</span>board</span>,</span> 2</span>)</span> +</span> 1</span> -</span> row</span>)</span>)</span></span>
            board</span>[</span>row</span>,</span> col</span>]</span> =</span> true</span></span>
            push!</span>(</span>INDICES</span>,</span> (</span>row</span>,</span> col</span>)</span>)</span></span>
        end</span></span>
    end</span></span>
end</span></span>
println</span>(</span>"</span>The board is represented as</span>"</span>)</span></span>
display</span>(</span>board</span>)</span></span>
println</span>(</span>"</span>Valid indices are</span>"</span>)</span></span>
println</span>(</span>INDICES</span>)</span></span></code></pre>The board is represented as</span></span>
5×9 BitMatrix:</span></span>
 1  0  1  0  1  0  1  0  1</span></span>
 0  1  0  1  0  1  0  1  0</span></span>
 0  0  1  0  1  0  1  0  0</span></span>
 0  0  0  1  0  1  0  0  0</span></span>
 0  0  0  0  1  0  0  0  0</span></span>
Valid indices are</span></span>
[(1, 1), (2, 2), (1, 3), (3, 3), (2, 4), (4, 4),</span></span>
(1, 5), (3, 5), (5, 5), (2, 6), (4, 6), (1, 7),</span></span>
(3, 7), (2, 8), (1, 9)]  </span></span></code></pre>
To make our life a bit easier, let's first write a function to render the board.
I love printstyled</code> by the way.
If you use the Julia REPL or a supported terminal, the pegs will be cyan.</p>
function</span> showboard</span>(</span>board</span>;</span> upsidedown</span>=</span>false</span>)</span></span>
    if</span> upsidedown</span></span>
        iiter </span>=</span> reverse</span>(</span>axes</span>(</span>board</span>,</span> 1</span>)</span>)</span></span>
    else</span></span>
        iiter </span>=</span> axes</span>(</span>board</span>,</span> 1</span>)</span></span>
    end</span></span>
    println</span>(</span>"</span>\n</span>------------------</span>"</span>)</span></span>
    for</span> i </span>in</span> iiter</span></span>
        for</span> j </span>in</span> axes</span>(</span>board</span>,</span> 2</span>)</span></span>
            if</span> board</span>[</span>i</span>,</span> j</span>]</span></span>
                printstyled</span>(</span>'</span>█</span>'</span>;</span> color</span>=</span>:cyan</span>)</span></span>
            elseif</span> (</span>i </span>%</span> 2</span> ==</span> j </span>%</span> 2</span>)</span> &&</span> (</span>j </span>>=</span> i</span>)</span> &&</span></span>
                    (</span>j </span><=</span> size</span>(</span>board</span>,</span> 2</span>)</span> +</span> 1</span> -</span> i</span>)</span></span>
                printstyled</span>(</span>'</span>o</span>'</span>;</span> color</span>=</span>:nothing</span>)</span></span>
            else</span></span>
                printstyled</span>(</span>'</span>░</span>'</span>;</span> color</span>=</span>:nothing</span>)</span></span>
            end</span></span>
        end</span></span>
        if</span> i </span>==</span> first</span>(</span>iiter</span>)</span> +</span> (</span>last</span>(</span>iiter</span>)</span> -</span> first</span>(</span>iiter</span>)</span>)</span>÷</span>2</span></span>
            print</span>(</span>"</span> pegs: </span>$</span>(</span>sum</span>(</span>board</span>)</span>)</span>"</span>)</span></span>
        end</span></span>
        print</span>(</span>"</span>\n</span>"</span>)</span></span>
    end</span></span>
    print</span>(</span>"</span>------------------</span>\n</span>"</span>)</span></span>
end</span></span></code></pre>board</span>[</span>1</span>,</span> 1</span>]</span> =</span> false</span> #</span> remove a peg for illustration</span></span>
showboard</span>(</span>board</span>)</span></span></code></pre>------------------</span></span>
o░█░█░█░█</span></span>
░█░█░█░█░</span></span>
░░█░█░█░░ pegs: 14</span></span>
░░░█░█░░░</span></span>
░░░░█░░░░</span></span>
------------------</span></span></code></pre>
This board is a potential starting position.
All holes are filled with pegs except the top left one.</p>
Next, we need to determine the potential moves.
In other words, given the current state of the board, what are the potential
states of the board after one jump?
For each peg, consider its neighbor positions (northwest, northeast, east,
southeast, southwest, or west).
The neighbor position must contain a peg, so we have something to jump.
In addition, the neighbor's neighbor in the same direction must be empty,
so we have a hole to jump to.</p>
Using the following figure, suppose i</code> is the location of the peg that will jump.
Then, locations marked n</code> are its neighbors, and n2</code> are the neighbors' neighbors at a valid index.
There are two potential moves: (1) jump from 3,5 to 1,3 and remove the peg at 2,4, and (2)
jump from 3,5 to 1,7 and remove the peg at 2,6.
As long as the applicable neighbor positions have a peg and the neighbors' neighbors are empty, these moves are valid.</p>
  1  2  3  4  5  6  7  8  9 </span></span>
1       n2          n2   </span></span>
2          n     n    </span></span>
3       n     i     n   </span></span>
4          n     n    </span></span>
5  </span></span></code></pre>
We must repeat this process, checking for valid moves for each peg in the board.
The following function returns a vector of BitMatrix representing board states reachable in one
move from the current board state (an input BitMatrix).
The input indices</code> is a vector of tuples representing valid indices for peg locations, so
we don't have to compute them for each call of the function.
mdist</code> is a helper function to find valid neighbor locations.</p>
function</span> mdist</span>(</span>a</span>,</span> b</span>)</span></span>
    return</span> abs</span>(</span>a</span>[</span>1</span>]</span> -</span> b</span>[</span>1</span>]</span>)</span> +</span> abs</span>(</span>a</span>[</span>2</span>]</span> -</span> b</span>[</span>2</span>]</span>)</span></span>
end</span></span>
</span>
function</span> getnextstates</span>(</span>board</span>,</span> indices</span>)</span></span>
    nextstates </span>=</span> Vector</span>{BitMatrix}</span>(</span>)</span></span>
    for</span> ind </span>in</span> indices</span></span>
        #</span> (1) there must be a peg to use to jump</span></span>
        if</span> !</span>board</span>[</span>ind</span>...</span>]</span></span>
            continue</span></span>
        end</span></span>
        neighbors </span>=</span> (</span>n </span>for</span> n </span>in</span> indices </span>if</span> ind</span>[</span>2</span>]</span> !=</span> n</span>[</span>2</span>]</span> &&</span></span>
                                           mdist</span>(</span>ind</span>,</span> n</span>)</span> <=</span> 2</span>)</span></span>
        for</span> n </span>in</span> neighbors</span></span>
            #</span> (2) the neighbor location must contain a peg</span></span>
            if</span> !</span>board</span>[</span>n</span>...</span>]</span></span>
                continue</span></span>
            end</span></span>
            n2 </span>=</span> (</span>n</span>[</span>1</span>]</span> -</span> (</span>ind</span>[</span>1</span>]</span> -</span> n</span>[</span>1</span>]</span>)</span>,</span> n</span>[</span>2</span>]</span> -</span> (</span>ind</span>[</span>2</span>]</span> -</span> n</span>[</span>2</span>]</span>)</span>)</span></span>
            #</span> (3) the neighbor's neighbor in the same direction</span></span>
            #</span> must be a valid location and empty</span></span>
            if</span> n2 </span>in</span> indices </span>&&</span> !</span>board</span>[</span>n2</span>...</span>]</span></span>
                #</span> If (1), (2), and (3) are satisfied,</span></span>
                #</span> make a copy of the current board.</span></span>
                newboard </span>=</span> copy</span>(</span>board</span>)</span></span>
                #</span> Remove the jumping peg from its current location.</span></span>
                newboard</span>[</span>ind</span>...</span>]</span> =</span> false</span></span>
                #</span> Remove the jumped peg.</span></span>
                newboard</span>[</span>n</span>...</span>]</span> =</span> false</span></span>
                #</span> Place the jumping peg in its new location.</span></span>
                newboard</span>[</span>n2</span>...</span>]</span> =</span> true</span></span>
                #</span> Save the new board state.</span></span>
                push!</span>(</span>nextstates</span>,</span> newboard</span>)</span></span>
            end</span></span>
        end</span></span>
    end</span></span>
    return</span> nextstates</span></span>
end</span></span></code></pre>
Now that we have a way to compute the states reachable from the current state,
we can apply depth-first search</a>
(DFS) to find a solution.
A full explanation of DFS is outside the scope of this note.
However, we use the iterative form (rather than recursive), and the general flow for this problem is
as follows:</p>


Initialize a vector of states to search (states</code>), beginning with the current state (board</code>).
Initialize an empty vector of states representing the path to the current state (pathtocurrentstate</code>).1</a></sup></p>
</li>

Pop a state from the vector of states to search, and add it to the path.</p>
</li>

Check if the state is a winner, i.e. the number of pegs is 1. If so, return the path.</p>
</li>

Generate the next possible states, and append them to the vector to be searched next.</p>
</li>

Go to 2.</p>
</li>
</ol>
function</span> findsolution</span>(</span>board</span>,</span> indices</span>)</span></span>
    states </span>=</span> [</span>copy</span>(</span>board</span>)</span>]</span></span>
    pathtocurrentstate </span>=</span> [</span>copy</span>(</span>board</span>)</span>]</span></span>
    pop!</span>(</span>pathtocurrentstate</span>)</span></span>
    while</span> !</span>isempty</span>(</span>states</span>)</span></span>
        currentstate </span>=</span> pop!</span>(</span>states</span>)</span></span>
        while</span> (</span>length</span>(</span>pathtocurrentstate</span>)</span> ></span> 0</span> &&</span></span>
               sum</span>(</span>pathtocurrentstate</span>[</span>end</span>]</span>)</span> <=</span> sum</span>(</span>currentstate</span>)</span>)</span></span>
            #</span> Use the sum as a measure of depth.</span></span>
            #</span> Remove any same or lower depth states before pushing</span></span>
            #</span> the current state to the path.</span></span>
            #</span> This is necessary when we recurse "up".</span></span>
            pop!</span>(</span>pathtocurrentstate</span>)</span></span>
        end</span></span>
        push!</span>(</span>pathtocurrentstate</span>,</span> currentstate</span>)</span></span>
        if</span> sum</span>(</span>currentstate</span>)</span> <=</span> 1</span></span>
            #</span> This is the winning state</span></span>
            return</span> pathtocurrentstate</span></span>
        end</span></span>
        nextstates </span>=</span> getnextstates</span>(</span>currentstate</span>,</span> indices</span>)</span></span>
        if</span> !</span>isempty</span>(</span>nextstates</span>)</span></span>
            #</span> If there are moves available from current_state,</span></span>
            #</span> push them to states (recursing down)</span></span>
            append!</span>(</span>states</span>,</span> nextstates</span>)</span></span>
        end</span></span>
    end</span></span>
    return</span> nothing</span></span>
end</span></span></code></pre>
Now that we have a way to find a solution, let's write a single function to
tie everything together.</p>
function</span> main</span>(</span>printsolution</span>=</span>true</span>)</span></span>
    #</span> Initialize board</span></span>
    INDICES </span>=</span> Vector</span>{Tuple{Int, Int}}</span>(</span>)</span></span>
    board </span>=</span> falses</span>(</span>5</span>,</span> 9</span>)</span></span>
    for</span> col </span>in</span> axes</span>(</span>board</span>,</span> 2</span>)</span></span>
        for</span> row </span>in</span> axes</span>(</span>board</span>,</span> 1</span>)</span></span>
            if</span> (</span>(</span>row </span>%</span> 2</span> ==</span> col </span>%</span> 2</span>)</span> &&</span> (</span>col </span>>=</span> row</span>)</span> &&</span></span>
                (</span>col </span><=</span> size</span>(</span>board</span>,</span> 2</span>)</span> +</span> 1</span> -</span> row</span>)</span>)</span></span>
                board</span>[</span>row</span>,</span> col</span>]</span> =</span> true</span></span>
                push!</span>(</span>INDICES</span>,</span> (</span>row</span>,</span> col</span>)</span>)</span></span>
            end</span></span>
        end</span></span>
    end</span></span>
</span>
    board</span>[</span>1</span>,</span>1</span>]</span> =</span> false</span></span>
    solution </span>=</span>  findsolution</span>(</span>board</span>,</span> INDICES</span>)</span></span>
    if</span> printsolution</span></span>
        if</span> isnothing</span>(</span>solution</span>)</span></span>
            println</span>(</span>"</span>We didn't find any solution.</span>"</span>)</span></span>
        else</span></span>
            println</span>(</span>"</span>Found the following solution:</span>"</span>)</span></span>
            for</span> state </span>in</span> solution</span></span>
                showboard</span>(</span>state</span>)</span></span>
            end</span></span>
        end</span></span>
    end</span></span>
    return</span> solution</span></span>
end</span></span></code></pre>main</span>(</span>)</span></span></code></pre>Found the following solution:</span></span>
</span>
------------------</span></span>
o░█░█░█░█</span></span>
░█░█░█░█░</span></span>
░░█░█░█░░ pegs: 14</span></span>
░░░█░█░░░</span></span>
░░░░█░░░░</span></span>
------------------</span></span>
</span>
------------------</span></span>
█░o░o░█░█</span></span>
░█░█░█░█░</span></span>
░░█░█░█░░ pegs: 13</span></span>
░░░█░█░░░</span></span>
░░░░█░░░░</span></span>
------------------</span></span>
</span>
...</span></span>
</span>
------------------</span></span>
o░o░o░o░o</span></span>
░o░o░o░o░</span></span>
░░o░o░o░░ pegs: 2</span></span>
░░░o░█░░░</span></span>
░░░░█░░░░</span></span>
------------------</span></span>
</span>
------------------</span></span>
o░o░o░o░o</span></span>
░o░o░o░o░</span></span>
░░o░o░█░░ pegs: 1</span></span>
░░░o░o░░░</span></span>
░░░░o░░░░</span></span>
------------------</span></span></code></pre>
You'll have to run the code yourself if you want the full solution 😀.</p>
Our method is reasonably fast,</p>
@time</span> main</span>(</span>false</span>)</span></span>
  0.003810</span> seconds </span>(</span>32.42</span> k allocations</span>:</span> 1.786</span> MiB</span>)</span></span></code></pre>
but there is quite some room for improvement, especially if we want to find all</em>
solutions.
getnextstates</code> copies a lot.
We could use an integer to directly represent the board state.
I believe this is what BitMatrix does under the hood, but it probably has some overhead.
We could cache the states we've already searched, including transformations
of the board, which is symmetric.
For more discussion on peg solitaire solution techniques or additional variants, refer to
this page</a> maintained by George Bell.</p>
^{1</sup>
The purpose of pathtocurrentstate = [copy(board)]</code> then pop!(pathtocurrentstate)</code>
is just to ensure pathtocurrentstate</code> has the same type as states</code>.</p>
</div>}


Derivation of the H-model dividend discount formula
2021-11-02T00:00:00+00:00
Introduction</a>
</h1>
Dividend discount models</a> (DDM) provide a basic mechanism to value companies by discounting expected dividend cash flows back to the current period.
Practitioners typically assume a series of dividend growth rates over a short horizon along with a constant terminal growth rate.
These assumptions imply a multi-stage model with growth rates $g_1,\ldots,g_A$ for the first $A$ periods, and a terminal growth rate $g_\infty$.
Given an initial dividend amount, $D_0$, and a discount rate, $r$, the present value today, $P_0$, is</p>
</span>
$$
P_0 = \sum_{t=1}^A \dfrac{D_0 \prod_{i=1}^t (1 + g_i)}{(1 + r)^t} +
D_A\dfrac{
\frac{1+g_{\infty}}{r-g_{\infty}}
}{
(1 + r)^A
}
\tag{1}
$$</p>
Determining the growth rates is a tricky task.
The terminal rate in particular cannot be too high, or
else the second term—often called the terminal value—will be unreasonably large.
Therefore, the earlier rates, $g_1,\ldots,g_A$, are often chosen to decrease towards a lower terminal rate.
In some cases, this strategy results in a cliff, where the drop from $g_A$ to $g_\infty$ is significant.
The drop is hard to justify qualitatively, although typically accepted to arrive at a reasonable valuation.</p>
The H-model</a>
</h1>
The H-model proposed by Fuller and Hsia (1984) seeks to alleviate these issues by assuming a linear decline in the growth rate during the first $A$ periods, so the modeler only need choose $g_1$ and $g_\infty$.
The model uses the following formula to determine the present value,</p>
</span>
$$
P_0 = \dfrac{D_0}{r - g_\infty} [(1 + g_\infty) + H(g_1 - g_\infty)]
\tag{2}
$$
where $H$ is half of the transition period—i.e., $H=A/2$.</p>
Equation (2)</a> appears quite nice, almost too nice.
Indeed, the H-model is only an approximation</em> of the present value of the assumed cash flows.1</a></sup>
The assumption of linearly declining growth rate does not admit much simplification of Equation (1)</a>.</p>
However, one of the goals of Fuller and Hsia (1984) is to provide a simple analytic formula for the discount rate, given a present value. That is,</p>
</span>
$$
r = \dfrac{D_0}{P_0}[(1 + g_\infty) + H(g_1 - g_\infty)] + g_\infty
\tag{3}
$$
Although an analytic formula is not necessary with modern computing power, most financial analysts in 1984 would have found solving Equation (1)</a> rather time-consuming.
Therefore, approximation error may have been a reasonable trade-off.</p>
Derivation</a>
</h2>
Unfortunately, Fuller and Hsia (1984) do not provide a formal derivation of the H-model in the paper.
A footnote contains the dreaded words, "available from the authors."2</a></sup>
However, with a bit a guess work, it's not too difficult to figure out how they likely arrived at Equation (2)</a>.</p>
Two-step approximation</a>
</h3>
The authors begin with a two-step model with two growth rates.
Dividends grow at the initial rate ($g_a$) for $A$ periods then at the terminal rate ($g_\infty$) for all future periods.
In the two-step model, the present value is given by equation 5 in the paper,</p>
</span>
$$
P_0 = D_0 \sum_{t=1}^A \left(\dfrac{1+g_a}{1+r} \right)^t +
D_A \sum_{t=A+1}^\infty \left(\dfrac{(1+g_\infty)^{t-A}}{(1+r)^t} \right)
\tag{4}
$$
By applying the formula for a growing annuity to the first term and the formula for a growing perpetuity to the second term, the present value can be written as</p>
</span>
$$
P_0 = D_0 \dfrac{1+g_a}{r-g_a}\left(1 - \left(\dfrac{1+g_a}{1+r} \right)^A \right) +
D_0 \left(\dfrac{1+g_a}{1+r} \right)^A \dfrac{1+g_\infty}{r-g_\infty}
\tag{5}
$$
With some algebra, Equation (5)</a> can be rearranged as</p>
</span>
$$
P_0 = D_0 \dfrac{1+g_a}{r-g_a}\left(1 - \left(\dfrac{1+g_a}{1+r} \right)^{A-1} \left(\dfrac{g_a-g_\infty}{r-g_\infty} \right) \right)
\tag{6}
$$
which is equation 6 in the paper.</p>
The authors claim that (6)</a> can be approximated by</p>
</span>
$$
P_0 \approx  \dfrac{D_0}{r-g_\infty}\left(1 + g_\infty + A(g_a - g_\infty)\right)
\tag{7}
$$</p>
The authors do not provide any hints for the approximation, and manipulating Equation (6)</a> is quite messy.
Instead, consider how extreme values of $g_a$ affect the "exact" present value in Equation (4)</a>.
If $g_a=g_\infty$, then the whole cash flow series is a perpetuity with constant growth, so $P_0=\frac{D_0(1+g_\infty)}{r-g_\infty}$.
If $g_a=r$, then both terms can be considerably simplified so $P_0=D_0A +\frac{D_0(1+g_\infty)}{r-g_\infty}$.</p>
Now, consider linearly interpolating between these two points for ($g_a,P_0$): from $\left(g_\infty,\frac{D_0(1+g_\infty)}{r-g_\infty}\right)$ to $\left(r,D_0A+\frac{D_0(1+g_\infty)}{r-g_\infty}\right)$.
The formula for a line connecting two points, $(x, f(x))$ and $(x_1, f(x_1))$, is
$$
f(x)-f(x_1)=m(x-x_1)
$$
where $m$ is the slope between the two points.
The slope between our two points for the present value is $\frac{D_0A}{r-g_\infty}$, so we can write the formula as
$$
f(x) - \frac{D_0(1+g_\infty)}{r-g_\infty} = \frac{D_0A}{r-g_\infty}(x-g_\infty)
$$
$$
f(x) = \frac{D_0}{r-g_\infty}(1+g_\infty+A(x-g_\infty))
$$
So, for a particular value of $g_a$, we have the approximation
$$
P_0 \approx f(g_a) = \frac{D_0}{r-g_\infty}(1+g_\infty+A(g_a-g_\infty))
$$</p>
The authors also write the approximation as the sum of two terms,
$$
P_0 \approx \frac{D_0}{r-g_\infty}(1+g_\infty)+ \frac{D_0}{r-g_\infty}A(g_a-g_\infty)
$$
and add the interpretation that the first term is the underlying perpetuity, while the second term is the "excess" dividend growth during the period from 0 to $A$ which itself grows at $g_\infty$ starting from period $A+1$.
The approximation ignores the discounting of the excess perpetuity back to time 0.</p>
The H-model formula</a>
</h3>
From this interpretation, one may consider a linearly declining growth rate over the $A$ period, instead of a constant $g_a$.
In that case, the "excess growth" would be halved, to $\frac{A}{2}(g_a - g_\infty)$ (again, ignore compounding).
Finally, with $H=A/2$, we have the H-model formula, Equation (2)</a>.</p>
Three-step approximation</a>
</h3>
The authors do not jump directly from the two-step approximation to the H-model formula in the paper.
Instead, they consider a three-step model, with growth rates of $g_a$ for $A$ periods, then $g_b$ for $B-A$ periods, then $g_\infty$ for all future periods.
In this model, they claim the present value can be approximated by
$$
P_0 \approx \frac{D_0}{r-g_\infty}(1+g_\infty+A(g_a-g_\infty)+(B-A)(g_b-g_\infty))
$$
again without explanation.3</a></sup>
Unfortunately, I was not able to find a clean mathematical way to arrive at this approximation.
Of course if $g_a=g_b=r$ or $g_a=g_b=g_\infty$, then the approximate formula matches the exact formula.
However, three points are required to form the plane representing the approximation.
I suspect the authors continue the geometric interpretation above that $A(g_a-g_\infty)$ is the excess growth in the first intermediate period, so $(B-A)(g_b-g_\infty)$ is the excess growth in the second intermediate period.</p>
Given the three-step approximation, the authors assume $g_b = \frac{g_a+g_n}{2}$, so the approximation reduces to
$$
P_0 \approx \frac{D_0}{r-g_\infty}\left(1+g_\infty+\frac{A+B}{2}(g_a-g_\infty)\right)
$$
Now the authors write $H=\frac{A+B}{2}$ to arrive at what they call the H-model.
The three-step approximation does not improve the accuracy of the H-model, although it shows another way to think about the underlying assumptions as being about the growth rate rather than the length of time over which the growth rate is declining.</p>
Approximation error</a>
</h2>
Let us now examine the accuracy of the approximations, including the H-model itself as compared to the "exact" model with linearly declining growth rate over $2H$ periods.</p>
Two-step</a>
</h3>
The following plot shows the present value as a function of $g_a$ with representative values of the remaining parameters: $D_0=1,A=5,g_\infty=0.01,r=0.15$.
The lines intersect at the points used to create the approximation—that is, where $g_a=g_\infty$ and $g_a=r$.
Interestingly, for reasonable parameter values ($g_\infty<g_a<r$), the approximation always overstates the present value.</p>

    
    
    H-model two step approximation</p>
</figcaption>
    
</figure>

Three-step</a>
</h3>
The three-step model is a bit harder to visualize, with present value now a function of the growth rates of the first two periods, $g_a$ and $g_b$.
The approximate present value is a plane in 3D space, and the exact present value is a curved surface.
The figure below shows the difference between the two (error) with the following parameters: $A=3,B=6,g_\infty=0.01,r=0.15$.
Again, for typical parameters ($g_\infty < g_a, g_b < r$), the approximation overstates the present value.</p>

    
    
    H-model three step approximation</p>
</figcaption>
    
</figure>

H-model versus "exact" linear decline in growth rate</a>
</h3>
Lastly, let us compare the H-model itself to the intended underlying assumption of linearly declining growth rate over $2H$ periods.
Similar to the two-step model, the approximation is exact when $g_a=g_\infty$, but the upper intersection point now depends on $H$.
Larger values of $H$ tend to result in overstating the present value, while lower values the opposite.</p>

    
    
    H-model exact linear decline approximation, H=2</p>
</figcaption>
    
</figure>


    
    
    H-model exact linear decline approximation, H=5</p>
</figcaption>
    
</figure>


    
    
    H-model exact linear decline approximation, H=10</p>
</figcaption>
    
</figure>
</p>

Why did I write this post?</a>
</h1>
That's a great question. I came across the H-model in my normal course of work.
It seemed like an elegant simplification of a more general model (linearly declining growth), so I tried to work out the derivation.
After unsuccessful attempts, I decided to dig up the original paper.
Finding out it was only an approximation was quite annoying, doubly so when I saw the authors didn't bother to include explanation—let alone derivation—of the approximate form.</p>
I guess I felt committed at this point, so I spent some time trying to deduce the path from the "exact" formula to the approximate one.
The two-step model eventually worked out nicely, but the three-step (or n-step) approximation appears to be heuristic.</p>
References</a>
</h1>
Fuller, Russell J. and Chi-Cheng Hsia (1984): "A Simplified Common Stock Valuation Model," Financial Analysts Journal</em>, Sep.--Oct., 1984, Vol. 40, No. 5, pp. 49--56. https://www.jstor.org/stable/4478774</p>
^{1</sup>
Many online resources explaining the H-model omit this important detail. The subjective choices of the growth and discount rates dwarf the materiality of approximation error, but it's nonetheless important for practitioners to understand how the model works.</p>
</div>
^{2</sup>
If you are unfamiliar with academic writing, this means it's not available. If it were available, it would be in the paper.</p>
</div>
^{3</sup>
In a footnote, they claim a general approximation for $N$ intermediate growth periods of the form $\frac{D_0}{r-g_\infty}(1+g_\infty+X_1(g_1-g_\infty)+X_2(g_2-g_\infty)+\ldots. +X_N(g_N-g_\infty))$, where $X_i$ is the length of each period.</p>
</div>}}}


Julia PoissonFE.jl Package
2020-06-02T00:00:00+00:00
I've released a Julia package, PoissonFE.jl</a>, to estimate Poisson regression with fixed effects.
Similar to my R package, poisFErobust</a>, it computes robust standard errors from Wooldridge (1999)</a>.</p>
The Julia package is quite a bit faster than R.
Aside from the raw speed improvement of Julia, it implements the likelihood function without the fixed effects parameters, so it should be faster than GLM.jl</a> for models with many fixed effect levels (thousands).
The estimates should match Stata's command, xtpoisson y x, fe vce(r)</code>.</p>
I continue to be impressed by Julia's ecosystem.
Estimating the coefficients only required implementing the likelihood function.
Automatic differentiation via ForwardDiff.jl</a> enables a simple function call to Optim.jl</a> which implements efficient line search algorithms.
There was no need to implement the gradient or Hessian by hand.</p>
References</a>
</h3>
Wooldridge, Jeffrey M. (1999): "Distribution-free estimation of some nonlinear panel data models," Journal of Econometrics</em>, 90, 77-97. https://doi.org/10.1016/S0304-4076%2898%2900033-5</a></p>


Some R-related gists
2019-03-05T00:00:00+00:00
pdf-merge-text.r</a></a>
</h2>
A common problem when scanning using a one-sided scanner with a document feeder is ending up with two PDFs: one for the front of each page and one for the back. This script writes LaTeX which combines those two PDFs, so that you end up with front1, back1, front2, back2, etc.</p>
The resulting pdfmergetex.tex file looks like the following:</p>
\</span>includepdf</span>[</span>pages=1</span>]</span>{file1.pdf}</span></span>
\</span>includepdf</span>[</span>pages=12</span>]</span>{file2.pdf}</span></span>
\</span>includepdf</span>[</span>pages=2</span>]</span>{file1.pdf}</span></span>
\</span>includepdf</span>[</span>pages=11</span>]</span>{file2.pdf}</span></span></code></pre>
which may be input to a LaTeX document like so:</p>
\</span>documentclass</span>[</span>12pt,letterpaper</span>]</span>{</span>article</span>}</span></span>
\</span>usepackage</span>[</span>utf8</span>]</span>{</span>inputenc</span>}</span></span>
\</span>usepackage</span>{</span>graphicx</span>}</span></span>
\</span>usepackage</span>[</span>left=1in,right=1in,top=1in,bottom=1in</span>]</span>{</span>geometry</span>}</span></span>
\</span>usepackage</span>{</span>pdfpages</span>}</span></span>
\</span>begin</span>{</span>document</span>}</span></span>
</span>
\</span>input</span>{</span>pdfmergetex.tex</span>}</span> %</span> Input the .tex generated by R here</span></span>
</span>
\</span>end</span>{</span>document</span>}</span></span></code></pre>
When compiled, the latter should produce a single, properly collated PDF. If the "backs" are in reverse order from what you expected, then try reverse = FALSE</code> in the inclpdf</code> call.</p>
inclpdf</span> <-</span> function</span>(</span>fronts</span>,</span> backs</span>,</span> npages</span>,</span> reverse</span> =</span> FALSE</span>)</span>{</span></span>
  if</span> (</span>reverse</span>)</span>{</span></span>
    for</span> (</span>i</span> in</span> 1</span>:</span>npages</span>)</span>{</span></span>
      cat</span>(</span>"</span>\\</span>includepdf[pages=</span>"</span>,</span> i</span>,</span> "</span>]{</span>"</span>,</span> fronts</span>,</span> "</span>}</span>\n</span>"</span>,</span></span>
          "</span>\\</span>includepdf[pages=</span>"</span>,</span> npages</span> -</span> i</span> +</span> 1</span>,</span> "</span>]{</span>"</span>,</span> backs</span>,</span> "</span>}</span>\n</span>"</span>,</span></span>
          sep</span> =</span> "</span>"</span>)</span></span>
    }</span></span>
  }</span> else</span> {</span></span>
    for</span> (</span>i</span> in</span> 1</span>:</span>npages</span>)</span>{</span></span>
    cat</span>(</span>"</span>\\</span>includepdf[pages=</span>"</span>,</span> i</span>,</span> "</span>]{</span>"</span>,</span> fronts</span>,</span> "</span>}</span>\n</span>"</span>,</span></span>
           "</span>\\</span>includepdf[pages=</span>"</span>,</span> i</span>,</span> "</span>]{</span>"</span>,</span> backs</span>,</span> "</span>}</span>\n</span>"</span>,</span></span>
        sep</span> =</span> "</span>"</span>)</span></span>
    }</span></span>
  }</span></span>
  </span></span>
}</span></span>
</span>
sink</span>(</span>"</span>C:/OCR/pdfmergetex.tex</span>"</span>)</span></span>
inclpdf</span>(</span>"</span>file1.pdf</span>"</span>,</span> "</span>file2.pdf</span>"</span>,</span> npages</span> =</span> 12</span>,</span> reverse</span> =</span> TRUE</span>)</span></span>
sink</span>(</span>)</span></span></code></pre>image-greyer.r</a></a>
</h2>
For every file ending in .png</code>, .jpg</code>, or .jpeg</code> (and not</em> beginning with blank-</code>) in the current working directory, the script will generate a grey image of the same dimensions with text of the image name.
The resulting images are named blank-<file name></code>. For example, if the original image name is e.png</code>, then the script will create the above image named blank-e.png</code>.
The script may be useful for generating lightweight placeholders for images in other documents, such as a LaTeX article.</p>
In the preamble of a LaTeX document, one could put</p>
\</span>usepackage</span>{</span>etoolbox</span>}</span></span>
\</span>newtoggle</span>{blank_images}</span></span>
\</span>toggletrue</span>{blank_images}</span></span>
%</span>\togglefalse{blank_images}</span></span></code></pre>
and in the body where one wishes to include the image</p>
\</span>iftoggle</span>{blank_images}{</span>%</span></span>
	\</span>includegraphics</span>[</span>scale=1</span>]</span>{blank-e.png}</span></span>
}{</span>%</span></span>
	\</span>includegraphics</span>[</span>scale=1</span>]</span>{e.png}</span></span>
}</span></span></code></pre>
As written, the above document will use the blank version blank-e.png</code>. If \togglefalse{blank_images}</code> is uncommented, then it will use the original e.png</code>.</p>
library</span>(</span>magick</span>)</span></span>
</span>
imagefiles</span> <-</span> list.files</span>(</span>pattern</span> =</span> "</span>(</span>\\</span>.png|</span>\\</span>.jpg|</span>\\</span>.jpeg)</span>"</span>)</span></span>
imagefiles</span> <-</span> imagefiles</span>[</span>!</span>grepl</span>(</span>"</span>blank-</span>"</span>,</span> imagefiles</span>)</span>]</span></span>
</span>
for</span> (</span>imgname</span> in</span> imagefiles</span>)</span>{</span></span>
  img</span> <-</span> image_read</span>(</span>imgname</span>)</span></span>
  img</span> <-</span> image_fill</span>(</span>img</span>,</span> "</span>lightgray</span>"</span>,</span> fuzz</span> =</span> 100</span>)</span></span>
  img</span> <-</span> image_annotate</span>(</span>img</span>,</span> imgname</span>)</span></span>
  </span></span>
  image_write</span>(</span>img</span>,</span> path</span> =</span> paste0</span>(</span>"</span>blank-</span>"</span>,</span> imgname</span>)</span>)</span></span>
}</span></span></code></pre>npp-r-function-parser.xml</a></a>
</h2>
Notepad++ has a Function List Panel</a> which, by default, lists the functions defined in the current document for common languages, including C and Python. This Gist provides a simple parser to provide the same functionality for R.</p>
functionList.xml</code>, found in %APPDATA%\Notepad++\ or in the Notepad++ install directory must be edited to include the Gist code.</p>
First, add the association in <associationMap>...</associationMap></code> next to the similar looking lines:</p>
<</span>association</span> langID</span>=</span>"</span>54</span>"</span> id</span>=</span>"</span>r_function</span>"</span>/></span></span></code></pre>
Second, add the parser code from the Gist (lines 5 through 13) in <parsers>...</parsers></code>. Again, one should be able to infer the location based on the other parsers defined in the file.</p>
The parser is quite simplistic and will not capture every possible way of defining an R function. However, it should capture the most common syntax. It may also include unwanted leading whitespace in the function's name in the Function List.</p>
<!--</span>Put this in associationMap</span>--></span></span>
<</span>association</span> langID</span>=</span>"</span>54</span>"</span> id</span>=</span>"</span>r_function</span>"</span>/></span></span>
</span>
<!--</span>Put this in parsers</span>--></span></span>
<</span>parser</span> id</span>=</span>"</span>r_function</span>"</span> displayName</span>=</span>"</span>R function</span>"</span> commentExpr</span>=</span>"</span>(#.*?$)</span>"</span>></span></span>
  <</span>function</span></span>
    mainExpr</span>=</span>"</span>(^\s*[a-zA-Z0-9_\.\$]+[\s<\-=]+function\s*\()</span>"</span></span>
    displayMode</span>=</span>"</span>$functionName</span>"</span>></span></span>
    <</span>functionName</span>></span></span>
      <</span>nameExpr</span> expr</span>=</span>"</span>^\s*([a-zA-Z0-9_\.\$]+)</span>"</span>/></span></span>
    </</span>functionName</span>></span></span>
  </</span>function</span>></span></span>
</</span>parser</span>></span></span></code></pre>


A key benefit of Julia
2019-01-15T00:00:00+00:00
Notice: this post relies on Unicode characters which may not be included in the font(s) on your current platform, especially mobile. Therefore, you may see empty boxes or question marks for certain characters.</em></p>
A key benefit of Julia</a> is the ability to write code which looks similar to the mathematical representation of an algorithm, with simple and expressive syntax, while still being performant.
Additionally, Julia accepts UTF-8 encoded Unicode characters as variable names, so if a paper defines some quantity,
$\mu_i$, then essentially the same Unicode representation, 𝛍ᵢ, could be used in the code.1</a></sup>
Although it may seem minor, there's a certain pedagogical advantage to having such a close match.
When reading through the code of a package for computing a statistical estimator, it's easier to parse

𝚲ᵢ' * 𝐖ᵢ * 𝐮ᵢ * 𝐮ᵢ' * 𝐖ᵢ * 𝚲ᵢ</code> than

t(Lambda_i) %*% W_i %*% u_i %*% t(u_i) %*% W_i %*% Lambda_i</code>

when the equivalent equation in the accompanying paper is

$$ \Lambda_i' W_i u_i u_i' W_i \Lambda_i $$</p>
Now, I don't recommend replacing all your descriptive variable and function names with fancy Unicode characters, but sometimes it does make sense to use the same mathematical symbols.
The Julia code below demonstrates this point by defining a function to compute standard errors and a hypothesis test from Wooldridge (1999)</a>.
The comments provide a mapping to definitions in the paper which share notation with the code.</p>
import</span> StatsFuns</span>.</span>chisqcdf</span>;</span> #</span> Run the following to install: import Pkg; Pkg.add("StatsFuns")</span></span>
using</span> LinearAlgebra</span>;</span></span>
poisfe_test </span>=</span> function</span>(</span>y</span>,</span> id</span>,</span> X</span>,</span> qcmle_coefs</span>)</span></span>
    #</span> No "hats" on variables are used for brevity.</span></span>
    #</span> We are already dealing with estimated quantities.</span></span>
    β </span>=</span> qcmle_coefs </span>#</span> Estimated parameters, p.79</span></span>
    K </span>=</span> length</span>(</span>β</span>)</span> #</span> Number of parameters, p.79</span></span>
    all_ids </span>=</span> unique</span>(</span>id</span>)</span></span>
    N </span>=</span> length</span>(</span>all_ids</span>)</span> #</span> Number of groups, p.83</span></span>
</span>
    #</span> Initialize arrays described later</span></span>
    𝐊 </span>=</span> zeros</span>(</span>Float64</span>,</span> K</span>,</span> K</span>)</span></span>
    𝐀 </span>=</span> zeros</span>(</span>Float64</span>,</span> K</span>,</span> K</span>)</span></span>
    𝐁 </span>=</span> zeros</span>(</span>Float64</span>,</span> K</span>,</span> K</span>)</span></span>
    𝚲 </span>=</span> Array</span>{Array{Float64}}</span>(</span>undef</span>,</span> N</span>)</span></span>
    𝐖 </span>=</span> Array</span>{Array{Float64}}</span>(</span>undef</span>,</span> N</span>)</span></span>
    𝐮 </span>=</span> Array</span>{Array{Float64}}</span>(</span>undef</span>,</span> N</span>)</span></span>
</span>
    #</span> Particular functional form (Poisson), p.79</span></span>
    μ </span>=</span> function</span>(</span>𝐱ᵢₜ</span>,</span> β</span>)</span></span>
        return</span> exp</span>(</span>𝐱ᵢₜ </span>⋅</span> β</span>)</span></span>
    end</span></span>
</span>
    for</span> i</span>=</span>1</span>:</span>N</span></span>
        this_index </span>=</span> searchsorted</span>(</span>id</span>,</span> all_ids</span>[</span>i</span>]</span>)</span></span>
        T </span>=</span> length</span>(</span>this_index</span>)</span></span>
        yᵢ </span>=</span> y</span>[</span>this_index</span>]</span> #</span> Tx1 vector of outcomes, p.79</span></span>
        𝐱ᵢ </span>=</span> X</span>[</span>this_index</span>,</span> :</span>]</span> #</span> TxK matrix of indep vars, p.79</span></span>
        nᵢ </span>=</span> sum</span>(</span>yᵢ</span>)</span> #</span> Scalar sum of outcomes, p.79</span></span>
</span>
        𝛍ᵢ </span>=</span> similar</span>(</span>yᵢ</span>)</span> #</span> Tx1 vector of conditional means, p.82</span></span>
        for</span> t </span>in</span> 1</span>:</span>T</span></span>
            𝛍ᵢ</span>[</span>t</span>]</span> =</span> μ</span>(</span>𝐱ᵢ</span>[</span>t</span>,</span> :</span>]</span>,</span> β</span>)</span></span>
        end</span></span>
        Σ𝛍ᵢ </span>=</span> sum</span>(</span>𝛍ᵢ</span>)</span> #</span> Scalar sum of conditional means, p.82</span></span>
</span>
        𝐩 </span>=</span> 𝛍ᵢ </span>/</span> Σ𝛍ᵢ </span>#</span> Tx1 vector of "choice probabilities," p.79 (2.5)</span></span>
        #</span> 𝚲ᵢ (TxK) is the derivative of 𝐩 wrt β; p.86</span></span>
        𝚲ᵢ </span>=</span> similar</span>(</span>𝐱ᵢ</span>)</span></span>
        for</span> k </span>in</span> 1</span>:</span>K</span></span>
            last_term </span>=</span> 0</span></span>
            for</span> s </span>in</span> 1</span>:</span>T</span></span>
                last_term </span>+=</span> 𝐱ᵢ</span>[</span>s</span>,</span> k</span>]</span> *</span> 𝐩</span>[</span>s</span>]</span></span>
            end</span></span>
            for</span> t </span>in</span> 1</span>:</span>T</span></span>
                𝚲ᵢ</span>[</span>t</span>,</span> k</span>]</span> =</span> 𝐩</span>[</span>t</span>]</span> *</span> (</span>𝐱ᵢ</span>[</span>t</span>,</span> k</span>]</span> -</span> last_term</span>)</span></span>
            end</span></span>
        end</span></span>
</span>
        𝐖ᵢ </span>=</span> Diagonal</span>(</span>(</span>1</span> ./</span> 𝐩</span>)</span>)</span>  #</span> TxT, p.82</span></span>
        𝐮ᵢ </span>=</span> yᵢ </span>-</span> nᵢ </span>*</span> 𝐩 </span>#</span> Tx1, p.82</span></span>
</span>
        #</span> Each group's 𝚲ᵢ, 𝐖ᵢ, and 𝐮ᵢ are stored in an array of arrays for</span></span>
        #</span> later computation of a hypothesis test, p.86</span></span>
        𝚲</span>[</span>i</span>]</span> =</span> 𝚲ᵢ</span></span>
        𝐖</span>[</span>i</span>]</span> =</span> 𝐖ᵢ</span></span>
        𝐮</span>[</span>i</span>]</span> =</span> 𝐮ᵢ</span></span>
</span>
        𝐊 </span>+=</span> 1</span>/</span>N </span>*</span> 𝚲ᵢ</span>'</span> *</span> (</span>nᵢ </span>*</span> 𝚲ᵢ</span>)</span> #</span> p.86 (3.16)</span></span>
        𝐀 </span>+=</span> 1</span>/</span>N </span>*</span> nᵢ </span>*</span> (</span>𝚲ᵢ</span>'</span> *</span> 𝐖ᵢ </span>*</span> 𝚲ᵢ</span>)</span> #</span> p.83 (3.9)</span></span>
        𝐁 </span>+=</span> 1</span>/</span>N </span>*</span> 𝚲ᵢ</span>'</span> *</span> 𝐖ᵢ </span>*</span> 𝐮ᵢ </span>*</span> 𝐮ᵢ</span>'</span> *</span> 𝐖ᵢ </span>*</span> 𝚲ᵢ </span>#</span> p.83 (3.10)</span></span>
    end</span></span>
</span>
    𝐀⁻¹ </span>=</span> inv</span>(</span>𝐀</span>)</span></span>
    se_rob </span>=</span> broadcast</span>(</span>sqrt</span>,</span> diag</span>(</span>(</span>𝐀⁻¹ </span>*</span> 𝐁 </span>*</span> 𝐀⁻¹</span>)</span> /</span> N </span>)</span>)</span> #</span> p.83, (3.11)</span></span>
</span>
    #</span> Test of the conditional mean specification (3.1) with the null</span></span>
    #</span> hypothesis (3.14), p.85</span></span>
    𝐫 </span>=</span> Array</span>{Float64, 2}</span>(</span>undef</span>,</span> N</span>,</span> K</span>)</span> #</span> p.86, (3.17)</span></span>
    for</span> i</span>=</span>1</span>:</span>N</span></span>
        𝐫</span>[</span>i</span>,</span>:</span>]</span> =</span> 𝐮</span>[</span>i</span>]</span>'</span> *</span> (</span>𝚲</span>[</span>i</span>]</span> -</span> 𝐖</span>[</span>i</span>]</span> *</span> 𝚲</span>[</span>i</span>]</span> *</span> 𝐀⁻¹ </span>*</span> 𝐊</span>'</span>)</span></span>
    end</span></span>
    ssr </span>=</span> sum</span>(</span>(</span>ones</span>(</span>N</span>)</span> -</span> 𝐫 </span>*</span> (</span>𝐫 </span>\</span> ones</span>(</span>N</span>)</span>)</span>)</span> .^</span> 2</span>)</span> #</span> p.86 (3.18)</span></span>
    p_value </span>=</span> 1</span> -</span> chisqcdf</span>(</span>K</span>,</span> N </span>-</span> ssr</span>)</span> #</span> p.86</span></span>
    return</span> se_rob</span>,</span> p_value</span></span>
end</span></span></code></pre>
The function may be tested with some synthetic data like the example below.</p>
y </span>=</span> [</span>1.0</span>,</span> 0.0</span>,</span> 0.0</span>,</span> 0.0</span>,</span> 7.0</span>,</span> 1.0</span>,</span> 0.0</span>,</span> 1.0</span>,</span> 0.0</span>,</span> 0.0</span>,</span> 6.0</span>,</span></span>
     2.0</span>,</span> 3.0</span>,</span> 0.0</span>,</span> 1.0</span>,</span> 6.0</span>,</span> 0.0</span>,</span> 7.0</span>,</span> 0.0</span>,</span> 21.0</span>]</span></span>
id </span>=</span> [</span>1</span>,</span> 1</span>,</span> 1</span>,</span> 1</span>,</span> 1</span>,</span> 1</span>,</span> 1</span>,</span> 1</span>,</span> 1</span>,</span> 1</span>,</span> 2</span>,</span> 2</span>,</span> 2</span>,</span> 2</span>,</span> 2</span>,</span> 2</span>,</span> 2</span>,</span> 2</span>,</span> 2</span>,</span> 2</span>]</span></span>
X </span>=</span> [</span>0.398106</span> 0.496961</span>;</span> -</span>0.612026</span> -</span>0.224875</span>;</span> 0.34112</span> -</span>1.11714</span>;</span></span>
     -</span>1.12936</span> -</span>0.394995</span>;</span> 1.43302</span> 1.54983</span>;</span> 1.9804</span> -</span>0.743514</span>;</span> -</span>0.367221</span> -</span>2.33171</span>;</span></span>
     -</span>1.04413</span> 0.812245</span>;</span> 0.56972</span> -</span>0.501311</span>;</span> -</span>0.135055</span> -</span>0.510887</span>;</span></span>
     2.40162</span> -</span>1.21536</span>;</span> -</span>0.03924</span> -</span>0.0225586</span>;</span> 0.689739</span> 0.701239</span>;</span></span>
     0.0280022</span> -</span>0.587482</span>;</span> -</span>0.743273</span> -</span>0.606728</span>;</span> 0.188792</span> 1.09664</span>;</span></span>
     -</span>1.80496</span> -</span>0.24751</span>;</span> 1.46555</span> -</span>0.159902</span>;</span> 0.153253</span> -</span>0.625778</span>;</span></span>
     2.17261</span> 0.900435</span>]</span></span>
qcmle_coefs </span>=</span> [</span>0.924499</span>;</span> 0.869371</span>]</span></span>
show</span>(</span>poisfe_test</span>(</span>y</span>,</span> id</span>,</span> X</span>,</span> qcmle_coefs</span>)</span>)</span></span></code></pre>([0.013508, 0.12758], 0.36787944117144233)</span></span></code></pre>
Despite many optimizations left on the table, the code above is still going to be blazing fast.
An equivalent in a low-level language is likely to be more verbose and difficult for non-experts to parse---especially those from the social sciences, the audience of the paper.
A high level language like R can be written in a more understandable way (see here</a>), but achieving high performance requires either vectorization (sometimes difficult, sometimes impossible) or dropping back down to a low-level language.
Julia seems to hit the sweet spot.</p>
References</a>
</h3>
Wooldridge, Jeffrey M. (1999): "Distribution-free estimation of some nonlinear panel data models," Journal of Econometrics</em>, 90, 77-97. https://doi.org/10.1016/S0304-4076%2898%2900033-5</a></p>
^{1</sup>
Many other languages will technically allow Unicode characters in variable names, but based on personal experience there's typically some type of extra configuration or risk of errors across platforms. Julia is more accommodating, and most Julia development environments---including the REPL---support tab completion of common Unicode characters, e.g. \alpha<tab></code> becomes α.</p>
</div>}


Robust standard errors for Poisson regression with fixed effects
2019-01-10T00:00:00+00:00
Last year I released an R package, poisFErobust</a>, which provides a function to compute standard errors for Poisson regression with fixed effects.
The standard errors are derived in Wooldridge (1999)</a> and are robust to conditional serial correlation of errors within groups.
The function also returns the p-value of the hypothesis test of the conditional mean assumption (3.1) as described in the paper, Section 3.3.</p>
The package is on CRAN, so it may be installed with</p>
install.packages</span>(</span>"</span>poisFErobust</span>"</span>)</span></span></code></pre>
The examples below show output when the assumption (3.1) is satisfied and when it is not satisfied.</p>
require</span>(</span>poisFErobust</span>)</span></span>
# ex.dt.good satisfies the conditional mean assumption</span></span>
data</span>(</span>"</span>ex.dt.good</span>"</span>)</span></span>
pois.fe.robust</span>(</span>outcome</span> =</span> "</span>y</span>"</span>,</span> xvars</span> =</span> c</span>(</span>"</span>x1</span>"</span>,</span> "</span>x2</span>"</span>)</span>,</span> group.name</span> =</span> "</span>id</span>"</span>,</span></span>
index.name</span> =</span> "</span>day</span>"</span>,</span> data</span> =</span> ex.dt.good</span>)</span></span></code></pre>$coefficients</span></span>
       x1        x2</span></span>
0.9899730 0.9917526</span></span>
</span>
$se.robust</span></span>
        x1         x2</span></span>
0.03112512 0.02481941</span></span>
</span>
$p.value</span></span>
[1] 0.6996001</span></span></code></pre># ex.dt.bad violates the conditional mean assumption</span></span>
data</span>(</span>"</span>ex.dt.bad</span>"</span>)</span></span>
pois.fe.robust</span>(</span>outcome</span> =</span> "</span>y</span>"</span>,</span> xvars</span> =</span> c</span>(</span>"</span>x1</span>"</span>,</span> "</span>x2</span>"</span>)</span>,</span> group.name</span> =</span> "</span>id</span>"</span>,</span></span>
index.name</span> =</span> "</span>day</span>"</span>,</span> data</span> =</span> ex.dt.bad</span>)</span></span></code></pre>$coefficients</span></span>
       x1        x2</span></span>
0.4800735 2.9866911</span></span>
</span>
$se.robust</span></span>
       x1        x2</span></span>
0.2864666 1.2743953</span></span>
</span>
$p.value</span></span>
[1] 0.02213269</span></span></code></pre>
The results should match those of Stata's xtpoisson y x, fe vce(r)</code>.</p>
The source code is available at https://bitbucket.org/ew-btb/poisson-fe-robust</a>.
Pull requests are welcome.</p>
References</a>
</h3>
Wooldridge, Jeffrey M. (1999): "Distribution-free estimation of some nonlinear panel data models," Journal of Econometrics</em>, 90, 77-97.</p>

The H-model formula </a> </h3>
(2)</a>.</p>

Approximation error
Two-step

Three-step

Why did I write this post?
References

Two-step

Three-step

Why did I write this post?
References

Three-step

Why did I write this post?
References

Why did I write this post?
References

Why did I write this post?
References

References

`References</a> </h3> Wooldridge, Jeffrey M. (1999): "Distribution-free estimation of some nonlinear panel data models," Journal of Econometrics</em>, 90, 77-97. https://doi.org/10.1016/S0304-4076%2898%2900033-5</a></p>`

`References</a> </h3> Wooldridge, Jeffrey M. (1999): "Distribution-free estimation of some nonlinear panel data models," Journal of Econometrics</em>, 90, 77-97.</p>`

My Blog

MILP Example with JuMP and HiGHS

Full Julia script

Z3.jl Optimization Example

Scraping Discord pins

Solving Peg Solitaire with Julia

Derivation of the H-model dividend discount formula

Julia PoissonFE.jl Package

Some R-related gists

A key benefit of Julia

Robust standard errors for Poisson regression with fixed effects