+
+The longest path problem asks: *what's a simple path of maximum edge weight between vertices in a graph?*
+What's special about it is the lack of a known "quick" algorithm for computing the path. Here, the word "quick" means
+that the execution time can be expressed as a polynomial function of the input size.
+Also, what's a *graph* in this case? To be more accurate, it is:
+- Undirected
+- Simple
+- *Cyclic*[^1]
+- Unweighted
+
+There are of course methods faster than brute-force (which is
+$$ \mathcal{\text{O}}(\text{n}! \cdot \text{n}) $$). One example is the Held-Karp algorithm,
+where the time complexity is $$ \mathcal{\text{O}}(2^\text{n} \cdot \text{n}^2) $$ (where $$ \text{n} $$ means
+the number of nodes, as in the entire article).
+> which still appears to be more desirable than brute force?
+
+That's correct, but a brute-force algorithm's space complexity depends linearly on the number of nodes, whereas
+Held-Karp's complexity grows as $$ \mathcal{\text{O}}(2^\text{n} \cdot \text{n}) $$. Which means that at 30
+nodes, the program would consume over 100 GB of RAM.
+Of course, there are certain
+low-level optimizations, such as capping the maximum number of nodes to 64
+and using bitmasks inside a `uint64_t` for edge storage.
+While these modifications do not affect the inherent time complexity,
+they could reduce the constant factors.
+
+But what about heuristics? They can be accurate or rough, fast or slow. Why not find a method to determine
+the near-exact longest path that finishes working within acceptable time limits?
+Now we bump into yet another obstacle: the longest path is notoriously hard to approximate. To put it informally, the
+main issue is that making optimal local choices doesn't necessarily lead us to the globally most extended path
+without the expensive exploration we're attempting to dodge.
+
+## The heuristic & general implementation
+
+All aspects of the problem point to neural networks. Let's try to train a model and see how well it performs.
+However, we're facing a core contradiction: how to get training data if there's no way to do it within
+a reasonable timespan? I won't be original here: the most realistic way is to leave it searching for the longest
+path in the background while you're using your computer. I've done exactly this and I'm pretty sure my dataset
+has some entries that have taken *tens* of minutes to process. These are of course the best examples to feed to
+the network. Here's how I save a given generation:
+```
+{
+ "num_nodes": 39,
+ "num_edges": 72,
+ "longest_path": 36,
+ "adjacency_list": [
+ [
+ 3,
+ 15,
+ 20,
+ 25
+ ],
+ [
+ 20,
+ 22,
+ ...
+}
+```
+
+This is not the only way to structure the dataset, but this one seems good enough.
+
+The *incredibly witty* algorithm that searches for the longest path is written in C. Here are the constraints that
+I use for the last chunk of the training data:
+```c
+#define MAX_NODES 64
+#define MIN_DEGREE 2.5
+#define MAX_DEGREE 4.1
+```
+
+On other runs, these constants are slightly different:
+we want to gather diverse entries.
+
+One might notice that `MIN_DEGREE` and `MAX_DEGREE` are counter-intuitively used as floats.
+To understand why that is the case, we should dive into what these really mean.
+First, it stochastically calculates a value that is proportional to total edge count:
+```c
+igraph_real_t target_degree = MIN_DEGREE + ((double)rand() / RAND_MAX) * (MAX_DEGREE - MIN_DEGREE)
+```
+
+Then, if $$ \text{p} $$ is the probability that two given nodes become linked and $$ \text{d} $$
+is `target_degree`,\
+$$ \text{p} = \begin{cases} 0, & \text{if } \text{n} \in \{0, 1\} \\ \text{min}\left(1.0, \frac{\text{d}}{\text{n} - 1}\right), & \text{if } \text{n} > 1 \end{cases} $$
+
+And it puts edges between every pair of nodes using $$ \text{p} $$.
+
+Like this, I've managed to accumulate over 800[^2] graphs and their corresponding longest paths.
+80% goes to training, 20% is reserved for validation.
+
+Now the important part, which is how we're going to implement the network. Fortunately, there's a very usable
+framework for working with graphs built into PyTorch.
+Before training on the collected data, I apply a pre-processor called `AddRandomWalkPE` to give the network a more
+global representation of the graph. All it does is it walks a given number of steps and returns the probability
+that a node is going to stroll into itself. Or, to be more exact:
+$$ (\text{P}^\text{k})_{\text{ii}} = \sum_{\text{i}_1} \sum_{\text{i}_2} \dots \sum_{\text{i}_{\text{k}-1}} \frac{\text{A}_{\text{i} \text{i}_1}}{\text{deg}(\text{i})} \cdot \frac{\text{A}_{\text{i}_1 \text{i}_2}}{\text{deg}(\text{i}_1)} \cdot \dots \cdot \frac{\text{A}_{\text{i}_{\text{k}-1} \text{i}}}{\text{deg}(\text{i}_{\text{k}-1})} $$
+
+$$ \text{i} $$ is the starting node,
+$$ \text{k} $$ is the walk length,
+$$ \text{A} $$ is a connection check (1 if connected, 0 otherwise),
+$$ \text{deg} $$ is a function that returns the degree of a given node (how many routes to choose from).
+
+Oh, and we're also going to throw in the degree of each node.
+It is of course not divided into inbound and unbound, because the graph is undirected.
+
+Here's the model that I've found to be effective enough:
+```python
+def __init__(self, hidden_channels):
+ super(Network, self).__init__()
+
+ self.input_proj = Linear(WALK_LENGTH + 1, hidden_channels)
+ self.proj_norm = GraphNorm(hidden_channels)
+
+ self.conv = GatedGraphConv(out_channels=hidden_channels, num_layers=10)
+ self.conv_norm = GraphNorm(hidden_channels)
+
+ self.mlp = Sequential(
+ Linear(hidden_channels * 2, hidden_channels),
+ LayerNorm(hidden_channels),
+ GELU(),
+ Dropout(p=0.3),
+ Linear(hidden_channels, hidden_channels // 2),
+ GELU(),
+ Linear(hidden_channels // 2, 1),
+ )
+```
+
+Keep in mind that $$ +1 $$ in the linear layer is that degree information I've just mentioned.
+
+And here are the actual calls:
+```python
+def forward(self, x, edge_index, batch):
+ h = GELU()(self.input_proj(x))
+ h = self.proj_norm(h, batch)
+
+ x_gated = self.conv(h, edge_index)
+ x_final = self.conv_norm(x_gated + h, batch)
+
+ pool_add = global_add_pool(x_final, batch)
+ pool_max = global_max_pool(x_final, batch)
+ x_pooled = torch.cat([pool_add, pool_max], dim=-1)
+
+ # Output the single continuous prediction float
+ return self.mlp(x_pooled).squeeze(-1)
+```
+
+Let's start with the easy parts. First, I feed the random walk into a linear layer and process it through
+an activation layer.
+After that, I normalize the node features separately for each individual graph using the batch vector.
+That's because graph 1 might have 5 nodes and graph 2 might have 100 nodes --
+we need to isolate their statistics so their different sizes don't distort each other's features.
+Now the "hard" parts.
+- `GatedGraphConv` applies a shared Gated Recurrent Unit (GRU) cell across all nodes to recursively
+update their structural representations during neighborhood message passing.
+Gated Recurrent Units (GRUs) use a reset gate and an update gate to regulate internal information flow.
+By selectively preserving or discarding historical state data at each step,
+GRUs capture long-term dependencies in
+sequential representations.
+
+Essentially, what this gives us is that our 128 features might implicitly represent something like, "Am I a leaf?",
+"How close am I to a densely connected hub?", or any other characteristic. Also, because it uses 10 layers,
+that means it learns stuff about friends, then friends of friends, and so on.
+- `conv_norm` accepts the sum of the result of `GatedGraphConv` and the very original blueprint.
+More precisely, it adds a tensor[^3] that is deterministically described by the original degree
+and stuff derived from the random walk. Then it normalizes
+the features.
+
+The rest of `forward` is in charge of boiling it all down into one value. `global_add_pool` walks down
+each graph's node rows and sums up their features, while `global_max_pool` grabs the peak values.
+Finally, `cat` glues the two outputs together. `mlp` is a sequence of layers
+that acts as the "funnel", squeezing everything down into a scalar, which is our predicted longest path.
+
+## Results
+I have specifically noted that these two have taken a pretty long time to finish:\
+`"num_nodes":61,"num_edges":89,"longest_path":47...`\
+`"num_nodes":45,"num_edges":81,"longest_path":39...`
+
+This is brand new data that the network has never been exposed to (beyond both training and validation),
+so let's test how well it approximates the longest path:
+
+
+> Graph #007 | True Path: 47.0 | Predicted: 45.7467 \
+> Graph #008 | True Path: 39.0 | Predicted: 41.6555
+
+
+As we can see, that's accurate enough. Keep in mind that the network finishes its calculations
+practically instantaneously, immeasurably faster than a deterministic method.
+
+## Closing notes
+Successful approximations from this network indicate that, in reality, when an NP-hard problem
+needs to be solved, we must resort to heuristics. Is this going to result in an exact solution?
+Definitely not. But is this approach going to suffice in practice? Well, that seems very likely.
+
+[^1]: Which is essential! There exists an algorithm for acyclic graphs with the time complexity of O(n).
+
+[^2]: Which is not a lot! But that's enough unless we are looking for perfect approximation, which we
+aren't in this instance.
+
+[^3]: Multi-dimensional array of numerical data.
+
+## Statistics, statistics, and statistics
+
+
+
+
+## Follow-up
+After some more experimentation, it turned out it was possible to add some data augmentation. The biggest
+advantage is that there's going to be less room for overfitting. In other words, the model should
+learn what a longest path is like in terms of existing patterns.
+I've tried embedding Gaussian noise, which is quite common for what I'm trying to achieve. However,
+a technique that makes more sense for graphs is not exactly that; it's dropping edges with a defined
+probability. In some sense, it is *noise* as well, because we confuse the model: for example, we might drop an edge that is necessary for the existence of the path we tell it serves as the longest.
+However, that's really the goal -- making it analyze the underlying structure of this path without
+focusing on unreliable details.
+
+Here's how I add it:
+```python
+class Network(torch.nn.Module):
+ def __init__(self, hidden_channels):
+ super(Network, self).__init__()
+
+ self.edge_dropout_p = 0.05
+
+ ...
+
+ def forward(self, x, edge_index, batch):
+ h = GELU()(self.input_proj(x))
+ h = self.proj_norm(h, batch)
+
+ edges = edge_index
+ edges, _ = dropout_edge(
+ edge_index,
+ p=self.edge_dropout_p,
+ force_undirected=True,
+ training=self.training
+ )
+
+ ...
+```
+
+Is it substantially better though?
+Here's what I witnessed with the previous setup:
+```
+Epoch 557 | Train Loss: 0.6867 | Val Loss: 0.8003 | Mean Abs Error: 0.8003
+Epoch 558 | Train Loss: 0.6996 | Val Loss: 0.8026 | Mean Abs Error: 0.8026
+Epoch 559 | Train Loss: 0.6926 | Val Loss: 0.7699 | Mean Abs Error: 0.7699
+Epoch 560 | Train Loss: 0.6276 | Val Loss: 0.7949 | Mean Abs Error: 0.7949
+Epoch 561 | Train Loss: 0.6610 | Val Loss: 0.7975 | Mean Abs Error: 0.7975
+Epoch 562 | Train Loss: 0.6952 | Val Loss: 0.8364 | Mean Abs Error: 0.8364
+Epoch 563 | Train Loss: 0.7464 | Val Loss: 0.8292 | Mean Abs Error: 0.8292
+Epoch 564 | Train Loss: 0.7076 | Val Loss: 0.7803 | Mean Abs Error: 0.7803
+Epoch 565 | Train Loss: 0.6764 | Val Loss: 0.7445 | Mean Abs Error: 0.7445
+Saved new best model with Val Loss: 0.7445
+Epoch 566 | Train Loss: 0.6699 | Val Loss: 0.8029 | Mean Abs Error: 0.8029
+Epoch 567 | Train Loss: 0.6857 | Val Loss: 0.7611 | Mean Abs Error: 0.7611
+Epoch 568 | Train Loss: 0.6820 | Val Loss: 0.7845 | Mean Abs Error: 0.7845
+Epoch 569 | Train Loss: 0.6510 | Val Loss: 0.8754 | Mean Abs Error: 0.8754
+...
+Epoch 788 | Train Loss: 0.6169 | Val Loss: 0.7690 | Mean Abs Error: 0.7690
+Epoch 789 | Train Loss: 0.6122 | Val Loss: 0.7681 | Mean Abs Error: 0.7681
+Epoch 790 | Train Loss: 0.6464 | Val Loss: 0.7660 | Mean Abs Error: 0.7660
+Epoch 791 | Train Loss: 0.6312 | Val Loss: 0.7657 | Mean Abs Error: 0.7657
+Epoch 792 | Train Loss: 0.6477 | Val Loss: 0.7664 | Mean Abs Error: 0.7664
+Epoch 793 | Train Loss: 0.5939 | Val Loss: 0.7661 | Mean Abs Error: 0.7661
+Epoch 794 | Train Loss: 0.6453 | Val Loss: 0.7662 | Mean Abs Error: 0.7662
+Epoch 795 | Train Loss: 0.6437 | Val Loss: 0.7643 | Mean Abs Error: 0.7643
+Epoch 796 | Train Loss: 0.6089 | Val Loss: 0.7630 | Mean Abs Error: 0.7630
+Epoch 797 | Train Loss: 0.6738 | Val Loss: 0.7629 | Mean Abs Error: 0.7629
+Epoch 798 | Train Loss: 0.5949 | Val Loss: 0.7632 | Mean Abs Error: 0.7632
+Epoch 799 | Train Loss: 0.6197 | Val Loss: 0.7632 | Mean Abs Error: 0.7632
+Epoch 800 | Train Loss: 0.6460 | Val Loss: 0.7635 | Mean Abs Error: 0.7635
+```
+
+And that was the last best model ever saved.
+
+Now the version with edge dropout. Running with the same seed!
+```
+Epoch 380 | Train Loss: 0.9379 | Val Loss: 0.7857 | Mean Abs Error: 0.7857
+Epoch 381 | Train Loss: 0.9091 | Val Loss: 0.8058 | Mean Abs Error: 0.8058
+Epoch 382 | Train Loss: 0.9151 | Val Loss: 0.8709 | Mean Abs Error: 0.8709
+Epoch 383 | Train Loss: 1.0247 | Val Loss: 0.8806 | Mean Abs Error: 0.8806
+Epoch 384 | Train Loss: 0.9175 | Val Loss: 0.8099 | Mean Abs Error: 0.8099
+Epoch 385 | Train Loss: 0.8757 | Val Loss: 0.7452 | Mean Abs Error: 0.7452
+Epoch 386 | Train Loss: 0.8859 | Val Loss: 1.0104 | Mean Abs Error: 1.0104
+Epoch 387 | Train Loss: 0.8694 | Val Loss: 0.8945 | Mean Abs Error: 0.8945
+Epoch 388 | Train Loss: 0.9448 | Val Loss: 0.7028 | Mean Abs Error: 0.7028
+Saved new best model with Val Loss: 0.7028
+Epoch 389 | Train Loss: 0.9180 | Val Loss: 0.7744 | Mean Abs Error: 0.7744
+Epoch 390 | Train Loss: 0.8837 | Val Loss: 0.7454 | Mean Abs Error: 0.7454
+Epoch 391 | Train Loss: 0.8545 | Val Loss: 0.7426 | Mean Abs Error: 0.7426
+Epoch 392 | Train Loss: 0.9028 | Val Loss: 0.7584 | Mean Abs Error: 0.7584
+Epoch 393 | Train Loss: 0.9434 | Val Loss: 0.8505 | Mean Abs Error: 0.8505
+...
+Epoch 790 | Train Loss: 0.6914 | Val Loss: 0.7665 | Mean Abs Error: 0.7665
+Epoch 791 | Train Loss: 0.7264 | Val Loss: 0.7670 | Mean Abs Error: 0.7670
+Epoch 792 | Train Loss: 0.6879 | Val Loss: 0.7644 | Mean Abs Error: 0.7644
+Epoch 793 | Train Loss: 0.6767 | Val Loss: 0.7646 | Mean Abs Error: 0.7646
+Epoch 794 | Train Loss: 0.6534 | Val Loss: 0.7650 | Mean Abs Error: 0.7650
+Epoch 795 | Train Loss: 0.7290 | Val Loss: 0.7658 | Mean Abs Error: 0.7658
+Epoch 796 | Train Loss: 0.7151 | Val Loss: 0.7654 | Mean Abs Error: 0.7654
+Epoch 797 | Train Loss: 0.6669 | Val Loss: 0.7645 | Mean Abs Error: 0.7645
+Epoch 798 | Train Loss: 0.6617 | Val Loss: 0.7625 | Mean Abs Error: 0.7625
+Epoch 799 | Train Loss: 0.7206 | Val Loss: 0.7611 | Mean Abs Error: 0.7611
+Epoch 800 | Train Loss: 0.6739 | Val Loss: 0.7611 | Mean Abs Error: 0.7611
+```
+
+That's just 5% and we can see better validation loss and a smoother balance between the two losses.
diff --git a/files/blog/assets/06-lpp/example.png b/files/blog/assets/06-lpp/example.png
new file mode 100644
index 0000000000000000000000000000000000000000..f79eaf557570bc25281bd5f6b9b9640793cc45a4
GIT binary patch
literal 258027
zcmeEuhhJ0a)@~eUbY?6#GBywx#|;@lK}BGwaYWHjReA?$O79r@i~^z}vLn4&=@0<{
zslkF65m0(ZA&3x)gc3r%YlkrByWjoY|KRvL9Cb|I{g$Jgt&{lorHVW#(5a!C)aa^9ypx%LF{M4)h7gR
z+y)$uc=|7$t4U+Mq@-jYLz$IE*+N^b=7Q+t@@J0!xbnOc`EB#}59CBeZ`{8WtZ;21
z4jx#&$^^L}6yK-T*iDly5T@DHoS
zhV&G=NZh1w+kkd%7kdV)Vx|#1m3(Z|r)bpL{a(~NJd|wgc&0?wxt+d~=NPRSDJ%oc
zmsdfM;Ms3lEg{tEK}!i2s;btLUiads%WkGgp7mBgW72fEQrPYy0Qf&}c6JhTwb(Ts
z%aFPqWkw|+-}Lj1Q=%@56L0n3;&-5dk^%%CzH}pSB=GzHL|Fw`LoX!}uMZMq_h%*U
zdrOcj9arlM#R_@FYXYKcF!F*Z86L7rylM@VM_uK#mIUf|wmz}TOrx#>$hh^SQ`o>w
zU|BQAK88@sT4ABr{U6HdGX8>HgvPN0qGOW?;)}0gKAc7q&=0WB@X?lBqoL1OFUf>v
z|BoDLJX+b#yI^^5uNr&^X3Q^a&+%L~={0goIS+D;h<3h-SyARBpu4<~?pn74au!V?
zA5zBa_?*^m0jkpAt&adj&x6vCF_eblZOq1O!}dUP-!7sJD5F=O_vb;djQ}WXZS{FJ
zP*GvjNm2+MNg{?wlg|isLZi$-*HQKq-3@rSAV?WiN28<)9e0uClw>Q?{J*fe=L*!t
zhuHQ#T_IVUdu<1e55mFs$1u03*y$9*`kxZ$n;a_8!*zT&Cu5&5&I_5`?_^$nCvR7~
zoKzjKm1{(QC{NtS8Wzr