blog 06: new post

files/blog/06-lpp.md

@@ -0,0 +1,321 @@
+[article]
+title = "Longest path problem w/ heuristics"
+author = "Cyril Ulyanov"
+date = "27-08-2026"
+---
+
+<summary>
+
+> ### TL;DR
+> [Longest path problem](https://en.wikipedia.org/wiki/Longest_path_problem) has no known "fast"
+> algorithm (unless [P=NP](https://en.wikipedia.org/wiki/P_versus_NP_problem)), but approximations suffice in practice.
+
+</summary>
+
+## The impasse
+
+<img src="/blog/assets/06-lpp/example.png" alt="Example graph" width="30%">
+
+The longest path problem asks: *what's a simple path of maximum edge weight between vertices in a graph?*
+What's special about it is the lack of a known "quick" algorithm for computing the path. Here, the word "quick" means
+that the execution time can be expressed as a polynomial function of the input size.
+Also, what's a *graph* in this case? To be more accurate, it is:
+- Undirected
+- Simple
+- *Cyclic*[^1]
+- Unweighted
+
+There are of course methods faster than brute-force (which is
+$$ \mathcal{\text{O}}(\text{n}! \cdot \text{n}) $$). One example is the Held-Karp algorithm,
+where the time complexity is $$ \mathcal{\text{O}}(2^\text{n} \cdot \text{n}^2) $$ (where $$ \text{n} $$ means
+the number of nodes, as in the entire article).
+> which still appears to be more desirable than brute force?
+
+That's correct, but a brute-force algorithm's space complexity depends linearly on the number of nodes, whereas
+Held-Karp's complexity grows as $$ \mathcal{\text{O}}(2^\text{n} \cdot \text{n}) $$. Which means that at 30
+nodes, the program would consume over 100 GB of RAM.
+Of course, there are certain
+low-level optimizations, such as capping the maximum number of nodes to 64
+and using bitmasks inside a `uint64_t` for edge storage.
+While these modifications do not affect the inherent time complexity,
+they could reduce the constant factors.
+
+But what about heuristics? They can be accurate or rough, fast or slow. Why not find a method to determine
+the near-exact longest path that finishes working within acceptable time limits?
+Now we bump into yet another obstacle: the longest path is notoriously hard to approximate. To put it informally, the
+main issue is that making optimal local choices doesn't necessarily lead us to the globally most extended path
+without the expensive exploration we're attempting to dodge.
+
+## The heuristic & general implementation
+
+All aspects of the problem point to neural networks. Let's try to train a model and see how well it performs.
+However, we're facing a core <u>contradiction</u>: how to get training data if there's no way to do it within
+a reasonable timespan? I won't be original here: the most realistic way is to leave it searching for the longest
+path in the background while you're using your computer. I've done exactly this and I'm pretty sure my dataset
+has some entries that have taken *tens* of minutes to process. These are of course the best examples to feed to
+the network. Here's how I save a given generation:
+```
+{
+  "num_nodes": 39,
+  "num_edges": 72,
+  "longest_path": 36,
+  "adjacency_list": [
+    [
+      3,
+      15,
+      20,
+      25
+    ],
+    [
+      20,
+      22,
+    ...
+}
+```
+
+This is not the only way to structure the dataset, but this one seems good enough.
+
+The *incredibly witty* algorithm that searches for the longest path is written in C. Here are the constraints that
+I use for the last chunk of the training data: 
+```c
+#define MAX_NODES 64
+#define MIN_DEGREE 2.5
+#define MAX_DEGREE 4.1
+```
+
+On other runs, these constants are slightly different:
+we want to gather diverse entries.
+
+One might notice that `MIN_DEGREE` and `MAX_DEGREE` are counter-intuitively used as floats.
+To understand why that is the case, we should dive into what these really mean.
+First, it stochastically calculates a value that is <u>proportional</u> to total edge count:
+```c
+igraph_real_t target_degree = MIN_DEGREE + ((double)rand() / RAND_MAX) * (MAX_DEGREE - MIN_DEGREE)
+```
+
+Then, if $$ \text{p} $$ is the probability that two given nodes become linked and $$ \text{d} $$
+is `target_degree`,\
+$$ \text{p} = \begin{cases} 0, & \text{if } \text{n} \in \{0, 1\} \\ \text{min}\left(1.0, \frac{\text{d}}{\text{n} - 1}\right), & \text{if } \text{n} > 1 \end{cases} $$
+
+And it puts edges between every pair of nodes using $$ \text{p} $$. 
+
+Like this, I've managed to accumulate over 800[^2] graphs and their corresponding longest paths.
+80% goes to training, 20% is reserved for validation.
+
+Now the important part, which is how we're going to implement the network. Fortunately, there's a very usable
+framework for working with graphs built into PyTorch.
+Before training on the collected data, I apply a pre-processor called `AddRandomWalkPE` to give the network a more
+global representation of the graph. All it does is it walks a given number of steps and returns the probability
+that a node is going to stroll into itself. Or, to be more exact:
+$$ (\text{P}^\text{k})_{\text{ii}} = \sum_{\text{i}_1} \sum_{\text{i}_2} \dots \sum_{\text{i}_{\text{k}-1}} \frac{\text{A}_{\text{i} \text{i}_1}}{\text{deg}(\text{i})} \cdot \frac{\text{A}_{\text{i}_1 \text{i}_2}}{\text{deg}(\text{i}_1)} \cdot \dots \cdot \frac{\text{A}_{\text{i}_{\text{k}-1} \text{i}}}{\text{deg}(\text{i}_{\text{k}-1})} $$
+
+$$ \text{i} $$ is the starting node,
+$$ \text{k} $$ is the walk length,
+$$ \text{A} $$ is a connection check (1 if connected, 0 otherwise),
+$$ \text{deg} $$ is a function that returns the degree of a given node (how many routes to choose from).
+
+Oh, and we're also going to throw in the degree of each node.
+It is of course not divided into inbound and unbound, because the graph is undirected.
+
+Here's the model that I've found to be effective enough:
+```python
+def __init__(self, hidden_channels):
+    super(Network, self).__init__()
+
+    self.input_proj = Linear(WALK_LENGTH + 1, hidden_channels)
+    self.proj_norm = GraphNorm(hidden_channels)
+
+    self.conv = GatedGraphConv(out_channels=hidden_channels, num_layers=10)
+    self.conv_norm = GraphNorm(hidden_channels)
+
+    self.mlp = Sequential(
+        Linear(hidden_channels * 2, hidden_channels),
+        LayerNorm(hidden_channels),
+        GELU(),
+        Dropout(p=0.3),
+        Linear(hidden_channels, hidden_channels // 2),
+        GELU(),
+        Linear(hidden_channels // 2, 1),
+    )
+```
+
+Keep in mind that $$ +1 $$ in the linear layer is that degree information I've just mentioned.
+
+And here are the actual calls:
+```python
+def forward(self, x, edge_index, batch):
+    h = GELU()(self.input_proj(x))
+    h = self.proj_norm(h, batch)
+
+    x_gated = self.conv(h, edge_index)
+    x_final = self.conv_norm(x_gated + h, batch)
+
+    pool_add = global_add_pool(x_final, batch)
+    pool_max = global_max_pool(x_final, batch)
+    x_pooled = torch.cat([pool_add, pool_max], dim=-1)
+
+    # Output the single continuous prediction float
+    return self.mlp(x_pooled).squeeze(-1)
+```
+
+Let's start with the easy parts. First, I feed the random walk into a linear layer and process it through
+an activation layer.
+After that, I normalize the node features separately for each individual graph using the batch vector.
+That's because graph 1 might have 5 nodes and graph 2 might have 100 nodes --
+we need to isolate their statistics so their different sizes don't distort each other's features.
+Now the "hard" parts.
+- `GatedGraphConv` applies a shared Gated Recurrent Unit (GRU) cell across all nodes to recursively
+update their structural representations during neighborhood message passing.
+Gated Recurrent Units (GRUs) use a reset gate and an update gate to regulate internal information flow.
+By selectively preserving or discarding historical state data at each step,
+GRUs capture long-term dependencies in
+sequential representations.
+
+Essentially, what this gives us is that our 128 features might <u>implicitly</u> represent something like, "Am I a leaf?",
+"How close am I to a densely connected hub?", or any other characteristic. Also, because it uses 10 layers,
+that means it learns stuff about friends, then friends of friends, and so on.
+- `conv_norm` accepts the sum of the result of `GatedGraphConv` and the very original blueprint.
+More precisely, it adds a tensor[^3] that is deterministically described by the original degree
+and stuff derived from the random walk. Then it normalizes
+the features.
+
+The rest of `forward` is in charge of boiling it all down into one value. `global_add_pool` walks down
+each graph's node rows and sums up their features, while `global_max_pool` grabs the peak values.
+Finally, `cat` glues the two outputs together. `mlp` is a sequence of layers
+that acts as the "funnel", squeezing everything down into a scalar, which is our predicted longest path.
+
+## Results
+I have specifically noted that these two have taken a pretty long time to finish:\
+`"num_nodes":61,"num_edges":89,"longest_path":47...`\
+`"num_nodes":45,"num_edges":81,"longest_path":39...`
+
+This is brand new data that the network has never been exposed to (beyond both training and validation),
+so let's test how well it approximates the longest path:
+
+
+> Graph #007 | True Path: 47.0 | Predicted: 45.7467 \
+> Graph #008 | True Path: 39.0 | Predicted: 41.6555
+
+
+As we can see, that's accurate enough. Keep in mind that the network finishes its calculations
+practically instantaneously, immeasurably faster than a deterministic method.
+
+## Closing notes
+Successful approximations from this network indicate that, in reality, when an NP-hard problem
+needs to be solved, we must resort to heuristics. Is this going to result in an exact solution?
+Definitely not. But is this approach going to suffice in practice? Well, that seems very likely.
+
+[^1]: Which is essential! There exists an algorithm for acyclic graphs with the time complexity of O(n).
+
+[^2]: Which is not a lot! But that's enough unless we are looking for perfect approximation, which we
+aren't in this instance.
+
+[^3]: Multi-dimensional array of numerical data.
+
+## Statistics, statistics, and statistics
+
+<img src="/blog/assets/06-lpp/graph.png" alt="Example graph" width="70%">
+<img src="/blog/assets/06-lpp/graph1.png" alt="Example graph" width="70%">
+
+## Follow-up
+After some more experimentation, it turned out it was possible to add some data augmentation. The biggest
+advantage is that there's going to be less room for overfitting. In other words, the model should
+learn what a longest path is like in terms of existing patterns.
+I've tried embedding Gaussian noise, which is quite common for what I'm trying to achieve. However,
+a technique that makes more sense for graphs is not exactly that; it's dropping edges with a defined
+probability. In some sense, it is *noise* as well, because we confuse the model: for example, we might drop an edge that is necessary for the existence of the path we tell it serves as the longest.
+However, that's really the goal -- making it analyze the underlying structure of this path without
+focusing on unreliable details.
+
+Here's how I add it:
+```python
+class Network(torch.nn.Module):
+    def __init__(self, hidden_channels):
+        super(Network, self).__init__()
+
+        self.edge_dropout_p = 0.05
+
+        ...
+
+    def forward(self, x, edge_index, batch):
+        h = GELU()(self.input_proj(x))
+        h = self.proj_norm(h, batch)
+
+        edges = edge_index
+        edges, _ = dropout_edge(
+            edge_index, 
+            p=self.edge_dropout_p,
+            force_undirected=True,
+            training=self.training
+        )
+
+        ...
+```
+
+Is it substantially better though?
+Here's what I witnessed with the previous setup:
+```
+Epoch 557 | Train Loss: 0.6867 | Val Loss: 0.8003 | Mean Abs Error: 0.8003
+Epoch 558 | Train Loss: 0.6996 | Val Loss: 0.8026 | Mean Abs Error: 0.8026
+Epoch 559 | Train Loss: 0.6926 | Val Loss: 0.7699 | Mean Abs Error: 0.7699
+Epoch 560 | Train Loss: 0.6276 | Val Loss: 0.7949 | Mean Abs Error: 0.7949
+Epoch 561 | Train Loss: 0.6610 | Val Loss: 0.7975 | Mean Abs Error: 0.7975
+Epoch 562 | Train Loss: 0.6952 | Val Loss: 0.8364 | Mean Abs Error: 0.8364
+Epoch 563 | Train Loss: 0.7464 | Val Loss: 0.8292 | Mean Abs Error: 0.8292
+Epoch 564 | Train Loss: 0.7076 | Val Loss: 0.7803 | Mean Abs Error: 0.7803
+Epoch 565 | Train Loss: 0.6764 | Val Loss: 0.7445 | Mean Abs Error: 0.7445
+Saved new best model with Val Loss: 0.7445
+Epoch 566 | Train Loss: 0.6699 | Val Loss: 0.8029 | Mean Abs Error: 0.8029
+Epoch 567 | Train Loss: 0.6857 | Val Loss: 0.7611 | Mean Abs Error: 0.7611
+Epoch 568 | Train Loss: 0.6820 | Val Loss: 0.7845 | Mean Abs Error: 0.7845
+Epoch 569 | Train Loss: 0.6510 | Val Loss: 0.8754 | Mean Abs Error: 0.8754
+...
+Epoch 788 | Train Loss: 0.6169 | Val Loss: 0.7690 | Mean Abs Error: 0.7690
+Epoch 789 | Train Loss: 0.6122 | Val Loss: 0.7681 | Mean Abs Error: 0.7681
+Epoch 790 | Train Loss: 0.6464 | Val Loss: 0.7660 | Mean Abs Error: 0.7660
+Epoch 791 | Train Loss: 0.6312 | Val Loss: 0.7657 | Mean Abs Error: 0.7657
+Epoch 792 | Train Loss: 0.6477 | Val Loss: 0.7664 | Mean Abs Error: 0.7664
+Epoch 793 | Train Loss: 0.5939 | Val Loss: 0.7661 | Mean Abs Error: 0.7661
+Epoch 794 | Train Loss: 0.6453 | Val Loss: 0.7662 | Mean Abs Error: 0.7662
+Epoch 795 | Train Loss: 0.6437 | Val Loss: 0.7643 | Mean Abs Error: 0.7643
+Epoch 796 | Train Loss: 0.6089 | Val Loss: 0.7630 | Mean Abs Error: 0.7630
+Epoch 797 | Train Loss: 0.6738 | Val Loss: 0.7629 | Mean Abs Error: 0.7629
+Epoch 798 | Train Loss: 0.5949 | Val Loss: 0.7632 | Mean Abs Error: 0.7632
+Epoch 799 | Train Loss: 0.6197 | Val Loss: 0.7632 | Mean Abs Error: 0.7632
+Epoch 800 | Train Loss: 0.6460 | Val Loss: 0.7635 | Mean Abs Error: 0.7635
+```
+
+And that was the last best model ever saved.
+
+Now the version with edge dropout. Running with the same seed!
+```
+Epoch 380 | Train Loss: 0.9379 | Val Loss: 0.7857 | Mean Abs Error: 0.7857
+Epoch 381 | Train Loss: 0.9091 | Val Loss: 0.8058 | Mean Abs Error: 0.8058
+Epoch 382 | Train Loss: 0.9151 | Val Loss: 0.8709 | Mean Abs Error: 0.8709
+Epoch 383 | Train Loss: 1.0247 | Val Loss: 0.8806 | Mean Abs Error: 0.8806
+Epoch 384 | Train Loss: 0.9175 | Val Loss: 0.8099 | Mean Abs Error: 0.8099
+Epoch 385 | Train Loss: 0.8757 | Val Loss: 0.7452 | Mean Abs Error: 0.7452
+Epoch 386 | Train Loss: 0.8859 | Val Loss: 1.0104 | Mean Abs Error: 1.0104
+Epoch 387 | Train Loss: 0.8694 | Val Loss: 0.8945 | Mean Abs Error: 0.8945
+Epoch 388 | Train Loss: 0.9448 | Val Loss: 0.7028 | Mean Abs Error: 0.7028
+Saved new best model with Val Loss: 0.7028
+Epoch 389 | Train Loss: 0.9180 | Val Loss: 0.7744 | Mean Abs Error: 0.7744
+Epoch 390 | Train Loss: 0.8837 | Val Loss: 0.7454 | Mean Abs Error: 0.7454
+Epoch 391 | Train Loss: 0.8545 | Val Loss: 0.7426 | Mean Abs Error: 0.7426
+Epoch 392 | Train Loss: 0.9028 | Val Loss: 0.7584 | Mean Abs Error: 0.7584
+Epoch 393 | Train Loss: 0.9434 | Val Loss: 0.8505 | Mean Abs Error: 0.8505
+...
+Epoch 790 | Train Loss: 0.6914 | Val Loss: 0.7665 | Mean Abs Error: 0.7665
+Epoch 791 | Train Loss: 0.7264 | Val Loss: 0.7670 | Mean Abs Error: 0.7670
+Epoch 792 | Train Loss: 0.6879 | Val Loss: 0.7644 | Mean Abs Error: 0.7644
+Epoch 793 | Train Loss: 0.6767 | Val Loss: 0.7646 | Mean Abs Error: 0.7646
+Epoch 794 | Train Loss: 0.6534 | Val Loss: 0.7650 | Mean Abs Error: 0.7650
+Epoch 795 | Train Loss: 0.7290 | Val Loss: 0.7658 | Mean Abs Error: 0.7658
+Epoch 796 | Train Loss: 0.7151 | Val Loss: 0.7654 | Mean Abs Error: 0.7654
+Epoch 797 | Train Loss: 0.6669 | Val Loss: 0.7645 | Mean Abs Error: 0.7645
+Epoch 798 | Train Loss: 0.6617 | Val Loss: 0.7625 | Mean Abs Error: 0.7625
+Epoch 799 | Train Loss: 0.7206 | Val Loss: 0.7611 | Mean Abs Error: 0.7611
+Epoch 800 | Train Loss: 0.6739 | Val Loss: 0.7611 | Mean Abs Error: 0.7611
+```
+
+That's just 5% and we can see better validation loss and a smoother balance between the two losses.