diff --git a/files/blog/06-lpp.md b/files/blog/06-lpp.md new file mode 100644 index 0000000..4d49b3a --- /dev/null +++ b/files/blog/06-lpp.md @@ -0,0 +1,321 @@ +[article] +title = "Longest path problem w/ heuristics" +author = "Cyril Ulyanov" +date = "27-08-2026" +--- + + + +> ### TL;DR +> [Longest path problem](https://en.wikipedia.org/wiki/Longest_path_problem) has no known "fast" +> algorithm (unless [P=NP](https://en.wikipedia.org/wiki/P_versus_NP_problem)), but approximations suffice in practice. + + + +## The impasse + +Example graph + +The longest path problem asks: *what's a simple path of maximum edge weight between vertices in a graph?* +What's special about it is the lack of a known "quick" algorithm for computing the path. Here, the word "quick" means +that the execution time can be expressed as a polynomial function of the input size. +Also, what's a *graph* in this case? To be more accurate, it is: +- Undirected +- Simple +- *Cyclic*[^1] +- Unweighted + +There are of course methods faster than brute-force (which is +$$ \mathcal{\text{O}}(\text{n}! \cdot \text{n}) $$). One example is the Held-Karp algorithm, +where the time complexity is $$ \mathcal{\text{O}}(2^\text{n} \cdot \text{n}^2) $$ (where $$ \text{n} $$ means +the number of nodes, as in the entire article). +> which still appears to be more desirable than brute force? + +That's correct, but a brute-force algorithm's space complexity depends linearly on the number of nodes, whereas +Held-Karp's complexity grows as $$ \mathcal{\text{O}}(2^\text{n} \cdot \text{n}) $$. Which means that at 30 +nodes, the program would consume over 100 GB of RAM. +Of course, there are certain +low-level optimizations, such as capping the maximum number of nodes to 64 +and using bitmasks inside a `uint64_t` for edge storage. +While these modifications do not affect the inherent time complexity, +they could reduce the constant factors. + +But what about heuristics? They can be accurate or rough, fast or slow. Why not find a method to determine +the near-exact longest path that finishes working within acceptable time limits? +Now we bump into yet another obstacle: the longest path is notoriously hard to approximate. To put it informally, the +main issue is that making optimal local choices doesn't necessarily lead us to the globally most extended path +without the expensive exploration we're attempting to dodge. + +## The heuristic & general implementation + +All aspects of the problem point to neural networks. Let's try to train a model and see how well it performs. +However, we're facing a core contradiction: how to get training data if there's no way to do it within +a reasonable timespan? I won't be original here: the most realistic way is to leave it searching for the longest +path in the background while you're using your computer. I've done exactly this and I'm pretty sure my dataset +has some entries that have taken *tens* of minutes to process. These are of course the best examples to feed to +the network. Here's how I save a given generation: +``` +{ + "num_nodes": 39, + "num_edges": 72, + "longest_path": 36, + "adjacency_list": [ + [ + 3, + 15, + 20, + 25 + ], + [ + 20, + 22, + ... +} +``` + +This is not the only way to structure the dataset, but this one seems good enough. + +The *incredibly witty* algorithm that searches for the longest path is written in C. Here are the constraints that +I use for the last chunk of the training data: +```c +#define MAX_NODES 64 +#define MIN_DEGREE 2.5 +#define MAX_DEGREE 4.1 +``` + +On other runs, these constants are slightly different: +we want to gather diverse entries. + +One might notice that `MIN_DEGREE` and `MAX_DEGREE` are counter-intuitively used as floats. +To understand why that is the case, we should dive into what these really mean. +First, it stochastically calculates a value that is proportional to total edge count: +```c +igraph_real_t target_degree = MIN_DEGREE + ((double)rand() / RAND_MAX) * (MAX_DEGREE - MIN_DEGREE) +``` + +Then, if $$ \text{p} $$ is the probability that two given nodes become linked and $$ \text{d} $$ +is `target_degree`,\ +$$ \text{p} = \begin{cases} 0, & \text{if } \text{n} \in \{0, 1\} \\ \text{min}\left(1.0, \frac{\text{d}}{\text{n} - 1}\right), & \text{if } \text{n} > 1 \end{cases} $$ + +And it puts edges between every pair of nodes using $$ \text{p} $$. + +Like this, I've managed to accumulate over 800[^2] graphs and their corresponding longest paths. +80% goes to training, 20% is reserved for validation. + +Now the important part, which is how we're going to implement the network. Fortunately, there's a very usable +framework for working with graphs built into PyTorch. +Before training on the collected data, I apply a pre-processor called `AddRandomWalkPE` to give the network a more +global representation of the graph. All it does is it walks a given number of steps and returns the probability +that a node is going to stroll into itself. Or, to be more exact: +$$ (\text{P}^\text{k})_{\text{ii}} = \sum_{\text{i}_1} \sum_{\text{i}_2} \dots \sum_{\text{i}_{\text{k}-1}} \frac{\text{A}_{\text{i} \text{i}_1}}{\text{deg}(\text{i})} \cdot \frac{\text{A}_{\text{i}_1 \text{i}_2}}{\text{deg}(\text{i}_1)} \cdot \dots \cdot \frac{\text{A}_{\text{i}_{\text{k}-1} \text{i}}}{\text{deg}(\text{i}_{\text{k}-1})} $$ + +$$ \text{i} $$ is the starting node, +$$ \text{k} $$ is the walk length, +$$ \text{A} $$ is a connection check (1 if connected, 0 otherwise), +$$ \text{deg} $$ is a function that returns the degree of a given node (how many routes to choose from). + +Oh, and we're also going to throw in the degree of each node. +It is of course not divided into inbound and unbound, because the graph is undirected. + +Here's the model that I've found to be effective enough: +```python +def __init__(self, hidden_channels): + super(Network, self).__init__() + + self.input_proj = Linear(WALK_LENGTH + 1, hidden_channels) + self.proj_norm = GraphNorm(hidden_channels) + + self.conv = GatedGraphConv(out_channels=hidden_channels, num_layers=10) + self.conv_norm = GraphNorm(hidden_channels) + + self.mlp = Sequential( + Linear(hidden_channels * 2, hidden_channels), + LayerNorm(hidden_channels), + GELU(), + Dropout(p=0.3), + Linear(hidden_channels, hidden_channels // 2), + GELU(), + Linear(hidden_channels // 2, 1), + ) +``` + +Keep in mind that $$ +1 $$ in the linear layer is that degree information I've just mentioned. + +And here are the actual calls: +```python +def forward(self, x, edge_index, batch): + h = GELU()(self.input_proj(x)) + h = self.proj_norm(h, batch) + + x_gated = self.conv(h, edge_index) + x_final = self.conv_norm(x_gated + h, batch) + + pool_add = global_add_pool(x_final, batch) + pool_max = global_max_pool(x_final, batch) + x_pooled = torch.cat([pool_add, pool_max], dim=-1) + + # Output the single continuous prediction float + return self.mlp(x_pooled).squeeze(-1) +``` + +Let's start with the easy parts. First, I feed the random walk into a linear layer and process it through +an activation layer. +After that, I normalize the node features separately for each individual graph using the batch vector. +That's because graph 1 might have 5 nodes and graph 2 might have 100 nodes -- +we need to isolate their statistics so their different sizes don't distort each other's features. +Now the "hard" parts. +- `GatedGraphConv` applies a shared Gated Recurrent Unit (GRU) cell across all nodes to recursively +update their structural representations during neighborhood message passing. +Gated Recurrent Units (GRUs) use a reset gate and an update gate to regulate internal information flow. +By selectively preserving or discarding historical state data at each step, +GRUs capture long-term dependencies in +sequential representations. + +Essentially, what this gives us is that our 128 features might implicitly represent something like, "Am I a leaf?", +"How close am I to a densely connected hub?", or any other characteristic. Also, because it uses 10 layers, +that means it learns stuff about friends, then friends of friends, and so on. +- `conv_norm` accepts the sum of the result of `GatedGraphConv` and the very original blueprint. +More precisely, it adds a tensor[^3] that is deterministically described by the original degree +and stuff derived from the random walk. Then it normalizes +the features. + +The rest of `forward` is in charge of boiling it all down into one value. `global_add_pool` walks down +each graph's node rows and sums up their features, while `global_max_pool` grabs the peak values. +Finally, `cat` glues the two outputs together. `mlp` is a sequence of layers +that acts as the "funnel", squeezing everything down into a scalar, which is our predicted longest path. + +## Results +I have specifically noted that these two have taken a pretty long time to finish:\ +`"num_nodes":61,"num_edges":89,"longest_path":47...`\ +`"num_nodes":45,"num_edges":81,"longest_path":39...` + +This is brand new data that the network has never been exposed to (beyond both training and validation), +so let's test how well it approximates the longest path: + + +> Graph #007 | True Path: 47.0 | Predicted: 45.7467 \ +> Graph #008 | True Path: 39.0 | Predicted: 41.6555 + + +As we can see, that's accurate enough. Keep in mind that the network finishes its calculations +practically instantaneously, immeasurably faster than a deterministic method. + +## Closing notes +Successful approximations from this network indicate that, in reality, when an NP-hard problem +needs to be solved, we must resort to heuristics. Is this going to result in an exact solution? +Definitely not. But is this approach going to suffice in practice? Well, that seems very likely. + +[^1]: Which is essential! There exists an algorithm for acyclic graphs with the time complexity of O(n). + +[^2]: Which is not a lot! But that's enough unless we are looking for perfect approximation, which we +aren't in this instance. + +[^3]: Multi-dimensional array of numerical data. + +## Statistics, statistics, and statistics + +Example graph +Example graph + +## Follow-up +After some more experimentation, it turned out it was possible to add some data augmentation. The biggest +advantage is that there's going to be less room for overfitting. In other words, the model should +learn what a longest path is like in terms of existing patterns. +I've tried embedding Gaussian noise, which is quite common for what I'm trying to achieve. However, +a technique that makes more sense for graphs is not exactly that; it's dropping edges with a defined +probability. In some sense, it is *noise* as well, because we confuse the model: for example, we might drop an edge that is necessary for the existence of the path we tell it serves as the longest. +However, that's really the goal -- making it analyze the underlying structure of this path without +focusing on unreliable details. + +Here's how I add it: +```python +class Network(torch.nn.Module): + def __init__(self, hidden_channels): + super(Network, self).__init__() + + self.edge_dropout_p = 0.05 + + ... + + def forward(self, x, edge_index, batch): + h = GELU()(self.input_proj(x)) + h = self.proj_norm(h, batch) + + edges = edge_index + edges, _ = dropout_edge( + edge_index, + p=self.edge_dropout_p, + force_undirected=True, + training=self.training + ) + + ... +``` + +Is it substantially better though? +Here's what I witnessed with the previous setup: +``` +Epoch 557 | Train Loss: 0.6867 | Val Loss: 0.8003 | Mean Abs Error: 0.8003 +Epoch 558 | Train Loss: 0.6996 | Val Loss: 0.8026 | Mean Abs Error: 0.8026 +Epoch 559 | Train Loss: 0.6926 | Val Loss: 0.7699 | Mean Abs Error: 0.7699 +Epoch 560 | Train Loss: 0.6276 | Val Loss: 0.7949 | Mean Abs Error: 0.7949 +Epoch 561 | Train Loss: 0.6610 | Val Loss: 0.7975 | Mean Abs Error: 0.7975 +Epoch 562 | Train Loss: 0.6952 | Val Loss: 0.8364 | Mean Abs Error: 0.8364 +Epoch 563 | Train Loss: 0.7464 | Val Loss: 0.8292 | Mean Abs Error: 0.8292 +Epoch 564 | Train Loss: 0.7076 | Val Loss: 0.7803 | Mean Abs Error: 0.7803 +Epoch 565 | Train Loss: 0.6764 | Val Loss: 0.7445 | Mean Abs Error: 0.7445 +Saved new best model with Val Loss: 0.7445 +Epoch 566 | Train Loss: 0.6699 | Val Loss: 0.8029 | Mean Abs Error: 0.8029 +Epoch 567 | Train Loss: 0.6857 | Val Loss: 0.7611 | Mean Abs Error: 0.7611 +Epoch 568 | Train Loss: 0.6820 | Val Loss: 0.7845 | Mean Abs Error: 0.7845 +Epoch 569 | Train Loss: 0.6510 | Val Loss: 0.8754 | Mean Abs Error: 0.8754 +... +Epoch 788 | Train Loss: 0.6169 | Val Loss: 0.7690 | Mean Abs Error: 0.7690 +Epoch 789 | Train Loss: 0.6122 | Val Loss: 0.7681 | Mean Abs Error: 0.7681 +Epoch 790 | Train Loss: 0.6464 | Val Loss: 0.7660 | Mean Abs Error: 0.7660 +Epoch 791 | Train Loss: 0.6312 | Val Loss: 0.7657 | Mean Abs Error: 0.7657 +Epoch 792 | Train Loss: 0.6477 | Val Loss: 0.7664 | Mean Abs Error: 0.7664 +Epoch 793 | Train Loss: 0.5939 | Val Loss: 0.7661 | Mean Abs Error: 0.7661 +Epoch 794 | Train Loss: 0.6453 | Val Loss: 0.7662 | Mean Abs Error: 0.7662 +Epoch 795 | Train Loss: 0.6437 | Val Loss: 0.7643 | Mean Abs Error: 0.7643 +Epoch 796 | Train Loss: 0.6089 | Val Loss: 0.7630 | Mean Abs Error: 0.7630 +Epoch 797 | Train Loss: 0.6738 | Val Loss: 0.7629 | Mean Abs Error: 0.7629 +Epoch 798 | Train Loss: 0.5949 | Val Loss: 0.7632 | Mean Abs Error: 0.7632 +Epoch 799 | Train Loss: 0.6197 | Val Loss: 0.7632 | Mean Abs Error: 0.7632 +Epoch 800 | Train Loss: 0.6460 | Val Loss: 0.7635 | Mean Abs Error: 0.7635 +``` + +And that was the last best model ever saved. + +Now the version with edge dropout. Running with the same seed! +``` +Epoch 380 | Train Loss: 0.9379 | Val Loss: 0.7857 | Mean Abs Error: 0.7857 +Epoch 381 | Train Loss: 0.9091 | Val Loss: 0.8058 | Mean Abs Error: 0.8058 +Epoch 382 | Train Loss: 0.9151 | Val Loss: 0.8709 | Mean Abs Error: 0.8709 +Epoch 383 | Train Loss: 1.0247 | Val Loss: 0.8806 | Mean Abs Error: 0.8806 +Epoch 384 | Train Loss: 0.9175 | Val Loss: 0.8099 | Mean Abs Error: 0.8099 +Epoch 385 | Train Loss: 0.8757 | Val Loss: 0.7452 | Mean Abs Error: 0.7452 +Epoch 386 | Train Loss: 0.8859 | Val Loss: 1.0104 | Mean Abs Error: 1.0104 +Epoch 387 | Train Loss: 0.8694 | Val Loss: 0.8945 | Mean Abs Error: 0.8945 +Epoch 388 | Train Loss: 0.9448 | Val Loss: 0.7028 | Mean Abs Error: 0.7028 +Saved new best model with Val Loss: 0.7028 +Epoch 389 | Train Loss: 0.9180 | Val Loss: 0.7744 | Mean Abs Error: 0.7744 +Epoch 390 | Train Loss: 0.8837 | Val Loss: 0.7454 | Mean Abs Error: 0.7454 +Epoch 391 | Train Loss: 0.8545 | Val Loss: 0.7426 | Mean Abs Error: 0.7426 +Epoch 392 | Train Loss: 0.9028 | Val Loss: 0.7584 | Mean Abs Error: 0.7584 +Epoch 393 | Train Loss: 0.9434 | Val Loss: 0.8505 | Mean Abs Error: 0.8505 +... +Epoch 790 | Train Loss: 0.6914 | Val Loss: 0.7665 | Mean Abs Error: 0.7665 +Epoch 791 | Train Loss: 0.7264 | Val Loss: 0.7670 | Mean Abs Error: 0.7670 +Epoch 792 | Train Loss: 0.6879 | Val Loss: 0.7644 | Mean Abs Error: 0.7644 +Epoch 793 | Train Loss: 0.6767 | Val Loss: 0.7646 | Mean Abs Error: 0.7646 +Epoch 794 | Train Loss: 0.6534 | Val Loss: 0.7650 | Mean Abs Error: 0.7650 +Epoch 795 | Train Loss: 0.7290 | Val Loss: 0.7658 | Mean Abs Error: 0.7658 +Epoch 796 | Train Loss: 0.7151 | Val Loss: 0.7654 | Mean Abs Error: 0.7654 +Epoch 797 | Train Loss: 0.6669 | Val Loss: 0.7645 | Mean Abs Error: 0.7645 +Epoch 798 | Train Loss: 0.6617 | Val Loss: 0.7625 | Mean Abs Error: 0.7625 +Epoch 799 | Train Loss: 0.7206 | Val Loss: 0.7611 | Mean Abs Error: 0.7611 +Epoch 800 | Train Loss: 0.6739 | Val Loss: 0.7611 | Mean Abs Error: 0.7611 +``` + +That's just 5% and we can see better validation loss and a smoother balance between the two losses. diff --git a/files/blog/assets/06-lpp/example.png b/files/blog/assets/06-lpp/example.png new file mode 100644 index 0000000..f79eaf5 Binary files /dev/null and b/files/blog/assets/06-lpp/example.png differ diff --git a/files/blog/assets/06-lpp/graph.png b/files/blog/assets/06-lpp/graph.png new file mode 100644 index 0000000..d876cd4 Binary files /dev/null and b/files/blog/assets/06-lpp/graph.png differ diff --git a/files/blog/assets/06-lpp/graph1.png b/files/blog/assets/06-lpp/graph1.png new file mode 100644 index 0000000..04a0033 Binary files /dev/null and b/files/blog/assets/06-lpp/graph1.png differ diff --git a/files/syntax/c.sublime-syntax b/files/syntax/c.sublime-syntax new file mode 100644 index 0000000..1028288 --- /dev/null +++ b/files/syntax/c.sublime-syntax @@ -0,0 +1,86 @@ +%YAML 1.2 +--- +name: c +file_extensions: [c, h] +scope: source.c + +contexts: + main: + - match: '\b(return|break|continue|goto|if|else|switch|case|default|while|do|for)\b' + scope: keyword.control + + - match: //.*$ + scope: comment + + - match: '\/\*' + push: comment_block + + - match: '\b(const|volatile|restrict|inline|extern|static|register|auto|typedef|struct|union|enum|sizeof)\b' + scope: keyword.other + + - match: '\{|\}|\(|\)' + scope: syntax.separator + + - match: '\*|\/|%|-|\+|<|>|=' + scope: op.arithmetic + + - match: '>>|<<|~|&|\||\^' + scope: op.bitwise + + - match: '!=|==|<=|>=|&&|\|\||!' + scope: op.logic + + - match: '&|\*' + scope: op.ref + + - match: '\b\?\b|:' + scope: op.special + + - match: ';' + scope: syntax.separator + + - match: '\b(void|char|short|int|long|float|double|signed|unsigned|_Bool|_Complex|_Imaginary)\b' + scope: entity.type + + - match: '\b(size_t|ssize_t|ptrdiff_t|intptr_t|uintptr_t|int8_t|int16_t|int32_t|int64_t|uint8_t|uint16_t|uint32_t|uint64_t)\b' + scope: entity.type.typedef + + - match: '\[|\]' + scope: op.brackets + + - match: '^\s*#\s*(include|define|undef|ifdef|ifndef|if|else|elif|endif|error|pragma)\b' + scope: keyword.control.import + + - match: '\b([a-zA-Z_]\w*)(?=\s*\()' + scope: entity.function + + - match: \b[0-9]+\.[0-9]+([fF]|[lL])?\b + scope: literal.float + + - match: '\b(0[xX][0-9a-fA-F]+|0[0-7]+|[0-9]+)([uU]?[lL]{0,2}|[lL]{0,2}[uU]?)?\b' + scope: literal.integer + + - match: "'" + push: char + + - match: '"' + push: string + + comment_block: + - meta_scope: comment + - match: '\*\/' + pop: true + + string: + - meta_scope: literal.string + - match: '\\.' + scope: constant.character.escape + - match: '"' + pop: true + + char: + - meta_scope: literal.char + - match: '\\.' + scope: constant.character.escape + - match: "'" + pop: true diff --git a/files/syntax/python.sublime-syntax b/files/syntax/python.sublime-syntax new file mode 100644 index 0000000..9561787 --- /dev/null +++ b/files/syntax/python.sublime-syntax @@ -0,0 +1,66 @@ +%YAML 1.2 +--- +name: python +file_extensions: [py, pyi, pyd, pyw] +scope: source.python + +contexts: + main: + # 1. Structural Symbols + - match: '\{|\}|\(|\)|\[|\]' + scope: syntax.separator + + - match: ':|;|\.' + scope: syntax.separator.delimiter + + - match: ',' + scope: syntax.separator.comma + + - match: '\b(if|else|elif|for|while|break|continue|return|yield|try|except|finally|raise|assert|pass|with|as)\b' + scope: keyword.control + + - match: '\b(def|class|lambda|global|nonlocal|import|from|del|in|is|and|or|not|async|await)\b' + scope: keyword.other + + - match: '\b(True|False|None)\b' + scope: constant.language + + - match: '\b(int|float|str|bool|list|dict|set|tuple|object|type|print|len|range|enumerate|zip|int|float)\b' + scope: entity.type + + - match: '\b(def|class)\s+([a-zA-Z_]\w*)' + captures: + 1: keyword.other + 2: entity.name.function + + - match: '\b([a-zA-Z_]\w*)(?=\s*\()' + scope: entity.function + + - match: '#.*$' + scope: comment + + - match: \b[0-9]+\.[0-9]+\b + scope: literal.float + + - match: \b[0-9]+\b + scope: literal.integer + + - match: "'" + push: char + + - match: '"' + push: string + + string: + - meta_scope: literal.string + - match: '\\.' + scope: constant.character.escape + - match: '"' + pop: true + + char: + - meta_scope: literal.char + - match: '\\.' + scope: constant.character.escape + - match: "'" + pop: true