Add mean shift clustering - #369
Merged
Merged
Conversation
Mean shift moves every seed towards the mean of the samples within one bandwidth of it until the seed settles, then keeps the strongest of the seeds that landed on the same mode. The number of clusters comes out of the data rather than being given. Every seed is moved at once rather than one at a time, so an iteration is a single pairwise matrix instead of one per seed. That costs O(seeds * samples) of space, the same shape DBSCAN already pays through radius_neighbors. fit/2 keeps one row per seed so the shapes stay static, marking the centers that lost with :infinity and reporting how many survived, and prune/1 drops them. This follows Scholar.Cluster.AffinityPropagation, which has the same problem of not knowing the cluster count until it has run. Validated against scikit-learn 1.6.1 on 131 datasets: 11 hand-written edge cases and 120 random ones spanning 2 to 25 samples, 1 to 4 features and 1 to 21 clusters. Labels, centers and cluster counts match on all of them, including the order scikit-learn puts the centers in, which breaks ties on the coordinates.
Both were left out of groups_for_modules when they landed, so ex_doc filed them outside the Models group the rest of the clustering algorithms sit in.
The seeds carry the while accumulator, so their type has to survive a pass of the loop. Samples of a wider type promoted the moved seeds through Nx.dot and the do-block then failed to match what it was given, which raised a CompileError for a f64 sample set with f32 seeds. Merge the two types up front. Also documents that :max_iterations counts moves, where scikit-learn's max_iter checks the limit after moving and so takes one step more than the number given.
Two independent divergences, both invisible until a fit is cut short by :max_iterations, and both changing the centers rather than only a reported number. The cap was checked before moving the seeds rather than after. scikit-learn tests its limit once the move is done, so a run capped at k takes k + 1 steps and reports k, while this took k steps and reported k. Passing the same limit to both gave centers one step less converged here, and :iterations agreed with n_iter_ only when the run converged on its own. The weight that decides which of two centers on the same mode survives was counted around where a seed landed. scikit-learn counts the neighborhood that produced the center, before the last move. The two agree at a fixed point, so this only showed up when the run was truncated, and it picked a different representative of the same mode. Measured over 200 datasets against scikit-learn 1.6.1, spanning max_iter of 1, 2, 3, 4, 7 and 300, of which 56 are cut short by the limit. Labels, centers, cluster counts and iteration counts went from 194, 177, 199 and 56 out of 200 to 200 out of 200 on all four.
Contributor
|
💚 💙 💜 💛 ❤️ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the Mean Shift item on #135.
Mean shift moves every seed towards the mean of the samples within one
bandwidth of it until the seed settles, then keeps the strongest of the seeds
that landed on the same mode. The number of clusters comes out of the data
rather than being given, which is what the bandwidth controls.
Shape of the implementation
Every seed moves at once rather than one at a time, so an iteration is a single
pairwise matrix instead of one per seed. That costs
O(seeds * samples)ofspace, the same shape
DBSCANalready pays throughradius_neighbors. Ak-d tree would not help here: the query mean shift needs is "every sample
within the bandwidth", whose result size is not known until it runs, and
Scholar.Neighbors.KDTreeonly answers k nearest anyway.fit/2keeps one row per seed so the shapes stay static, marking the centersthat lost with
:infinityand reporting how many survived, andprune/1dropsthem. This follows
Scholar.Cluster.AffinityPropagation, which has the sameproblem of not knowing the cluster count until it has run.
prune/1raisesrather than building an empty tensor when nothing survived.
Validation
Checked against scikit-learn 1.6.1 on 131 datasets: 11 hand-written edge cases
and 120 random ones spanning 2 to 25 samples, 1 to 4 features, bandwidths from
0.3 to 6.0, and 1 to 21 resulting clusters. Labels, centers and cluster counts
match on all of them, on the default backend and under EXLA.
That includes the order scikit-learn puts the centers in, which sorts by how
many samples a center gathered and breaks ties on the coordinates. The edge
cases cover a bandwidth wide enough to hold everything, one narrower than the
closest pair, samples that are all the same point, a single sample, one
feature, samples sitting exactly at the bandwidth, and seeds that never reach a
sample, which scikit-learn drops and so does this.
max_iterationsandcluster_alleach have a test where the option changesthe answer, and there is a
jit_applytest.Left out on purpose
bin_seedingandestimate_bandwidthare not here. Binning produces a numberof seeds that is only known at runtime, which does not fit a static shape, so
the cheap way in is the
:seedsoption, the wayTrimapaccepts precomputedtriplets. Both are worth a follow-up issue.
I added
HDBSCANandOPTICSbecause were neveradded to
groups_for_modules, so ex_doc filed them outside the Models group.