use drupal trackchange instead of hw mark property to resolve migrati…

[rzhang152]

…on process count confliction (resolves #7)

[ctgraham]

$file_changed looks abandoned here. It is passed to parseXmlFile() but unused there.

[rzhang152]

file_changed is still referred in parseXmlFile() to capture the value in the source data row (...// Process each item
foreach ($items as $index => $item) {
$row_data = [
'media_id' => $media_id,
'file_changed' => $file_changed,
];
....
). It is necessary to include for checksum of the source row array as a row hash in the migration map table to detect a file change.

[ctgraham]

How will this addition impact existing migrated items?

[rzhang152]

Checked with Claude to consult how drupal track_changes works:

track_changes is a property read by Drupal\migrate\Plugin\migrate\source\SourcePluginBase. When set to true:

For each source row, Migrate computes a hash of the row's data. This hash is stored in the map table's hash column.
On subsequent runs, the newly computed hash for each row is compared against the stored hash.
If they differ, the row is flagged as needing update (MigrateIdMapInterface::STATUS_NEEDS_UPDATE) even if it was already STATUS_IMPORTED, so it gets reprocessed.
If they match, the row is skipped as unchanged.

while the tradeoff or impact I think will be for a very large dataset, it can slow down migration run due to hash.

I also asked Claude in case if we don't use any mechanism to detect change:

Without any mechanism, drupal migration will simply check the map table for the row's source MediaID. If no matching row exists in the map table, it's treated as new and gets imported; while if a matching row exists and its status is STATUS_IMPORTED, the row is skipped, no matter file change in the source (in our case file_changed value).

I think it will be better we add the track_changes property for a clean incremental migration. but there's a performance tradeoff as mentioned above. I am open to your thought.

[ctgraham]

I ran this through Claude as well, as:

In Drupal Migrate can I use a custom hashing mechanism for the track_changes source option? I have a custom source plugin which will read files and extract contents into source fields, but I really only need to compute the content hash on the files' properties, like size and modification time. Fetching the data for every file every time to check for changes would be too operationally expensive.

The response was essentially:

• Don't use track_changes; the hash mechanism isn't customizable
• Short circuit the expensive work in prepareRow() with a return FALSE; when there is nothing to do

So:

  /**
   * {@inheritdoc}
   */
  protected function initializeIterator() {
    $this->filesList = $this->listMediaXmlFiles();
    return new \ArrayIterator($this->filesList);
  }

  /**
   * prepare each source row after initializeIterator() and trigger POST_RAW_SAVE
   * {@inheritdoc}
   */
  public function prepareRow(Row $row) {
    foreach ($files as $file) {
        $row->setSourceProperty('file_modified', $file->getChangedTime());
        $row->setSourceProperty('file_uri', $file->getFileUri());
        $row->rehash();
          // Compare against what's stored for this row already.
          $map_row = $this->migration->getIdMap()->getRowBySource($row->getSourceIdValues());
          if ($map_row && $map_row['hash'] === $row->getHash()) {
            // Nothing changed on disk — skip without ever reading file contents.
            return FALSE;
          }
    }
    // New or change files
    foreach ($this->parseXmlFile($file) as $property) {
        $row->setSourceProperty($property->name, $property->value);
    }
    return parent::prepareRow($row);
  }