Trigger reindex from dbmirror2 replication data

XMLWordPrintable

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Normal
    • None
    • Affects Version/s: None
    • Component/s: Indexer
    • None
    • Country

      SIR is currently notified about changes in the database through RabbitMQ, which is fed via SQL triggers (using the pg_amqp extension). This adds operational complexity, and some managed services do not allow installing the pg_amqp extension. This extension doesn't appear to be maintained any more either, with the last commit from 7 years ago.

      An alternative would be to have SIR triggers the reindex process from the replicated data produced by dbmirror2 and stored in PostgreSQL directly. The dbmirror2.pending_data rows track insertions, updates, and deletions on replicated tables, and can also be used to determine which columns changed for updates. Although this table is cleared hourly during replication packet generation, we could have a trigger copy the relevant rows to another table sir.* stored in PostgreSQL directly and to be used by SIR.


      Some more debatable implementation details below:

      Since the MusicBrainz replication scripts work by importing packet data into the dbmirror2.pending_data table, this alternative should work just as well on mirrors.

      There are at least two issues with this approach that I can see (but they are solvable):

      • If we want to continue allowing search indexing with standalone MusicBrainz instances in development, we'd have to start running the master replication setup on standalone servers, so that changes are tracked in the dbmirror2.* tables. However, standalone servers don't produce replication packets, so these changes would grow indefinitely. We could resolve this by introducing a standalone-only trigger that deletes the dbmirror2 data immediately after copying it for SIR, or provide a cron script that clears the tables hourly.
      • SIR installs triggers on two non-replicated tables: editor and recording_first_release_date. This could be resolved by maintaining custom triggers for these tables (which would need to be installed on any master, standalone, or mirror server using search indexing).

            Assignee:
            Unassigned
            Reporter:
            Michael Wiencek
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:

                Version Package