-
Type:
Task
-
Resolution: Unresolved
-
Priority:
High
-
Component/s: Nagios
-
None
I'm not sure how easy this would be, but there are some pieces of data which would make this possible:
The incremental JSON dumps work by consuming replication packets.
The current_replication_sequence is the number of the latest replication packet which was produced by MB:
ssh -A queen 'docker exec postgres-slave psql -U musicbrainz -c "select current_replication_sequence from musicbrainz.replication_control;" musicbrainz_db' | head -n 3 | tail -n 1 | awk '{$1=$1};1'
The last_processed_replication_sequence is the last sequence which the JSON dumps have processed and generated an incremental dump for:
ssh -A williams 'docker exec postgres-williams psql -U musicbrainz -c "select last_processed_replication_sequence from json_dump.control;" musicbrainz_json_dump' | head -n 3 | tail -n 1 | awk '{$1=$1};1')
It would be nice to have flashing red alert for when:
last_processed_replication_sequence < (current_replication_sequence - 1)
It would also be nice to alert if the last processed sequence can't be fetched from the API (if this request 404s):
curl -I "https://metabrainz.org/api/musicbrainz/json-dumps/json-dump-$((last_processed_replication_sequence - 1))/artist.tar.xz?token=$TOKEN"
For the full JSON dumps, we should alert if the date in LATEST is out of date. The full dumps run every Wednesday and Saturday at 00:10:00 UTC, and usually finish by 05:00:00 (but sometimes a couple hours later, depending on server load I guess), then sync to FTP. So perhaps a check that occurs every Wednesday and Saturday at 12:00:00 UTC that alerts if LATEST is not equal to the current day.