-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Normal
-
None
-
None
I am setting up a MusicBrainz mirror server with data replication. The Set Up Search Indexes step of the instructions instructs us to run a Docker compose command to reindex the Sir system. When I run that, I get thousands of error messages. As far as I can tell, the re-indexing fails.
How to reproduce:
- Set up a MusicBrainz mirror server following the MusicBrainz docker instructions.
- As instructed by the Set Up Search Indexes step of the instructions, run this command to reindex the Sir system
docker compose exec indexer python -m sir reindex
Observed behaviour:
The command generates thousands of lines of diagnostics. See below for details of the messages. The initial error message seems to be:
pysolr.SolrError: Failed to connect to server at http://search:8983/solr/annotation/update/: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
Then there are about 1549 further pysolr.SolrError error messages mentioning "[Errno 111] Connection refused" and 'NameResolutionError("...: Failed to resolve 'search' ([Errno -2] Name or service not known)")'.
The diagnostics related to normal progress include: "Importing annotation...", "Importing area...".
The diagnostics end without any clear statement that the re-index operation succeeded or failed.
Expected behaviour:
No error indications from the reindexing.
A moderate amount of diagnostics, with an indication of success at the end.
Diagnosted related to importing "annotation", "artist", "area", "cdstub", "editor", "event", "instrument", "label", "place", "recording", "release", "release-group", "series", "tag", "url", and "work". (Source: sir/schema/{}init{}.py:589-607.)
Discussion:
I observed similar failures both upon initially loading a data dump into an empty database on my Musicbrainz-Docker instance, and upon running a re-index on the database after loading the pre-built search indexes based on the latest data dump.
My config file at /code/config.ini reads,
[database]
dbname = musicbrainz_db
host = ${MUSICBRAINZ_POSTGRES_SERVER}
password = ${POSTGRES_PASSWORD}
port = 5432
user = ${POSTGRES_USER}
[solr]
uri = http://${MUSICBRAINZ_SEARCH_SERVER}
batch_size = 200
[sir]
import_threads = 8
index_limit = 200000
live_index_batch_size = 100
process_delay = 15
query_batch_size = 5000
wscompat = on
[rabbitmq]
host = ${MUSICBRAINZ_RABBITMQ_SERVER}
user = sir
password = sir
vhost = /search-index-rebuilder
prefetch_count = 350
[sentry]
dsn =
I re-ran the reindex command with the --debug switch, capturing stdout and stderr. It generated 14 MB of text. I can upload that if it is helpful. Below are some excerpts.
There are 8 occurrences of the following initial error message and cascaded diagnostics.
Solr-3 2025-11-23 09:25:31,597 ERROR: Error while submitting data to Solr:
Traceback (most recent call last):
File "/usr/local/lib/python3.13/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
conn,
...<10 lines>...
**response_kw,
)
File "/usr/local/lib/python3.13/site-packages/urllib3/connectionpool.py", line 534, in _make_request
response = conn.getresponse()
File "/usr/local/lib/python3.13/site-packages/urllib3/connection.py", line 516, in getresponse
httplib_response = super().getresponse()
File "/usr/local/lib/python3.13/site-packages/sentry_sdk/integrations/stdlib.py", line 131, in getresponse
rv = real_getresponse(self, *args, **kwargs)
File "/usr/local/lib/python3.13/http/client.py", line 1430, in getresponse
response.begin()
~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.13/http/client.py", line 331, in begin
version, status, reason = self._read_status()
~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.13/http/client.py", line 292, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/socket.py", line 719, in readinto
return self._sock.recv_into(b)
~~~~~~~~~~~~~~~~~~~~^^^
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.13/site-packages/requests/adapters.py", line 667, in send
resp = conn.urlopen(
method=request.method,
...<9 lines>...
chunked=chunked,
)
File "/usr/local/lib/python3.13/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
)
File "/usr/local/lib/python3.13/site-packages/urllib3/util/retry.py", line 474, in increment
raise reraise(type(error), error, _stacktrace)
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/urllib3/util/util.py", line 38, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.13/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
conn,
...<10 lines>...
**response_kw,
)
File "/usr/local/lib/python3.13/site-packages/urllib3/connectionpool.py", line 534, in _make_request
response = conn.getresponse()
File "/usr/local/lib/python3.13/site-packages/urllib3/connection.py", line 516, in getresponse
httplib_response = super().getresponse()
File "/usr/local/lib/python3.13/site-packages/sentry_sdk/integrations/stdlib.py", line 131, in getresponse
rv = real_getresponse(self, *args, **kwargs)
File "/usr/local/lib/python3.13/http/client.py", line 1430, in getresponse
response.begin()
~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.13/http/client.py", line 331, in begin
version, status, reason = self._read_status()
~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.13/http/client.py", line 292, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/socket.py", line 719, in readinto
return self._sock.recv_into(b)
~~~~~~~~~~~~~~~~~~~~^^^
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.13/site-packages/pysolr.py", line 414, in _send_request
resp = requests_method(
url,
...<4 lines>...
auth=self.auth,
)
File "/usr/local/lib/python3.13/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.13/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.13/site-packages/requests/adapters.py", line 682, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/code/sir/indexing.py", line 338, in send_data_to_solr
solr_connection.add(data)
~~~~~~~~~~~~~~~~~~~^^^^^^
File "/usr/local/lib/python3.13/site-packages/pysolr.py", line 1083, in add
return self._update(
~~~~~~~~~~~~^
m,
^^
...<8 lines>...
min_rf=min_rf,
^^^^^^^^^^^^^^
)
^
File "/usr/local/lib/python3.13/site-packages/pysolr.py", line 572, in _update
return self._send_request(
~~~~~~~~~~~~~~~~~~^
"post",
^^^^^^^
...<2 lines>...
{"Content-type": "application/json; charset=utf-8"},
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/usr/local/lib/python3.13/site-packages/pysolr.py", line 429, in _send_request
raise SolrError(error_message % (url, err))
pysolr.SolrError: Failed to connect to server at http://search:8983/solr/annotation/update/: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
I suspect that the other error messages might be consequences of this error, or have the same cause. So if this error could be fixed, then the rest might be fixed also.
Here are a subset of the diagnostic messages mentioning "Solr-1", which I take to be one of the 8 worker processes:
Solr-1 2025-11-23 09:24:21,533 DEBUG: Setting up a connection to http://search:8983/solr
Solr-1 2025-11-23 09:24:21,533 DEBUG: Pinging http://search:8983/solr/annotation/admin/ping
Solr-1 2025-11-23 09:24:21,600 DEBUG: Connection to the Solr core at http://search:8983/solr/annotation
...
Solr-1 2025-11-23 09:25:31,659 ERROR: Error while submitting data to Solr:
Traceback (most recent call last):
File "/usr/local/lib/python3.13/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
conn,
...<10 lines>...
**response_kw,
)
File "/usr/local/lib/python3.13/site-packages/urllib3/connectionpool.py", line 534, in _make_request
response = conn.getresponse()
File "/usr/local/lib/python3.13/site-packages/urllib3/connection.py", line 516, in getresponse
httplib_response = super().getresponse()
File "/usr/local/lib/python3.13/site-packages/sentry_sdk/integrations/stdlib.py", line 131, in getresponse
rv = real_getresponse(self, *args, **kwargs)
File "/usr/local/lib/python3.13/http/client.py", line 1430, in getresponse
response.begin()
~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.13/http/client.py", line 331, in begin
version, status, reason = self._read_status()
~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.13/http/client.py", line 292, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/socket.py", line 719, in readinto
return self._sock.recv_into(b)
~~~~~~~~~~~~~~~~~~~~^^^
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.13/site-packages/requests/adapters.py", line 667, in send
resp = conn.urlopen(
method=request.method,
...<9 lines>...
chunked=chunked,
)
File "/usr/local/lib/python3.13/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
)
File "/usr/local/lib/python3.13/site-packages/urllib3/util/retry.py", line 474, in increment
raise reraise(type(error), error, _stacktrace)
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/urllib3/util/util.py", line 38, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.13/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
conn,
...<10 lines>...
**response_kw,
)
File "/usr/local/lib/python3.13/site-packages/urllib3/connectionpool.py", line 534, in _make_request
response = conn.getresponse()
File "/usr/local/lib/python3.13/site-packages/urllib3/connection.py", line 516, in getresponse
httplib_response = super().getresponse()
File "/usr/local/lib/python3.13/site-packages/sentry_sdk/integrations/stdlib.py", line 131, in getresponse
rv = real_getresponse(self, *args, **kwargs)
File "/usr/local/lib/python3.13/http/client.py", line 1430, in getresponse
response.begin()
~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.13/http/client.py", line 331, in begin
version, status, reason = self._read_status()
~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.13/http/client.py", line 292, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/socket.py", line 719, in readinto
return self._sock.recv_into(b)
~~~~~~~~~~~~~~~~~~~~^^^
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.13/site-packages/pysolr.py", line 414, in _send_request
resp = requests_method(
url,
...<4 lines>...
auth=self.auth,
)
File "/usr/local/lib/python3.13/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.13/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.13/site-packages/requests/adapters.py", line 682, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/code/sir/indexing.py", line 338, in send_data_to_solr
solr_connection.add(data)
~~~~~~~~~~~~~~~~~~~^^^^^^
File "/usr/local/lib/python3.13/site-packages/pysolr.py", line 1083, in add
return self._update(
~~~~~~~~~~~~^
m,
^^
...<8 lines>...
min_rf=min_rf,
^^^^^^^^^^^^^^
)
^
File "/usr/local/lib/python3.13/site-packages/pysolr.py", line 572, in _update
return self._send_request(
~~~~~~~~~~~~~~~~~~^
"post",
^^^^^^^
...<2 lines>...
{"Content-type": "application/json; charset=utf-8"},
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/usr/local/lib/python3.13/site-packages/pysolr.py", line 429, in _send_request
raise SolrError(error_message % (url, err))
pysolr.SolrError: Failed to connect to server at http://search:8983/solr/annotation/update/: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
...
Solr-1 2025-11-23 09:25:31,954 DEBUG: Sent 200 new documents. Total: 200
... [several repetitions of the above ERROR message then DEBUG message]...
Solr-1 2025-11-23 09:33:16,618 DEBUG: Sent 200 new documents. Total: 38800
Solr-1 2025-11-23 09:33:17,420 DEBUG: <pysolr.Solr object at 0x7ffff6aa7b60>: Sending remaining data & stopping
...
Solr-1 2025-11-23 09:33:17,464 ERROR: Error while submitting data to Solr:
Traceback (most recent call last):
File "/usr/local/lib/python3.13/site-packages/urllib3/connection.py", line 198, in _new_conn
sock = connection.create_connection(
(self._dns_host, self.port),
...<2 lines>...
socket_options=self.socket_options,
)
File "/usr/local/lib/python3.13/site-packages/urllib3/util/connection.py", line 85, in create_connection
raise err
File "/usr/local/lib/python3.13/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
~~~~~~~~~~~~^^^^
ConnectionRefusedError: [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.13/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
conn,
...<10 lines>...
**response_kw,
)
File "/usr/local/lib/python3.13/site-packages/urllib3/connectionpool.py", line 493, in _make_request
conn.request(
~~~~~~~~~~~~^
method,
^^^^^^^
...<6 lines>...
enforce_content_length=enforce_content_length,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/usr/local/lib/python3.13/site-packages/urllib3/connection.py", line 445, in request
self.endheaders()
~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.13/http/client.py", line 1333, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/http/client.py", line 1093, in _send_output
self.send(msg)
~~~~~~~~~^^^^^
File "/usr/local/lib/python3.13/http/client.py", line 1037, in send
self.connect()
~~~~~~~~~~~~^^
File "/usr/local/lib/python3.13/site-packages/urllib3/connection.py", line 276, in connect
self.sock = self._new_conn()
~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.13/site-packages/urllib3/connection.py", line 213, in _new_conn
raise NewConnectionError(
self, f"Failed to establish a new connection: {e}"
) from e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ffff47df530>: Failed to establish a new connection: [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.13/site-packages/requests/adapters.py", line 667, in send
resp = conn.urlopen(
method=request.method,
...<9 lines>...
chunked=chunked,
)
File "/usr/local/lib/python3.13/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
)
File "/usr/local/lib/python3.13/site-packages/urllib3/util/retry.py", line 519, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='search', port=8983): Max retries exceeded with url: /solr/annotation/update/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ffff47df530>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.13/site-packages/pysolr.py", line 414, in _send_request
resp = requests_method(
url,
...<4 lines>...
auth=self.auth,
)
File "/usr/local/lib/python3.13/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.13/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.13/site-packages/requests/adapters.py", line 700, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='search', port=8983): Max retries exceeded with url: /solr/annotation/update/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ffff47df530>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/code/sir/indexing.py", line 338, in send_data_to_solr
solr_connection.add(data)
~~~~~~~~~~~~~~~~~~~^^^^^^
File "/usr/local/lib/python3.13/site-packages/pysolr.py", line 1083, in add
return self._update(
~~~~~~~~~~~~^
m,
^^
...<8 lines>...
min_rf=min_rf,
^^^^^^^^^^^^^^
)
^
File "/usr/local/lib/python3.13/site-packages/pysolr.py", line 572, in _update
return self._send_request(
~~~~~~~~~~~~~~~~~~^
"post",
^^^^^^^
...<2 lines>...
{"Content-type": "application/json; charset=utf-8"},
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/usr/local/lib/python3.13/site-packages/pysolr.py", line 429, in _send_request
raise SolrError(error_message % (url, err))
pysolr.SolrError: Failed to connect to server at http://search:8983/solr/annotation/update/: HTTPConnectionPool(host='search', port=8983): Max retries exceeded with url: /solr/annotation/update/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ffff47df530>: Failed to establish a new connection: [Errno 111] Connection refused'))
...
Solr-1 2025-11-23 09:33:17,533 DEBUG: Committing changes to Solr
Solr-1 2025-11-23 09:33:18,016 DEBUG: Setting up a connection to http://search:8983/solr
Solr-1 2025-11-23 09:33:18,016 DEBUG: Pinging http://search:8983/solr/area/admin/ping