Uploaded image for project: 'ListenBrainz'
  1. ListenBrainz
  2. LB-1695

listenbrainz-web-prod container failed

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Normal Normal
    • None
    • web pages
    • None

      The site had a downtime today with a lot of DB connection issues similar to the following:

      (Background on this error at: https://sqlalche.me/e/20/e3q8)
      2024-12-02 07:10:53,266 listenbrainz.webserver ERROR Exception on /1/submit-listens [POST]
      Traceback (most recent call last):
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
      self.dialect.do_execute(
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute
      cursor.execute(statement, parameters)
      psycopg2.OperationalError: server closed the connection unexpectedly
      This probably means the server terminated abnormally
      before or while processing the request.

      The above exception was the direct cause of the following exception:

      Traceback (most recent call last):
      File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1455, in wsgi_app
      response = self.full_dispatch_request()
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 869, in full_dispatch_request
      rv = self.handle_user_exception(e)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 867, in full_dispatch_request
      rv = self.dispatch_request()
      ^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 852, in dispatch_request
      return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/code/listenbrainz/listenbrainz/webserver/decorators.py", line 18, in decorator
      resp = make_response(f(*args, **kwargs))
      ^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/brainzutils/ratelimit.py", line 262, in rate_limited
      return f(*args, **kwargs)
      ^^^^^^^^^^^^^^^^^^
      File "/code/listenbrainz/listenbrainz/webserver/views/api.py", line 75, in submit_listen
      user = validate_auth_header(fetch_email=True)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/code/listenbrainz/listenbrainz/webserver/views/api_tools.py", line 486, in validate_auth_header
      user = db_user.get_by_token(db_conn, auth_token, fetch_email=fetch_email)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/code/listenbrainz/listenbrainz/db/user.py", line 199, in get_by_token
      result = db_conn.execute(sqlalchemy.text("""
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1416, in execute
      return meth(
      ^^^^^
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 516, in _execute_on_connection
      return connection._execute_clauseelement(
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1639, in _execute_clauseelement
      ret = self._execute_context(
      ^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1848, in _execute_context
      return self._exec_single_context(
      ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1988, in _exec_single_context
      self._handle_dbapi_exception(
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2343, in _handle_dbapi_exception
      raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
      self.dialect.do_execute(
      File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute
      cursor.execute(statement, parameters)
      sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
      This probably means the server terminated abnormally
      before or while processing the request.

      [SQL:
      SELECT id,created,musicbrainz_id,auth_token,last_login,latest_import,gdpr_agreed,musicbrainz_row_id,login_id,email
      FROM "user"
      WHERE auth_token = %(auth_token)s
      ]
      [parameters:

      {'auth_token': 'c5f0300b-b0da-45e6-82e8-8e2641ca54de'}

      ]

          Loading...
          Uploaded image for project: 'ListenBrainz'
          1. ListenBrainz
          2. LB-1695

          listenbrainz-web-prod container failed

            • Icon: Bug Bug
            • Resolution: Fixed
            • Icon: Normal Normal
            • None
            • web pages
            • None

              The site had a downtime today with a lot of DB connection issues similar to the following:

              (Background on this error at: https://sqlalche.me/e/20/e3q8)
              2024-12-02 07:10:53,266 listenbrainz.webserver ERROR Exception on /1/submit-listens [POST]
              Traceback (most recent call last):
              File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
              self.dialect.do_execute(
              File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute
              cursor.execute(statement, parameters)
              psycopg2.OperationalError: server closed the connection unexpectedly
              This probably means the server terminated abnormally
              before or while processing the request.

              The above exception was the direct cause of the following exception:

              Traceback (most recent call last):
              File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1455, in wsgi_app
              response = self.full_dispatch_request()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 869, in full_dispatch_request
              rv = self.handle_user_exception(e)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 867, in full_dispatch_request
              rv = self.dispatch_request()
              ^^^^^^^^^^^^^^^^^^^^^^^
              File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 852, in dispatch_request
              return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/code/listenbrainz/listenbrainz/webserver/decorators.py", line 18, in decorator
              resp = make_response(f(*args, **kwargs))
              ^^^^^^^^^^^^^^^^^^
              File "/usr/local/lib/python3.11/site-packages/brainzutils/ratelimit.py", line 262, in rate_limited
              return f(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^
              File "/code/listenbrainz/listenbrainz/webserver/views/api.py", line 75, in submit_listen
              user = validate_auth_header(fetch_email=True)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/code/listenbrainz/listenbrainz/webserver/views/api_tools.py", line 486, in validate_auth_header
              user = db_user.get_by_token(db_conn, auth_token, fetch_email=fetch_email)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/code/listenbrainz/listenbrainz/db/user.py", line 199, in get_by_token
              result = db_conn.execute(sqlalchemy.text("""
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1416, in execute
              return meth(
              ^^^^^
              File "/usr/local/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 516, in _execute_on_connection
              return connection._execute_clauseelement(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1639, in _execute_clauseelement
              ret = self._execute_context(
              ^^^^^^^^^^^^^^^^^^^^^^
              File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1848, in _execute_context
              return self._exec_single_context(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1988, in _exec_single_context
              self._handle_dbapi_exception(
              File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2343, in _handle_dbapi_exception
              raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
              File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1969, in _exec_single_context
              self.dialect.do_execute(
              File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 922, in do_execute
              cursor.execute(statement, parameters)
              sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
              This probably means the server terminated abnormally
              before or while processing the request.

              [SQL:
              SELECT id,created,musicbrainz_id,auth_token,last_login,latest_import,gdpr_agreed,musicbrainz_row_id,login_id,email
              FROM "user"
              WHERE auth_token = %(auth_token)s
              ]
              [parameters:

              {'auth_token': 'c5f0300b-b0da-45e6-82e8-8e2641ca54de'}

              ]

                    rob Robert Kaye
                    rob Robert Kaye
                    Votes:
                    0 Vote for this issue
                    Watchers:
                    1 Start watching this issue

                      Created:
                      Updated:

                        Version Package

                          rob Robert Kaye
                          rob Robert Kaye
                          Votes:
                          0 Vote for this issue
                          Watchers:
                          1 Start watching this issue

                            Created:
                            Updated:

                              Version Package