diff --git a/bus_alt_connection/README.rst b/bus_alt_connection/README.rst new file mode 100644 index 000000000..f219115b1 --- /dev/null +++ b/bus_alt_connection/README.rst @@ -0,0 +1,187 @@ +================== +Bus Alt Connection +================== + +.. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + !! This file is generated by oca-gen-addon-readme !! + !! changes will be overwritten. !! + !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + +.. |badge1| image:: https://img.shields.io/badge/maturity-Beta-yellow.png + :target: https://odoo-community.org/page/development-status + :alt: Beta +.. |badge2| image:: https://img.shields.io/badge/licence-AGPL--3-blue.png + :target: http://www.gnu.org/licenses/agpl-3.0-standalone.html + :alt: License: AGPL-3 +.. |badge3| image:: https://img.shields.io/badge/github-OCA%2Fserver--tools-lightgray.png?logo=github + :target: https://github.com/OCA/server-tools/tree/12.0/bus_alt_connection + :alt: OCA/server-tools +.. |badge4| image:: https://img.shields.io/badge/weblate-Translate%20me-F47D42.png + :target: https://translation.odoo-community.org/projects/server-tools-12-0/server-tools-12-0-bus_alt_connection + :alt: Translate me on Weblate +.. |badge5| image:: https://img.shields.io/badge/runbot-Try%20me-875A7B.png + :target: https://runbot.odoo-community.org/runbot/149/12.0 + :alt: Try me on Runbot + +|badge1| |badge2| |badge3| |badge4| |badge5| + +This module makes it possible to use PgBouncer_ as a connection pooler +for odoo. + +.. _PgBouncer: https://pgbouncer.github.io/ + +Why isn't odoo's connection pooling good enough? +================================================ + +Odoo's builtin connection pooling works at process level: each Odoo process +has its own ConnectionPool_, limited to ``db_maxconn``. + +It does the job of re-using open connections available in the pool. +But it never closes these connections, `unless reaching db_maxconn`_. + +.. _ConnectionPool: https://github.com/odoo/odoo/blob/12.0/odoo/sql_db.py#L525 +.. _`unless reaching db_maxconn`: https://github.com/odoo/odoo/blob/12.0/odoo/sql_db.py#L593 + +In practice, we observe that each odoo worker will end up +with up to 3 open connection in its pool. +With 10 http workers, that's up to 30 connection continuously open just +for one single instance. + +Here comes PgBouncer +==================== + +PgBouncer will help to limit this number of open connections, +by sharing a pool of connections at the instance level, between +all workers. Odoo workers will still have up to 3 open connections, +but these will be connections to PgBouncer, that on its side will +close unnecessary connections to pg. + +This has proven to help performances on Odoo deployments with +multiple instances. + +It allows you to define how resources should be shared, +according to your priorities, e.g. : + +* key odoo instance on host A can open up to 30 connections +* while odoo instance on host B, dedicated to reports, + can open up to 10 connections only + +And most importantly, it helps you to ensure that +``max_connections`` will never be reached on pg server side. + + +Why is this module needed? +========================== + +When configuring PgBouncer, you can choose between 2 transaction pooling modes: + +* `pool_mode = session` +* `pool_mode = transaction` + +If we choose `pool_mode = session`, then one server connection will be tied +to a given odoo process until its death, which is exactly what we're trying +to change. Thus, to release the server connection once the transaction is +complete, we use `pool_mode = transaction`. + +This works fine, except for Odoo's longpolling features that relies +on the `LISTEN/NOTIFY`_ mechanism from pg, which is `not compatible`_ with that +mode. + +.. _`LISTEN/NOTIFY`: https://www.postgresql.org/docs/9.6/static/sql-notify.html +.. _`not compatible`: https://wiki.postgresql.org/wiki/PgBouncer + + +To be more precise, `NOTIFY` statements are properly transfered by PgBouncer +in that mode; only the `LISTEN` statement isn't (because it needs to keep the +server connection open). + +So for the unique "listening" connection per instance that requires this +statement (here_), we need odoo to connect directly to the pg server, bypassing +PgBouncer. + +That's what this module implements, by overriding the relevant method +of the Dispatcher_. + +.. _here: https://github.com/odoo/odoo/blob/12.0/addons/bus/models/bus.py#L166 +.. _Dispatcher: https://github.com/odoo/odoo/blob/12.0/addons/bus/models/bus.py#L105 + +**Table of contents** + +.. contents:: + :local: + +Installation +============ + +You don't need to install this module in the database(s) to enable it. + +But you need to load it server-wide: + +* By starting Odoo with ``--load=web,bus_alt_connection`` + +* Or by updating its configuration file: + +.. code-block:: ini + + [options] + (...) + server_wide_modules = web,bus_alt_connection + +Configuration +============= + +You need to define how to connect directly to the database: + +* Either by defining environment variables: + + - ``IMDISPATCHER_DB_HOST=db-01`` + - ``IMDISPATCHER_DB_PORT=5432`` + +* Or in Odoo's configuration file: + +.. code-block:: ini + + [options] + (...) + imdispatcher_db_host = db-01 + imdispatcher_db_port = 5432 + +Bug Tracker +=========== + +Bugs are tracked on `GitHub Issues `_. +In case of trouble, please check there if your issue has already been reported. +If you spotted it first, help us smashing it by providing a detailed and welcomed +`feedback `_. + +Do not contact contributors directly about support or help with technical issues. + +Credits +======= + +Authors +~~~~~~~ + +* Trobz + +Contributors +~~~~~~~~~~~~ + +* Nils Hamerlinck + +Maintainers +~~~~~~~~~~~ + +This module is maintained by the OCA. + +.. image:: https://odoo-community.org/logo.png + :alt: Odoo Community Association + :target: https://odoo-community.org + +OCA, or the Odoo Community Association, is a nonprofit organization whose +mission is to support the collaborative development of Odoo features and +promote its widespread use. + +This module is part of the `OCA/server-tools `_ project on GitHub. + +You are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute. diff --git a/bus_alt_connection/__init__.py b/bus_alt_connection/__init__.py new file mode 100644 index 000000000..1bcfe0e2f --- /dev/null +++ b/bus_alt_connection/__init__.py @@ -0,0 +1,3 @@ +# Copyright 2019 Trobz +# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl). +from . import models diff --git a/bus_alt_connection/__manifest__.py b/bus_alt_connection/__manifest__.py new file mode 100644 index 000000000..9133093b9 --- /dev/null +++ b/bus_alt_connection/__manifest__.py @@ -0,0 +1,15 @@ +# Copyright 2019 Trobz +# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl). + +{ + 'name': 'Bus Alt Connection', + 'summary': 'Needed when using PgBouncer as a connection pooler', + 'version': '12.0.1.0.0', + 'author': 'Trobz,Odoo Community Association (OCA)', + 'website': 'https://github.com/OCA/server-tools', + 'category': 'Extra Tools', + 'license': 'AGPL-3', + 'depends': ['bus'], + 'installable': True, + 'auto_install': False, +} diff --git a/bus_alt_connection/models/__init__.py b/bus_alt_connection/models/__init__.py new file mode 100644 index 000000000..3dc9bd993 --- /dev/null +++ b/bus_alt_connection/models/__init__.py @@ -0,0 +1,3 @@ +# Copyright 2019 Trobz +# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl). +from . import bus diff --git a/bus_alt_connection/models/bus.py b/bus_alt_connection/models/bus.py new file mode 100644 index 000000000..d9c545e16 --- /dev/null +++ b/bus_alt_connection/models/bus.py @@ -0,0 +1,77 @@ +# Copyright 2019 Trobz +# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl). + +import os +import json +import logging +import select + +import psycopg2 +from psycopg2.extensions import ISOLATION_LEVEL_AUTOCOMMIT + +import odoo +from odoo.tools import config + +from odoo.addons.bus.models.bus import hashable, TIMEOUT +import odoo.addons.bus.models.bus + + +_logger = logging.getLogger(__name__) + + +def _connection_info_for(db_name): + db_or_uri, connection_info = odoo.sql_db.connection_info_for(db_name) + + for p in ('host', 'port'): + cfg = (os.environ.get('ODOO_IMDISPATCHER_DB_%s' % p.upper()) or + config.get('imdispatcher_db_' + p)) + + if cfg: + connection_info[p] = cfg + + return connection_info + + +class ImDispatch(odoo.addons.bus.models.bus.ImDispatch): + + def loop(self): + """ Dispatch postgres notifications to the relevant + polling threads/greenlets """ + connection_info = _connection_info_for('postgres') + _logger.info("Bus.loop listen imbus on db postgres " + "(via %(host)s:%(port)s)", + connection_info) + conn = psycopg2.connect(**connection_info) + conn.set_isolation_level(ISOLATION_LEVEL_AUTOCOMMIT) + with conn.cursor() as cr: + cr.execute("listen imbus") + conn.commit() + while True: + if select.select([conn], [], [], TIMEOUT) == ([], [], []): + pass + else: + conn.poll() + channels = [] + while conn.notifies: + channels.extend(json.loads(conn + .notifies.pop().payload)) + # dispatch to local threads/greenlets + events = set() + for channel in channels: + events.update(self.channels.pop(hashable(channel), + set())) + for event in events: + event.set() + + +odoo.addons.bus.models.bus.ImDispatch = ImDispatch + +# we can replace the existing dispatcher because its thread +# has not been started yet; indeed, since a2ed3d it only starts +# on first /poll request: +# https://github.com/odoo/odoo/commit/a2ed3d3d5bdb6025a1ba14ad557a115a86413e65 + +if not odoo.multi_process or odoo.evented: + dispatch = ImDispatch() + odoo.addons.bus.models.bus.dispatch = dispatch + odoo.addons.bus.controllers.main.dispatch = dispatch diff --git a/bus_alt_connection/readme/CONFIGURE.rst b/bus_alt_connection/readme/CONFIGURE.rst new file mode 100644 index 000000000..cd5021912 --- /dev/null +++ b/bus_alt_connection/readme/CONFIGURE.rst @@ -0,0 +1,15 @@ +You need to define how to connect directly to the database: + +* Either by defining environment variables: + + - ``IMDISPATCHER_DB_HOST=db-01`` + - ``IMDISPATCHER_DB_PORT=5432`` + +* Or in Odoo's configuration file: + +.. code-block:: ini + + [options] + (...) + imdispatcher_db_host = db-01 + imdispatcher_db_port = 5432 diff --git a/bus_alt_connection/readme/CONTRIBUTORS.rst b/bus_alt_connection/readme/CONTRIBUTORS.rst new file mode 100644 index 000000000..198a2f3bd --- /dev/null +++ b/bus_alt_connection/readme/CONTRIBUTORS.rst @@ -0,0 +1 @@ +* Nils Hamerlinck diff --git a/bus_alt_connection/readme/DESCRIPTION.rst b/bus_alt_connection/readme/DESCRIPTION.rst new file mode 100644 index 000000000..ea95393d2 --- /dev/null +++ b/bus_alt_connection/readme/DESCRIPTION.rst @@ -0,0 +1,79 @@ +This module makes it possible to use PgBouncer_ as a connection pooler +for odoo. + +.. _PgBouncer: https://pgbouncer.github.io/ + +Why isn't odoo's connection pooling good enough? +================================================ + +Odoo's builtin connection pooling works at process level: each Odoo process +has its own ConnectionPool_, limited to ``db_maxconn``. + +It does the job of re-using open connections available in the pool. +But it never closes these connections, `unless reaching db_maxconn`_. + +.. _ConnectionPool: https://github.com/odoo/odoo/blob/12.0/odoo/sql_db.py#L525 +.. _`unless reaching db_maxconn`: https://github.com/odoo/odoo/blob/12.0/odoo/sql_db.py#L593 + +In practice, we observe that each odoo worker will end up +with up to 3 open connection in its pool. +With 10 http workers, that's up to 30 connection continuously open just +for one single instance. + +Here comes PgBouncer +==================== + +PgBouncer will help to limit this number of open connections, +by sharing a pool of connections at the instance level, between +all workers. Odoo workers will still have up to 3 open connections, +but these will be connections to PgBouncer, that on its side will +close unnecessary connections to pg. + +This has proven to help performances on Odoo deployments with +multiple instances. + +It allows you to define how resources should be shared, +according to your priorities, e.g. : + +* key odoo instance on host A can open up to 30 connections +* while odoo instance on host B, dedicated to reports, + can open up to 10 connections only + +And most importantly, it helps you to ensure that +``max_connections`` will never be reached on pg server side. + + +Why is this module needed? +========================== + +When configuring PgBouncer, you can choose between 2 transaction pooling modes: + +* `pool_mode = session` +* `pool_mode = transaction` + +If we choose `pool_mode = session`, then one server connection will be tied +to a given odoo process until its death, which is exactly what we're trying +to change. Thus, to release the server connection once the transaction is +complete, we use `pool_mode = transaction`. + +This works fine, except for Odoo's longpolling features that relies +on the `LISTEN/NOTIFY`_ mechanism from pg, which is `not compatible`_ with that +mode. + +.. _`LISTEN/NOTIFY`: https://www.postgresql.org/docs/9.6/static/sql-notify.html +.. _`not compatible`: https://wiki.postgresql.org/wiki/PgBouncer + + +To be more precise, `NOTIFY` statements are properly transfered by PgBouncer +in that mode; only the `LISTEN` statement isn't (because it needs to keep the +server connection open). + +So for the unique "listening" connection per instance that requires this +statement (here_), we need odoo to connect directly to the pg server, bypassing +PgBouncer. + +That's what this module implements, by overriding the relevant method +of the Dispatcher_. + +.. _here: https://github.com/odoo/odoo/blob/12.0/addons/bus/models/bus.py#L166 +.. _Dispatcher: https://github.com/odoo/odoo/blob/12.0/addons/bus/models/bus.py#L105 diff --git a/bus_alt_connection/readme/INSTALL.rst b/bus_alt_connection/readme/INSTALL.rst new file mode 100644 index 000000000..c6a9fcbcf --- /dev/null +++ b/bus_alt_connection/readme/INSTALL.rst @@ -0,0 +1,13 @@ +You don't need to install this module in the database(s) to enable it. + +But you need to load it server-wide: + +* By starting Odoo with ``--load=web,bus_alt_connection`` + +* Or by updating its configuration file: + +.. code-block:: ini + + [options] + (...) + server_wide_modules = web,bus_alt_connection diff --git a/bus_alt_connection/static/description/icon.png b/bus_alt_connection/static/description/icon.png new file mode 100644 index 000000000..3a0328b51 Binary files /dev/null and b/bus_alt_connection/static/description/icon.png differ diff --git a/bus_alt_connection/static/description/index.html b/bus_alt_connection/static/description/index.html new file mode 100644 index 000000000..14b5312dc --- /dev/null +++ b/bus_alt_connection/static/description/index.html @@ -0,0 +1,498 @@ + + + + + + +Bus Alt Connection + + + +
+

Bus Alt Connection

+ + +

Beta License: AGPL-3 OCA/server-tools Translate me on Weblate Try me on Runbot

+

This module makes it possible to use PgBouncer as a connection pooler +for odoo.

+
+

Why isn’t odoo’s connection pooling good enough?

+

Odoo’s builtin connection pooling works at process level: each Odoo process +has its own ConnectionPool, limited to db_maxconn.

+

It does the job of re-using open connections available in the pool. +But it never closes these connections, unless reaching db_maxconn.

+

In practice, we observe that each odoo worker will end up +with up to 3 open connection in its pool. +With 10 http workers, that’s up to 30 connection continuously open just +for one single instance.

+
+
+

Here comes PgBouncer

+

PgBouncer will help to limit this number of open connections, +by sharing a pool of connections at the instance level, between +all workers. Odoo workers will still have up to 3 open connections, +but these will be connections to PgBouncer, that on its side will +close unnecessary connections to pg.

+

This has proven to help performances on Odoo deployments with +multiple instances.

+

It allows you to define how resources should be shared, +according to your priorities, e.g. :

+
    +
  • key odoo instance on host A can open up to 30 connections
    • +
  • while odoo instance on host B, dedicated to reports, +can open up to 10 connections only
    • +
+

And most importantly, it helps you to ensure that +max_connections will never be reached on pg server side.

+
+
+

Why is this module needed?

+

When configuring PgBouncer, you can choose between 2 transaction pooling modes:

+
    +
  • pool_mode = session
    • +
  • pool_mode = transaction
    • +
+

If we choose pool_mode = session, then one server connection will be tied +to a given odoo process until its death, which is exactly what we’re trying +to change. Thus, to release the server connection once the transaction is +complete, we use pool_mode = transaction.

+

This works fine, except for Odoo’s longpolling features that relies +on the LISTEN/NOTIFY mechanism from pg, which is not compatible with that +mode.

+

To be more precise, NOTIFY statements are properly transfered by PgBouncer +in that mode; only the LISTEN statement isn’t (because it needs to keep the +server connection open).

+

So for the unique “listening” connection per instance that requires this +statement (here), we need odoo to connect directly to the pg server, bypassing +PgBouncer.

+

That’s what this module implements, by overriding the relevant method +of the Dispatcher.

+

Table of contents

+
+
+

Installation

+

You don’t need to install this module in the database(s) to enable it.

+

But you need to load it server-wide:

+
    +
  • By starting Odoo with --load=web,bus_alt_connection
    • +
  • Or by updating its configuration file:
    • +
+
+[options]
+(...)
+server_wide_modules = web,bus_alt_connection
+
+
+
+

Configuration

+

You need to define how to connect directly to the database:

+
    +

  • Either by defining environment variables:

    +
    +
      +
    • IMDISPATCHER_DB_HOST=db-01
      • +
    • IMDISPATCHER_DB_PORT=5432
      • +
    +
    +
    • +

  • Or in Odoo’s configuration file:

    +
    • +
+
+[options]
+(...)
+imdispatcher_db_host = db-01
+imdispatcher_db_port = 5432
+
+
+
+

Bug Tracker

+

Bugs are tracked on GitHub Issues. +In case of trouble, please check there if your issue has already been reported. +If you spotted it first, help us smashing it by providing a detailed and welcomed +feedback.

+

Do not contact contributors directly about support or help with technical issues.

+
+
+

Credits

+
+

Authors

+
    +
  • Trobz
    • +
+
+
+

Contributors

+ +
+
+

Maintainers

+

This module is maintained by the OCA.

+Odoo Community Association +

OCA, or the Odoo Community Association, is a nonprofit organization whose +mission is to support the collaborative development of Odoo features and +promote its widespread use.

+

This module is part of the OCA/server-tools project on GitHub.

+

You are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute.

+
+
+
+ +