[ADD] sentry module (#761)

* [ADD] sentry module

* [FIX] updated sentry module according to PR comments

sentry/README.rst

@@ -0,0 +1,168 @@
+.. image:: https://img.shields.io/badge/licence-AGPL--3-blue.svg
+   :target: http://www.gnu.org/licenses/agpl-3.0-standalone.html
+   :alt: License: AGPL-3
+
+======
+Sentry
+======
+
+This module allows painless `Sentry <https://sentry.io/>`__ integration with
+Odoo.
+
+Installation
+============
+
+The module can be installed just like any other Odoo module, by adding the
+module's directory to Odoo *addons_path*. In order for the module to correctly
+wrap the Odoo WSGI application, it also needs to be loaded as a server-wide
+module. This can be done with the ``server_wide_modules`` parameter in your
+Odoo config file or with the ``--load`` command-line parameter.
+
+This module additionally requires the raven_ Python package to be available on
+the system. It can be installed using pip::
+
+    pip install raven
+
+Configuration
+=============
+
+The following additional configuration options can be added to your Odoo
+configuration file:
+
+=============================  ====================================================================  ==========================================================
+        Option                                          Description                                                         Default
+=============================  ====================================================================  ==========================================================
+``sentry_dsn``                 Sentry *Data Source Name*. You can find this value in your Sentry     ``''``
+                               project configuration. Typically it looks something like this:
+                               *https://<public_key>:<secret_key>@sentry.example.com/<project id>*
+                               This is the only required option in order to use the module.
+
+``sentry_enabled``             Whether or not Sentry logging is enabled.                             ``True``
+
+``sentry_logging_level``       The minimal logging level for which to send reports to Sentry.        ``warn``
+                               Possible values: *notset*, *debug*, *info*, *warn*, *error*,
+                               *critical*. It is recommended to have this set to at least *warn*,
+                               to avoid spamming yourself with Sentry events.
+
+``sentry_exclude_loggers``     A string of comma-separated logger names which should be excluded     ``werkzeug``
+                               from Sentry.
+
+``sentry_ignored_exceptions``  A string of comma-separated exceptions which should be ignored.       ``odoo.exceptions.AccessDenied,
+                               You can use a star symbol (*) at the end, to ignore all exceptions    odoo.exceptions.AccessError,
+                               from a module, eg.: *odoo.exceptions.**.                              odoo.exceptions.DeferredException,
+                                                                                                     odoo.exceptions.MissingError,
+                                                                                                     odoo.exceptions.RedirectWarning,
+                                                                                                     odoo.exceptions.UserError,
+                                                                                                     odoo.exceptions.ValidationError,
+                                                                                                     odoo.exceptions.Warning,
+                                                                                                     odoo.exceptions.except_orm``
+
+``sentry_processors``          A string of comma-separated processor classes which will be applied   ``raven.processors.SanitizePasswordsProcessor,
+                               on an event before sending it to Sentry.                              odoo.addons.sentry.logutils.SanitizeOdooCookiesProcessor``
+
+``sentry_transport``           Transport class which will be used to send events to Sentry.          ``threaded``
+                               Possible values: *threaded*: spawns an async worker for processing
+                               messages, *synchronous*: a synchronous blocking transport;
+                               *requests_threaded*: an asynchronous transport using the *requests*
+                               library; *requests_synchronous* - blocking transport using the
+                               *requests* library.
+
+``sentry_include_context``     If enabled, additional context data will be extracted from current    ``True``
+                               HTTP request and user session (if available). This has no effect
+                               for Cron jobs, as no request/session is available inside a Cron job.
+
+``sentry_odoo_dir``            Absolute path to your Odoo installation directory. This is optional
+                               and will only be used to extract the Odoo Git commit, which will be
+                               sent to Sentry, to allow to distinguish between Odoo updates.
+=============================  ====================================================================  ==========================================================
+
+Other `client arguments
+<https://docs.sentry.io/clients/python/advanced/#client-arguments>`_ can be
+configured by prepending the argument name with *sentry_* in your Odoo config
+file. Currently supported additional client arguments are: ``install_sys_hook,
+include_paths, exclude_paths, machine, auto_log_stacks, capture_locals,
+string_max_length, list_max_length, site, include_versions, environment``.
+
+Example Odoo configuration
+--------------------------
+
+Below is an example of Odoo configuration file with *Odoo Sentry* options::
+
+    [options]
+    sentry_dsn = https://<public_key>:<secret_key>@sentry.example.com/<project id>
+    sentry_enabled = true
+    sentry_logging_level = warn
+    sentry_exclude_loggers = werkzeug
+    sentry_ignore_exceptions = odoo.exceptions.AccessDenied,odoo.exceptions.AccessError,odoo.exceptions.MissingError,odoo.exceptions.RedirectWarning,odoo.exceptions.UserError,odoo.exceptions.ValidationError,odoo.exceptions.Warning,odoo.exceptions.except_orm
+    sentry_processors = raven.processors.SanitizePasswordsProcessor,odoo.addons.sentry.logutils.SanitizeOdooCookiesProcessor
+    sentry_transport = threaded
+    sentry_include_context = true
+    sentry_environment = production
+    sentry_auto_log_stacks = false
+    sentry_odoo_dir = /home/odoo/odoo/
+
+Usage
+=====
+
+Once configured and installed, the module will report any logging event at and
+above the configured Sentry logging level, no additional actions are necessary.
+
+.. image:: https://odoo-community.org/website/image/ir.attachment/5784_f2813bd/datas
+   :alt: Try me on Runbot
+   :target: https://runbot.odoo-community.org/runbot/149/10.0
+
+Known issues / Roadmap
+======================
+
+* **No database separation** -- This module functions by intercepting all Odoo
+  logging records in a running Odoo process. This means that once installed in
+  one database, it will intercept and report errors for all Odoo databases,
+  which are used on that Odoo server.
+
+* **Frontend integration** -- In the future, it would be nice to add
+  Odoo client-side error reporting to this module as well, by integrating
+  `raven-js <https://github.com/getsentry/raven-js>`_. Additionally, `Sentry user
+  feedback form <https://docs.sentry.io/learn/user-feedback/>`_ could be
+  integrated into the Odoo client error dialog window to allow users shortly
+  describe what they were doing when things went wrong.
+
+Bug Tracker
+===========
+
+Bugs are tracked on `GitHub Issues
+<https://github.com/OCA/server-tools/issues>`_. In case of trouble, please
+check there if your issue has already been reported. If you spotted it first,
+help us smash it by providing detailed and welcomed feedback.
+
+Credits
+=======
+
+Images
+------
+
+* `Module Icon <https://sentry.io/branding/>`_
+
+Contributors
+------------
+
+* Mohammed Barsi <barsintod@gmail.com>
+* Andrius Preimantas <andrius@versada.eu>
+* Naglis Jonaitis <naglis@versada.eu>
+
+Maintainer
+----------
+
+.. image:: https://odoo-community.org/logo.png
+   :alt: Odoo Community Association
+   :target: https://odoo-community.org
+
+This module is maintained by the OCA.
+
+OCA, or the Odoo Community Association, is a nonprofit organization whose
+mission is to support the collaborative development of Odoo features and
+promote its widespread use.
+
+To contribute to this module, please visit https://odoo-community.org.
+
+
+.. _raven: https://github.com/getsentry/raven-python

sentry/__init__.py

@@ -0,0 +1,75 @@
+# -*- coding: utf-8 -*-
+# Copyright 2016-2017 Versada <https://versada.eu/>
+# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl).
+
+import logging
+
+from odoo.service import wsgi_server
+from odoo.tools import config as odoo_config
+
+from . import const
+from .logutils import LoggerNameFilter, OdooSentryHandler
+
+_logger = logging.getLogger(__name__)
+try:
+    import raven
+    from raven.middleware import Sentry
+except ImportError:
+    _logger.debug('Cannot import "raven". Please make sure it is installed.')
+
+
+def get_odoo_commit(odoo_dir):
+    '''Attempts to get Odoo git commit from :param:`odoo_dir`.'''
+    if not odoo_dir:
+        return
+    try:
+        return raven.fetch_git_sha(odoo_dir)
+    except raven.exceptions.InvalidGitRepository:
+        _logger.debug(
+            u'Odoo directory: "%s" not a valid git repository', odoo_dir)
+
+
+def initialize_raven(config, client_cls=raven.Client):
+    '''
+    Setup an instance of :class:`raven.Client`.
+
+    :param config: Sentry configuration
+    :param client: class used to instantiate the raven client.
+    '''
+    options = {
+        'release': get_odoo_commit(config.get('sentry_odoo_dir')),
+    }
+    for option in const.SENTRY_OPTIONS:
+        value = config.get('sentry_%s' % option.key, option.default)
+        if callable(option.converter):
+            value = option.converter(value)
+        options[option.key] = value
+
+    client = client_cls(**options)
+
+    enabled = config.get('sentry_enabled', True)
+    level = config.get('sentry_logging_level', const.DEFAULT_LOG_LEVEL)
+    exclude_loggers = const.split_multiple(
+        config.get('sentry_exclude_loggers', const.DEFAULT_EXCLUDE_LOGGERS)
+    )
+    if level not in const.LOG_LEVEL_MAP:
+        level = const.DEFAULT_LOG_LEVEL
+
+    if enabled:
+        handler = OdooSentryHandler(
+            config.get('sentry_include_context', True),
+            client=client,
+            level=const.LOG_LEVEL_MAP[level],
+        )
+        if exclude_loggers:
+            handler.addFilter(LoggerNameFilter(
+                exclude_loggers, name='sentry.logger.filter'))
+        raven.conf.setup_logging(handler)
+        wsgi_server.application = Sentry(
+            wsgi_server.application, client=client)
+
+    return client
+
+
+sentry_client = initialize_raven(odoo_config)
+sentry_client.captureMessage('Starting Odoo Server')

sentry/__manifest__.py

@@ -0,0 +1,24 @@
+# -*- coding: utf-8 -*-
+# Copyright 2016-2017 Versada <https://versada.eu/>
+# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl).
+{
+    'name': 'Sentry',
+    'summary': 'Report Odoo errors to Sentry',
+    'version': '10.0.1.0.0',
+    'category': 'Extra Tools',
+    'website': 'https://odoo-community.org/',
+    'author': 'Mohammed Barsi,'
+              'Versada,'
+              'Odoo Community Association (OCA)',
+    'license': 'AGPL-3',
+    'application': False,
+    'installable': True,
+    'external_dependencies': {
+        'python': [
+            'raven',
+        ]
+    },
+    'depends': [
+        'base',
+    ],
+}

sentry/const.py

@@ -0,0 +1,84 @@
+# -*- coding: utf-8 -*-
+# Copyright 2016-2017 Versada <https://versada.eu/>
+# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl).
+
+import collections
+import logging
+
+import odoo.loglevels
+
+_logger = logging.getLogger(__name__)
+try:
+    import raven
+    from raven.conf import defaults
+except ImportError:
+    _logger.debug('Cannot import "raven". Please make sure it is installed.')
+
+
+def split_multiple(string, delimiter=',', strip_chars=None):
+    '''Splits :param:`string` and strips :param:`strip_chars` from values.'''
+    if not string:
+        return []
+    return [v.strip(strip_chars) for v in string.split(delimiter)]
+
+
+SentryOption = collections.namedtuple(
+    'SentryOption', ['key', 'default', 'converter'])
+
+# Mapping of Odoo logging level -> Python stdlib logging library log level.
+LOG_LEVEL_MAP = dict([
+    (getattr(odoo.loglevels, 'LOG_%s' % x), getattr(logging, x))
+    for x in ('CRITICAL', 'ERROR', 'WARNING', 'INFO', 'DEBUG', 'NOTSET')
+])
+DEFAULT_LOG_LEVEL = 'warn'
+
+DEFAULT_TRANSPORT = 'threaded'
+TRANSPORT_CLASS_MAP = {
+    'requests_synchronous': raven.transport.RequestsHTTPTransport,
+    'requests_threaded': raven.transport.ThreadedRequestsHTTPTransport,
+    'synchronous': raven.transport.HTTPTransport,
+    'threaded': raven.transport.ThreadedHTTPTransport,
+}
+
+ODOO_USER_EXCEPTIONS = [
+    'odoo.exceptions.AccessDenied',
+    'odoo.exceptions.AccessError',
+    'odoo.exceptions.DeferredException',
+    'odoo.exceptions.MissingError',
+    'odoo.exceptions.RedirectWarning',
+    'odoo.exceptions.UserError',
+    'odoo.exceptions.ValidationError',
+    'odoo.exceptions.Warning',
+    'odoo.exceptions.except_orm',
+]
+DEFAULT_IGNORED_EXCEPTIONS = ','.join(ODOO_USER_EXCEPTIONS)
+
+PROCESSORS = (
+    'raven.processors.SanitizePasswordsProcessor',
+    'odoo.addons.sentry.logutils.SanitizeOdooCookiesProcessor',
+)
+DEFAULT_PROCESSORS = ','.join(PROCESSORS)
+
+EXCLUDE_LOGGERS = (
+    'werkzeug',
+)
+DEFAULT_EXCLUDE_LOGGERS = ','.join(EXCLUDE_LOGGERS)
+
+SENTRY_OPTIONS = [
+    SentryOption('dsn', '', str.strip),
+    SentryOption('install_sys_hook', False, None),
+    SentryOption('transport', DEFAULT_TRANSPORT, TRANSPORT_CLASS_MAP.get),
+    SentryOption('include_paths', '', split_multiple),
+    SentryOption('exclude_paths', '', split_multiple),
+    SentryOption('machine', defaults.NAME, None),
+    SentryOption('auto_log_stacks', defaults.AUTO_LOG_STACKS, None),
+    SentryOption('capture_locals', defaults.CAPTURE_LOCALS, None),
+    SentryOption('string_max_length', defaults.MAX_LENGTH_STRING, None),
+    SentryOption('list_max_length', defaults.MAX_LENGTH_LIST, None),
+    SentryOption('site', None, None),
+    SentryOption('include_versions', True, None),
+    SentryOption(
+        'ignore_exceptions', DEFAULT_IGNORED_EXCEPTIONS, split_multiple),
+    SentryOption('processors', DEFAULT_PROCESSORS, split_multiple),
+    SentryOption('environment', None, None),
+]

sentry/logutils.py

@@ -0,0 +1,104 @@
+# -*- coding: utf-8 -*-
+# Copyright 2016-2017 Versada <https://versada.eu/>
+# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl).
+
+import logging
+import urlparse
+
+import odoo.http
+
+_logger = logging.getLogger(__name__)
+try:
+    from raven.handlers.logging import SentryHandler
+    from raven.processors import SanitizePasswordsProcessor
+    from raven.utils.wsgi import get_environ, get_headers
+except ImportError:
+    _logger.debug('Cannot import "raven". Please make sure it is installed.')
+
+
+def get_request_info(request):
+    '''
+    Returns context data extracted from :param:`request`.
+
+    Heavily based on flask integration for Sentry: https://git.io/vP4i9.
+    '''
+    urlparts = urlparse.urlsplit(request.url)
+    return {
+        'url': '%s://%s%s' % (urlparts.scheme, urlparts.netloc, urlparts.path),
+        'query_string': urlparts.query,
+        'method': request.method,
+        'headers': dict(get_headers(request.environ)),
+        'env': dict(get_environ(request.environ)),
+    }
+
+
+def get_extra_context():
+    '''
+    Extracts additional context from the current request (if such is set).
+    '''
+    request = odoo.http.request
+    try:
+        session = getattr(request, 'session', {})
+    except RuntimeError:
+        ctx = {}
+    else:
+        ctx = {
+            'tags': {
+                'database': session.get('db', None),
+            },
+            'user': {
+                'login': session.get('login', None),
+                'uid': session.get('uid', None),
+            },
+            'extra': {
+                'context': session.get('context', {}),
+            },
+        }
+        if request.httprequest:
+            ctx.update({
+                'request': get_request_info(request.httprequest),
+            })
+    return ctx
+
+
+class LoggerNameFilter(logging.Filter):
+    '''
+    Custom :class:`logging.Filter` which allows to filter loggers by name.
+    '''
+
+    def __init__(self, loggers, name=''):
+        super(LoggerNameFilter, self).__init__(name=name)
+        self._exclude_loggers = set(loggers)
+
+    def filter(self, event):
+        return event.name not in self._exclude_loggers
+
+
+class OdooSentryHandler(SentryHandler):
+    '''
+    Customized :class:`raven.handlers.logging.SentryHandler`.
+
+    Allows to add additional Odoo and HTTP request data to the event which is
+    sent to Sentry.
+    '''
+
+    def __init__(self, include_extra_context, *args, **kwargs):
+        super(OdooSentryHandler, self).__init__(*args, **kwargs)
+        self.include_extra_context = include_extra_context
+
+    def emit(self, record):
+        if self.include_extra_context:
+            self.client.context.merge(get_extra_context())
+        return super(OdooSentryHandler, self).emit(record)
+
+
+class SanitizeOdooCookiesProcessor(SanitizePasswordsProcessor):
+    '''
+    Custom :class:`raven.processors.Processor`.
+
+    Allows to sanitize sensitive Odoo cookies, namely the "session_id" cookie.
+    '''
+
+    FIELDS = frozenset([
+        'session_id',
+    ])

sentry/tests/__init__.py

@@ -0,0 +1,8 @@
+# -*- coding: utf-8 -*-
+# Copyright 2016-2017 Versada <https://versada.eu/>
+# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl).
+
+from . import (
+    test_client,
+    test_logutils,
+)

sentry/tests/test_client.py

@@ -0,0 +1,125 @@
+# -*- coding: utf-8 -*-
+# Copyright 2016-2017 Versada <https://versada.eu/>
+# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl).
+
+import logging
+import sys
+import unittest
+
+import raven
+
+from odoo import exceptions
+
+from .. import initialize_raven
+from ..logutils import OdooSentryHandler
+
+
+def log_handler_by_class(logger, handler_cls):
+    for handler in logger.handlers:
+        if isinstance(handler, handler_cls):
+            yield handler
+
+
+def remove_logging_handler(logger_name, handler_cls):
+    '''Removes handlers of specified classes from a :class:`logging.Logger`
+    with a given name.
+
+    :param string logger_name: name of the logger
+
+    :param handler_cls: class of the handler to remove. You can pass a tuple of
+        classes to catch several classes
+    '''
+    logger = logging.getLogger(logger_name)
+    for handler in log_handler_by_class(logger, handler_cls):
+        logger.removeHandler(handler)
+
+
+class InMemoryClient(raven.Client):
+    '''A :class:`raven.Client` subclass which simply stores events in a list.
+
+    Extended based on the one found in raven-python to avoid additional testing
+    dependencies: https://git.io/vyGO3
+    '''
+
+    def __init__(self, **kwargs):
+        self.events = []
+        super(InMemoryClient, self).__init__(**kwargs)
+
+    def is_enabled(self):
+        return True
+
+    def send(self, **kwargs):
+        self.events.append(kwargs)
+
+    def has_event(self, event_level, event_msg):
+        for event in self.events:
+            if (event.get('level') == event_level and
+                    event.get('message') == event_msg):
+                return True
+        return False
+
+
+class TestClientSetup(unittest.TestCase):
+
+    def setUp(self):
+        super(TestClientSetup, self).setUp()
+        self.logger = logging.getLogger(__name__)
+
+        # Sentry is enabled by default, so the default handler will be added
+        # when the module is loaded. After that, subsequent calls to
+        # setup_logging will not re-add our handler. We explicitly remove
+        # OdooSentryHandler handler so we can test with our in-memory client.
+        remove_logging_handler('', OdooSentryHandler)
+
+    def assertEventCaptured(self, client, event_level, event_msg):
+        self.assertTrue(
+            client.has_event(event_level, event_msg),
+            msg=u'Event: "%s" was not captured' % event_msg
+        )
+
+    def assertEventNotCaptured(self, client, event_level, event_msg):
+        self.assertFalse(
+            client.has_event(event_level, event_msg),
+            msg=u'Event: "%s" was captured' % event_msg
+        )
+
+    def test_initialize_raven_sets_dsn(self):
+        config = {
+            'sentry_enabled': False,
+            'sentry_dsn': 'http://public:secret@example.com/1',
+        }
+        client = initialize_raven(config, client_cls=InMemoryClient)
+        self.assertEqual(client.remote.base_url, 'http://example.com')
+
+    def test_capture_event(self):
+        config = {
+            'sentry_enabled': True,
+            'sentry_dsn': 'http://public:secret@example.com/1',
+        }
+        level, msg = logging.WARNING, 'Test event, can be ignored'
+        client = initialize_raven(config, client_cls=InMemoryClient)
+        self.logger.log(level, msg)
+        self.assertEventCaptured(client, level, msg)
+
+    def test_ignore_exceptions(self):
+        config = {
+            'sentry_enabled': True,
+            'sentry_dsn': 'http://public:secret@example.com/1',
+            'sentry_ignore_exceptions': 'odoo.exceptions.UserError',
+        }
+        level, msg = logging.WARNING, 'Test UserError'
+        client = initialize_raven(config, client_cls=InMemoryClient)
+
+        handlers = list(
+            log_handler_by_class(logging.getLogger(), OdooSentryHandler)
+        )
+        self.assertTrue(handlers)
+        handler = handlers[0]
+        try:
+            raise exceptions.UserError(msg)
+        except exceptions.UserError:
+            exc_info = sys.exc_info()
+        record = logging.LogRecord(
+            __name__, level, __file__, 42, msg, (), exc_info)
+        handler.emit(record)
+        self.assertEventNotCaptured(client, level, msg)

sentry/tests/test_logutils.py

@@ -0,0 +1,78 @@
+# -*- coding: utf-8 -*-
+# Copyright 2016-2017 Versada <https://versada.eu/>
+# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl).
+
+import unittest
+
+import mock
+
+from ..logutils import SanitizeOdooCookiesProcessor
+
+
+class TestOdooCookieSanitizer(unittest.TestCase):
+
+    def test_cookie_as_string(self):
+        data = {
+            'request': {
+                'cookies': 'website_lang=en_us;'
+                           'session_id=hello;'
+                           'Session_ID=hello;'
+                           'foo=bar',
+            },
+        }
+
+        proc = SanitizeOdooCookiesProcessor(mock.Mock())
+        result = proc.process(data)
+
+        self.assertTrue('request' in result)
+        http = result['request']
+        self.assertEqual(
+            http['cookies'],
+            'website_lang=en_us;'
+            'session_id={m};'
+            'Session_ID={m};'
+            'foo=bar'.format(
+                m=proc.MASK,
+            ),
+        )
+
+    def test_cookie_as_string_with_partials(self):
+        data = {
+            'request': {
+                'cookies': 'website_lang=en_us;session_id;foo=bar',
+            },
+        }
+
+        proc = SanitizeOdooCookiesProcessor(mock.Mock())
+        result = proc.process(data)
+
+        self.assertTrue('request' in result)
+        http = result['request']
+        self.assertEqual(
+            http['cookies'],
+            'website_lang=en_us;session_id;foo=bar'.format(m=proc.MASK),
+        )
+
+    def test_cookie_header(self):
+        data = {
+            'request': {
+                'headers': {
+                    'Cookie': 'foo=bar;'
+                              'session_id=hello;'
+                              'Session_ID=hello;'
+                              'a_session_id_here=hello',
+                },
+            },
+        }
+
+        proc = SanitizeOdooCookiesProcessor(mock.Mock())
+        result = proc.process(data)
+
+        self.assertTrue('request' in result)
+        http = result['request']
+        self.assertEqual(
+            http['headers']['Cookie'],
+            'foo=bar;'
+            'session_id={m};'
+            'Session_ID={m};'
+            'a_session_id_here={m}'.format(m=proc.MASK))