Browse Source

[ADD] base_import_match: Import deduplicator.

By default, when importing data (like CSV import) with the ``base_import``
module, Odoo follows this rule:

1. If you import the XMLID of a record, make an **update**.
2. If you do not, **create** a new record.

This module allows you to set additional rules to match if a given import is an
update or a new record.

This is useful when you need to sync heterogeneous databases, and the field you
use to match records in those databases with Odoo's is not the XMLID but the
name, VAT, email, etc.

After installing this module, the import logic will be changed to:

1. If you import the XMLID of a record, make an **update**.
2. If you do not:
   1. If there are import match rules for the model you are importing:
       1. Discard the rules that require fields you are not importing.
       2. Traverse the remaining rules one by one in order to find a match in
          the database.
          1. If one match is found:
             1. Stop traversing the rest of valid rules.
             2. **Update** that record.
          2. If zero or multiple matches are found:
             1. Continue with the next rule.
          3. If all rules are exhausted and no single match is found:
             1. **Create** a new record.
   2. If there are no match rules for your model:
      1. **Create** a new record.
pull/649/head
Jairo Llopis 9 years ago
committed by Pedro M. Baeza
parent
commit
c837ee502f
  1. 109
      base_import_match/README.rst
  2. 5
      base_import_match/__init__.py
  3. 24
      base_import_match/__openerp__.py
  4. 37
      base_import_match/demo/base_import_match.yml
  5. 83
      base_import_match/i18n/es.po
  6. 5
      base_import_match/models/__init__.py
  7. 220
      base_import_match/models/base_import.py
  8. BIN
      base_import_match/static/description/icon.png
  9. 5
      base_import_match/tests/__init__.py
  10. 2
      base_import_match/tests/import_data/res_partner_email.csv
  11. 2
      base_import_match/tests/import_data/res_partner_name.csv
  12. 2
      base_import_match/tests/import_data/res_partner_parent_name_is_company.csv
  13. 2
      base_import_match/tests/import_data/res_partner_vat.csv
  14. 2
      base_import_match/tests/import_data/res_users_login.csv
  15. 68
      base_import_match/tests/test_import.py
  16. 70
      base_import_match/views/base_import_match_view.xml

109
base_import_match/README.rst

@ -0,0 +1,109 @@
.. image:: https://img.shields.io/badge/licence-AGPL--3-blue.svg
:target: http://www.gnu.org/licenses/agpl-3.0-standalone.html
:alt: License: AGPL-3
=================
Base Import Match
=================
By default, when importing data (like CSV import) with the ``base_import``
module, Odoo follows this rule:
#. If you import the XMLID of a record, make an **update**.
#. If you do not, **create** a new record.
This module allows you to set additional rules to match if a given import is an
update or a new record.
This is useful when you need to sync heterogeneous databases, and the field you
use to match records in those databases with Odoo's is not the XMLID but the
name, VAT, email, etc.
After installing this module, the import logic will be changed to:
#. If you import the XMLID of a record, make an **update**.
#. If you do not:
#. If there are import match rules for the model you are importing:
#. Discard the rules that require fields you are not importing.
#. Traverse the remaining rules one by one in order to find a match in
the database.
#. If one match is found:
#. Stop traversing the rest of valid rules.
#. **Update** that record.
#. If zero or multiple matches are found:
#. Continue with the next rule.
#. If all rules are exhausted and no single match is found:
#. **Create** a new record.
#. If there are no match rules for your model:
#. **Create** a new record.
Configuration
=============
To configure this module, you need to:
#. Go to *Settings > Technical > Database Structure > Import Match*.
#. *Create*.
#. Choose a *Model*.
#. Choose the *Fields* that conform an unique key in that model.
#. *Save*.
In that list view, you can sort rules by drag and drop.
Usage
=====
To use this module, you need to:
#. Follow steps in **Configuration** section above.
#. Go to any list view.
#. Press *Import* and follow the import procedure as usual.
.. image:: https://odoo-community.org/website/image/ir.attachment/5784_f2813bd/datas
:alt: Try me on Runbot
:target: https://runbot.odoo-community.org/runbot/149/8.0
Roadmap / Known Issues
======================
* Add a filter to let you apply some rules only to incoming imports that match
a given criteria (like a domain, but for import data).
* Matching by VAT for ``res.partner`` records will only work when the partner
has no contacts, because otherwise Odoo reflects the parent company's VAT in
the contact, and that results in multiple matches. Fixing the above point
should make this work.
Bug Tracker
===========
Bugs are tracked on `GitHub Issues
<https://github.com/OCA/server-tools/issues>`_. In case of trouble, please
check there if your issue has already been reported. If you spotted it first,
help us smashing it by providing a detailed and welcomed `feedback
<https://github.com/OCA/
server-tools/issues/new?body=module:%20
base_import_match%0Aversion:%20
8.0%0A%0A**Steps%20to%20reproduce**%0A-%20...%0A%0A**Current%20behavior**%0A%0A**Expected%20behavior**>`_.
Credits
=======
Contributors
------------
* Jairo Llopis <yajo.sk8@gmail.com>
Maintainer
----------
.. image:: https://odoo-community.org/logo.png
:alt: Odoo Community Association
:target: https://odoo-community.org
This module is maintained by the OCA.
OCA, or the Odoo Community Association, is a nonprofit organization whose
mission is to support the collaborative development of Odoo features and
promote its widespread use.
To contribute to this module, please visit https://odoo-community.org.

5
base_import_match/__init__.py

@ -0,0 +1,5 @@
# -*- coding: utf-8 -*-
# © 2016 Grupo ESOC Ingeniería de Servicios, S.L.U. - Jairo Llopis
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html).
from . import models

24
base_import_match/__openerp__.py

@ -0,0 +1,24 @@
# -*- coding: utf-8 -*-
# © 2016 Grupo ESOC Ingeniería de Servicios, S.L.U. - Jairo Llopis
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html).
{
"name": "Base Import Match",
"summary": "Try to avoid duplicates before importing",
"version": "8.0.1.0.0",
"category": "Tools",
"website": "https://grupoesoc.es",
"author": "Grupo ESOC Ingeniería de Servicios, "
"Odoo Community Association (OCA)",
"license": "AGPL-3",
"application": False,
"installable": True,
"depends": [
"base_import",
],
"data": [
"views/base_import_match_view.xml",
],
"demo": [
"demo/base_import_match.yml",
],
}

37
base_import_match/demo/base_import_match.yml

@ -0,0 +1,37 @@
# -*- coding: utf-8 -*-
# © 2016 Grupo ESOC Ingeniería de Servicios, S.L.U. - Jairo Llopis
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl).
- !context {noupdate: True}
- !record {id: res_partner_vat, model: base_import.match}:
model_id: base.model_res_partner
sequence: 10
field_ids:
- base.field_res_partner_vat
- !record {id: res_partner_parent_name_is_company, model: base_import.match}:
model_id: base.model_res_partner
sequence: 20
field_ids:
- base.field_res_partner_name
- base.field_res_partner_parent_id
- base.field_res_partner_is_company
- !record {id: res_partner_email, model: base_import.match}:
model_id: base.model_res_partner
sequence: 30
field_ids:
- base.field_res_partner_email
- !record {id: res_partner_name, model: base_import.match}:
model_id: base.model_res_partner
sequence: 40
field_ids:
- base.field_res_partner_name
- !record {id: res_users_login, model: base_import.match}:
model_id: base.model_res_users
sequence: 50
field_ids:
- base.field_res_users_login

83
base_import_match/i18n/es.po

@ -0,0 +1,83 @@
# Translation of Odoo Server.
# This file contains the translation of the following modules:
# * base_import_match
#
msgid ""
msgstr ""
"Project-Id-Version: Odoo Server 8.0\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2016-05-11 12:24+0200\n"
"PO-Revision-Date: 2016-05-11 12:26+0200\n"
"Language-Team: \n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=2; plural=(n != 1);\n"
"X-Generator: Poedit 1.8.7.1\n"
"Last-Translator: Jairo Llopis <j.llopis@grupoesoc.es>\n"
"Language: es_ES\n"
#. module: base_import_match
#: model:ir.model,name:base_import_match.model_base_import_match
msgid "Deduplicate settings prior to CSV imports."
msgstr "Configuración para deduplicar antes de importar CSV."
#. module: base_import_match
#: field:base_import.match,display_name:0
msgid "Display Name"
msgstr "Nombre a mostrar"
#. module: base_import_match
#: sql_constraint:base_import.match:0
msgid "Duplicated match!"
msgstr "¡Coincidencia duplicada!"
#. module: base_import_match
#: field:base_import.match,field_ids:0
msgid "Fields"
msgstr "Campos"
#. module: base_import_match
#: help:base_import.match,field_ids:0
msgid "Fields that will define an unique key."
msgstr "Campos que definirán una clave única."
#. module: base_import_match
#: view:base_import.match:base_import_match.match_search_view
msgid "Group By"
msgstr "Agrupar por"
#. module: base_import_match
#: view:base_import.match:base_import_match.match_form_view
#: view:base_import.match:base_import_match.match_search_view
#: view:base_import.match:base_import_match.match_tree_view
#: model:ir.actions.act_window,name:base_import_match.match_action
#: model:ir.ui.menu,name:base_import_match.match_menu
msgid "Import Match"
msgstr "Coincidencia de importación"
#. module: base_import_match
#: help:base_import.match,model_id:0
msgid "In this model you will apply the match."
msgstr "En este modelo se aplicará la coincidencia."
#. module: base_import_match
#: field:base_import.match,__last_update:0
msgid "Last Modified on"
msgstr "Última actualización por"
#. module: base_import_match
#: view:base_import.match:base_import_match.match_search_view
#: field:base_import.match,model_id:0 field:base_import.match,model_name:0
msgid "Model"
msgstr "Modelo"
#. module: base_import_match
#: field:base_import.match,name:0
msgid "Name"
msgstr "Nombre"
#. module: base_import_match
#: field:base_import.match,sequence:0
msgid "Sequence"
msgstr "Secuencia"

5
base_import_match/models/__init__.py

@ -0,0 +1,5 @@
# -*- coding: utf-8 -*-
# © 2016 Grupo ESOC Ingeniería de Servicios, S.L.U. - Jairo Llopis
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html).
from . import base_import

220
base_import_match/models/base_import.py

@ -0,0 +1,220 @@
# -*- coding: utf-8 -*-
# © 2016 Grupo ESOC Ingeniería de Servicios, S.L.U. - Jairo Llopis
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html).
from openerp import api, fields, models
from openerp.exceptions import except_orm as ValueError # TODO remove in v9
from openerp import SUPERUSER_ID # TODO remove in v10
class BaseImportMatch(models.Model):
_name = "base_import.match"
_description = "Deduplicate settings prior to CSV imports."
_order = "sequence, name"
_sql_constraints = [
("name_unique", "UNIQUE(name)", "Duplicated match!"),
]
name = fields.Char(
compute="_compute_name",
store=True,
index=True)
sequence = fields.Integer(index=True)
model_id = fields.Many2one(
"ir.model",
"Model",
required=True,
ondelete="cascade",
domain=[("osv_memory", "=", False)],
help="In this model you will apply the match.")
model_name = fields.Char(
related="model_id.model",
store=True,
index=True)
field_ids = fields.Many2many(
"ir.model.fields",
string="Fields",
required=True,
domain="[('model_id', '=', model_id)]",
help="Fields that will define an unique key.")
@api.multi
@api.onchange("model_id")
def _onchange_model_id(self):
self.field_ids = False
@api.model
def create(self, vals):
"""Wrap the model after creation."""
result = super(BaseImportMatch, self).create(vals)
self._load_autopatch(result.model_name)
return result
@api.multi
def unlink(self):
"""Unwrap the model after deletion."""
models = set(self.mapped("model_name"))
result = super(BaseImportMatch, self).unlink()
for model in models:
self._load_autopatch(model)
return result
@api.multi
def write(self, vals):
"""Wrap the model after writing."""
result = super(BaseImportMatch, self).write(vals)
if "model_id" in vals or "model_name" in vals:
for s in self:
self._load_autopatch(s.model_name)
return result
# TODO convert to @api.model_cr in v10
def _register_hook(self, cr):
"""Autopatch on init."""
models = set(
self.browse(
cr,
SUPERUSER_ID,
self.search(cr, SUPERUSER_ID, list()))
.mapped("model_name"))
for model in models:
self._load_autopatch(cr, SUPERUSER_ID, model)
@api.multi
@api.depends("model_id", "field_ids")
def _compute_name(self):
"""Automatic self-descriptive name for the setting records."""
for s in self:
s.name = "{}: {}".format(
s.model_id.display_name,
" + ".join(s.field_ids.mapped("display_name")))
@api.model
def _load_wrapper(self):
"""Create a new load patch method."""
@api.model
def wrapper(self, fields, data):
"""Try to identify rows by other pseudo-unique keys.
It searches for rows that have no XMLID specified, and gives them
one if any :attr:`~.field_ids` combination is found. With a valid
XMLID in place, Odoo will understand that it must *update* the
record instead of *creating* a new one.
"""
newdata = list()
# Mock Odoo to believe the user is importing the ID field
if "id" not in fields:
fields.append("id")
# Needed to work with relational fields
clean_fields = [
models.fix_import_export_id_paths(f)[0] for f in fields]
# Get usable rules to perform matches
usable = self.env["base_import.match"]._usable_for_load(
self._name, clean_fields)
for row in (dict(zip(clean_fields, r)) for r in data):
# All rows need an ID
if "id" not in row:
row["id"] = u""
# Skip rows with ID, they do not need all this
elif row["id"]:
continue
# Store records that match a combination
match = self
for combination in usable:
match |= self.search(
[(field.name, "=", row[field.name])
for field in combination.field_ids])
# When a single match is found, stop searching
if len(match) != 1:
break
# Only one record should have been found
try:
match.ensure_one()
# You hit this because...
# a. No match. Odoo must create a new record.
# b. Multiple matches. No way to know which is the right
# one, so we let Odoo create a new record or raise
# the corresponding exception.
# In any case, we must do nothing.
except ValueError:
continue
# Give a valid XMLID to this row
row["id"] = match._BaseModel__export_xml_id()
# Store the modified row, in the same order as fields
newdata.append(tuple(row[f] for f in clean_fields))
# Leave the rest to Odoo itself
del data
return wrapper.origin(self, fields, newdata)
# Flag to avoid confusions with other possible wrappers
wrapper.__base_import_match = True
return wrapper
@api.model
def _load_autopatch(self, model_name):
"""[Un]apply patch automatically."""
self._load_unpatch(model_name)
if self.search([("model_name", "=", model_name)]):
self._load_patch(model_name)
@api.model
def _load_patch(self, model_name):
"""Apply patch for :param:`model_name`'s load method.
:param str model_name:
Model technical name, such as ``res.partner``.
"""
self.env[model_name]._patch_method(
"load", self._load_wrapper())
@api.model
def _load_unpatch(self, model_name):
"""Apply patch for :param:`model_name`'s load method.
:param str model_name:
Model technical name, such as ``res.partner``.
"""
model = self.env[model_name]
# Unapply patch only if there is one
try:
if model.load.__base_import_match:
model._revert_method("load")
except AttributeError:
pass
@api.model
def _usable_for_load(self, model_name, fields):
"""Return a set of elements usable for calling ``load()``.
:param str model_name:
Technical name of the model where you are loading data.
E.g. ``res.partner``.
:param list(str|bool) fields:
List of field names being imported.
"""
result = self
available = self.search([("model_name", "=", model_name)])
# Use only criteria with all required fields to match
for record in available:
if all(f.name in fields for f in record.field_ids):
result += record
return result

BIN
base_import_match/static/description/icon.png

After

Width: 128  |  Height: 128  |  Size: 9.2 KiB

5
base_import_match/tests/__init__.py

@ -0,0 +1,5 @@
# -*- coding: utf-8 -*-
# © 2016 Grupo ESOC Ingeniería de Servicios, S.L.U. - Jairo Llopis
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html).
from . import test_import

2
base_import_match/tests/import_data/res_partner_email.csv

@ -0,0 +1,2 @@
email,name
michel.fletcher@agrolait.example.com,Michel Fletcher Changed

2
base_import_match/tests/import_data/res_partner_name.csv

@ -0,0 +1,2 @@
function,name
Function Changed,Michel Fletcher

2
base_import_match/tests/import_data/res_partner_parent_name_is_company.csv

@ -0,0 +1,2 @@
name,is_company,parent_id/id,email
Michel Fletcher,False,base.res_partner_2,changed@agrolait.example.com

2
base_import_match/tests/import_data/res_partner_vat.csv

@ -0,0 +1,2 @@
name,vat
Federal Changed,BE0477472701

2
base_import_match/tests/import_data/res_users_login.csv

@ -0,0 +1,2 @@
login,name
demo,Demo User Changed

68
base_import_match/tests/test_import.py

@ -0,0 +1,68 @@
# -*- coding: utf-8 -*-
# © 2016 Grupo ESOC Ingeniería de Servicios, S.L.U. - Jairo Llopis
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html).
from os import path
from openerp.tests.common import TransactionCase
PATH = path.join(path.dirname(__file__), "import_data", "%s.csv")
OPTIONS = {
"headers": True,
"quoting": '"',
"separator": ",",
}
class ImportCase(TransactionCase):
def _base_import_record(self, res_model, file_name):
"""Create and return a ``base_import.import`` record."""
with open(PATH % file_name) as demo_file:
return self.env["base_import.import"].create({
"res_model": res_model,
"file": demo_file.read(),
"file_name": "%s.csv" % file_name,
"file_type": "csv",
})
def test_res_partner_vat(self):
"""Change name based on VAT."""
federal = self.env.ref("base.res_partner_26")
federal.vat = "BE0477472701"
record = self._base_import_record("res.partner", "res_partner_vat")
record.do(["name", "vat"], OPTIONS)
federal.env.invalidate_all()
self.assertEqual(federal.name, "Federal Changed")
def test_res_partner_parent_name_is_company(self):
"""Change email based on parent_id, name and is_company."""
record = self._base_import_record(
"res.partner", "res_partner_parent_name_is_company")
record.do(["name", "is_company", "parent_id/id", "email"], OPTIONS)
self.assertEqual(
self.env.ref("base.res_partner_address_4").email,
"changed@agrolait.example.com")
def test_res_partner_email(self):
"""Change name based on email."""
record = self._base_import_record("res.partner", "res_partner_email")
record.do(["email", "name"], OPTIONS)
self.assertEqual(
self.env.ref("base.res_partner_address_4").name,
"Michel Fletcher Changed")
def test_res_partner_name(self):
"""Change function based on name."""
record = self._base_import_record("res.partner", "res_partner_name")
record.do(["function", "name"], OPTIONS)
self.assertEqual(
self.env.ref("base.res_partner_address_4").function,
"Function Changed")
def test_res_users_login(self):
"""Change name based on login."""
record = self._base_import_record("res.users", "res_users_login")
record.do(["login", "name"], OPTIONS)
self.assertEqual(
self.env.ref("base.user_demo").name,
"Demo User Changed")

70
base_import_match/views/base_import_match_view.xml

@ -0,0 +1,70 @@
<?xml version="1.0" encoding="utf-8"?>
<!-- © 2016 Grupo ESOC Ingeniería de Servicios, S.L.U. - Jairo Llopis
License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html). -->
<openerp>
<data>
<record id="match_form_view" model="ir.ui.view">
<field name="name">Import match form view</field>
<field name="model">base_import.match</field>
<field name="arch" type="xml">
<form string="Import Match">
<sheet>
<h1>
<field name="name"/>
</h1>
<group>
<field name="model_id"/>
<field
name="field_ids"
widget="many2many_tags"
options="{
'no_create': True,
}"/>
<field name="sequence"/>
</group>
</sheet>
</form>
</field>
</record>
<record id="match_tree_view" model="ir.ui.view">
<field name="name">Import match tree view</field>
<field name="model">base_import.match</field>
<field name="arch" type="xml">
<tree string="Import Match">
<field name="name"/>
<field name="sequence" invisible="True"/>
</tree>
</field>
</record>
<record id="match_search_view" model="ir.ui.view">
<field name="name">Import match search view</field>
<field name="model">base_import.match</field>
<field name="arch" type="xml">
<search string="Import Match">
<field name="name"/>
<field name="model_id"/>
<field name="field_ids"/>
<separator/>
<group expand="0" string="Group By">
<filter string="Model" context="{'group_by': 'model_id'}"/>
</group>
</search>
</field>
</record>
<act_window
name="Import Match"
res_model="base_import.match"
id="match_action"/>
<menuitem
id="match_menu"
action="match_action"
parent="base.next_id_9"/>
</data>
</openerp>
Loading…
Cancel
Save