Browse Source

[MIG] base_import_odoo

10.0
Holger Brunn 6 years ago
committed by OCA-git-bot
parent
commit
c38baed60a
  1. 6
      .travis.yml
  2. 17
      base_import_odoo/CONFIGURE.rst
  3. 1
      base_import_odoo/CONTRIBUTORS.rst
  4. 8
      base_import_odoo/DESCRIPTION.rst
  5. 99
      base_import_odoo/README.rst
  6. 5
      base_import_odoo/ROADMAP.rst
  7. 7
      base_import_odoo/USAGE.rst
  8. 2
      base_import_odoo/__init__.py
  9. 5
      base_import_odoo/__manifest__.py
  10. 18
      base_import_odoo/demo/import_odoo_database.xml
  11. 140
      base_import_odoo/demo/import_odoo_database_field.xml
  12. 67
      base_import_odoo/demo/import_odoo_database_model.xml
  13. 18
      base_import_odoo/demo/ir_attachment.xml
  14. 12
      base_import_odoo/demo/res_partner.xml
  15. 14
      base_import_odoo/demo/res_users.xml
  16. 6
      base_import_odoo/models/__init__.py
  17. 45
      base_import_odoo/models/import_odoo_database.py
  18. 4
      base_import_odoo/models/import_odoo_database_field.py
  19. 9
      base_import_odoo/models/import_odoo_database_model.py
  20. 4
      base_import_odoo/models/ir_model_data.py
  21. 2
      base_import_odoo/tests/__init__.py
  22. 7
      base_import_odoo/tests/test_base_import_odoo.py
  23. 180
      base_import_odoo/views/import_odoo_database.xml
  24. 90
      base_import_odoo/views/import_odoo_database_field.xml
  25. 20
      base_import_odoo/views/menu.xml
  26. 1
      requirements.txt

6
.travis.yml

@ -25,14 +25,16 @@ env:
matrix: matrix:
- LINT_CHECK="1" - LINT_CHECK="1"
- TRANSIFEX="1" - TRANSIFEX="1"
- TESTS="1" ODOO_REPO="odoo/odoo" EXCLUDE="database_cleanup,auth_session_timeout,auth_brute_force"
- TESTS="1" ODOO_REPO="OCA/OCB" EXCLUDE="database_cleanup,auth_session_timeout,auth_brute_force"
- TESTS="1" ODOO_REPO="odoo/odoo" EXCLUDE="database_cleanup,auth_session_timeout,auth_brute_force,base_import_odoo"
- TESTS="1" ODOO_REPO="OCA/OCB" EXCLUDE="database_cleanup,auth_session_timeout,auth_brute_force,base_import_odoo"
- TESTS="1" ODOO_REPO="odoo/odoo" INCLUDE="database_cleanup" - TESTS="1" ODOO_REPO="odoo/odoo" INCLUDE="database_cleanup"
- TESTS="1" ODOO_REPO="OCA/OCB" INCLUDE="database_cleanup" - TESTS="1" ODOO_REPO="OCA/OCB" INCLUDE="database_cleanup"
- TESTS="1" ODOO_REPO="odoo/odoo" INCLUDE="auth_session_timeout" - TESTS="1" ODOO_REPO="odoo/odoo" INCLUDE="auth_session_timeout"
- TESTS="1" ODOO_REPO="OCA/OCB" INCLUDE="auth_session_timeout" - TESTS="1" ODOO_REPO="OCA/OCB" INCLUDE="auth_session_timeout"
- TESTS="1" ODOO_REPO="odoo/odoo" INCLUDE="auth_brute_force" - TESTS="1" ODOO_REPO="odoo/odoo" INCLUDE="auth_brute_force"
- TESTS="1" ODOO_REPO="OCA/OCB" INCLUDE="auth_brute_force" - TESTS="1" ODOO_REPO="OCA/OCB" INCLUDE="auth_brute_force"
- TESTS="1" ODOO_REPO="odoo/odoo" INCLUDE="base_import_odoo"
- TESTS="1" ODOO_REPO="OCA/OCB" INCLUDE="base_import_odoo"
virtualenv: virtualenv:
system_site_packages: true system_site_packages: true

17
base_import_odoo/CONFIGURE.rst

@ -0,0 +1,17 @@
Go to Settings / Remote Odoo import / Import configurations and create a configuration.
After filling in your credentials, select models you want to import from the remote database. If you only want to import a subset of the records, add an appropriate domain.
The import will copy records of all models listed, and handle links to records of models which are not imported depending on the existing field mappings. Field mappings to local records also are a stopping condition. Without those, the import will have to create some record for all required x2x fields, which you probably don't want.
Probably you'll want to map records of model ``res.company``, and at least the admin user.
The module doesn't import one2many fields, if you want to have those, add the model the field in question points to to the list of imported models, possibly with a domain.
If you don't fill in a remote ID, the addon will use the configured local ID for every record of the model (this way, you can for example map all users in the remote system to some import user in the current system).
For fields that have a uniqueness constraint (like ``res.users#login``), create a field mapping if type ``unique``, then the import will generate a unique value for this field.
For models using references with two fields (like ``ir.attachment``), create a field mapping of type ``by reference`` and select the two fields involved. The import will transform known ids (=ids of models you import) to the respective local id, and clean out the model/id fields for unknown models/ids.
You can add custom defaults per model in case your database has different required fields than the source database. For ``res.partner``, you'll most certainly fill in ``{'name': '/'}`` or somethign similar.

1
base_import_odoo/CONTRIBUTORS.rst

@ -0,0 +1 @@
* Holger Brunn <hbrunn@therp.nl>

8
base_import_odoo/DESCRIPTION.rst

@ -0,0 +1,8 @@
This module was written to import data from another Odoo database. The idea is that you define which models to import from the other database, and add eventual mappings for records you don't want to import.
Use cases
=========
- merging databases
- one way sync
- aggregating management data from distributed systems

99
base_import_odoo/README.rst

@ -1,98 +1 @@
.. image:: https://img.shields.io/badge/licence-AGPL--3-blue.svg
:target: http://www.gnu.org/licenses/agpl-3.0-standalone.html
:alt: License: AGPL-3
================================
Import from remote Odoo database
================================
This module was written to import data from another Odoo database. The idea is that you define which models to import from the other database, and add eventual mappings for records you don't want to import.
Use cases
=========
- merging databases
- one way sync
- aggregating management data from distributed systems
Configuration
=============
Go to Settings / Remote Odoo import / Import configurations and create a configuration.
After filling in your credentials, select models you want to import from the remote database. If you only want to import a subset of the records, add an appropriate domain.
The import will copy records of all models listed, and handle links to records of models which are not imported depending on the existing field mappings. Field mappings to local records also are a stopping condition. Without those, the import will have to create some record for all required x2x fields, which you probably don't want.
Probably you'll want to map records of model `res.company`, and at least the admin user.
The module doesn't import one2many fields, if you want to have those, add the model the field in question points to to the list of imported models, possibly with a domain.
If you don't fill in a remote ID, the addon will use the configured local ID for every record of the model (this way, you can for example map all users in the remote system to some import user in the current system).
For fields that have a uniqueness constraint (like `res.users#login`), create a field mapping if type `unique`, then the import will generate a unique value for this field.
For models using references with two fields (like `ir.attachment`), create a field mapping of type `by reference` and select the two fields involved. The import will transform known ids (=ids of models you import) to the respective local id, and clean out the model/id fields for unknown models/ids.
Usage
=====
To use this module, you need to:
#. go to an import configuration and hit the button ``Run import``
#. be patient, this creates a cronjob which will start up to a minutes afterwards
#. reload the form, as soon as the cronjob runs you'll see a field ``Progress`` that lets you inspect what was imported already
#. note that the cronjob also resets the password as soon as it has read it. So for a subsequent import, you'll have to fill it in again
#. running an import a second time won't duplicate data, it should recognize records imported earlier and just update them
.. image:: https://odoo-community.org/website/image/ir.attachment/5784_f2813bd/datas
:alt: Try me on Runbot
:target: https://runbot.odoo-community.org/runbot/149/8.0
Known issues / Roadmap
======================
* Yes of course this duplicates a lot of connector functionality. Rewrite this with the connector framework, probably collaborate with https://github.com/OCA/connector-odoo2odoo
* Do something with workflows
* Support reference fields, while being at it refactor _run_import_map_values to call a function per field type
* Probably it's safer and faster to disable recomputation during import, and recompute all fields afterwards
* Add duplicate handling strategy 'Overwrite older'
Bug Tracker
===========
Bugs are tracked on `GitHub Issues
<https://github.com/OCA/server-tools/issues>`_. In case of trouble, please
check there if your issue has already been reported. If you spotted it first,
help us smashing it by providing a detailed and welcomed feedback.
Credits
=======
Images
------
* Odoo Community Association: `Icon <https://github.com/OCA/maintainer-tools/blob/master/template/module/static/description/icon.svg>`_.
Contributors
------------
* Holger Brunn <hbrunn@therp.nl>
Do not contact contributors directly about help with questions or problems concerning this addon, but use the `community mailing list <mailto:community@mail.odoo.com>`_ or the `appropriate specialized mailinglist <https://odoo-community.org/groups>`_ for help, and the bug tracker linked in `Bug Tracker`_ above for technical issues.
Maintainer
----------
.. image:: https://odoo-community.org/logo.png
:alt: Odoo Community Association
:target: https://odoo-community.org
This module is maintained by the OCA.
OCA, or the Odoo Community Association, is a nonprofit organization whose
mission is to support the collaborative development of Odoo features and
promote its widespread use.
To contribute to this module, please visit https://odoo-community.org.
/

5
base_import_odoo/ROADMAP.rst

@ -0,0 +1,5 @@
* Yes of course this duplicates a lot of connector functionality. Rewrite this with the connector framework, probably collaborate with https://github.com/OCA/connector-odoo2odoo
* Do something with workflows
* Support reference fields, while being at it refactor _run_import_map_values to call a function per field type
* Probably it's safer and faster to disable recomputation during import, and recompute all fields afterwards
* Add duplicate handling strategy 'Overwrite older'

7
base_import_odoo/USAGE.rst

@ -0,0 +1,7 @@
To use this module, you need to:
#. go to an import configuration and hit the button ``Run import``
#. be patient, this creates a cronjob which will start up to a minutes afterwards
#. reload the form, as soon as the cronjob runs you'll see a field ``Progress`` that lets you inspect what was imported already
#. note that the cronjob also resets the password as soon as it has read it. So for a subsequent import, you'll have to fill it in again
#. running an import a second time won't duplicate data, it should recognize records imported earlier and just update them

2
base_import_odoo/__init__.py

@ -1,4 +1,4 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
# © 2017 Therp BV <http://therp.nl>
# Copyright 2017-2018 Therp BV <http://therp.nl>
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html). # License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html).
from . import models from . import models

5
base_import_odoo/__openerp__.py → base_import_odoo/__manifest__.py

@ -1,13 +1,14 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
# © 2017 Therp BV <http://therp.nl>
# Copyright 2017-2018 Therp BV <http://therp.nl>
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html). # License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html).
{ {
"name": "Import from Odoo", "name": "Import from Odoo",
"version": "8.0.1.0.1",
"version": "10.0.1.0.0",
"author": "Therp BV,Odoo Community Association (OCA)", "author": "Therp BV,Odoo Community Association (OCA)",
"license": "AGPL-3", "license": "AGPL-3",
"category": "Tools", "category": "Tools",
"summary": "Import records from another Odoo instance", "summary": "Import records from another Odoo instance",
"website": "https://github.com/OCA/server-tools",
"depends": [ "depends": [
'mail', 'mail',
], ],

18
base_import_odoo/demo/import_odoo_database.xml

@ -1,11 +1,9 @@
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<openerp>
<data noupdate="1">
<record id="demodb" model="import.odoo.database">
<field name="url">http://localhost:8069</field>
<field name="database">demodb</field>
<field name="user">admin</field>
<field name="password" eval="False" />
</record>
</data>
</openerp>
<odoo noupdate="1">
<record id="demodb" model="import.odoo.database">
<field name="url">http://localhost:8069</field>
<field name="database">demodb</field>
<field name="user">admin</field>
<field name="password" eval="False" />
</record>
</odoo>

140
base_import_odoo/demo/import_odoo_database_field.xml

@ -1,72 +1,70 @@
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<openerp>
<data noupdate="1">
<record id="mapping_partner_id_root" model="import.odoo.database.field">
<field name="database_id" ref="demodb" />
<field name="mapping_type">fixed</field>
<field name="model_id" ref="base.model_res_partner" />
<field name="local_id" ref="base.partner_root" />
<field name="remote_id" ref="base.partner_root" />
</record>
<record id="mapping_partner_id_company" model="import.odoo.database.field">
<field name="database_id" ref="demodb" />
<field name="mapping_type">fixed</field>
<field name="model_id" ref="base.model_res_partner" />
<field name="local_id" ref="base.main_partner" />
<field name="remote_id" ref="base.main_partner" />
</record>
<record id="mapping_partner_id_public" model="import.odoo.database.field">
<field name="database_id" ref="demodb" />
<field name="mapping_type">fixed</field>
<field name="model_id" ref="base.model_res_partner" />
<field name="local_id" ref="base.public_partner" />
<field name="remote_id" ref="base.public_partner" />
</record>
<record id="mapping_user_root" model="import.odoo.database.field">
<field name="database_id" ref="demodb" />
<field name="mapping_type">fixed</field>
<field name="model_id" ref="base.model_res_users" />
<field name="local_id" ref="mapped_admin" />
<field name="remote_id" ref="base.user_root" />
</record>
<record id="mapping_user_public" model="import.odoo.database.field">
<field name="database_id" ref="demodb" />
<field name="mapping_type">fixed</field>
<field name="model_id" ref="base.model_res_users" />
<field name="local_id" ref="base.public_user" />
<field name="remote_id" ref="base.public_user" />
</record>
<record id="mapping_company_id" model="import.odoo.database.field">
<field name="database_id" ref="demodb" />
<field name="mapping_type">fixed</field>
<field name="model_id" ref="base.model_res_company" />
<field name="local_id" ref="base.main_company" />
<field name="remote_id" ref="base.main_company" />
</record>
<record id="mapping_login" model="import.odoo.database.field">
<field name="database_id" ref="demodb" />
<field name="mapping_type">unique</field>
<field name="model_id" ref="base.model_res_users" />
<field name="field_ids" eval="[(4, ref('base.field_res_users_login'))]" />
</record>
<record id="mapping_models" model="import.odoo.database.field">
<field name="database_id" ref="demodb" />
<field name="mapping_type">by_field</field>
<field name="model_id" ref="base.model_ir_model" />
<field name="field_ids" eval="[(4, ref('base.field_ir_model_name'))]" />
</record>
<record id="mapping_groups" model="import.odoo.database.field">
<field name="database_id" ref="demodb" />
<field name="mapping_type">by_field</field>
<field name="model_id" ref="base.model_res_groups" />
<field name="field_ids" eval="[(4, ref('base.field_res_groups_name'))]" />
</record>
<record id="mapping_attachment" model="import.odoo.database.field">
<field name="database_id" ref="demodb" />
<field name="mapping_type">by_reference</field>
<field name="model_id" ref="base.model_ir_attachment" />
<field name="model_field_id" ref="base.field_ir_attachment_res_model" />
<field name="id_field_id" ref="base.field_ir_attachment_res_id" />
</record>
</data>
</openerp>
<odoo noupdate="1">
<record id="mapping_partner_id_root" model="import.odoo.database.field">
<field name="database_id" ref="demodb" />
<field name="mapping_type">fixed</field>
<field name="model_id" ref="base.model_res_partner" />
<field name="local_id" ref="base.partner_root" />
<field name="remote_id" ref="base.partner_root" />
</record>
<record id="mapping_partner_id_company" model="import.odoo.database.field">
<field name="database_id" ref="demodb" />
<field name="mapping_type">fixed</field>
<field name="model_id" ref="base.model_res_partner" />
<field name="local_id" ref="base.main_partner" />
<field name="remote_id" ref="base.main_partner" />
</record>
<record id="mapping_partner_id_public" model="import.odoo.database.field">
<field name="database_id" ref="demodb" />
<field name="mapping_type">fixed</field>
<field name="model_id" ref="base.model_res_partner" />
<field name="local_id" ref="base.public_partner" />
<field name="remote_id" ref="base.public_partner" />
</record>
<record id="mapping_user_root" model="import.odoo.database.field">
<field name="database_id" ref="demodb" />
<field name="mapping_type">fixed</field>
<field name="model_id" ref="base.model_res_users" />
<field name="local_id" ref="mapped_admin" />
<field name="remote_id" ref="base.user_root" />
</record>
<record id="mapping_user_public" model="import.odoo.database.field">
<field name="database_id" ref="demodb" />
<field name="mapping_type">fixed</field>
<field name="model_id" ref="base.model_res_users" />
<field name="local_id" ref="base.public_user" />
<field name="remote_id" ref="base.public_user" />
</record>
<record id="mapping_company_id" model="import.odoo.database.field">
<field name="database_id" ref="demodb" />
<field name="mapping_type">fixed</field>
<field name="model_id" ref="base.model_res_company" />
<field name="local_id" ref="base.main_company" />
<field name="remote_id" ref="base.main_company" />
</record>
<record id="mapping_login" model="import.odoo.database.field">
<field name="database_id" ref="demodb" />
<field name="mapping_type">unique</field>
<field name="model_id" ref="base.model_res_users" />
<field name="field_ids" eval="[(4, ref('base.field_res_users_login'))]" />
</record>
<record id="mapping_models" model="import.odoo.database.field">
<field name="database_id" ref="demodb" />
<field name="mapping_type">by_field</field>
<field name="model_id" ref="base.model_ir_model" />
<field name="field_ids" eval="[(4, ref('base.field_ir_model_name'))]" />
</record>
<record id="mapping_groups" model="import.odoo.database.field">
<field name="database_id" ref="demodb" />
<field name="mapping_type">by_field</field>
<field name="model_id" ref="base.model_res_groups" />
<field name="field_ids" eval="[(4, ref('base.field_res_groups_name'))]" />
</record>
<record id="mapping_attachment" model="import.odoo.database.field">
<field name="database_id" ref="demodb" />
<field name="mapping_type">by_reference</field>
<field name="model_id" ref="base.model_ir_attachment" />
<field name="model_field_id" ref="base.field_ir_attachment_res_model" />
<field name="id_field_id" ref="base.field_ir_attachment_res_id" />
</record>
</odoo>

67
base_import_odoo/demo/import_odoo_database_model.xml

@ -1,35 +1,34 @@
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<openerp>
<data noupdate="1">
<record id="model_partner" model="import.odoo.database.model">
<field name="sequence">1</field>
<field name="model_id" ref="base.model_res_partner" />
<field name="database_id" ref="demodb" />
<field name="domain">[(1, '=', 1)]</field>
</record>
<record id="model_alias" model="import.odoo.database.model">
<field name="sequence">2</field>
<field name="model_id" ref="mail.model_mail_alias" />
<field name="database_id" ref="demodb" />
<field name="domain">[(1, '=', 1)]</field>
</record>
<record id="model_users" model="import.odoo.database.model">
<field name="sequence">3</field>
<field name="model_id" ref="base.model_res_users" />
<field name="database_id" ref="demodb" />
<field name="domain">[(1, '=', 1)]</field>
</record>
<record id="model_groups" model="import.odoo.database.model">
<field name="sequence">4</field>
<field name="model_id" ref="base.model_res_groups" />
<field name="database_id" ref="demodb" />
<field name="domain">[(1, '=', 1)]</field>
</record>
<record id="model_attachment" model="import.odoo.database.model">
<field name="sequence">5</field>
<field name="model_id" ref="base.model_ir_attachment" />
<field name="database_id" ref="demodb" />
<field name="domain">[('res_model', 'in', ['res.users'])]</field>
</record>
</data>
</openerp>
<odoo noupdate="1">
<record id="model_partner" model="import.odoo.database.model">
<field name="sequence">1</field>
<field name="model_id" ref="base.model_res_partner" />
<field name="database_id" ref="demodb" />
<field name="domain">[(1, '=', 1)]</field>
<field name="defaults">{'name': '/'}</field>
</record>
<record id="model_alias" model="import.odoo.database.model">
<field name="sequence">2</field>
<field name="model_id" ref="mail.model_mail_alias" />
<field name="database_id" ref="demodb" />
<field name="domain">[(1, '=', 1)]</field>
</record>
<record id="model_users" model="import.odoo.database.model">
<field name="sequence">3</field>
<field name="model_id" ref="base.model_res_users" />
<field name="database_id" ref="demodb" />
<field name="domain">[(1, '=', 1)]</field>
</record>
<record id="model_groups" model="import.odoo.database.model">
<field name="sequence">4</field>
<field name="model_id" ref="base.model_res_groups" />
<field name="database_id" ref="demodb" />
<field name="domain">[(1, '=', 1)]</field>
</record>
<record id="model_attachment" model="import.odoo.database.model">
<field name="sequence">5</field>
<field name="model_id" ref="base.model_ir_attachment" />
<field name="database_id" ref="demodb" />
<field name="domain">[('res_model', 'in', ['res.users'])]</field>
</record>
</odoo>

18
base_import_odoo/demo/ir_attachment.xml

@ -1,11 +1,9 @@
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<openerp>
<data>
<record id="attachment_demo" model="ir.attachment">
<field name="name">Demo attachment</field>
<field name="res_model">res.users</field>
<field name="res_id" ref="base.user_demo" />
<field name="datas">aGVsbG8gd29ybGQK</field>
</record>
</data>
</openerp>
<odoo>
<record id="attachment_demo" model="ir.attachment">
<field name="name">Demo attachment</field>
<field name="res_model">res.users</field>
<field name="res_id" ref="base.user_demo" />
<field name="datas">aGVsbG8gd29ybGQK</field>
</record>
</odoo>

12
base_import_odoo/demo/res_partner.xml

@ -1,8 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<openerp>
<data>
<record id="base.res_partner_1" model="res.partner">
<field name="user_id" ref="base.user_demo" />
</record>
</data>
</openerp>
<odoo>
<record id="base.res_partner_1" model="res.partner">
<field name="user_id" ref="base.user_demo" />
</record>
</odoo>

14
base_import_odoo/demo/res_users.xml

@ -1,9 +1,7 @@
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<openerp>
<data noupdate="1">
<record id="mapped_admin" model="res.users" context="{'no_reset_password': True}">
<field name="name">Mapped admin</field>
<field name="login">mapped_admin</field>
</record>
</data>
</openerp>
<odoo noupdate="1">
<record id="mapped_admin" model="res.users" context="{'no_reset_password': True}">
<field name="name">Mapped admin</field>
<field name="login">mapped_admin</field>
</record>
</odoo>

6
base_import_odoo/models/__init__.py

@ -1,7 +1,7 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
# © 2017 Therp BV <http://therp.nl>
# Copyright 2017-2018 Therp BV <http://therp.nl>
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html). # License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html).
from . import ir_model_data
from . import import_odoo_database from . import import_odoo_database
from . import import_odoo_database_model
from . import import_odoo_database_field from . import import_odoo_database_field
from . import import_odoo_database_model
from . import ir_model_data

45
base_import_odoo/models/import_odoo_database.py

@ -1,5 +1,5 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
# © 2017 Therp BV <http://therp.nl>
# Copyright 2017-2018 Therp BV <http://therp.nl>
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html). # License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html).
import logging import logging
try: try:
@ -11,7 +11,8 @@ except ImportError: # pragma: no cover
import psycopg2 import psycopg2
import traceback import traceback
from urlparse import urlparse from urlparse import urlparse
from openerp import _, api, exceptions, fields, models, tools
from odoo import _, api, exceptions, fields, models, tools
from odoo.tools.safe_eval import safe_eval
from collections import namedtuple from collections import namedtuple
_logger = logging.getLogger('base_import_odoo') _logger = logging.getLogger('base_import_odoo')
@ -86,11 +87,20 @@ class ImportOdooDatabase(models.Model):
'cronjob_id': self._create_cronjob().id, 'cronjob_id': self._create_cronjob().id,
}) })
@api.model
def _run_import_cron(self, ids):
return self.browse(ids)._run_import()
@api.multi @api.multi
def _run_import(self, commit=True, commit_threshold=100): def _run_import(self, commit=True, commit_threshold=100):
"""Run the import as cronjob, commit often""" """Run the import as cronjob, commit often"""
self.ensure_one() self.ensure_one()
if not self.password: if not self.password:
self.write({
'status_data': dict(
self.status_data, error='No password provided',
),
})
return return
# model name: [ids] # model name: [ids]
remote_ids = {} remote_ids = {}
@ -118,7 +128,7 @@ class ImportOdooDatabase(models.Model):
model = model_line.model_id model = model_line.model_id
remote_ids[model.model] = remote.execute( remote_ids[model.model] = remote.execute(
model.model, 'search', model.model, 'search',
tools.safe_eval(model_line.domain) if model_line.domain else []
safe_eval(model_line.domain) if model_line.domain else []
) )
remote_counts[model.model] = len(remote_ids[model.model]) remote_counts[model.model] = len(remote_ids[model.model])
self.write({ self.write({
@ -210,6 +220,21 @@ class ImportOdooDatabase(models.Model):
self.id, model._name.replace('.', '_'), _id or 0, self.id, model._name.replace('.', '_'), _id or 0,
) )
record = self._create_record_filter_fields(model, record) record = self._create_record_filter_fields(model, record)
model_defaults = {}
if context.model_line.defaults:
model_defaults.update(safe_eval(context.model_line.defaults))
for key, value in model_defaults.items():
record.setdefault(key, value)
if context.model_line.postprocess:
safe_eval(
context.model_line.postprocess, {
'vals': record,
'env': self.env,
'_id': _id,
'remote': context.remote,
},
mode='exec',
)
new = self.env.ref('base_import_odoo.%s' % xmlid, False) new = self.env.ref('base_import_odoo.%s' % xmlid, False)
if new and new.exists(): if new and new.exists():
if self.duplicates == 'overwrite_empty': if self.duplicates == 'overwrite_empty':
@ -284,7 +309,7 @@ class ImportOdooDatabase(models.Model):
else: else:
logged = True logged = True
_logger.debug( _logger.debug(
'Got %s(%d[%d]) from idmap', model._model, _id,
'Got %s(%d[%d]) from idmap', model._name, _id,
record['id'] or 0, record['id'] or 0,
) )
if not _id: if not _id:
@ -294,7 +319,7 @@ class ImportOdooDatabase(models.Model):
elif not logged: elif not logged:
logged = True logged = True
_logger.debug( _logger.debug(
'Got %s(%d[%d]) from dummies', model._model, _id, record['id'],
'Got %s(%d[%d]) from dummies', model._name, _id, record['id'],
) )
if not _id: if not _id:
xmlid = self.env['ir.model.data'].search([ xmlid = self.env['ir.model.data'].search([
@ -309,17 +334,17 @@ class ImportOdooDatabase(models.Model):
logged = True logged = True
_logger.debug( _logger.debug(
'Got %s(%d[%d]) from mappings', 'Got %s(%d[%d]) from mappings',
model._model, _id, record['id'],
model._name, _id, record['id'],
) )
if not _id and create_dummy: if not _id and create_dummy:
_id = self._run_import_create_dummy( _id = self._run_import_create_dummy(
context, model, record, context, model, record,
forcecreate=record['id'] not in
forcecreate=record['id'] and record['id'] not in
self.status_data['ids'].get(model._name, []) self.status_data['ids'].get(model._name, [])
) )
elif _id and not logged: elif _id and not logged:
_logger.debug( _logger.debug(
'Got %s(%d[%d]) from xmlid', model._model, _id, record['id'],
'Got %s(%d[%d]) from xmlid', model._name, _id, record['id'],
) )
return _id return _id
@ -550,7 +575,7 @@ class ImportOdooDatabase(models.Model):
name: field name: field
for name, field for name, field
in self.env[context.model_line.model_id.model]._fields.items() in self.env[context.model_line.model_id.model]._fields.items()
if not field.compute or field.related
if not field.compute or field.inverse
} }
@api.multi @api.multi
@ -655,7 +680,7 @@ class ImportOdooDatabase(models.Model):
return self.env['ir.cron'].create({ return self.env['ir.cron'].create({
'name': self.display_name, 'name': self.display_name,
'model': self._name, 'model': self._name,
'function': '_run_import',
'function': '_run_import_cron',
'doall': True, 'doall': True,
'args': str((self.ids,)), 'args': str((self.ids,)),
}) })

4
base_import_odoo/models/import_odoo_database_field.py

@ -1,7 +1,7 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
# © 2017 Therp BV <http://therp.nl>
# Copyright 2017-2018 Therp BV <http://therp.nl>
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html). # License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html).
from openerp import api, fields, models
from odoo import api, fields, models
class ImportOdooDatabaseField(models.Model): class ImportOdooDatabaseField(models.Model):

9
base_import_odoo/models/import_odoo_database_model.py

@ -1,7 +1,7 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
# © 2017 Therp BV <http://therp.nl>
# Copyright 2017-2018 Therp BV <http://therp.nl>
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html). # License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html).
from openerp import fields, models
from odoo import fields, models
class ImportOdooDatabaseModel(models.Model): class ImportOdooDatabaseModel(models.Model):
@ -18,3 +18,8 @@ class ImportOdooDatabaseModel(models.Model):
ondelete='cascade', ondelete='cascade',
) )
domain = fields.Char(help='Optional filter to import only a subset') domain = fields.Char(help='Optional filter to import only a subset')
defaults = fields.Char(help='Optional defaults dict to avoid empty values')
postprocess = fields.Text(
help='Optional python code for postprocessing. Your code has access '
'to `vals` which is the dictionary passed to create/write, and `env`.',
)

4
base_import_odoo/models/ir_model_data.py

@ -1,7 +1,7 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
# © 2017 Therp BV <http://therp.nl>
# Copyright 2017-2018 Therp BV <http://therp.nl>
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html). # License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html).
from openerp import fields, models
from odoo import fields, models
class IrModelData(models.Model): class IrModelData(models.Model):

2
base_import_odoo/tests/__init__.py

@ -1,4 +1,4 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
# © 2017 Therp BV <http://therp.nl>
# Copyright 2017-2018 Therp BV <http://therp.nl>
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html). # License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html).
from . import test_base_import_odoo from . import test_base_import_odoo

7
base_import_odoo/tests/test_base_import_odoo.py

@ -1,8 +1,8 @@
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
# © 2017 Therp BV <http://therp.nl>
# Copyright 2017-2018 Therp BV <http://therp.nl>
# License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html). # License AGPL-3.0 or later (http://www.gnu.org/licenses/agpl.html).
from mock import patch from mock import patch
from openerp.tests.common import TransactionCase, post_install, at_install
from odoo.tests.common import TransactionCase, post_install, at_install
from ..models.import_odoo_database import ImportContext, field_context from ..models.import_odoo_database import ImportContext, field_context
@ -104,7 +104,8 @@ class TestBaseImportOdoo(TransactionCase):
self.assertFalse(demodb.cronjob_running) self.assertFalse(demodb.cronjob_running)
# in our setting we won't get dummies, so we test this manually # in our setting we won't get dummies, so we test this manually
import_context = ImportContext( import_context = ImportContext(
None, None, [], {}, {}, [], {}, field_context(None, None, None)
None, self.env.ref('base_import_odoo.model_partner'),
[], {}, {}, [], {}, field_context(None, None, None)
) )
dummy_id = demodb._run_import_create_dummy( dummy_id = demodb._run_import_create_dummy(
import_context, self.env['res.partner'], {'id': 424242}, import_context, self.env['res.partner'], {'id': 424242},

180
base_import_odoo/views/import_odoo_database.xml

@ -1,93 +1,95 @@
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<openerp>
<data>
<record id="view_import_odoo_database_tree" model="ir.ui.view">
<field name="model">import.odoo.database</field>
<field name="arch" type="xml">
<tree>
<field name="url" />
<field name="database" />
</tree>
</field>
</record>
<record id="view_import_odoo_database_form" model="ir.ui.view">
<field name="model">import.odoo.database</field>
<field name="arch" type="xml">
<form>
<header>
<button type="object" name="action_import" string="Run import" class="oe_highlight" attrs="{'invisible': [('cronjob_running', '=', True)]}" />
<button type="object" name="exists" string="Refresh" class="oe_highlight" attrs="{'invisible': [('cronjob_running', '=', False)]}" />
<field name="cronjob_running" invisible="1" />
</header>
<sheet>
<field name="status_html" attrs="{'invisible': [('status_html', '=', False)]}" />
<group col="4" name="credentials">
<field name="url" widget="url" attrs="{'readonly': [('cronjob_running', '=', True)]}" />
<field name="database" attrs="{'readonly': [('cronjob_running', '=', True)]}" />
<field name="user" attrs="{'readonly': [('cronjob_running', '=', True)]}" />
<field name="password" password="True" attrs="{'readonly': [('cronjob_running', '=', True)], 'required': [('cronjob_running', '=', False)]}" />
<field name="duplicates" />
<field name="cronjob_id" attrs="{'invisible': [('cronjob_id', '=', False)]}" />
</group>
<field name="import_line_ids" attrs="{'readonly': [('cronjob_running', '=', True)]}">
<tree editable="top">
<field name="sequence" widget="handle" />
<field name="model_id" />
<field name="domain" />
</tree>
</field>
<field name="import_field_mappings" attrs="{'readonly': [('cronjob_running', '=', True)]}"/>
</sheet>
</form>
</field>
</record>
<template id="view_import_odoo_database_qweb">
<script type="text/javascript">
function base_import_database_open(model, model_name, database_id)
<odoo>
<record id="view_import_odoo_database_tree" model="ir.ui.view">
<field name="model">import.odoo.database</field>
<field name="arch" type="xml">
<tree>
<field name="url" />
<field name="database" />
</tree>
</field>
</record>
<record id="view_import_odoo_database_form" model="ir.ui.view">
<field name="model">import.odoo.database</field>
<field name="arch" type="xml">
<form>
<header>
<button type="object" name="action_import" string="Run import" class="oe_highlight" attrs="{'invisible': [('cronjob_running', '=', True)]}" />
<button type="object" name="exists" string="Refresh" class="oe_highlight" attrs="{'invisible': [('cronjob_running', '=', False)]}" />
<field name="cronjob_running" invisible="1" />
</header>
<sheet>
<field name="status_html" attrs="{'invisible': [('status_html', '=', False)]}" />
<group col="4" name="credentials">
<field name="url" widget="url" attrs="{'readonly': [('cronjob_running', '=', True)]}" />
<field name="database" attrs="{'readonly': [('cronjob_running', '=', True)]}" />
<field name="user" attrs="{'readonly': [('cronjob_running', '=', True)]}" />
<field name="password" password="True" attrs="{'readonly': [('cronjob_running', '=', True)], 'required': [('cronjob_running', '=', False)]}" />
<field name="duplicates" />
<field name="cronjob_id" attrs="{'invisible': [('cronjob_id', '=', False)]}" />
</group>
<label for="import_line_ids" />
<field name="import_line_ids" attrs="{'readonly': [('cronjob_running', '=', True)]}">
<tree editable="top">
<field name="sequence" widget="handle" />
<field name="model_id" />
<field name="domain" />
<field name="defaults" />
<field name="postprocess" placeholder="# `vals` is the dictionary to be passed to create/write, and you have access to `env`, `_id`, `remote`" />
</tree>
</field>
<label for="import_field_mappings" />
<field name="import_field_mappings" attrs="{'readonly': [('cronjob_running', '=', True)]}"/>
</sheet>
</form>
</field>
</record>
<template id="view_import_odoo_database_qweb">
<script type="text/javascript">
function base_import_database_open(model, model_name, database_id)
{
return new openerp.web.Model('ir.model.data')
.query(['res_id'])
.filter([
['module', '=', 'base_import_odoo'],
['model', '=', model],
['import_database_id', '=', database_id],
])
.all()
.then(function(data)
{ {
return new openerp.web.Model('ir.model.data')
.query(['res_id'])
.filter([
['module', '=', 'base_import_odoo'],
['model', '=', model],
['import_database_id', '=', database_id],
])
.all()
.then(function(data)
{
return openerp.webclient.action_manager.do_action({
'name': model_name,
'type': 'ir.actions.act_window',
'views': [[false, 'list'], [false, 'form']],
'res_model': model,
'domain': [['id', 'in', _.map(data, function(x) {return x.res_id})]]
});
return openerp.webclient.action_manager.do_action({
'name': model_name,
'type': 'ir.actions.act_window',
'views': [[false, 'list'], [false, 'form']],
'res_model': model,
'domain': [['id', 'in', _.map(data, function(x) {return x.res_id})]]
}); });
}
</script>
<h2 t-if="object.cronjob_running">Import progress</h2>
<h2 t-if="not object.cronjob_running">Import results</h2>
<div style="display: flex; flex-flow: row wrap">
<div style="margin-right: .5em" t-foreach="object.import_line_ids" t-as="import_line">
<t t-set="model_name" t-value="import_line.model_id.model" />
<t t-set="model_display_name" t-value="import_line.model_id.name" />
<h3 t-esc="model_display_name" />
<a href="#" t-att-onclick="'base_import_database_open(&quot;%s&quot;, &quot;%s&quot;, %s)' % (model_name, model_display_name, object.id)">
<span t-esc="object.status_data.get('done', {}).get(model_name, 0)" />/<span t-esc="object.status_data.get('counts', {}).get(model_name, 0)" /> done
</a>
</div>
});
}
</script>
<h2 t-if="object.cronjob_running">Import progress</h2>
<h2 t-if="not object.cronjob_running">Import results</h2>
<div style="display: flex; flex-flow: row wrap">
<div style="margin-right: .5em" t-foreach="object.import_line_ids" t-as="import_line">
<t t-set="model_name" t-value="import_line.model_id.model" />
<t t-set="model_display_name" t-value="import_line.model_id.name" />
<h3 t-esc="model_display_name" />
<a href="#" t-att-onclick="'base_import_database_open(&quot;%s&quot;, &quot;%s&quot;, %s)' % (model_name, model_display_name, object.id)">
<span t-esc="object.status_data.get('done', {}).get(model_name, 0)" />/<span t-esc="object.status_data.get('counts', {}).get(model_name, 0)" /> done
</a>
</div> </div>
<t t-if="object.status_data.get('error')"><pre t-esc="object.status_data['error']" /></t>
<div t-if="object.status_data.get('dummies')">
The following remote ids don't have a mapping but have to be imported anyways due to not null constraints.
<dl>
<t t-foreach="object.status_data['dummies']" t-as="model_name">
<dt t-esc="model_name" />
<dd t-esc="object.status_data['dummies'][model_name]" />
</t>
</dl>
To fix this, create mappings for the remote ids listed, or if this is not feasible, map the whole model. You might also have a too specific domain on your import model definition.
</div>
</template>
</data>
</openerp>
</div>
<t t-if="object.status_data.get('error')"><pre t-esc="object.status_data['error']" /></t>
<div t-if="object.status_data.get('dummies')">
The following remote ids don't have a mapping but have to be imported anyways due to not null constraints.
<dl>
<t t-foreach="object.status_data['dummies']" t-as="model_name">
<dt t-esc="model_name" />
<dd t-esc="object.status_data['dummies'][model_name]" />
</t>
</dl>
To fix this, create mappings for the remote ids listed, or if this is not feasible, map the whole model. You might also have a too specific domain on your import model definition.
</div>
</template>
</odoo>

90
base_import_odoo/views/import_odoo_database_field.xml

@ -1,47 +1,45 @@
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<openerp>
<data>
<record id="view_import_odoo_database_field_form" model="ir.ui.view">
<field name="model">import.odoo.database.field</field>
<field name="arch" type="xml">
<form>
<group>
<field name="model_id" />
<field name="model" invisible="True" />
<field name="mapping_type" />
</group>
<div attrs="{'invisible': [('mapping_type', '!=', 'fixed')]}" class="oe_edit_only">
When a record of this model is imported, it will be replaced with the record you select as local ID. If you select a field and/or a remote ID, this replacement is only effective when setting the specified field and/or when the remote value is the specified record.
</div>
<div attrs="{'invisible': [('mapping_type', '!=', 'by_field')]}" class="oe_edit_only">
Select fields which must be equal to treat a pair of remote and local records of this model as being equal.
</div>
<div attrs="{'invisible': [('mapping_type', '!=', 'by_reference')]}" class="oe_edit_only">
Select the field that stores the model name and the one that stores the linked ID. The IDs for previously imported records will be mapped to the local ID, for unknown models or IDs, the fields are set to NULL.
</div>
<div attrs="{'invisible': [('mapping_type', '!=', 'unique')]}" class="oe_edit_only">
Select fields for which to generate unique values during import. You'll need this for res.users#login for example.
</div>
<group>
<field name="local_id" attrs="{'invisible': [('mapping_type', '!=', 'fixed')], 'required': [('mapping_type', '=', 'fixed')]}" />
<field name="remote_id" attrs="{'invisible': [('mapping_type', '!=', 'fixed')]}" />
<field name="field_ids" attrs="{'invisible': [('mapping_type', 'not in', ['fixed', 'by_field', 'unique'])], 'required': [('mapping_type', 'in', ['by_field', 'unique'])]}" widget="many2many_tags" domain="[mapping_type == 'fixed' and ('relation', '=', model) or ('model_id', '=', model_id)]"/>
<field name="model_field_id" attrs="{'invisible': [('mapping_type', '!=', 'by_reference')], 'required': [('mapping_type', '=', 'by_reference')]}" domain="[('ttype', '=', 'char'), ('model_id.model', '=', model)]" />
<field name="id_field_id" attrs="{'invisible': [('mapping_type', '!=', 'by_reference')], 'required': [('mapping_type', '=', 'by_reference')]}" domain="[('ttype', '=', 'integer'), ('model_id.model', '=', model)]" />
</group>
</form>
</field>
</record>
<record id="view_import_odoo_database_field_tree" model="ir.ui.view">
<field name="model">import.odoo.database.field</field>
<field name="arch" type="xml">
<tree>
<field name="sequence" widget="handle" />
<field name="model_id" />
<field name="mapping_type" />
<field name="field_ids" />
</tree>
</field>
</record>
</data>
</openerp>
<odoo>
<record id="view_import_odoo_database_field_form" model="ir.ui.view">
<field name="model">import.odoo.database.field</field>
<field name="arch" type="xml">
<form>
<group>
<field name="model_id" />
<field name="model" invisible="True" />
<field name="mapping_type" />
</group>
<div attrs="{'invisible': [('mapping_type', '!=', 'fixed')]}" class="oe_edit_only">
When a record of this model is imported, it will be replaced with the record you select as local ID. If you select a field and/or a remote ID, this replacement is only effective when setting the specified field and/or when the remote value is the specified record.
</div>
<div attrs="{'invisible': [('mapping_type', '!=', 'by_field')]}" class="oe_edit_only">
Select fields which must be equal to treat a pair of remote and local records of this model as being equal.
</div>
<div attrs="{'invisible': [('mapping_type', '!=', 'by_reference')]}" class="oe_edit_only">
Select the field that stores the model name and the one that stores the linked ID. The IDs for previously imported records will be mapped to the local ID, for unknown models or IDs, the fields are set to NULL.
</div>
<div attrs="{'invisible': [('mapping_type', '!=', 'unique')]}" class="oe_edit_only">
Select fields for which to generate unique values during import. You'll need this for res.users#login for example.
</div>
<group>
<field name="local_id" attrs="{'invisible': [('mapping_type', '!=', 'fixed')], 'required': [('mapping_type', '=', 'fixed')]}" />
<field name="remote_id" attrs="{'invisible': [('mapping_type', '!=', 'fixed')]}" />
<field name="field_ids" attrs="{'invisible': [('mapping_type', 'not in', ['fixed', 'by_field', 'unique'])], 'required': [('mapping_type', 'in', ['by_field', 'unique'])]}" widget="many2many_tags" domain="[mapping_type == 'fixed' and ('relation', '=', model) or ('model_id', '=', model_id)]"/>
<field name="model_field_id" attrs="{'invisible': [('mapping_type', '!=', 'by_reference')], 'required': [('mapping_type', '=', 'by_reference')]}" domain="[('ttype', '=', 'char'), ('model_id.model', '=', model)]" />
<field name="id_field_id" attrs="{'invisible': [('mapping_type', '!=', 'by_reference')], 'required': [('mapping_type', '=', 'by_reference')]}" domain="[('ttype', '=', 'integer'), ('model_id.model', '=', model)]" />
</group>
</form>
</field>
</record>
<record id="view_import_odoo_database_field_tree" model="ir.ui.view">
<field name="model">import.odoo.database.field</field>
<field name="arch" type="xml">
<tree>
<field name="sequence" widget="handle" />
<field name="model_id" />
<field name="mapping_type" />
<field name="field_ids" />
</tree>
</field>
</record>
</odoo>

20
base_import_odoo/views/menu.xml

@ -1,12 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<openerp>
<data>
<menuitem id="menu_main" parent="base.menu_administration" name="Remote Odoo import" sequence="5" />
<act_window id="action_import_odoo_database"
res_model="import.odoo.database"
name="Import configurations"
view_mode="tree,form"
/>
<menuitem id="menu_import_odoo_database" action="action_import_odoo_database" parent="menu_main" />
</data>
</openerp>
<odoo>
<menuitem id="menu_main" parent="base.menu_administration" name="Remote Odoo import" sequence="5" />
<act_window id="action_import_odoo_database"
res_model="import.odoo.database"
name="Import configurations"
view_mode="tree,form"
/>
<menuitem id="menu_import_odoo_database" action="action_import_odoo_database" parent="menu_main" />
</odoo>

1
requirements.txt

@ -11,3 +11,4 @@ raven
checksumdir checksumdir
python-stdnum python-stdnum
pyopenssl pyopenssl
odoorpc
Loading…
Cancel
Save