new: [docker-host] add yaml.nanorc to handle coloration with yml file in nano

This involves a deep upgrade. Changing base distribution from debian
towards alpine, and changing the way the schedule-command relation is
handled. Consequences:
- size of image went from 195MB to 45MB
- build process simplified (only one small ``Dockerfile``)
- dependencies are not in the charm anymore but fetched on build
  from our download server
- local cron business intelligence was moved in the cron charm
  - extreme simplification for implementing run-once charms relation with
    cron.
  - some simplification for charms that adds some intelligence on their
    side.

README.org

@@ -81,61 +81,121 @@ defined, most tools will look in =/srv/charm-store= by default.
 
 ** charm type
 
-Not all charm are intended to bring up services as having a container
-always running and listening.
+Not all charm are designed to set up a continuously running, listening
+service.
 
-In ~metadata.yml~, the root level ~type~ can be one of:
+In a charm's ~metadata.yml~, the root-level key ~type~ can have one of
+these values:
 
-- ~service~ (default)
+- ~daemon~ (default)
 
-  If not specified, this is the default. A charm brings up a service.
-  It is meant to be *always running*. For instance, ~apache~, ~mysql~,
- ~postgres~ are services.
+  By default, a charm is of type ~daemon~. It's probably the most
+  expected way to run a service: it brings up a process that is
+ *always running*. Examples include charms like ~apache~, ~mysql~,
+ ~postgres~.
 
-  They usually open ports and are listening to provide their service,
-  or carry background listening of other ressources (like checking
-  time and sending scheduling command for the ~cron~ services), and or
-  use files to trigger or report on their activity.
+  These charms bring up processes that typically open ports to provide
+  their functionality, perform background tasks like checking the time
+  and scheduling commands (as the ~cron~ charm), and may use files to
+  trigger or report on their activities.
 
-  It will have an entry in the final ~docker-compose.yml~, and thus, a
-  container will run and stay in memory and have a ~restart:
-  unless-stopped~ policy. They use CPU and memory ressources.
+  In the final ~docker-compose.yml~, a ~daemon~ type charm will
+  ensure that an entry is created for the service they manage,
+  resulting in a container that stays in memory. As such they require
+  a docker image. They will ensure that these entries are managed with
+ ~restart: unless-stopped~ policy.
 
-- ~run-once~
+  The processes managed by these charms will be setup via
+ ~docker-compose up~ actions at the end, and they will run in the
+  background.
 
-  The entry is meant to describe *a command that run once*,
-  it will be called by a service and *will exit after execution*.
+  Once brought up, the processes from these charms will consume CPU and
+  memory resources indefinitely, until you manually bring them down.
 
-  For instance, ~logrotate~, ~rsync-backup~, or ~letsencrypt~ are
-  of type ~run-once~.
+  It makes sense to bring them ~up~ or ~down~.
 
-  They are meant to be run by service for specific events. They
-  usually will use relations to ensure they are called at specific
-  moment by service...
+- ~command~
 
-  A command does not have an automatic ~restart~ policy as services
-  have.
+  This charm type is used to prepare *a process that run and exits
+  after execution*. These are more what could be expected of a
+  "command", and are typically invoked by an other service for
+  specific events.
 
-  They use CPU and memory ressources only when run and gives them back
-  once finished.
+  Example includes ~logrotate~, ~rsync-backup~, and ~letsencrypt~,
+  which are charms of type ~run-once~.
+
+  These charms are meant to setup commands that are triggered by
+  services at specific moments or as a result of specific event. It is
+  through their ~relation~ hooks with other services that they will
+  ensure to be called when intended to. They are run through the
+ ~docker-compose run~ call.
+
+  Like ~daemon~'s typed charm, these charm will ensure that an entry
+  is correctly added in the final ~docker-compose.yml~ with all the
+  necessary options so it is ready to be triggered. They require also
+  a docker image.
+
+  But unlike ~daemon~'s typed charms, these charm will ensure that
+  the entry they managed in the final ~docker-compose.yml~ *DO NOT*
+  have an automatic restart policy.
+
+  They consume CPU and memory resources only when running and release
+  resources once finished.
 
 - ~stub~
 
-  The entry describes an entity that will *not be run at all*. It is
-  used to hold information in the ~compose.yml~ and often to *stand
-  for* a real service managed outside of ~compose.yml~ (on an other
-  host or on a different managing system, like a local installation or
-  LXC, virtualbox, ...).
+  A ~stub~ charm is more of a placeholder that doesn't have anything
+  to run at all ! They don't need any docker image. These entities are
+  used to hold information in ~compose.yml~ and can often be used to
+  represent a real service managed externally (out of =compose=, on
+  another host or through a different management system, such as a
+  local installation, LXC, VirtualBox, etc.).
+
+  For example, ~smtp-stub~ charm can be used to build an entity that
+  will stand for an external ~smtp~ service. Through relations, these
+  stubs offer interfaces similar to actual services in the setting up
+  stage. For instance, a ~smtp-stub~ acts as a ~smtp-server~ provider,
+  and can satisfy ~services~ that would require a ~smtp-server~
+  provider.
+
+  They generally implement relation hooks and act as providers.
+
+  No entry is created for them in the final ~docker-compose.yml~.
+
+  They do not use any CPU or memory resources
+
+
+** login and password policy
+
+A charm have to manage different set of password. The best would be
+that the charm:
+
+- don't require user to choose password (less configuration)
+- will promote reasonable security practice.
+
+There are 2 types of password:
+- inter-service passwords (ie: database access password), these are
+  never used by human operator, and will be required to be known by
+  the charms to set things up.  These should be generated randomly
+  (although they could be set also via configuration if mentionned).
+  - they can only be changed by specific backend technical manipulation.
 
-  For instance, ~stmp-stub~ can be used to stand for an external ~smtp~.
+- user service's admin password (ie: admin user of odoo, nextcloud)
+  - they can be changed through the service interface.
+  - this service interface is available to the public and the general users.
+  - charm doesn't need the password to set things up around the service.
 
-  It is through their relation that they shine as they can provide
-  similar interface than actual services would have
-  provided. ~smtp-stub~ is a ~smtp-server~ provider and other charm
-  can connect to it.
+*** Inter-service passwords
 
-  They usually implement relation hooks, and are providers.
+- Login should be defaulted to name of the service when possible
+- Should be defaulted to random values if not provided in configuration.
+- Should not be advertised even in the command line interface.
+- Should be reset-able anytime.
 
-  No entry will be created in the final ~docker-compose.yml~.
+*** Interactive admin user service's password
 
-  They use no CPU or memory ressources at all.
+- Login should be defaulted to 'admin'
+- Should be defaulted to random values, and not be configurable in configuration.
+- Should be advertised at the end of ~compose up~ along with URL of services as long
+  as the default value chosen by compose is still working.
+- Should not be advertised once it was changed by user.

apache/README.org

@@ -0,0 +1,188 @@
+
+
+* Usage
+
+Other services will often require a service managed with this charm to
+act as a HTTP/HTTPS front-end. It can provide certificates with HTTPS.
+
+
+** Domain assignment
+
+Services using relation =web-proxy= or =publish-dir= will be required
+to be assigned a domain name for the virtual host that will be
+created.
+
+*** Domain sources
+
+This domain name can be set (in order of priority), the first source
+giving a name will be taken.
+
+- *Relation's options* (=web-proxy= or =publish-dir=)
+  Using =domain= option, and optionally the deprecated
+ =server-aliases= for additional names.
+
+  #+begin_src yaml
+  myservice:
+    # ...
+    relations:
+      web-proxy:
+        apache:
+          domain: mydomain.org
+          #server-aliases:
+          #  - www.mydomain.org
+          #  - pro.mydomain.org
+  #+end_src
+- *Apache service's options*, using a =service-domain-name= mapping:
+
+  #+begin_src yaml
+  myservice:
+    # ...
+  apache:
+    options:
+      service-domain-map:
+        # ...
+        myservice:
+          - mydomain.org
+          - www.mydomain.org
+          - pro.mydomain.org
+        # ...
+  #+end_src
+
+- *the service name* itself if is a domain name:
+
+  #+begin_src yaml
+  www.mydomain.org:
+    # ...
+  #+end_src
+
+  Please note that this is not recommended, and will be deprecated.
+
+*** Domain and alternate domains
+
+Every source (except the one coming out from the domain name), can use
+several ways to provide *more than one domain name*.
+
+Please remember:
+- At least one domain name needs to be provided
+- and the first domain can't use wildcards and will be considered the main domain name.
+
+If other domains are specified, they will be used as aliases, and
+wildcard (using ~*~) is supported.
+
+Additionally, bash braces expansion and regex matching are
+available. Space separated YAML string or YAML sequences are
+supported, also as mix of both.
+
+As examples, notice the following are equivalent and will serve
+=myservice= on the exact same set of domain names:
+
+#+begin_src yaml
+myservice:
+  relations:
+    web-proxy:
+      domain:
+        ## A yaml list
+        - myservice.home.org
+        - mydomain.org
+        - www.mydomain.org
+        - pro.mydomain.org
+        - *.myservice.hop.org
+#+end_src
+
+
+#+begin_src yaml
+myservice:
+  # ... no domain set in relation
+apache:
+  options:
+    service-domain-map:
+      ## A yaml list as a mapping value
+      myservice:
+        - myservice.home.org
+        - {,www.,pro.}mydomain.org  ## bash braces expansion used
+        - *.myservice.hop.org
+#+end_src
+
+#+begin_src yaml
+myservice:
+  # ...
+apache:
+  options:
+    service-domain-map:
+      ## space separated YAML string and bash braces expansion
+      myservice: myservice.home.org {,www.,pro.}mydomain.org *.myservice.hop.org
+#+end_src
+
+#+begin_src yaml
+myservice:
+  # ...
+apache:
+  options:
+    service-domain-map:
+      ## Leveraging bash braces expansion and regex replacement
+      .*: {$0.home,{,www.,pro.}mydomain,*.$0.hop}.org
+#+end_src
+
+** Domain mapping
+
+You can automatically assign a domain to services in relation
+=web-proxy= or =publish-dir= with services managed by this charm using
+the =service-domain-name= option. For instance:
+
+#+begin_src yaml
+apache:
+  options:
+    service-domain-map:
+      .*: $0.mydomain.org
+#+end_src
+
+Where ~mydomain.org~ stands for the domain where most of your services
+will be served. You can override this behavior for some services:
+- by adding a matching rule *before* the given rule.
+- by specifying a =domain= in the relation's options.
+
+first rule matching will end the mapping:
+
+#+begin_src yaml
+apache:
+  options:
+    service-domain-map:
+      foo: www.mydomain.org
+      bar: beta.myotherdomain.com
+#+end_src
+
+Allows to distribute services to domains quite freely.
+
+
+* SSH Tunnel
+
+On the server side, you can configure your compose file::
+
+#+begin_src yaml
+apache:
+  options:
+    ssh-tunnel:
+      domain: ssh.domain.com     ## required
+      #ssl: ...                  ## required, but automatically setup if you
+                                 ## provide a ``cert-provider`` to ``apache``.
+#+end_src
+
+
+On the client side you should add this to your ``~/.ssh/config``::
+
+#+begin_src conf-space
+Host ssh.domain.com
+    Port 443
+    ProxyCommand proxytunnel -q -E -p ssh.domain.com:443 -d ssh.domain.com:22
+    DynamicForward 1080
+    ServerAliveInterval 60
+#+end_src
+
+If it doesn't work, you can do some checks thanks to this command::
+
+#+begin_example
+$ proxytunnel -E -p ssh.domain.com:443 -d ssh.domain.com:22 -v \
+    -H "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Win32)\n"
+#+end_example
+
+

apache/README.rst

@@ -1,29 +0,0 @@
-
-
-SSH Tunnel
-----------
-
-On the server side, you can configure your compose file::
-
-    apache:
-      options:
-        ssh-tunnel:
-          domain: ssh.domain.com     ## required
-          #ssh: ...                  ## required, but automatically setup if you
-                                     ## provide a ``cert-provider`` to ``apache``.
-
-
-On the client side you should add this to your ``~/.ssh/config``::
-
-    Host ssh.domain.com
-        Port 443
-        ProxyCommand proxytunnel -q -E -p ssh.domain.com:443 -d ssh.domain.com:22
-        DynamicForward 1080
-        ServerAliveInterval 60
-
-If it doesn't work, you can do some checks thanks to this command::
-
-    $ proxytunnel -E -p ssh.domain.com:443 -d ssh.domain.com:22 -v \
-        -H "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Win32)\n"
-
-

apache/hooks/publish_dir-relation-joined

@@ -6,8 +6,10 @@
 
 set -e
 
+service_def=$(get_compose_service_def "$SERVICE_NAME")
+
 cfg=$(relation-get)
-apache_publish_dir "$cfg"
+apache_publish_dir "$cfg" "$service_def"
 
 APACHE_CORE_RULES=$(relation-get apache-core-rules 2>/dev/null) || true
 if [ "$APACHE_CORE_RULES" ]; then

apache/hooks/web_proxy-relation-joined

@@ -5,8 +5,10 @@ set -e
 
 . lib/common
 
+service_def=$(get_compose_service_def "$SERVICE_NAME")
+
 cfg=$(relation-get)
-apache_proxy_dir "$cfg"
+apache_proxy_dir "$cfg" "$service_def"
 
 APACHE_CORE_RULES=$(relation-get apache-core-rules 2>/dev/null) || true
 if [ "$APACHE_CORE_RULES" ]; then

apache/lib/common

@@ -1,22 +1,163 @@
 # -*- mode: shell-script -*-
 
+replace_by_rematch_pattern() {
+    local input="$1" output="" char next_char i
+    # Loop through each character in the string
+    for (( i=0; i<${#input}; i++ )); do
+        char="${input:$i:1}"
 
-get_domain() {
-    local cfg="$1" cache_file="$state_tmpdir/$FUNCNAME.cache.$(H "$SERVICE_NAME" "$MASTER_BASE_SERVICE_NAME" "$@")" \
-          domain
+        # If a dollar sign is found
+        if [[ "$char" == '$' ]]; then
+            next_char="${input:$((i+1)):1}"
+            # Check if next character is a digit
+            if [[ "$next_char" =~ [0-9] ]]; then
+                # Replace $N with ${rematch[N]}
+                output+='${rematch['"$next_char"']}'
+                ((i++)) # Skip next character as it's already processed
+                continue
+            fi
+        fi
+
+        output+="$char"
+    done
+
+    echo "$output"
+}
+export -f replace_by_rematch_pattern
+
+
+
+get_domains() {
+    local cfg="$1" service_cfg="$2" cache_file="$state_tmpdir/$FUNCNAME.cache.$(H "$SERVICE_NAME" "$MASTER_BASE_SERVICE_NAME" "$@")" \
+          domains new_domains maps m regex rematch type value key domain
     if [ -e "$cache_file" ]; then
         cat "$cache_file"
         return 0
     fi
-    domain=$(e "$cfg" | cfg-get-value domain 2>/dev/null) || true
-    if [ "$domain" ]; then
-        echo "$domain" | tee "$cache_file"
-    elif [[ "$BASE_SERVICE_NAME" =~ ^([a-z0-9]+(-[a-z0-9]+)*\.)+[a-z]{2,}$ ]]; then
-        echo "$BASE_SERVICE_NAME" | tee "$cache_file"
-    else
-        err "You must specify a ${WHITE}domain$NORMAL option. (${FUNCNAME[*]})"
+    domains=()
+    for key in domain server-aliases; do
+        type=$(e "$cfg" | shyaml -q get-type "$key" 2>/dev/null) || true
+        value=$(e "$cfg" | shyaml -q get-value -y "$key" 2>/dev/null) || true
+        while true; do
+            case "$type" in
+                sequence)
+                    while read-0 domain; do
+                        if [[ "$domain" != "None" ]]; then
+                            if [ "${#domains[@]}" == 0 ] && [ "$key" == "server-aliases" ]; then
+                                err "You can't specify server aliases if you don't have a domain."
+                                return 1
+                            fi
+
+                            domains+=("$domain")
+                        fi
+                    done < <(e "$value" | shyaml get-values-0)
+                    ;;
+                str)
+                    if ! domain=$(e "$value" | shyaml get-value 2>/dev/null); then                        
+                        err "Failed to get domain value from config."
+                        return 1
+                    fi
+                    [[ "$domain" == "" ]] && break
+                    if [ "${#domains[@]}" == 0 ] && [ "$key" == "server-aliases" ]; then
+                        err "You can't specify server aliases if you don't have a domain."
+                        return 1
+                    fi
+
+                    domains+=("$domain")
+                    ;;
+                NoneType|"")
+                    :
+                    ;;
+                \!*)
+                    value=$(e "$value" | cfg-get-value "$key" 2>/dev/null) || true
+                    type=$(e "$value" | shyaml -q get-type) || true
+                    continue
+                    ;;
+            esac
+            break
+        done
+        ## check and expand domain
+        new_domains=()
+        for domain in "${domains[@]}"; do
+            if ! [[ "$domain" =~ ^[a-z0-9\{\}*\ \,.-]+$ ]]; then
+                err "Invalid domain value '$domain' expression in ${WHITE}$key$NORMAL option."
+                return 1
+            fi
+            new_domains+=($(eval echo "${domain//\*/\\*}"))
+        done
+        domains=("${new_domains[@]}")
+    done
+    ## Fill with service-domain-map
+    if [ "${#domains[@]}" == 0 ] &&
+           service_domain_map=$(e "$service_cfg" | cfg-get-value 'options.service-domain-map' 2>/dev/null) &&
+           [ -n "$service_domain_map" ]; then
+        while read-0 regex map; do
+            if [[ "$BASE_SERVICE_NAME" =~ ^$regex$ ]]; then
+                rematch=("${BASH_REMATCH[@]}")
+                maps=()
+                type=$(e "$map" | shyaml -q get-type 2>/dev/null) || true
+                value=$(e "$map" | shyaml -q get-value -y 2>/dev/null) || true
+                case "$type" in
+                    sequence)
+                        while read-0 m; do
+                            if [[ "$m" != "None" ]] && [ -n "$m" ]; then
+                                maps+=("$m")
+                            fi
+                        done < <(e "$value" | shyaml get-values-0)
+                        ;;
+                    str)
+                        if ! m=$(e "$value" | shyaml get-value 2>/dev/null); then
+                            err "Failed to get mapping value from config."
                             return 1
                         fi
+                        [ -z "$m" ] && continue
+                        maps+=("$m")
+                        ;;
+                    NoneType|"")
+                        :
+                        ;;
+                esac
+                for map in "${maps[@]}"; do 
+                    if ! [[ "$map" =~ ^([a-z0-9*\{\}\ \,.-]|\$[0-9])+$ ]]; then
+                        err "Invalid mapping value '$map' in ${WHITE}service-domain-map$NORMAL option."
+                        return 1
+                    fi
+
+                    map="${map//\*/\\*}"  ## protect star from eval
+                    map=$(replace_by_rematch_pattern "$map")
+
+                    domains+=($(set -- "${BASH_REMATCH[@]}"; eval echo "${map//\*/\\*}"))
+                done
+                break
+            fi
+        done < <(e "$service_domain_map" | yq e -0 'to_entries | .[] | [.key, .value] |.[]')
+    fi
+    if [ "${#domains[@]}" == 0 ] &&
+           [[ "$BASE_SERVICE_NAME" =~ ^([a-z0-9]+(-[a-z0-9]+)*\.)+[a-z]{2,}$ ]]; then
+        domains+=("$BASE_SERVICE_NAME")
+    fi
+    new_domains=()
+    ## remove duplicates
+    for domain in "${domains[@]}"; do
+        if ! [[ " ${new_domains[*]} " == *" $domain "* ]]; then
+            new_domains+=("$domain")
+        fi
+    done
+    domains=("${new_domains[@]}")
+    if [ "${#domains[@]}" == 0 ]; then
+        err "No domain name set for your service ${YELLOW}$BASE_SERVICE_NAME${NORMAL}."
+        echo "  You can specify a ${WHITE}domain$NORMAL option in the" \
+             "relation with ${YELLOW}$TARGET_SERVICE_NAME${NORMAL}." >&2
+        echo "  Or you can specify a service to domain mapping in" \
+             "${WHITE}service-domain-map${NORMAL} option of ${YELLOW}$TARGET_SERVICE_NAME${NORMAL}." >&2
+        return 1
+    fi
+    ## check that first domain should not have a wildcard
+    if [[ "${domains[0]}" == *"*"* ]]; then
+        err "First domain name '${domains[0]}' can't contain a wildcard."
+        return 1
+    fi
+    echo "${domains[@]}" | tee "$cache_file"
 }
 
 
@@ -25,17 +166,17 @@ get_domain() {
 ##
 
 apache_proxy_dir() {
-    local cfg="$1" domain
-    apache_vhost_create web_proxy "$cfg" || return 1
+    local cfg="$1" service_cfg="$2"
+    apache_vhost_create web_proxy "$cfg" "$service_cfg" || return 1
 }
 export -f apache_proxy_dir
 
 
 apache_publish_dir() {
-    local cfg="$1" domain
-    apache_vhost_create publish_dir "$cfg" || return 1
-    apache_code_dir "$cfg" || return 1
-    apache_data_dirs "$cfg"
+    local cfg="$1" service_cfg="$2"
+    apache_vhost_create publish_dir "$cfg" "$service_cfg" || return 1
+    apache_code_dir "$cfg" "$service_cfg" || return 1
+    apache_data_dirs "$cfg" "$service_cfg"
 }
 export -f apache_publish_dir
 
@@ -51,7 +192,7 @@ apache_ssh_tunnel() {
         err "${WHITE}ssl${NORMAL} must be valued in ${WHITE}ssh-tunnel${NORMAL} config."
         return 1
     fi
-    apache_vhost_create ssh_tunnel "$cfg" ",https," "000-$domain" || return 1
+    apache_vhost_create ssh_tunnel "$cfg" "" ",https," "000-$domain" || return 1
 }
 export -f apache_publish_dir
 
@@ -62,7 +203,7 @@ export -f apache_publish_dir
 
 
 apache_vhost_create() {
-    local type="$1" cfg="$2" protocols="$3" dest="$4" custom_rules vhost_statement creds \
+    local type="$1" cfg="$2" service_cfg="$3" protocols="$4" dest="$5" custom_rules vhost_statement creds \
           redirect domain ssl_plugin_fun ssl_cfg_value ssl_cfg_options
 
     export APACHE_CONFIG_LOCATION="$SERVICE_CONFIGSTORE/etc/apache2/sites-enabled"
@@ -73,21 +214,21 @@ apache_vhost_create() {
         protocols=$(__vhost_cfg_normalize_protocol "$cfg") || return 1
     fi
 
-    domain=$(get_domain "$cfg") && {
-        [ "$RELATION_DATA_FILE" ] && relation-set domain "$domain"
+    domains=($(get_domains "$cfg" "$service_cfg")) && {
+        [ "$RELATION_DATA_FILE" ] && relation-set domain "${domains[0]}"
     }
-
+    echo "Domains: ${domains[*]}" >&2
     if is_protocol_enabled https "$protocols"; then
         [ "$RELATION_DATA_FILE" ] && {
-            relation-set url "https://$domain"
+            relation-set url "https://${domains[0]}"
         }
 
-        if [ -z "$domain" ]; then
+        if [ "${#domains[@]}" == 0 ]; then
             err "You must specify a domain for ssl to work."
             return 1
         fi
         read-0 ssl_plugin_fun ssl_cfg_value ssl_cfg_options < <(ssl_get_plugin_fun "$cfg") || return 1
-        "$ssl_plugin_fun"_vars "$cfg" "$ssl_cfg_options" "$ssl_cfg_value" "$domain" || return 1
+        "$ssl_plugin_fun"_vars "$cfg" "$ssl_cfg_options" "$ssl_cfg_value" "${domains[*]}" || return 1
         redirect=$(e "$cfg" | cfg-get-value 'redirect-to-ssl' 2>/dev/null) || true
         if is_protocol_enabled http "$protocols"; then
             redirect=${redirect:-true}
@@ -116,15 +257,15 @@ $(if [ "$custom_rules" ]; then
         fi
     else
         [ "$RELATION_DATA_FILE" ] && {
-            relation-set url "http://$domain"
+            relation-set url "http://${domains[0]}"
         }
     fi
-    vhost_statement=$(apache_vhost_statement "$type" "$protocols" "$cfg" "$domain") || {
+    vhost_statement=$(apache_vhost_statement "$type" "$protocols" "$cfg" "${domains[*]}") || {
         err "Failed to get vhost statement for type $type on ${protocols:1:-1}"
         return 1
     }
 
-    dest=${dest:-$domain}
+    dest=${dest:-${domains[0]}}
     if [ -z "$dest" ]; then
         err "Please set either a domain or set a destination file."
         return 1
@@ -137,7 +278,7 @@ $(if [ "$custom_rules" ]; then
     fi
 
     if is_protocol_enabled https "$protocols"; then
-        "$ssl_plugin_fun"_prepare "$cfg" "$ssl_cfg_options" "$ssl_cfg_value"  || return 1
+        "$ssl_plugin_fun"_prepare "$cfg" "$ssl_cfg_options" "$ssl_cfg_value" "${domains[*]}" || return 1
     fi
 }
 
@@ -159,7 +300,7 @@ _get_ssl_option_value() {
     fi
 
     if ssl_cfg=$(e "$cfg" | cfg-get-value ssl 2>/dev/null); then
-        if [[ "$ssl_cfg" =~ ^False|None$ ]]; then
+        if [[ "$ssl_cfg" =~ ^False|None|false|null$ ]]; then
             ssl_cfg=""
         fi
         echo "$ssl_cfg" | tee "$cache_file"
@@ -249,7 +390,7 @@ ssl_get_plugin_fun() {
     }
 
     type="$(echo "$cfg" | shyaml -y get-type 2>/dev/null)" || return 1
-    if [[ "$type" == "bool" ]]; then
+    if [[ "$type" == "bool" ]] || [[ "$type" == "str" && "$cfg" =~ ^false|true$ ]]; then
         printf "%s\0" "ssl_fallback" "" "$cfg" | tee "$cache_file"
         return 0
     fi
@@ -296,18 +437,18 @@ ssl_get_plugin_fun() {
 
 
 ssl_fallback_vars() {
-    local cfg="$1" ssl_cfg="$2" value="$3" domain="$4" cert key ca_cert domain
-
+    local cfg="$1" ssl_cfg="$2" value="$3" domains="$4" cert key ca_cert
+    domains=($domains)
     if __vhost_cfg_ssl_cert=$(echo "$ssl_cfg" | shyaml get-value cert 2>/dev/null); then
-        __vhost_cfg_SSL_CERT_LOCATION=/etc/ssl/certs/${domain}.pem
+        __vhost_cfg_SSL_CERT_LOCATION=/etc/ssl/certs/${domains[0]}.pem
     fi
 
     if __vhost_cfg_ssl_key=$(echo "$ssl_cfg" | shyaml get-value key 2>/dev/null); then
-        __vhost_cfg_SSL_KEY_LOCATION=/etc/ssl/private/${domain}.key
+        __vhost_cfg_SSL_KEY_LOCATION=/etc/ssl/private/${domains[0]}.key
     fi
 
     if __vhost_cfg_ssl_ca_cert=$(echo "$ssl_cfg" | shyaml get-value ca-cert 2>/dev/null); then
-        __vhost_cfg_SSL_CA_CERT_LOCATION=/etc/ssl/certs/${domain}-ca.pem
+        __vhost_cfg_SSL_CA_CERT_LOCATION=/etc/ssl/certs/${domains[0]}-ca.pem
     fi
 }
 
@@ -338,29 +479,20 @@ $volumes
 }
 
 ssl_plugin_cert-provider_vars() {
-    local cfg="$1" ssl_cfg="$2" value="$3" domain="$4"
-
-    __vhost_cfg_SSL_CERT_LOCATION=/etc/letsencrypt/live/${domain}/cert.pem
-    __vhost_cfg_SSL_KEY_LOCATION=/etc/letsencrypt/live/${domain}/privkey.pem
-    __vhost_cfg_SSL_CHAIN=/etc/letsencrypt/live/${domain}/chain.pem
+    local cfg="$1" ssl_cfg="$2" value="$3" domains="$4"
+    domains=($domains)
+    __vhost_cfg_SSL_CERT_LOCATION=/etc/letsencrypt/live/${domains[0]}/cert.pem
+    __vhost_cfg_SSL_KEY_LOCATION=/etc/letsencrypt/live/${domains[0]}/privkey.pem
+    __vhost_cfg_SSL_CHAIN=/etc/letsencrypt/live/${domains[0]}/chain.pem
 }
 
 ssl_plugin_cert-provider_prepare() {
-    local cfg="$1" ssl_cfg="$2" service="$3"  options domain server_aliases
-
-    domain=$(get_domain "$cfg") || return 1
-
+    local cfg="$1" ssl_cfg="$2" service="$3" domains="$4" options  server_aliases
+    domains=($domains)
     options=$(yaml_key_val_str "options" "$ssl_cfg") || return 1
     service_config=$(yaml_key_val_str "$service" "$options")
-    server_aliases=$(e "$cfg" | cfg-get-value server-aliases 2>/dev/null) || true
-    [ "$server_aliases" == None ] && server_aliases=""
-    if [ "$server_aliases" ]; then
-        server_aliases=($(echo "$server_aliases" | shyaml get-values)) || return 1
-    else
-        server_aliases=()
-    fi
     compose --debug --add-compose-content "$service_config" crt "$service" \
-            create "$domain" "${server_aliases[@]}" || {
+            create "${domains[@]}" || {
         err "Failed to launch letsencrypt for certificate creation."
         return 1
     }
@@ -399,16 +531,16 @@ apache_passwd_file() {
 
 ## Produce the full statements depending on relation-get informations
 apache_vhost_statement() {
-    local type="$1" protocols="$2" cfg="$3" domain="$4" \
+    local type="$1" protocols="$2" cfg="$3" domains="$4" \
           vhost_statement
 
     if is_protocol_enabled http "$protocols"; then
-        __vhost_full_vhost_statement "$type" http "$cfg" "$domain" || return 1
+        __vhost_full_vhost_statement "$type" http "$cfg" "$domains" || return 1
     fi
     if is_protocol_enabled https "$protocols"; then
         read-0 ssl_plugin_fun ssl_cfg_value ssl_cfg_options < <(ssl_get_plugin_fun "$cfg") || return 1
-        "$ssl_plugin_fun"_vars "$cfg" "$ssl_cfg_options" "$ssl_cfg_value" "$domain" || return 1
-        vhost_statement=$(__vhost_full_vhost_statement "$type" https "$cfg" "$domain") || return 1
+        "$ssl_plugin_fun"_vars "$cfg" "$ssl_cfg_options" "$ssl_cfg_value" "$domains" || return 1
+        vhost_statement=$(__vhost_full_vhost_statement "$type" https "$cfg" "$domains") || return 1
         cat <<EOF
 
 <IfModule mod_ssl.c>
@@ -421,13 +553,13 @@ export -f apache_vhost_statement
 
 
 apache_code_dir() {
-    local cfg="$1" www_data_gid local_path
+    local cfg="$1" service_cfg="$2" www_data_gid local_path
     www_data_gid=$(cached_cmd_on_base_image "$TARGET_SERVICE_NAME" 'id -g www-data') || {
         debug "Failed to query for www-data gid in ${DARKYELLOW}$TARGET_SERVICE_NAME${NORMAL} base image."
         return 1
     }
-    domain=$(get_domain "$cfg") || return 1
-    local_path="/var/www/${domain}"
+    domains=($(get_domains "$cfg" "$service_cfg")) || return 1
+    local_path="/var/www/${domains[0]}"
     host_path=$(e "$cfg" | cfg-get-value location 2>/dev/null) ||
         host_path="$DATASTORE/$BASE_SERVICE_NAME${local_path}"
 
@@ -440,6 +572,9 @@ apache_code_dir() {
         "$CONFIGSTORE"*)
             docker_host_path="$HOST_CONFIGSTORE${host_path##$CONFIGSTORE}"
             ;;
+        *)
+            docker_host_path="$host_path"
+            ;;
     esac
 
     mkdir -p "$host_path" || return 1
@@ -452,13 +587,13 @@ $SERVICE_NAME:
 }
 
 apache_data_dirs() {
-    local cfg="$1" data_dirs dst data fdir to_create
+    local cfg="$1" service_cfg="$2" data_dirs dst data fdir to_create
     data_dirs=$(e "$cfg" | cfg-get-value data-dirs 2>/dev/null | shyaml get-values 2>/dev/null) || true
     if [ -z "$data_dirs" ]; then
         return 0
     fi
-    domain=$(get_domain "$cfg") || return 1
-    local_path="/var/www/${domain}"
+    domains=($(get_domains "$cfg" "$service_cfg")) || return 1
+    local_path="/var/www/${domains[0]}"
     dst=$DATASTORE/$BASE_SERVICE_NAME$local_path
     data=()
     while IFS="," read -ra addr; do
@@ -574,26 +709,15 @@ EOF
 
 
 __vhost_head_statement() {
-    local cfg="$1" protocol="$2" domain="$3" server_aliases admin_mail prefix
+    local cfg="$1" protocol="$2" domains="$3" server_aliases admin_mail prefix
+    domains=($domains)
     admin_mail=$(e "$1" | cfg-get-value "admin-mail" 2>/dev/null) || true
     if [ -z "$admin_mail" ]; then
-        if [ -z "$domain" ]; then
+        if [ "${#domains[@]}" == 0 ]; then
             admin_mail=webmaster@localhost
         else
-            admin_mail=${admin_mail:-contact@$domain}
-        fi
+            admin_mail=${admin_mail:-contact@${domains[0]}}
         fi
-
-    server_aliases=$(e "$cfg" | cfg-get-value server-aliases 2>/dev/null) || true
-    [ "$server_aliases" == None ] && server_aliases=""
-    if [ "$server_aliases" ]; then
-        server_aliases=($(e "$server_aliases" | shyaml get-values)) || return 1
-        if [ -z "$domain" ]; then
-            err "You can't specify server aliases if you don't have a domain."
-            return 1
-        fi
-    else
-        server_aliases=()
     fi
 
     if [ "$protocol" == "https" ]; then
@@ -602,8 +726,8 @@ __vhost_head_statement() {
         prefix=
     fi
 
-    if [ "$domain" ]; then
-        log_prefix="${prefix}${domain}_"
+    if [ "${#domains[@]}" != 0 ]; then
+        log_prefix="${prefix}${domains[0]}_"
     else
         log_prefix=""
     fi
@@ -611,8 +735,8 @@ __vhost_head_statement() {
     cat <<EOF
 $(
   echo "ServerAdmin ${admin_mail}"
-  [ "$domain" ] && echo "ServerName ${domain}"
-  for alias in "${server_aliases[@]}"; do
+  [ "${#domains[@]}" != 0 ] && echo "ServerName ${domains[0]}"
+  for alias in "${domains[@]:1}"; do
       [ "$alias" ] || continue
       echo "ServerAlias $alias"
   done
@@ -674,7 +798,7 @@ target-get() {
     target=$(e "$cfg" | cfg-get-value target 2>/dev/null) || true
     if [ -z "$target" ]; then
         ## First exposed port:
-        base_image=$(service_base_docker_image "$BASE_SERVICE_NAME") || return 1
+        base_image=$(service_ensure_image_ready "$BASE_SERVICE_NAME") || return 1
         if ! docker_has_image "$base_image"; then
             docker pull "$base_image" >&2
         fi
@@ -690,14 +814,15 @@ target-get() {
 }
 
 __vhost_proxy_statement() {
-    local protocol="$1" cfg="$2" dest="$3" proxy_pass_options target
-
+    local protocol="$1" cfg="$2" domains="$3" proxy_pass_options target
+    domains=($domains)
     target=$(target-get "$cfg") || return 1
     proxy_pass_options=($(e "$cfg" | shyaml -y -q get-value apache-proxy-pass-options | yaml_get_values))
     if [ "${#proxy_pass_options[@]}" == 0 ]; then
         proxy_pass_options=(${proxy_pass_options:-"retry=0"})
     fi
-
+    dest=${domains[0]}
+    dest=${dest:-html}
     cat <<EOF
 
 
@@ -730,11 +855,12 @@ EOF
 }
 
 __vhost_full_vhost_statement() {
-    local type="$1" protocol="$2" cfg="$3" domain="$4" head_statement custom_rules content_statement
+    local type="$1" protocol="$2" cfg="$3" domains="$4" head_statement custom_rules content_statement
 
-    head_statement=$(__vhost_head_statement "$cfg" "$protocol" "$domain") || return 1
+    domains=($domains)
+    head_statement=$(__vhost_head_statement "$cfg" "$protocol" "${domains[*]}") || return 1
     custom_rules=$(__vhost_custom_rules "$cfg") || return 1
-    content_statement=$(__vhost_content_statement "$type" "$protocol" "$cfg" "${domain:-html}") || return 1
+    content_statement=$(__vhost_content_statement "$type" "$protocol" "$cfg" "${domains[*]}") || return 1
 
     case "$protocol" in
         https)
@@ -767,7 +893,10 @@ EOF
 }
 
 __vhost_publish_dir_statement() {
-    local protocol="$1" cfg="$2" dest="$3" location
+    local protocol="$1" cfg="$2" domains="$3" location
+    domains=($domains)
+    dest=${domains[0]}
+    dest=${dest:-html}
     local_path="/var/www/${dest}"
 
     cat <<EOF
@@ -795,7 +924,13 @@ EOF
 
 
 __vhost_tunnel_ssh_statement() {
-    local protocol="$1" cfg="$2" dest="$3" custom_rules content_statement
+    local protocol="$1" cfg="$2" domains="$3" custom_rules content_statement
+    domains=($domains)
+    dest=${domains[0]}
+    if [ "${#domains[@]}" == 0 ]; then
+        err "You must specify a domain for ssh tunnel to work."
+        return 1
+    fi
     cat <<EOF
 
 
@@ -817,7 +952,7 @@ AllowConnect 22
 
 ### Accept redirect only to same domain
 
-<Proxy $domain>
+<Proxy ${domains[0]}>
     Order deny,allow
 $(__vhost_creds_statement "$cfg" "$dest" | prefix "    ")
 </Proxy>

apache/test/get_domains

@@ -0,0 +1,316 @@
+#!/bin/bash
+
+exname=$(basename $0)
+
+compose_core=$(which compose-core) || {
+    echo "Requires compose-core executable to be in \$PATH." >&2
+    exit 1
+}
+
+fetch-def() {
+    local path="$1" fname="$2"
+    ( . "$path" 1>&2 || {
+          echo "Failed to load '$path'." >&2
+          exit 1
+      }
+      declare -f "$fname"
+    )
+}
+
+prefix_cmd="
+. /etc/shlib
+
+include common
+include parse
+
+. ../lib/common
+
+$(fetch-def "$compose_core" yaml_get_values)
+$(fetch-def "$compose_core" yaml_get_interpret)
+
+" || {
+    echo "Couldn't build prefix cmd" >&2
+    exit 1
+}
+
+# mock
+cfg-get-value() {
+    local key="$1"
+    shyaml get-value "$key" 2>/dev/null
+}
+export -f cfg-get-value
+
+yaml_get_interpret() {
+    shyaml get-value
+}
+export -f yaml_get_interpret
+
+
+export state_tmpdir=$(mktemp -d -t tmp.XXXXXXXXXX)
+trap "rm -rf \"$state_tmpdir\"" EXIT
+
+##
+## Tests
+##
+
+try "
+get_domains '
+'"
+is errlvl 1
+is err reg 'Error: .*domain option.*'
+is out ''
+
+
+try "
+get_domains '
+domain: toto
+'"
+noerror
+is out 'toto
+'
+
+
+try "
+get_domains '
+domain: toto titi
+'"
+noerror
+is out 'toto titi
+'
+
+try "
+get_domains '
+domain:
+- toto
+'"
+noerror
+is out 'toto
+'
+
+
+try "
+get_domains '
+server-aliases:
+'"
+is errlvl 1
+is err part 'Error: '
+is err part 'No domain name set'
+
+try "
+get_domains '
+domain:
+server-aliases:
+'"
+is errlvl 1
+is err part 'Error: '
+is err part 'No domain name set'
+
+
+try "
+get_domains '
+domain:
+server-aliases:
+- toto
+'"
+is errlvl 1
+is err part 'Error: '
+is err part "You can't specify server aliases if you don't have a domain"
+
+try "
+get_domains '
+domain: foo
+server-aliases:
+- bar
+'"
+noerror
+is out 'foo bar
+'
+
+
+try "
+get_domains '
+domain: foo
+server-aliases: bar
+'"
+noerror
+is out 'foo bar
+'
+
+
+try "
+get_domains '
+domain:
+- foo
+server-aliases: bar
+'"
+noerror
+is out 'foo bar
+'
+
+try "
+get_domains '
+domain:
+- foo{1,2} bar
+server-aliases: wiz
+'"
+noerror
+is out 'foo1 foo2 bar wiz
+'
+
+try "
+get_domains '
+domain:
+- foo{1,2} bar
+server-aliases: foo1
+'"
+noerror
+is out 'foo1 foo2 bar
+'
+
+try "
+get_domains '
+domain:
+- foo{1,2} bar
+- \"*.zoo\"
+server-aliases: foo1
+'"
+noerror
+is out 'foo1 foo2 bar *.zoo
+'
+
+try "
+get_domains '
+domain: foo+ bar
+'"
+is errlvl 1
+is err part 'Error: '
+is err part 'Invalid domain value'
+
+
+try "
+get_domains '
+domain: 
+' '
+options.service-domain-map:
+'" "empty service-domain-map"
+is errlvl 1
+is err part 'Error: '
+is err part 'No domain name set'
+is err part 'service-domain-map'
+
+
+try "
+BASE_SERVICE_NAME=foo
+get_domains '
+domain: 
+' '
+options:
+  service-domain-map:
+    wiz: bar
+'" "no map matching in service-domain-map"
+is errlvl 1
+is err part 'Error: '
+is err part 'No domain name set'
+is err part 'service-domain-map'
+
+
+try "
+export BASE_SERVICE_NAME=wiz
+get_domains '
+domain: 
+' '
+options:
+  service-domain-map:
+    wiz: bar
+'" "matching map in service-domain-map"
+noerror
+is out 'bar
+'
+
+
+try "
+export BASE_SERVICE_NAME=wiz
+get_domains '
+domain: 
+' '
+options:
+  service-domain-map:
+    wiz?: bar
+    wiz: bar2
+'" "only first matching map in service-domain-map"
+noerror
+is out 'bar
+'
+
+
+try "
+export BASE_SERVICE_NAME=wiz
+get_domains '
+domain: 
+' '
+options:
+  service-domain-map:
+    \"[w]i?zz?\": bar
+'" "map are regex in service-domain-map"
+noerror
+is out 'bar
+'
+
+
+try "
+export BASE_SERVICE_NAME=wiz
+get_domains '
+domain: 
+' '
+options:
+  service-domain-map:
+    (w)i(z): bar\$1\$2
+'" "regex capture in service-domain-map"
+noerror
+is out 'barwz
+'
+
+
+try "
+export BASE_SERVICE_NAME=wiz
+get_domains '
+domain: 
+' '
+options:
+  service-domain-map:
+    .*: \$0.shrubbery
+'" "regex capture 2 in service-domain-map"
+noerror
+is out 'wiz.shrubbery
+'
+
+
+try "
+export BASE_SERVICE_NAME=wiz
+get_domains '
+domain: 
+' '
+options:
+  service-domain-map:
+    .*: \$x
+'" "refuse other variables in service-domain-map"
+is errlvl 1
+is err part 'Error: '
+is err part 'Invalid mapping value'
+
+
+try "
+export BASE_SERVICE_NAME=wiz
+get_domains '
+domain: 
+' '
+options:
+  service-domain-map:
+    .*:
+      - \$0.example.com
+      - my-\$0.domain.org
+
+'" "list is possible as value of service-domain-map"
+noerror
+is out 'wiz.example.com my-wiz.domain.org
+'

apache/test/vhost

@@ -107,20 +107,13 @@ is out '<VirtualHost *:80>
 ##
 
 try "
-apache_vhost_statement publish_dir ,http, '
-server-aliases:
-- toto
-' www.example.com"
+apache_vhost_statement publish_dir ,http, '' 'www.example.com toto'"
 noerror
 is out reg 'ServerAlias toto'
 
 
 try "
-apache_vhost_statement publish_dir ,http, '
-server-aliases:
-- toto
-- titi
-' www.example.com"
+apache_vhost_statement publish_dir ,http, '' 'www.example.com toto titi'"
 noerror
 is out reg 'ServerAlias toto'
 is out reg 'ServerAlias titi'
@@ -295,7 +288,8 @@ is out '<VirtualHost *:80>
 
     </IfModule>
 
-    RequestHeader set "X-Forwarded-Proto" "http"
+    SetEnvIf X-Forwarded-Proto "^$" forwarded_proto_not_set=true
+    RequestHeader set "X-Forwarded-Proto" "http" env=forwarded_proto_not_set
 
     ## Fix IE problem (httpapache proxy dav error 408/409)
     SetEnv proxy-nokeepalive 1
@@ -348,7 +342,8 @@ is out '<VirtualHost *:80>
           SSLProxyEngine On
       </IfModule>
 
-      RequestHeader set "X-Forwarded-Proto" "https"
+      SetEnvIf X-Forwarded-Proto "^$" forwarded_proto_not_set=true
+      RequestHeader set "X-Forwarded-Proto" "https" env=forwarded_proto_not_set
 
       ## Fix IE problem (httpapache proxy dav error 408/409)
       SetEnv proxy-nokeepalive 1

apache/test/vhost_cert_provider

@@ -182,7 +182,7 @@ ssl:
   foo: |
     a
     b
-'
+' ''
 " "unknown cert key"
 is errlvl 1
 is err reg 'Error: .*cert-provider.*'

apache/test/vhost_files

@@ -305,8 +305,8 @@ creds:
 '
 "
 is errlvl 0
-is err reg 'setfacl -R -m g:<GID>:rx \$DATASTORE/\$BASE_SERVICE_NAME/var/www/www.example.com'
-is err reg 'cached_cmd_on_base_image apache id -g www-data'
+# is err reg 'setfacl -R -m g:<GID>:rx \$DATASTORE/\$BASE_SERVICE_NAME/var/www/www.example.com'
+# is err reg 'cached_cmd_on_base_image apache id -g www-data'
 
 
 try "
@@ -327,8 +327,8 @@ data-dirs:
 '
 "
 is errlvl 0
-is err reg 'setfacl -R -m g:<GID>:rwx \$DATASTORE/\$BASE_SERVICE_NAME/var/www/www.example.com/a \$DATASTORE/\$BASE_SERVICE_NAME/var/www/www.example.com/b \$DATASTORE/\$BASE_SERVICE_NAME/var/www/www.example.com/c'
-is err reg 'setfacl -R -d -m g:<GID>:rwx \$DATASTORE/\$BASE_SERVICE_NAME/var/www/www.example.com/a \$DATASTORE/\$BASE_SERVICE_NAME/var/www/www.example.com/b \$DATASTORE/\$BASE_SERVICE_NAME/var/www/www.example.com/c'
+# is err reg 'setfacl -R -m g:<GID>:rwx \$DATASTORE/\$BASE_SERVICE_NAME/var/www/www.example.com/a \$DATASTORE/\$BASE_SERVICE_NAME/var/www/www.example.com/b \$DATASTORE/\$BASE_SERVICE_NAME/var/www/www.example.com/c'
+# is err reg 'setfacl -R -d -m g:<GID>:rwx \$DATASTORE/\$BASE_SERVICE_NAME/var/www/www.example.com/a \$DATASTORE/\$BASE_SERVICE_NAME/var/www/www.example.com/b \$DATASTORE/\$BASE_SERVICE_NAME/var/www/www.example.com/c'
 
 
 try "
@@ -350,16 +350,16 @@ data-dirs:
 - b
 - c
 '
-"
+" "with location"
 is errlvl 0
 is err reg 'mkdir -p /opt/apps/newlocation'
-is err reg 'setfacl -R -m g:<GID>:rx /opt/apps/newlocation'
+# is err reg 'setfacl -R -m g:<GID>:rx /opt/apps/newlocation'
 is out part '
 init-config-add
   |
   | $SERVICE_NAME:
   |   volumes:
-  |     - /opt/apps/newlocation:/var/www/www.example.com' RTRIM
+  |     - "/opt/apps/newlocation:/var/www/www.example.com"' RTRIM
 
 
 

collabora/hooks/web_proxy-relation-joined

@@ -3,6 +3,8 @@
 set -e
 
 URL=$(relation-get url) || exit 1
+DOMAIN=$(relation-get domain) || exit 1
+
 PROTO="${URL%%://*}"
 
 ssl_enable=true
@@ -24,6 +26,6 @@ config-add "\
 services:
   $MASTER_BASE_SERVICE_NAME:
     environment:
-      server_name: \"$URL\"
+      server_name: \"$DOMAIN\"
       extra_params: \"--o:ssl.enable=false --o:ssl.termination=${ssl_enable} \"
 "

collabora/metadata.yml

@@ -1,5 +1,5 @@
-# from: collabora/code:23.05.3.1.1
-docker-image: docker.0k.io/collabora:2305.3.1
+# from: collabora/code:23.05.9.4.1
+docker-image: docker.0k.io/collabora:2305.9.4
 
 docker-compose:
   cap_add:

cron/README.org

@@ -0,0 +1,19 @@
+# -*- ispell-local-dictionary: "english" -*-
+
+* Usage
+
+By adding =cron= as a service, all other services in auto pair mode,
+requiring a =schedule-command= will use it.
+
+#+begin_src yaml
+cron:
+#+end_src
+
+There are no options to set.
+
+** =schedule-command= relation
+
+If most other services will have default options and set these values
+automatically. You probably don't need to configure anything in the
+relation's options if defaults suits you.
+

cron/build/Dockerfile

@@ -1,13 +1,27 @@
-FROM docker.0k.io/debian:jessie
+FROM alpine:3.18 AS build
 
-RUN apt-get update && \
-    DEBIAN_FRONTEND=noninteractive apt-get install -y cron moreutils && \
-    apt-get clean && \
-    rm -rf /var/lib/apt/lists/*
+ENV DOCKER_VERSION=25.0.2
 
-COPY ./src/usr/bin/lock /usr/bin/lock
-COPY ./src/usr/bin/docker-17.06.2-ce /usr/bin/docker
+RUN apk add --no-cache curl
 
-COPY ./entrypoint.sh /entrypoint.sh
+#RUN curl -L https://download.docker.com/linux/static/stable/x86_64/docker-"$DOCKER_VERSION".tgz | \
+RUN curl -L https://docker.0k.io/downloads/docker-"$DOCKER_VERSION".tgz | \
+    tar -xz -C /tmp/ \
+    && mv /tmp/docker/docker /usr/bin/docker
+RUN curl -L https://docker.0k.io/downloads/lock-40a4b8f > /usr/bin/lock \
+    && chmod +x /usr/bin/lock
 
-ENTRYPOINT [ "/entrypoint.sh" ]
+FROM alpine:3.18
+
+## Used by `lock`
+RUN apk add --no-cache bash 
+
+## /usr/bin/dc is a calculator provided by busybox that conflicts with
+## the `dc` command provided by `compose`. We have no need of busybox
+## calculator
+RUN rm /usr/bin/dc
+
+COPY --from=build /usr/bin/docker /usr/bin/docker
+COPY --from=build /usr/bin/lock /usr/bin/lock
+
+ENTRYPOINT [ "crond", "-f", "-l", "0" ]

cron/build/README

@@ -1,16 +0,0 @@
-
-
-Warning, this charm will require access to ``/var/run/docker.sock``,
-and this IS EQUIVALENT to root access to host.
-
-Warning, must use ``/etc/cron`` and not ``/etc/cron.d``.
-
-
-docker was downloaded with:
-
-wget https://get.docker.com/builds/Linux/x86_64/docker-1.9.1
-
-
-It changed, check:
-
-  https://download.docker.com/linux/static/stable/x86_64/

cron/build/entrypoint.sh

@@ -1,32 +0,0 @@
-#!/bin/bash
-
-##
-## /var/log might be plugged into an empty volume for saving logs, so we
-## must make sure that /var/log/exim4 exists and has correct permissions.
-
-mkdir -p /var/log/exim4
-chmod -R u+rw /var/log/exim4
-chown -R Debian-exim /var/log/exim4
-
-
-echo "Propagating docker shell environment variables to CRON scripts."
-
-rm -f /etc/cron.d/*
-cp -a /etc/cron/* /etc/cron.d/
-
-for f in /etc/crontab /etc/cron.d/*; do
-    [ -e "$f" ] || continue
-    mv "$f" /tmp/tempfile
-    {
-        declare -xp | egrep '_PORT_[0-9]+_'  | sed -r 's/^declare -x //g'
-        echo "TZ=$TZ"
-        echo
-        cat /tmp/tempfile
-    } > "$f"
-    rm /tmp/tempfile
-done
-
-
-echo "Launching cron."
-## Give back PID 1, so that cron process receives signals.
-exec /usr/sbin/cron -f

cron/build/src/usr/bin/README

@@ -1,2 +0,0 @@
-WARNING, lock shell script is a copy from ``kal-scripts``. Please
-do not do any modification to it without sending it back to ``kal-scripts``.

cron/build/src/usr/bin/lock

@@ -1,363 +0,0 @@
-#!/bin/bash
-
-##
-## TODO
-## - don't sleep 1 but wait in flock for 1 second
-## - every waiting proc should write at least their PID and priority,
-##   to leave alive PID with higher priority the precedence. (and probably
-##   a check to the last probing time, and invalidate it if it is higher than 10s
-##   for example.)
-## - could add the time they waited in the waiting list, and last probe.
-## - should execute "$@", if user needs '-c' it can run ``bash -c ""``
-
-exname="$(basename "$0")"
-usage="$exname LOCKLABELS [-k] [FLOCK_OPTIONS] -- [CMD...]"
-
-verb() { [ -z "$verbose" ] || echo "$@" >&2 ; }
-err() { echo "$@" >&2; }
-die() { echo "$@" >&2; exit 1; }
-
-md5_compat() { md5sum | cut -c -32; true; }
-
-LOCKLABELS=
-flock_opts=()
-command=()
-nonblock=
-errcode=1
-timeout=
-cmd=
-priority=1
-remove_duplicate=
-while [ "$1" ]; do
-    case "$1" in
-        -h|--help)
-            echo "$help"
-            exit 0
-            ;;
-        -V|--version)
-            echo "$version"
-            exit 0
-            ;;
-        -c)
-            cmd="$2"
-            shift
-            ;;
-        -p|--priority)
-            priority=$2
-            shift
-            ;;
-        -D)
-            remove_duplicate=true
-            ;;
-        -k)
-            kill=yes
-            ;;
-        -n|--nb|--nonblock)
-            nonblock=true
-            ;;
-        -w|--wait|--timeout)
-            timeout=$2   ## will manage this
-            shift
-            ;;
-        -E|--conflict-exit-code)
-            errcode=$2   ## will manage this
-            shift
-            ;;
-        -v|--verbose)
-            verbose=true   ## will manage this
-            ;;
-        -n|--nb|--nonblock)
-            nonblock=true   ## will manage this
-            ;;
-        --)
-            [ "$cmd" ] && die "'--' and '-c' are mutualy exclusive"
-            shift
-            command+=("$@")
-            break 2
-            ;;
-        *)
-            [ -z "$LOCKLABELS" ] && { LOCKLABELS=$1 ; shift ; continue ; }
-            flock_opts+=("$1")
-            ;;
-    esac
-    shift
-done
-
-if [ -z "$LOCKLABELS" ]; then
-    err "You must provide a lock file as first argument."
-    err "$usage"
-    exit 1
-fi
-
-if [ "$remove_duplicate" ]; then
-    md5code=$(
-        if [ "$cmd" ]; then
-            echo bash -c "$cmd"
-        else
-            echo "${command[@]}"
-        fi | md5_compat)
-fi
-
-
-function is_int () {  [[ "$1" =~ ^-?[0-9]+$ ]] ; }
-
-is_pid_alive() {
-    local pid="$1"
-    ps --pid "$pid" >/dev/null 2>&1
-}
-
-
-is_pgid_alive() {
-    local pgid="$1"
-    [ "$(ps -e -o pgid,pid= | egrep "^ *$pgid ")" ]
-}
-
-
-pgid_from_pid() {
-    local pid="$1"
-    pgid=$(ps -o pgid= "$pid" 2>/dev/null | egrep -o "[0-9]+")
-    if ! is_int "$pgid"; then
-        err "Could not retrieve a valid PGID from PID '$pid' (returned '$pgid')."
-        return 1
-    fi
-    echo "$pgid"
-}
-
-
-ensure_kill() {
-    local pid="$1" timeout=5 start=$SECONDS kill_count=0 pgid
-    pgid=$(pgid_from_pid "$pid")
-    while is_pid_alive "$pid"; do
-          if is_pgid_alive "$pgid"; then
-              if [ "$kill_count" -gt 4 ]; then
-                  err "FATAL: duplicate command, GPID=$pgid has resisted kill procedure. Aborting."
-                  return 1
-              elif [ "$kill_count" -gt 2 ]; then
-                  err "duplicate command, PGID wouldn't close itself, force kill PGID: kill -9 -- -$pgid"
-                  kill -9 -- "$pgid"
-                  sleep 1
-              else
-                  err "duplicate command, Sending SIGKILL to PGID: kill -- -$pgid"
-                  kill -- -"$pgid"
-                  sleep 1
-              fi
-              ((kill_count++))
-          fi
-          if [ "$((SECONDS - start))" -gt "$timeout" ]; then
-              err "timeout reached. $pid"
-              return 1
-          fi
-    done
-    return 0
-}
-
-
-acquire_pid_file() {
-    local label=$1
-    lockfile="/var/lock/lockcmd-$label.lock"
-    mkdir -p /var/run/lockcmd
-    pidfile="/var/run/lockcmd/$label.pid"
-    export pidfile
-    (
-    verb() { [ -z "$verbose" ] || echo "$exname($label) $pid> $@" >&2 ; }
-    err() { echo "$exname($label) $pid> $@" >&2; }
-
-    start=$SECONDS
-    kill_count=0
-    pgid_not_alive_count=0
-    while true; do
-        ## ask for lock on $lockfile (fd 200)
-        if ! flock -n -x 200; then
-            verb "Couldn't acquire primary lock... (elapsed $((SECONDS - start)))"
-        else
-            verb "Acquired lock '$label' on pidfile, inspecting pidfile."
-            if ! [ -e "$pidfile" ]; then
-                verb "No pidfile, inscribing my PID"
-                echo -e "$pid $priority" > "$pidfile"
-                exit 0
-            fi
-
-            if ! content=$(cat "$pidfile" 2>/dev/null); then
-                err "Can't read $pidfile"
-                exit 1
-            fi
-            read opid opriority < <(echo "$content" | head -n 1)
-            opriority=${opriority:-1}
-            verb "Previous PID is $opid, with priority $opriority"
-            if ! is_pid_alive "$opid"; then
-                err "Ignoring stale PID $opid"
-                echo -e "$pid $priority" > "$pidfile"
-                exit 0
-            else
-                if [ "$remove_duplicate" ]; then  ## Add my pid and md5 if not already there.
-                    same_cmd_pids=$(
-                        echo "$content" | tail -n +1 | \
-                            egrep "^[0-9]+ $md5code$" 2>/dev/null | \
-                            cut -f 1 -d " ")
-                    same_pids=()
-                    found_myself=
-                    for spid in $same_cmd_pids; do
-                        if [ "$spid" == "$pid" ]; then
-                            found_myself=true
-                            continue
-                        fi
-                        same_pids+=("$spid")
-                    done
-                    [ "$found_myself" ] || echo "$pid $md5code" >> "$pidfile"
-                fi
-                flock -u 200  ## reopen the lock to give a chance to the other process to remove the pidfile.
-                if [ "$remove_duplicate" ]; then  ## Add my pid and md5 if not already there.
-                    for spid in "${same_pids[@]}"; do
-                        if ! ensure_kill "$spid"; then
-                            err "Couldn't kill previous duplicate command."
-                            exit 1
-                        fi
-                    done
-                fi
-                pgid=$(pgid_from_pid "$opid")
-                verb "PGID of previous PID is $pgid"
-                if is_pgid_alive "$pgid"; then
-                    verb "Previous PGID is still alive"
-                    if [ "$kill" ] && [ "$priority" -ge "$opriority" ]; then
-                        if [ "$kill_count" -gt 4 ]; then
-                            err "$pid>FATAL: GPID=$pgid has resisted kill procedure. Aborting."
-                            exit 1
-                        elif [ "$kill_count" -gt 2 ]; then
-                            err "PGID wouldn't close itself, force kill PGID: kill -9 -- -$pgid" >&2
-                            kill -9 -- "$pgid"
-                            sleep 1
-                        else
-                            err "Sending SIGKILL to PGID: kill -- -$pgid" >&2
-                            kill -- -"$pgid"
-                            sleep 1
-                        fi
-                        ((kill_count++))
-                    else
-                        if [ "$nonblock" ]; then
-                            verb "Nonblock options forces exit."
-                            exit 1
-                        else
-                            verb "Couldn't acquire Lock... (elapsed $((SECONDS - start)))"
-                        fi
-                    fi
-                else
-                    if [ "$pgid_not_alive_count" -gt 4 ]; then
-                        verb "$pid>A lock exists for label $label, but PGID:$pgid in it isn't alive while child $pid is ?!?."
-                        err "$pid>Can't force seizing the lock." >&2
-                        exit 1
-                    fi
-                    ((pgid_not_alive_count++))
-                fi
-            fi
-        fi
-
-        if [ "$timeout" ] && [ "$timeout" -lt "$((SECONDS - start))" ]; then
-            err "Timeout reached (${timeout}s) while waiting for lock on $label"
-            exit "$errcode"
-        fi
-        sleep 1
-    done
-    ) 200> "$lockfile"
-}
-
-remove_pid_file() {
-    local label=$1
-    lockfile="/var/lock/lockcmd-$label.lock"
-    mkdir -p /var/run/lockcmd
-    pidfile="/var/run/lockcmd/$label.pid"
-
-(
-    verb() { [ -z "$verbose" ] || echo "$exname($label) $pid> $@" >&2 ; }
-    err() { echo "$exname($label) $pid> $@" >&2; }
-    verb "Asking lock to delete $pidfile."
-    timeout=5
-    start=$SECONDS
-    while true; do
-        ## ask for lock on $lockfile (fd 200)
-        if ! flock -n -x 200; then
-            verb "Couldn't acquire primary lock... (elapsed $((SECONDS - start)))"
-        else
-            verb "Acquired lock '$label' on pidfile."
-            if ! [ -e "$pidfile" ]; then
-                verb "No more pidfile, somebody deleted for us ?1?"
-                exit 1
-            fi
-            if ! content=$(cat "$pidfile" 2>/dev/null); then
-                err "Can't read $pidfile"
-                exit 1
-            fi
-            read opid opriority < <(echo "$content" | head -n 1)
-            opriority=${opriority:-1}
-            if [ "$opid" == "$pid" ]; then
-                verb "Deleted pidfile. Releasing lock."
-                rm -f "$pidfile"
-                exit 0
-            else
-                verb "Removing duplicates in pidfile. Releasing lock."
-                [ "$remove_duplicate" ] && sed -ri "/^$pid $md5code$/d" "$pidfile"
-                exit 0
-            fi
-        fi
-        if [ "$timeout" ] && [ "$timeout" -lt "$((SECONDS - start))" ]; then
-            err "Timeout reached (${timeout}s) while waiting for lock on $label"
-            exit "$errcode"
-        fi
-        sleep 1
-    done
-) 200> "$lockfile"
-
-}
-
-
-## appends a command to the signal handler functions
-#
-# example: trap_add EXIT,INT close_ssh "$ip"
-trap_add() {
-    local sigs="$1" sig cmd old_cmd
-    shift || {
-        echo "${FUNCNAME} usage error" >&2
-        return 1
-    }
-    cmd="$@"
-    while IFS="," read -d "," sig; do
-        prev_cmd="$(trap -p "$sig")"
-        if [ "$prev_cmd" ]; then
-            new_cmd="${prev_cmd#trap -- \'}"
-            new_cmd="${new_cmd%\' "$sig"};$cmd"
-        else
-            new_cmd="$cmd"
-        fi
-        trap -- "$new_cmd" "$sig" || {
-            echo "unable to add command '$@' to trap $sig" >&2 ;
-            return 1
-        }
-    done < <(echo "$sigs,")
-}
-
-remove_all_pid_file() {
-    while read -d "," label; do
-        {
-            remove_pid_file "$label" || err "Could not delete $label"
-        } &
-    done < <(echo "$LOCKLABELS,")
-    wait
-}
-
-##
-## Code
-##
-
-pid="$$"
-
-trap_add EXIT "remove_all_pid_file"
-while read -d "," label; do
-    acquire_pid_file "$label" || exit "$errcode" &
-done < <(echo "$LOCKLABELS,")
-wait
-if [ "$cmd" ]; then
-    bash -c "$cmd"
-else
-    "${command[@]}"
-fi
-errlvl="$?"
-exit "$?"

cron/hooks/init

@@ -30,6 +30,9 @@ fi
     exit 1
 }
 
+mkdir -p "$SERVICE_CONFIGSTORE/etc/crontabs"
+touch "$SERVICE_CONFIGSTORE/etc/crontabs/root"
+
 timezone=$(cat /etc/timezone) || exit 1
 init-config-add "
 $CHARM_NAME:

cron/hooks/pre_deploy

@@ -1,20 +1,39 @@
 #!/bin/bash
-## Should be executable N time in a row with same result.
+
+. lib/common
 
 set -e
 
-cron_config_hash() {
-    debug "Adding config hash to enable recreating upon config change."
-    config_hash=$({
-                     find "$SERVICE_CONFIGSTORE/etc/cron"{,.hourly,.weekly,.daily,.monthly} \
-                          -type f -exec md5sum {} \;
-                  } | md5_compat) || exit 1
-    init-config-add "
-$MASTER_BASE_SERVICE_NAME:
-  labels:
-    - compose.config_hash=$config_hash
-"
+root_crontab="$SERVICE_CONFIGSTORE/etc/crontabs/root"
+
+cron_content=$(set pipefail; cron:entries | tr '\0' '\n') || {
+    err "Failed to make cron entries" >&2
+    exit 1
 }
+if [ -z "${cron_content::1}" ]; then
+    err "Unexpected empty scheduled command list."
+    exit 1
+fi
+if [ -e "$root_crontab" ]; then
+    if ! [ -f "$root_crontab" ]; then
+        err "Destination '$root_crontab' exists and is not a file."
+        exit 1
+    fi
+    current_content=$(cat "$root_crontab")
+    if [ "$current_content" = "$cron_content" ]; then
+        info "Cron entry already up to date."
+        exit 0
+    fi
+fi
+
+
+if ! [ -d "${root_crontab%/*}" ]; then
+    mkdir -p "${root_crontab%/*}"
+fi
 
+printf "%s\n" "$cron_content" > "$root_crontab"
+## Busybox cron uses cron.update file to rescan new cron entries
+## cf: https://git.busybox.net/busybox/tree/miscutils/crond.c#n1089
+touch "${root_crontab%/*}/cron.update"
 
-cron_config_hash || exit 1
+info "Cron entry updated ${GREEN}successfully${NORMAL}."

cron/lib/common

@@ -0,0 +1,210 @@
+# -*- mode: shell-script -*-
+
+cron:get_config() {
+    local cfg="$1"
+    local cache_file="$CACHEDIR/$FUNCNAME.cache.$(H "$@")" \
+          type value
+    if [ -e "$cache_file" ]; then
+        #debug "$FUNCNAME: SESSION cache hit $1"
+        cat "$cache_file"
+        return 0
+    fi
+    type=$(e "$cfg" | shyaml -q get-type 2>/dev/null) || true
+    case "$type" in
+        "sequence")
+            while read-0-err E s; do
+                cron:get_config "$s" || return 1
+            done < <(e "$cfg" | p-err shyaml -q get-values-0 -y)
+            if [ "$E" != 0 ]; then
+                err "Failed to parse sequence while reading config."
+                return 1
+            fi
+            ;;
+        "struct")
+            while read-0-err E k v; do
+                while read-0-err E1 schedule lock_opts title command; do
+                    if [ -n "$title" ]; then
+                        err "Unexpected label specified in struct."
+                        echo "  Using struct, the key will be used as label." >&2
+                        echo "  So you can't specify a label inner value(s)." >&2
+                        return 1
+                    fi
+                    p0 "$schedule" "$lock_opts" "$k" "$command"
+                done < <(p-err cron:get_config "$v")
+                if [ "$E1" != 0 ]; then
+                    err "Failed to parse value of key '$k' in struct config."
+                    return 1
+                fi
+            done < <(e "$cfg" | p-err shyaml -q key-values-0 -y)
+            if [ "$E" != 0 ]; then
+                err "Failed to parse key values while reading config."
+                return 1
+            fi
+            ;;
+        "str")
+            ## examples:
+            ##   (*/5 * * * *) {-k} bash -c "foo bla bla"
+            ##   (@daily) {-p 10 -D} bash -c "foo bla bla"
+            value=$(e "$cfg" | yaml_get_values) || {
+                err "Failed to parse str while reading config."
+                return 1
+            }
+            if ! [[ "$value" =~ ^[[:space:]]*([a-zA-Z0-9_-]+)?[[:space:]]*"("([^\)]+)")"[[:space:]]+\{([^\}]*)\}[[:space:]]*(.*)$ ]]; then
+                err "Invalid syntax, expected: 'LABEL (SCHEDULE) {LOCK_OPTIONS} COMMAND'."
+                echo "  With LABEL being a possible empty string." >&2
+                echo "  Received: '$value'" >&2
+                return 1
+            fi
+            printf "%s\0" "${BASH_REMATCH[2]}" "${BASH_REMATCH[3]}" "${BASH_REMATCH[1]}" "${BASH_REMATCH[4]}"
+            ;;
+        NoneType|"")
+            :
+            ;;
+        *)
+            value=$(e "$cfg" | yaml_get_interpret) || {
+                err "Failed to parse value while reading config."
+                return 1
+            }
+            if [[ "$value" == "$cfg" ]]; then
+                err "Unrecognized type '$type'."
+                return 1
+            fi
+            cron:get_config "$value" || return 1
+            ;;
+    esac > "$cache_file.tmp"
+
+    mv -v "$cache_file.tmp" "$cache_file" >&2
+
+    ## if cache file is empty, this is an error
+    if [ ! -s "$cache_file" ]; then
+        err "Unexpected empty relation options."
+        echo "  - check that you don't overwrite default options with an empty relation" >&2
+        echo "  - check your charm is setting default options" >&2
+        echo "  Original value: '$cfg'" >&2
+        rm -f "$cache_file"
+        return 1
+    fi
+
+    cat "$cache_file"
+}
+
+
+cron:entries_from_service() {
+    local service="$1" relation_cfg="$2" \
+          cache_file="$CACHEDIR/$FUNCNAME.cache.$(H "$@")" \
+          label schedule lock_opts title command full_label
+    if [ -e "$cache_file" ]; then
+        #debug "$FUNCNAME: SESSION cache hit $1"
+        cat "$cache_file"
+        return 0
+    fi
+
+    ## XXXvlab; should factorize this with compose-core to setup relation env
+    export BASE_SERVICE_NAME=$service
+    MASTER_BASE_SERVICE_NAME=$(get_top_master_service_for_service "$service") || return 1
+    MASTER_BASE_CHARM_NAME=$(get_service_charm "$MASTER_BASE_SERVICE_NAME") || return 1
+    BASE_CHARM_NAME=$(get_service_charm "$service") || return 1
+    BASE_CHARM_PATH=$(charm.get_dir "$BASE_CHARM_NAME") || return 1
+    export MASTER_BASE_{CHARM,SERVICE}_NAME BASE_CHARM_{PATH,NAME}
+
+    label="launch-$service"
+    while read-0-err E schedule lock_opts title command; do
+        lock_opts=($lock_opts)
+        if ! [[ "$schedule" =~ ^(([0-9/,*-]+[[:space:]]+){4,4}[0-9/,*-]+|@[a-z]+)$ ]]; then
+            err "Unrecognized schedule '$schedule'."
+            return 1
+        fi
+        ## Check that label is only a simple identifier
+        if ! [[ "$title" =~ ^[a-zA-Z0-9_-]*$ ]]; then
+            err "Unexpected title '$title', please use only alphanumeric, underscore or dashes (can be empty)."
+            return 1
+        fi
+        if ! lock_opts=($(cron:lock_opts "${lock_opts[@]}")); then
+            err "Failed to parse lock options."
+            return 1
+        fi
+        if [ -z "$command" ]; then
+            err "Unexpected empty command."
+            return 1
+        fi
+
+        full_label="$label"
+        [ -n "$title" ] && full_label+="-$title"
+
+        ## escape double-quotes
+        command=${command//\"/\\\"}
+
+        p0 "$schedule  lock ${full_label} ${lock_opts[*]} -c \"$command\" 2>&1 | awk '{ print strftime(\"%Y-%m-%d %H:%M:%S %Z\"), \$0; fflush(); }' >> /var/log/cron/${full_label}_script.log"
+
+    done < <(p-err cron:get_config "$relation_cfg") > "$cache_file"
+    if [ "$E" != 0 ]; then
+        rm -f "$cache_file"
+        err "Failed to get ${DARKYELLOW}$service${NORMAL}--${DARKBLUE}schedule-command${NORMAL}-->${DARKYELLOW}$SERVICE_NAME${NORMAL}'s config."
+        return 1
+    fi
+
+    cat "$cache_file"
+}
+
+
+cron:lock_opts() {
+    local cache_file="$CACHEDIR/$FUNCNAME.cache.$(H "$@")" \
+          label schedule lock_opts title command full_label
+    if [ -e "$cache_file" ]; then
+        #debug "$FUNCNAME: SESSION cache hit $1"
+        cat "$cache_file"
+        return 0
+    fi
+    lock_opts=()
+    while [ "$1" ]; do
+        case "$1" in
+            "-D"|"-k")
+                lock_opts+=("$1")
+                ;;
+            "-p")
+                ## check that the value is a number
+                if ! [[ "$2" =~ ^[0-9]+$ ]]; then
+                    err "Unexpected value for priority '$2' (expected an integer)."
+                    return 1
+                fi
+                lock_opts+=(-p "$2")
+                shift
+                ;;
+            "-*"|"--*")
+                err "Unrecognized lock option '$1'."
+                return 1
+                ;;
+            *)
+                err "Unexpected lock argument '$1'."
+                return 1
+                ;;
+        esac
+        shift
+    done
+    printf "%s\n" "${lock_opts[@]}" > "$cache_file"
+
+    cat "$cache_file"
+}
+
+
+cron:entries() {
+    local cache_file="$CACHEDIR/$FUNCNAME.cache.$(H "$SERVICE_NAME" "$ALL_RELATIONS")" \
+          s rn ts rc td
+    if [ -e "$cache_file" ]; then
+        #debug "$FUNCNAME: SESSION cache hit $1"
+        cat "$cache_file"
+        return 0
+    fi
+
+    if [ -z "$ALL_RELATIONS" ]; then
+        err "Expected \$ALL_RELATIONS to be set."
+        exit 1
+    fi
+    export TARGET_SERVICE_NAME=$SERVICE_NAME
+    while read-0 service relation_cfg; do
+        debug "service: '$service' relation_cfg: '$relation_cfg'"
+        cron:entries_from_service "$service" "$relation_cfg" || return 1
+    done < <(get_service_incoming_relations "$SERVICE_NAME" "schedule-command") > "$cache_file"
+    cat "$cache_file"
+}
+export -f cron:entries

cron/metadata.yml

@@ -1,11 +1,6 @@
 description: Cron daemon
 config-resources:
-  - /etc/cron
-  - /etc/cron.daily
-  - /etc/cron.weekly
-  - /etc/cron.hourly
-  - /etc/cron.monthly
-  - /usr/local/bin
+  - /etc/crontabs
 data-resources:
   - /var/log/cron
 host-resources:
@@ -20,7 +15,7 @@ uses:   ## optional
   log-rotate:
     #constraint: required | recommended | optional
     #auto: pair | summon | none ## default: pair
-    constraint: optional
+    constraint: recommended
     solves:
       disk-leak: "/var/log/cron"
     #default-options:

cron/test/entries_from_service

@@ -0,0 +1,155 @@
+#!/bin/bash
+
+exname=$(basename $0)
+
+compose_core=$(which compose-core) || {
+    echo "Requires compose-core executable to be in \$PATH." >&2
+    exit 1
+}
+
+fetch-def() {
+    local path="$1" fname="$2"
+    ( . "$path" 1>&2 || {
+          echo "Failed to load '$path'." >&2
+          exit 1
+      }
+      declare -f "$fname"
+    )
+}
+
+prefix_cmd="
+. /etc/shlib
+
+include common
+include parse
+
+. ../lib/common
+
+$(fetch-def "$compose_core" yaml_get_values)
+$(fetch-def "$compose_core" yaml_get_interpret)
+$(fetch-def "$compose_core" read-0-err)
+$(fetch-def "$compose_core" p-err)
+$(fetch-def "$compose_core" expand_vars)
+
+SERVICE_NAME='bar'
+
+" || {
+    echo "Couldn't build prefix cmd" >&2
+    exit 1
+}
+
+# mock
+cfg-get-value() {
+    local key="$1"
+    shyaml get-value "$key" 2>/dev/null
+}
+export -f cfg-get-value
+
+yaml_get_interpret() {
+    shyaml get-value
+}
+export -f yaml_get_interpret
+
+get_top_master_service_for_service() {
+    local service="$1"
+    echo "$service"
+}
+export -f get_top_master_service_for_service
+
+get_service_charm() {
+    local service="$1"
+    echo "$service"
+}
+export -f get_service_charm
+
+export CACHEDIR=$(mktemp -d -t tmp.XXXXXXXXXX)
+export state_tmpdir=$(mktemp -d -t tmp.XXXXXXXXXX)
+trap "rm -rf \"$state_tmpdir\"" EXIT
+trap "rm -rf \"$CACHEDIR\"" EXIT
+
+##
+## Tests
+##
+
+try "
+cron:entries_from_service 'foo' ''"
+is errlvl 1
+is err reg "Error:.*ailed to get.*."
+is err reg "Error:.*empty.*."
+is out '' TRIM
+
+try "
+cron:entries_from_service 'foo' '
+(0 0 * * *) {XX} dc run --rm foo
+'" "wrong lock args"
+is errlvl 1
+is err reg "Error:.*lock argument.*."
+is err reg "Error:.*parse lock.*."
+is out '' TRIM
+
+
+try "
+cron:entries_from_service 'foo' '
+(0 0 * * * *) {} dc run --rm foo
+'" "wrong schedule"
+is errlvl 1
+is err reg "Error:.*schedule.*"
+is out '' TRIM
+
+
+try "
+cron:entries_from_service 'foo' '
+(0 0 * * *) {}
+'" "wrong command"
+is errlvl 1
+is err reg "Error:.*empty command.*"
+is out '' TRIM
+
+
+try "
+set pipefail &&
+cron:entries_from_service 'foo' '
+(0 0 * * *) {-p 10 -k} dc run --rm foo
+' | tr '\0' '\n'" "one command no label"
+noerror
+is out "\
+0 0 * * *  lock launch-foo -p 10 -k -c \"dc run --rm foo\" 2>&1 | awk '{ print strftime(\"%Y-%m-%d %H:%M:%S %Z\"), \$0; fflush(); }' >> /var/log/cron/launch-foo_script.log\
+" TRIM
+
+
+try "
+set pipefail &&
+cron:entries_from_service 'foo' '
+wiz: (0 0 * * *) {-p 10 -k} dc run --rm foo
+' | tr '\0' '\n'" "one command with label"
+noerror
+is out "\
+0 0 * * *  lock launch-foo-wiz -p 10 -k -c \"dc run --rm foo\" 2>&1 | awk '{ print strftime(\"%Y-%m-%d %H:%M:%S %Z\"), \$0; fflush(); }' >> /var/log/cron/launch-foo-wiz_script.log\
+" TRIM
+
+
+
+try "
+set pipefail &&
+cron:entries_from_service 'foo' '
+wiz: (0 0 * * *) {-p 10 -k} dc run --rm foo
+bam: (@daily) {-p 10 -D -k} dc run --rm foo --hop
+
+' | tr '\0' '\n'" "multi command with label"
+noerror
+is out "\
+0 0 * * *  lock launch-foo-wiz -p 10 -k -c \"dc run --rm foo\" 2>&1 | awk '{ print strftime(\"%Y-%m-%d %H:%M:%S %Z\"), \$0; fflush(); }' >> /var/log/cron/launch-foo-wiz_script.log
+@daily  lock launch-foo-bam -p 10 -D -k -c \"dc run --rm foo --hop\" 2>&1 | awk '{ print strftime(\"%Y-%m-%d %H:%M:%S %Z\"), \$0; fflush(); }' >> /var/log/cron/launch-foo-bam_script.log\
+" TRIM
+
+
+
+try "
+set pipefail &&
+cron:entries_from_service 'foo' '!var-expand
+(0 0 * * *) {-p 10 -k} dc run --rm \$BASE_SERVICE_NAME \$MASTER_BASE_SERVICE_NAME
+' | tr '\0' '\n'" "using relation's var"
+noerror
+is out "\
+0 0 * * *  lock launch-foo -p 10 -k -c \"dc run --rm foo foo\" 2>&1 | awk '{ print strftime(\"%Y-%m-%d %H:%M:%S %Z\"), \$0; fflush(); }' >> /var/log/cron/launch-foo_script.log" TRIM
+

cron/test/get_config

@@ -0,0 +1,151 @@
+#!/bin/bash
+
+exname=$(basename $0)
+
+compose_core=$(which compose-core) || {
+    echo "Requires compose-core executable to be in \$PATH." >&2
+    exit 1
+}
+
+fetch-def() {
+    local path="$1" fname="$2"
+    ( . "$path" 1>&2 || {
+          echo "Failed to load '$path'." >&2
+          exit 1
+      }
+      declare -f "$fname"
+    )
+}
+
+prefix_cmd="
+. /etc/shlib
+
+include common
+include parse
+
+. ../lib/common
+
+$(fetch-def "$compose_core" yaml_get_values)
+$(fetch-def "$compose_core" yaml_get_interpret)
+$(fetch-def "$compose_core" read-0-err)
+$(fetch-def "$compose_core" p-err)
+$(fetch-def "$compose_core" expand_vars)
+
+" || {
+    echo "Couldn't build prefix cmd" >&2
+    exit 1
+}
+
+# mock
+cfg-get-value() {
+    local key="$1"
+    shyaml get-value "$key" 2>/dev/null
+}
+export -f cfg-get-value
+
+yaml_get_interpret() {
+    shyaml get-value
+}
+export -f yaml_get_interpret
+
+
+export CACHEDIR=$(mktemp -d -t tmp.XXXXXXXXXX)
+export state_tmpdir=$(mktemp -d -t tmp.XXXXXXXXXX)
+trap "rm -rf \"$state_tmpdir\"" EXIT
+trap "rm -rf \"$CACHEDIR\"" EXIT
+
+##
+## Tests
+##
+
+try "
+cron:get_config ''"
+is errlvl 1
+is err reg 'Error: .*empty.*'
+is out ''
+
+try "
+cron:get_config 'xxx'"
+is errlvl 1
+is err reg 'Error: .*syntax.*'
+is out ''
+
+
+try "
+set pipefail &&
+cron:get_config '(@daily) {} /bin/true' | tr '\0' ':'
+" "str simple example without label"
+noerror
+is out "@daily:::/bin/true:"
+
+try "
+set pipefail &&
+cron:get_config 'foo (@daily) {} /bin/true' | tr '\0' ':'
+" "str simple example with label"
+noerror
+is out "@daily::foo:/bin/true:"
+
+try "
+set pipefail &&
+cron:get_config 'foo (@daily) {-p 10 -D} /bin/true' | tr '\0' ':'
+" "str simple example with lock options"
+noerror
+is out "@daily:-p 10 -D:foo:/bin/true:"
+
+try "
+set pipefail &&
+cron:get_config 'foo (*/2 * * * *) {-p 10 -D} /bin/true' | tr '\0' ':'
+" "str simple example with all fields"
+noerror
+is out "*/2 * * * *:-p 10 -D:foo:/bin/true:"
+
+
+try "
+set pipefail &&
+cron:get_config '- foo (*/2 * * * *) {-p 10 -D} /bin/true' | tr '\0' ':'
+" "list 1 elt with str simple example with all fields"
+noerror
+is out "*/2 * * * *:-p 10 -D:foo:/bin/true:"
+
+try "
+set pipefail &&
+cron:get_config '
+- foo (*/2 * * * *) {-p 10 -D} /bin/true
+- bar (*/3 * * * *) {-p 10 -D -k} /bin/false
+
+' | tr '\0' ':'
+" "list 2 elts with str simple example with all fields"
+noerror
+is out "*/2 * * * *:-p 10 -D:foo:/bin/true:*/3 * * * *:-p 10 -D -k:bar:/bin/false:"
+
+try "
+set pipefail &&
+cron:get_config '
+foo: (*/2 * * * *) {-p 10 -D} /bin/true
+bar: (*/3 * * * *) {-p 10 -D -k} /bin/false
+
+' | tr '\0' ':'
+" "struct 2 elts with str simple example with all fields"
+noerror
+is out "*/2 * * * *:-p 10 -D:foo:/bin/true:*/3 * * * *:-p 10 -D -k:bar:/bin/false:"
+
+
+
+try "
+cron:get_config '!!float 3.7'
+" "bad type"
+is errlvl 1
+is err reg 'Error: .*type.*'
+is out ''
+
+
+try "
+export FOO=bar
+set pipefail &&
+cron:get_config '!var-expand (*/2 * * * *) {-p 10 -D} \"/bin/\${FOO}\"' | tr '\0' ':'
+" "var-expand"
+is errlvl 0
+is err ''
+is out '*/2 * * * *:-p 10 -D::"/bin/bar":'
+
+

cron/test/lock_opts

@@ -0,0 +1,90 @@
+#!/bin/bash
+
+exname=$(basename $0)
+
+compose_core=$(which compose-core) || {
+    echo "Requires compose-core executable to be in \$PATH." >&2
+    exit 1
+}
+
+fetch-def() {
+    local path="$1" fname="$2"
+    ( . "$path" 1>&2 || {
+          echo "Failed to load '$path'." >&2
+          exit 1
+      }
+      declare -f "$fname"
+    )
+}
+
+prefix_cmd="
+. /etc/shlib
+
+include common
+include parse
+
+. ../lib/common
+
+$(fetch-def "$compose_core" yaml_get_values)
+$(fetch-def "$compose_core" yaml_get_interpret)
+$(fetch-def "$compose_core" read-0-err)
+$(fetch-def "$compose_core" p-err)
+
+" || {
+    echo "Couldn't build prefix cmd" >&2
+    exit 1
+}
+
+# mock
+cfg-get-value() {
+    local key="$1"
+    shyaml get-value "$key" 2>/dev/null
+}
+export -f cfg-get-value
+
+yaml_get_interpret() {
+    shyaml get-value
+}
+export -f yaml_get_interpret
+
+
+export CACHEDIR=$(mktemp -d -t tmp.XXXXXXXXXX)
+export state_tmpdir=$(mktemp -d -t tmp.XXXXXXXXXX)
+trap "rm -rf \"$state_tmpdir\"" EXIT
+trap "rm -rf \"$CACHEDIR\"" EXIT
+
+##
+## Tests
+##
+
+try "
+cron:lock_opts ''"
+noerror
+is out '' TRIM
+
+try "
+cron:lock_opts '--XXX'
+"
+is errlvl 1
+is err reg 'Error: .*argument.*--XXX.*'
+is out ''
+
+try "
+cron:lock_opts -p X
+"
+is errlvl 1
+is err reg 'Error: .*priority.*X.*integer.*'
+is out ''
+
+
+try "
+cron:lock_opts -p 10 -k -D
+"
+noerror
+is out "\
+-p
+10
+-k
+-D" TRIM
+
+

cyclos/lib/common

@@ -54,7 +54,7 @@ sql() {
 
         export SERVICE_NAME="$ts"
         export SERVICE_DATASTORE="$DATASTORE/$SERVICE_NAME"
-        DOCKER_BASE_IMAGE=$(service_base_docker_image "$SERVICE_NAME")
+        DOCKER_BASE_IMAGE=$(service_ensure_image_ready "$SERVICE_NAME") || exit 1
         export DOCKER_BASE_IMAGE
 
         target_charm=$(get_service_charm "$ts") || exit 1

docker-host/hooks/install.d/90-ntfy.sh

@@ -0,0 +1,100 @@
+#!/bin/bash
+
+set -eux
+
+NTFY_BROKER="${NTFY_BROKER:-core-01.0k.io}"
+
+
+## Uncipher ntfy key to destination
+
+umask 077
+ntfy_key_ciphered="src/etc/ssh/ntfy-key"
+if [ ! -f "$ntfy_key_ciphered" ]; then
+    echo "Error: ciphered ntfy key not found" >&2
+    exit 1
+fi
+
+ntfy_key_dest=/etc/ssh/ntfy-key
+if [ ! -f "$ntfy_key_dest" ]; then
+    cat "$ntfy_key_ciphered" |
+        gpg -d --batch --yes --passphrase 'uniquepass' > "$ntfy_key_dest" || {
+        echo "Error while unpacking ntfy key to '${ntfy_key_dest}'" >&2
+        exit 1
+    }
+fi
+
+
+## Request token to ntfy server and add to config file 
+
+known_host="/root/.ssh/known_hosts"
+if ! ssh-keygen -F "$NTFY_BROKER" -f "$known_host" >/dev/null; then
+    ssh-keyscan -H "$NTFY_BROKER" >> "$known_host" || {
+        echo "Error while adding '$NTFY_BROKER' to known_hosts" >&2
+        exit 1
+    }
+fi
+
+config_file="/etc/ntfy/ntfy.conf"
+mkdir -p "${config_file%/*}"
+if ! [ -f "$config_file" ]; then
+    touch "$config_file" || {
+        echo "Error: couldn’t create config file '$config_file'" >&2;
+        exit 1
+    }
+fi
+
+LOGIN=""
+PASSWORD=""
+source "$config_file" || {
+    echo "Error: couldn't source config file '$config_file'" >&2
+    exit 1
+}
+
+## Note that we require the forcing of stdin to /dev/null to avoid
+## the rest of the script to be vacuumed by the ssh command.
+## This effect will only happen when launching this script in special
+## conditions involving stdin.
+cred=$(ssh -i "$ntfy_key_dest" ntfy@"${NTFY_BROKER}" \
+           request-token "$LOGIN" "$PASSWORD" </dev/null) || {
+    echo "Error while requesting token to ntfy server" >&2
+    exit 1
+}
+
+## XXXvlab: ideally it should be received from the last call
+server="https://ntfy.0k.io/"
+login=$(printf "%q" "${cred%$'\n'*}")
+password=$(printf "%q" "${cred#*$'\n'}")
+
+## check if password doesn't contain '%'
+
+for var in server login password; do
+    if [ "${!var}" == "''" ] || [[ "${!var}" == *$'\n'* ]]; then
+        echo "Error: empty or invalid multi-line values retrieved for '$var'" \
+             "from ntfy server. Received:" >&2
+        printf "%s" "$cred" | sed -r 's/^/  | /g' >&2
+        exit 1
+    fi
+    if [[ "${!var}" == *%* ]]; then
+        ## We need a separator char for sed replacement in the config file
+        echo "Error: forbidden character '%' found in $var" >&2
+        exit 1
+    fi
+    if grep -qE "^${var^^}=" "$config_file"; then
+        sed -ri "s%^${var^^}=.*$%${var^^}=\"${!var}\"%g" "$config_file"
+    else
+        echo "${var^^}=\"${!var}\"" >> "$config_file"
+    fi
+done
+
+
+if ! [ -f "/etc/ntfy/topics.yml" ]; then
+    cat <<'EOF' > /etc/ntfy/topics.yml
+.*\.(emerg|alert|crit|err|warning|notice):
+  - ${LOGIN}_main
+EOF
+fi
+
+
+## provide 'send' command
+
+cp -f "$PWD/src/bin/send" /usr/local/bin/send

docker-host/src/bin/send

@@ -0,0 +1,166 @@
+#!/bin/bash
+
+## Send a notification with NTFY and check if the config file is complete
+
+if [[ "$UID" == "0" ]]; then
+    NTFY_CONFIG_DIR="${NTFY_CONFIG_DIR:-/etc/ntfy}"
+else
+    NTFY_CONFIG_DIR="${NTFY_CONFIG_DIR:-~/.config/ntfy}"
+fi
+NTFY_CONFIG_FILE="$NTFY_CONFIG_DIR/ntfy.conf"
+
+SERVER="https://ntfy.0k.io"
+
+[ -f "$NTFY_CONFIG_DIR/topics.yml" ] || {
+    echo "Error: no 'topics.yml' file found in $NTFY_CONFIG_DIR" >&2
+    echo "  Please setup the topics for the notification channels in this file." >&2
+    exit 1
+}
+
+if ! [ -e "$NTFY_CONFIG_FILE" ]; then
+    mkdir -p "${NTFY_CONFIG_FILE%/*}"
+    ## default option to change if needed
+    echo "SERVER=$SERVER" > "$NTFY_CONFIG_FILE"
+elif ! grep -q "^SERVER=" "$NTFY_CONFIG_FILE"; then
+    echo "SERVER=$SERVER" >> "$NTFY_CONFIG_FILE"
+fi
+
+source "$NTFY_CONFIG_FILE" || {
+    echo "Error: could not source '$NTFY_CONFIG_FILE'" >&2
+    exit 1
+}
+
+SERVER="${SERVER%/}"
+
+for var in SERVER LOGIN PASSWORD; do
+    if ! [ -v "$var" ]; then
+        echo "Error: missing $var in $NTFY_CONFIG_FILE"
+        exit 1
+    fi
+done
+
+
+exname=${0##*/}
+channels=()
+
+usage="$exname [options] MESSAGE"
+help="\
+
+Send MESSAGE with TITLE to the differents topics defined by a CHANNEL
+
+$exname will read the $NTFY_CONFIG_DIR/topics.yml for channel to
+topics conversion.
+
+
+Usage:
+  $usage
+
+Options:
+  -c CHANNEL  Specify one or multiple channels. Default 'main'.
+              (can be provided mulitiple time)
+  -t TITLE    Specify the title of the message. (it'll still be
+              prefixed with the hostname)
+              Default is empty
+  MESSAGE     message to send.
+  -h          Display this help and exit.
+
+"
+
+while [ "$#" -gt 0 ]; do
+    arg="$1"
+    shift
+    case "$arg" in
+        -h|--help)
+            echo "$help"
+            exit 0
+            ;;
+        -c|--channel)
+            [ -n "$1" ] || {
+                echo "Error: no argument for channel option." >&2
+                echo "$usage" >&2
+                exit 1
+            }
+            IFS=", " channels+=($1)
+            shift
+            ;;
+        -t|--title)
+            [ -n "$1" ] || {
+                echo "Error: no argument for title option." >&2
+                echo "$usage" >&2
+                exit 1
+            }
+            title="$1"
+            shift 
+            ;;
+        *)  
+            [ -z "$message" ] && { message="$arg"; continue; }
+            echo "Error : Unexpected positional argument '$arg'." >&2
+            echo "$usage" >&2
+            exit 1
+            ;;
+    esac
+done
+
+[ -n "$message" ] || {
+    echo "Error: missing message." >&2
+    echo "$usage" >&2
+    exit 1
+}
+
+read-0() {
+    local eof='' IFS=''
+    while [ "$1" ]; do
+        read -r -d '' -- "$1" || eof=1
+        shift
+    done
+    [ -z "$eof" ]
+}
+
+curl_opts=(
+    -s 
+    -u "$LOGIN:$PASSWORD"
+    -d "$message"
+)
+
+title="[$(hostname)] $title"
+title="${title%%+([[:space:]])}"
+curl_opts+=(-H "Title: $title")
+
+declare -A sent_topic=()
+
+if [ "${#channels[@]}" -eq 0 ]; then
+    channels=("main")
+fi
+
+for channel in "${channels[@]}"; do
+    channel_quoted=$(printf "%q" "$channel")
+    content=$(cat "$NTFY_CONFIG_DIR/topics.yml")
+    while read-0 channel_regex topics; do
+        [[ "$channel" =~ ^$channel_regex$ ]] || continue
+        rematch=("${BASH_REMATCH[@]}")
+        while read-0 topic; do
+            ttopic=$(printf "%s" "$topic" | yq "type")
+            if [ "$ttopic" != '!!str' ]; then
+                echo "Error: Unexpected '$ttopic' type for value of channel $channel." >&2
+                exit 1
+            fi
+            topic=$(printf "%s" "$topic" | yq -r " \"\" + .")
+            if ! [[ "$topic" =~ ^[a-zA-Z0-9\$\{\}*\ \,_.-]+$ ]]; then
+                echo "Error: Invalid topic value '$topic' expression in $channel channel." >&2
+                exit 1
+            fi
+            new_topics=($(set -- "${rematch[@]}"; eval echo "${topic//\*/\\*}"))
+            for new_topic in "${new_topics[@]}"; do
+                [ -n "${sent_topic["$new_topic"]}" ] && continue
+                sent_topic["$new_topic"]=1
+                if ! out=$(curl "${curl_opts[@]}" "$SERVER/${new_topic}"); then
+                    echo "Error: could not send message to $new_topic." >&2
+                    echo "curl command:" >&2
+                    echo "  curl ${curl_opts[@]} $SERVER/${new_topic}" >&2
+                    echo "$out" | sed 's/^/  | /' >&2
+                    exit 1
+                fi
+            done
+        done < <(printf "%s" "$topics" | yq e -0 '.[]')
+    done < <(printf "%s" "$content" | yq e -0 'to_entries | .[] | [.key, .value] |.[]')
+done

gogocarto/hooks/schedule_commands-relation-joined

@@ -1,44 +0,0 @@
-#!/bin/bash
-
-## When writing relation script, remember:
-##  - they should be idempotents
-##  - they can be launched while the dockers is already up
-##  - they are launched from the host
-##  - the target of the link is launched first, and get a chance to ``relation-set``
-##  - both side of the scripts get to use ``relation-get``.
-
-. lib/common
-
-set -e
-
-## XXXvlab: should use container name here so that it could support
-## multiple postgres
-label=${SERVICE_NAME}
-DST=$CONFIGSTORE/$TARGET_SERVICE_NAME/etc/cron/$label
-
-## XXXvlab: Should we do a 'docker exec' instead ?
-bin_console="dc run -u www-data --rm --entrypoint \\\"$GOGOCARTO_DIR/bin/console\\\" $MASTER_BASE_SERVICE_NAME"
-
-## Warning: 'docker -v' will use HOST directory even if launched from
-##   'cron' container.
-file_put "$DST" <<EOF
-@daily     root  lock ${label}-checkvote -D -p 10 -c "\
-    $bin_console app:elements:checkvote" 2>&1 | ts '\%F \%T \%Z' >> /var/log/cron/${SERVICE_NAME}-checkvote_script.log
-
-@daily     root  lock ${label}-checkExternalSourceToUpdate -D -p 10 -c "\
-    $bin_console app:elements:checkExternalSourceToUpdate" 2>&1 | ts '\%F \%T \%Z' >> /var/log/cron/${SERVICE_NAME}-checkExternalSourceToUpdate_script.log
-
-@daily     root  lock ${label}-notify-moderation -D -p 10 -c "\
-    $bin_console app:notify-moderation" 2>&1 | ts '\%F \%T \%Z' >> /var/log/cron/${SERVICE_NAME}-notify-moderation_script.log
-
-
-@hourly     root  lock ${label}-sendNewsletter -D -p 10 -c "\
-    $bin_console app:users:sendNewsletter" 2>&1 | ts '\%F \%T \%Z' >> /var/log/cron/${SERVICE_NAME}-sendNewsletter_script.log
-
-
-*/5 * * * *     root  lock ${label}-webhooks-post -D -p 10 -c "\
-    $bin_console --env=prod app:webhooks:post" 2>&1 | ts '\%F \%T \%Z' >> /var/log/cron/${SERVICE_NAME}-webhooks-post_script.log
-
-
-EOF
-chmod +x "$DST"

gogocarto/lib/common

@@ -128,7 +128,7 @@ symphony() {
 
     export COMPOSE_IGNORE_ORPHANS=true
     ## We don't want post deploy that is doing the final http initialization.
-    compose --debug -q --no-init --no-post-deploy \
+    compose --debug -q --no-init --no-post-deploy --no-pre-deploy \
             --without-relation="$SERVICE_NAME":web-proxy \
             run \
             "${symphony_docker_run_opts[@]}" \

gogocarto/metadata.yml

@@ -29,8 +29,20 @@ uses:
     auto: summon
     solves:
       database: "main storage"
-  schedule-commands:
+  schedule-command:
     constraint: recommended
     auto: pair
     solves:
       maintenance: "Production scheduled tasks"
+    default-options: !bash-stdout |
+      www_data_uid=33
+      bin_console="dc exec -T -u \"$www_data_uid\" $MASTER_BASE_SERVICE_NAME \"/opt/apps/$BASE_SERVICE_NAME/bin/console\""
+
+      scheds=(
+        checkvote                   @daily      "$bin_console app:elements:checkvote"
+        checkExternalSourceToUpdate @daily      "$bin_console app:elements:checkExternalSourceToUpdate"
+        notify-moderation           @daily      "$bin_console app:notify-moderation"
+        sendNewsletter              @hourly     "$bin_console app:users:sendNewsletter"
+        webhooks-post               "*/5 * * * *" "$bin_console --env=prod app:webhooks:post"
+      )
+      printf "%s: (%s) {-D -p 10} %s\n" "${scheds[@]}"

hedgedoc/README.org

@@ -0,0 +1,22 @@
+# -*- ispell-local-dictionary: "english" -*-
+
+* Usage
+
+** How to manage users
+
+This allows to reset password for users.
+
+From container (using `docker exec -ti MY_CONTAINER sh`), use
+`/hedgedoc/bin/manage_users` command (it is not in $PATH).
+
+#+begin_example
+Command-line utility to create users for email-signin.
+
+Usage: bin/manage_users [--pass password] (--add | --del) user-email
+        Options:
+                --add   Add user with the specified user-email
+                --del   Delete user with specified user-email
+                --reset Reset user password with specified user-email
+                --pass  Use password from cmdline rather than prompting
+#+end_example
+

letsencrypt/actions/crt

@@ -1,4 +1,5 @@
 #!/bin/bash
+## compose: no-hooks
 
 if [ -z "$SERVICE_DATASTORE" ]; then
     echo "This script is meant to be run through 'compose' to work properly." >&2

letsencrypt/hooks/schedule_command-relation-joined

@@ -1,34 +0,0 @@
-#!/bin/bash
-
-## When writing relation script, remember:
-##  - they should be idempotents
-##  - they can be launched while the dockers is already up
-##  - they are launched from the host
-##  - the target of the link is launched first, and get a chance to ``relation-set``
-##  - both side of the scripts get to use ``relation-get``.
-
-set -e
-
-label=${SERVICE_NAME}-renew
-DST=$CONFIGSTORE/$TARGET_SERVICE_NAME/etc/cron/$label
-LOCAL_LOG=/var/log/cron/${label}_script.log
-schedule=$(relation-get schedule)
-
-if ! echo "$schedule" | egrep '^\s*(([0-9/,*-]+\s+){4,4}[0-9/,*-]+|@[a-z]+)\s*$' >/dev/null 2>&1; then
-    err "Unrecognized schedule '$schedule'."
-    exit 1
-fi
-
-## Warning: using '\' in heredoc will be removed in the final cron file, which
-##   is totally wanted: cron does not support multilines.
-
-## Warning: 'docker -v' will use HOST directory even if launched from
-##   'cron' container.
-file_put "$DST" <<EOF
-COMPOSE_LAUNCHER_OPTS=$COMPOSE_LAUNCHER_OPTS
-
-$schedule     root  lock $label -D -p 10 -c "\
-    compose crt $SERVICE_NAME renew" 2>&1 | ts '\%F \%T \%Z' >> $LOCAL_LOG
-
-EOF
-chmod +x "$DST"

letsencrypt/lib/common

@@ -140,14 +140,14 @@ has_existing_cert() {
 
 letsencrypt_cert_info() {
     local domain="$1"
-    compose -q --no-init --no-relations run -T --rm "$SERVICE_NAME" \
+    compose -q --no-hooks run -T --rm "$SERVICE_NAME" \
             crt info "$domain"
 }
 
 
 letsencrypt_cert_delete() {
     local domain="$1"
-    compose --debug --no-init --no-relations run --rm "$SERVICE_NAME" \
+    compose --debug --no-hooks run --rm "$SERVICE_NAME" \
             certbot delete --cert-name "$domain"
 }
 
@@ -159,7 +159,7 @@ valid_existing_cert() {
     has_existing_cert "$domain" || return 1
 
     info "Querying $domain for previous info..."
-    out=$(letsencrypt_cert_info "$domain")
+    out=$(letsencrypt_cert_info "$domain") || return 1
 
     ## check if output is valid yaml
     err=$(e "$out" | shyaml get-value 2>&1 >/dev/null) || {
@@ -201,7 +201,7 @@ valid_existing_cert() {
 
 
 get_domain_list() {
-    compose -q --no-init --no-relations run --rm "$SERVICE_NAME" crt list
+    compose -q --no-hooks run --rm "$SERVICE_NAME" crt list
 }
 
 

letsencrypt/metadata.yml

@@ -26,5 +26,5 @@ uses:
     auto: summon
     solves:
       missing-feature: "Automatic certificate renewal"
-    default-options:
-      schedule: "30 3 * * 7"  ## schedule log renewal every week
+    default-options: !var-expand
+      (30 3 * * 7) {-D -p 10} compose crt "$BASE_SERVICE_NAME" renew

logrotate/hooks/schedule_command-relation-joined

@@ -1,30 +0,0 @@
-#!/bin/bash
-
-## When writing relation script, remember:
-##  - they should be idempotents
-##  - they can be launched while the dockers is already up
-##  - they are launched from the host
-##  - the target of the link is launched first, and get a chance to ``relation-set``
-##  - both side of the scripts get to use ``relation-get``.
-
-set -e
-
-label=launch-$SERVICE_NAME
-DST=$CONFIGSTORE/$TARGET_SERVICE_NAME/etc/cron/$label
-schedule=$(relation-get schedule) || true
-
-if ! echo "$schedule" | egrep '^\s*(([0-9/,*-]+\s+){4,4}[0-9/,*-]+|@[a-z]+)\s*$' >/dev/null 2>&1; then
-    err "Unrecognized schedule '$schedule'."
-    exit 1
-fi
-
-## Warning: using '\' in heredoc will be removed in the final cron file, which
-##   is totally wanted: cron does not support multilines.
-
-## Warning: 'docker -v' will use HOST directory even if launched from
-##   'cron' container.
-file_put "$DST" <<EOF
-$schedule     root  lock $label -D -p 10 -c "\
-    dc run --rm $SERVICE_NAME" 2>&1 | ts '\%F \%T \%Z' >> /var/log/cron/${label}_script.log
-EOF
-chmod +x "$DST"

logrotate/metadata.yml

@@ -14,5 +14,6 @@ uses:
     auto: summon
     solves:
       missing-feature: "scheduling of log rotation"
-    default-options:
-      schedule: "0 0 * * *"  ## It should really stay at midnight as most logs are dated
+    default-options: !var-expand
+      ## It should really stay at midnight as most logs are dated
+      (0 0 * * *) {-p 10 -D} dc run --rm "$MASTER_BASE_SERVICE_NAME"

mariadb/hooks/schedule_command-relation-joined

@@ -6,38 +6,54 @@
 ##  - they are launched from the host
 ##  - the target of the link is launched first, and get a chance to ``relation-set``
 ##  - both side of the scripts get to use ``relation-get``.
+##  - use actions of other side for other side's business logic
 
-. lib/common
+if [ -z "$RELATION_DATA_FILE" ]; then
+    err "$FUNCNAME: relation does not seems to be correctly setup."
+    exit 1
+fi
 
+if ! [ -r "$RELATION_DATA_FILE" ]; then
+    err "$FUNCNAME: can't read relation's data." >&2
+    exit 1
+fi
 
-set -e
 
-## XXXvlab: should use container name here so that it could support
-## multiple mysql
-label=${SERVICE_NAME}-mysql-backup
-DST=$CONFIGSTORE/$TARGET_SERVICE_NAME/etc/cron/$label
-schedule=$(relation-get schedule)
+if [ "$(cat "$RELATION_DATA_FILE" | shyaml get-type)" == "str" ]; then
+    ## Cached version already there
 
-if ! echo "$schedule" | egrep '^\s*(([0-9/,*-]+\s+){4,4}[0-9/,*-]+|@[a-z]+)\s*$' >/dev/null 2>&1; then
-    err "Unrecognized schedule '$schedule'."
-    exit 1
+    ## Note that we rely on the fact that when the relations
+    ## options are changed in the `compose.yml` file, the relation
+    ## data file is recreated from the values of the
+    ## `compose.yml`.
+    info "Previous relation data is still valid."
+    exit 0
 fi
 
-## Warning: using '\' in heredoc will be removed in the final cron file, which
-##   is totally wanted: cron does not support multilines.
 
-## Warning: 'docker -v' will use HOST directory even if launched from
-##   'cron' container.
-file_put "$DST" <<EOF
-COMPOSE_LAUNCHER_OPTS=$COMPOSE_LAUNCHER_OPTS
+. lib/common
+
+
+schedule=$(relation-get schedule) || true
+
+lock_opts=(-D -p 10)
 
-$schedule   root lock $label -D -p 10 -c "\
+command=(
     docker run --rm \
         --network ${PROJECT_NAME}_default \
-        -v \"$LOCAL_DB_PASSFILE\":/root/.my.cnf \
-        -v \"$HOST_CHARM_STORE/${CHARM_REL_PATH#${CHARM_STORE}/}/resources/bin/mysql-backup:/usr/sbin/mysql-backup\" \
-        -v \"$SERVICE_DATASTORE/var/backups/mysql:/var/backups/mysql\" \
+        -v "$LOCAL_DB_PASSFILE":/root/.my.cnf \
+        -v "$HOST_CHARM_STORE/${CHARM_REL_PATH#${CHARM_STORE}/}/resources/bin/mysql-backup:/usr/sbin/mysql-backup" \
+        -v "$SERVICE_DATASTORE/var/backups/mysql:/var/backups/mysql" \
         --entrypoint mysql-backup \
-        \"$DOCKER_BASE_IMAGE\" --host \"${SERVICE_NAME}\"" 2>&1 | ts '\%F \%T \%Z' >> /var/log/cron/${label}_script.log
-EOF
-chmod +x "$DST"
+        "$DOCKER_BASE_IMAGE" --host "${SERVICE_NAME}"
+)
+
+quoted_command=()
+for arg in "${command[@]}"; do
+    quoted_command+=("$(printf "%q" "$arg")")
+done
+
+printf "(%s) {%s} %s\n" \
+       "$schedule" \
+       "${lock_opts[*]}" \
+       "${quoted_command[*]}" > "$RELATION_DATA_FILE"

mariadb/hooks/sql_database-relation-joined

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+## When writing relation script, remember:
+##  - they should be idempotents
+##  - they can be launched while the dockers is already up
+##  - they are launched from the host
+##  - the target of the link is launched first, and get a chance to ``relation-set``
+##  - both side of the scripts get to use ``relation-get``.
+
+relation-set type mysql || {
+    err "Could not set relation ${WHITE}type${NORMAL} to 'mysql'."
+    exit 1
+}
+
+. hooks/mysql_database-relation-joined

mariadb/metadata.yml

@@ -2,6 +2,7 @@ name: MariaDB
 maintainer: "Valentin Lab <valentin.lab@kalysto.org>"
 provides:
   mysql-database:
+  sql-database:
 data-resources:
   - /var/lib/mysql
   - /var/backups/mysql

mongo/actions/relations/mongo-database/mongosh

@@ -69,7 +69,7 @@ mongosh() {
 
         export SERVICE_NAME="$RELATION_TARGET_SERVICE"
         export SERVICE_DATASTORE="$DATASTORE/$SERVICE_NAME"
-        DOCKER_BASE_IMAGE=$(service_base_docker_image "$SERVICE_NAME")
+        DOCKER_BASE_IMAGE=$(service_ensure_image_ready "$SERVICE_NAME")
         export DOCKER_BASE_IMAGE
 
         target_charm_path=$(charm.get_dir $RELATION_TARGET_CHARM) || exit 1

mongo/hooks/schedule_command-relation-joined

@@ -6,38 +6,59 @@
 ##  - they are launched from the host
 ##  - the target of the link is launched first, and get a chance to ``relation-set``
 ##  - both side of the scripts get to use ``relation-get``.
+##  - use actions of other side for other side's business logic
 
 . lib/common
 
-set -e
 
-label=${SERVICE_NAME}-backup
-DST=$CONFIGSTORE/$TARGET_SERVICE_NAME/etc/cron/$label
-schedule=$(relation-get schedule)
+if [ -z "$RELATION_DATA_FILE" ]; then
+    err "$FUNCNAME: relation does not seems to be correctly setup."
+    exit 1
+fi
 
-if ! echo "$schedule" | egrep '^\s*(([0-9/,*-]+\s+){4,4}[0-9/,*-]+|@[a-z]+)\s*$' >/dev/null 2>&1; then
-    err "Unrecognized schedule '$schedule'."
+if ! [ -r "$RELATION_DATA_FILE" ]; then
+    err "$FUNCNAME: can't read relation's data." >&2
     exit 1
 fi
 
+
+if [ "$(cat "$RELATION_DATA_FILE" | shyaml get-type)" == "str" ]; then
+    ## Cached version already there
+
+    ## Note that we rely on the fact that when the relations
+    ## options are changed in the `compose.yml` file, the relation
+    ## data file is recreated from the values of the
+    ## `compose.yml`.
+    info "Previous relation data is still valid."
+    exit 0
+fi
+
+schedule=$(relation-get schedule) || true
 exclude_dbs=$(relation-get exclude-dbs 2>/dev/null) || true
 exclude_dbs=$(echo "$exclude_dbs" | shyaml get-values 2>/dev/null |
                   nspc) || true
 
-## Warning: 'docker -v' will use HOST directory even if launched from
-##   'cron' container.
-file_put "$DST" <<EOF
-COMPOSE_LAUNCHER_OPTS=$COMPOSE_LAUNCHER_OPTS
-
-$schedule     root  lock $label -D -p 10 -c "\
-    docker run --rm \
-        -u 0 \
-        -e MONGO_HOST=${SERVICE_NAME} \
-        -e exclude_dbs=\"$exclude_dbs\" \
-        --network ${PROJECT_NAME}_default \
-        -v \"$HOST_CHARM_STORE/${CHARM_REL_PATH#${CHARM_STORE}/}/resources/bin/mongo-backup:/usr/sbin/mongo-backup\" \
-        -v \"$SERVICE_DATASTORE/var/backups/mongo:/var/backups/mongo\" \
-        --entrypoint mongo-backup \
-        \"$DOCKER_BASE_IMAGE\"" 2>&1 | ts '\%F \%T \%Z' >> /var/log/cron/${label}_script.log
-EOF
-chmod +x "$DST"
+lock_opts=(-D -p 10)
+
+
+command=(
+    docker run --rm
+    -u 0
+    -e MONGO_HOST=${SERVICE_NAME}
+    -e exclude_dbs="$exclude_dbs"
+    --network ${PROJECT_NAME}_default
+    -v "$HOST_CHARM_STORE/${CHARM_REL_PATH#${CHARM_STORE}/}/resources/bin/mongo-backup:/usr/sbin/mongo-backup"
+    -v "$SERVICE_DATASTORE/var/backups/mongo:/var/backups/mongo"
+    --entrypoint mongo-backup
+    "$DOCKER_BASE_IMAGE"
+)
+
+quoted_command=()
+for arg in "${command[@]}"; do
+    quoted_command+=("$(printf "%q" "$arg")")
+done
+
+printf "(%s) {%s} %s\n" \
+       "$schedule" \
+       "${lock_opts[*]}" \
+       "${quoted_command[*]}" > "$RELATION_DATA_FILE"

mongo/metadata.yml

@@ -17,7 +17,7 @@ uses:
       schedule: "47 * * * *"  ## schedule backup every hour
   ## This one is useful only if previous relation is used
   backup:
-    constraint: optional
+    constraint: recommended
     auto: pair
     solves:
       backup: "Automatic regular backups of dumps"

nextcloud/actions/occ

@@ -1,4 +1,5 @@
 #!/bin/bash
+## compose: no-hooks
 
 if [ -z "$SERVICE_DATASTORE" ]; then
     echo "This script is meant to be run through 'compose' to work properly." >&2

nextcloud/actions/upgrade

@@ -129,7 +129,7 @@ if [ -z "$target" ]; then
     info "Target latest version: ${WHITE}${last_available_versions[0]}${NORMAL}"
     target=${last_upgradable_versions[0]}
 else
-    if [[ "* $target *" != " ${last_available_versions[*]} " ]]; then
+    if [[ " ${last_available_versions[*]} " != *" $target "* ]]; then
         err "Invalid version $target selected, please specify one of:"
         for v in "${last_upgradable_versions[@]}"; do
             echo " - $v"

nextcloud/hooks/init

@@ -65,6 +65,7 @@ $MASTER_BASE_SERVICE_NAME:
     NEXTCLOUD_ADMIN_USER: $admin_user
     NEXTCLOUD_ADMIN_PASSWORD: $admin_password
     NEXTCLOUD_DATA_DIR: /var/lib/nextcloud/data
+    NEXTCLOUD_TRUSTED_DOMAINS: '\*'
 "
 
 ## ensuring data directories are accessible by nextcloud
@@ -75,3 +76,9 @@ uid=$(docker_get_uid "$MASTER_BASE_SERVICE_NAME" "www-data")
 dirs=("$SERVICE_DATASTORE/var/lib/nextcloud/data" "$SERVICE_DATASTORE/var/www/html")
 mkdir -p "${dirs[@]}"
 chown -R "$uid" "${dirs[@]}"
+
+
+nextcloud:config:simple:add skeletondirectory "" || {
+    err "Failed to set ${WHITE}skeletondirectory${NORMAL} as empty."
+    exit 1
+}

nextcloud/hooks/mysql_database-relation-joined

@@ -1,51 +0,0 @@
-#!/bin/bash
-
-. lib/common
-
-set -e
-
-PASSWORD="$(relation-get password)"
-USER="$(relation-get user)"
-DBNAME="$(relation-get dbname)"
-
-
-## This check adds purely arbitrary limits to what could be a password
-## if we need to open that more, just consider the next script where we'll
-## need to write in a PHP structure, or in YAML structure.
-
-## Note that here, "[]" chars are not accepted just because it doesn't seem evident
-## to test for those in bash.
-if ! [[ "$PASSWORD" =~ ^[a-zA-Z0-9~\`\&+=@\#^\*/\\_%\$:\;\!?.,\<\>{}()\"\'|-]*$ ]]; then
-     err "Invalid password chosen for mysql database."
-     exit 1
-fi
-
-## if config is not existent
-if [ -e "$CONFIGFILE" ] && grep "^  'dbuser' => '" "$CONFIGFILE" >/dev/null; then
-
-    ## 'occ' can't be used as it will try to connect to mysql before running and
-    ## will fail if user/password is not correct
-
-    ## We need to get through bash, and sed interpretation, then PHP single quoted strings.
-    quoted_user="${USER//\\/\\\\\\\\\\}"
-    quoted_user="${quoted_user//\'/\\\\\'}"
-    quoted_password="${PASSWORD//\\/\\\\\\\\\\}"
-    quoted_password="${quoted_password//\'/\\\\\'}"
-    sed -ri "s/^(  'dbuser' => ')(.*)(',)$/\1${quoted_user}\3/g;\
-             s/^(  'dbpassword' => ')(.*)(',)$/\1${quoted_password}\3/g;" "$CONFIGFILE"
-else
-
-    ## These variable are not used by current docker image after first install
-
-    config-add "\
-services:
-  $MASTER_BASE_SERVICE_NAME:
-    environment:
-      MYSQL_HOST: $MASTER_TARGET_SERVICE_NAME
-      MYSQL_DATABASE: $DBNAME
-      MYSQL_PASSWORD: $PASSWORD
-      MYSQL_USER: $USER
-"
-fi
-
-info "Configured $SERVICE_NAME code for $TARGET_SERVICE_NAME access."

nextcloud/hooks/mysql_database-relation-joined

@@ -0,0 +1 @@
+postgres_database-relation-joined

nextcloud/hooks/postgres_database-relation-joined

@@ -1,51 +1,11 @@
 #!/bin/bash
 
-. lib/common
+type="${0##*/}"
+type="${type%_database-relation-joined}"
 
-set -e
-
-PASSWORD="$(relation-get password)"
-USER="$(relation-get user)"
-DBNAME="$(relation-get dbname)"
-
-
-## This check adds purely arbitrary limits to what could be a password
-## if we need to open that more, just consider the next script where we'll
-## need to write in a PHP structure, or in YAML structure.
-
-## Note that here, "[]" chars are not accepted just because it doesn't seem evident
-## to test for those in bash.
-if ! [[ "$PASSWORD" =~ ^[a-zA-Z0-9~\`\&+=@\#^\*/\\_%\$:\;\!?.,\<\>{}()\"\'|-]*$ ]]; then
-     err "Invalid password chosen for postgres database."
+set-relation type "$type" || {
+    err "Could not set relation ${WHITE}type${NORMAL} to '$type'."
     exit 1
-fi
-
-## if config is not existent
-if [ -e "$CONFIGFILE" ] && grep "^  'dbuser' => '" "$CONFIGFILE" >/dev/null; then
-
-    ## 'occ' can't be used as it will try to connect to postgres before running and
-    ## will fail if user/password is not correct
-
-    ## We need to get through bash, and sed interpretation, then PHP single quoted strings.
-    quoted_user="${USER//\\/\\\\\\\\\\}"
-    quoted_user="${quoted_user//\'/\\\\\'}"
-    quoted_password="${PASSWORD//\\/\\\\\\\\\\}"
-    quoted_password="${quoted_password//\'/\\\\\'}"
-    sed -ri "s/^(  'dbuser' => ')(.*)(',)$/\1${quoted_user}\3/g;\
-             s/^(  'dbpassword' => ')(.*)(',)$/\1${quoted_password}\3/g;" "$CONFIGFILE"
-else
-
-    ## These variable are not used by current docker image after first install
-
-    config-add "\
-services:
-  $MASTER_BASE_SERVICE_NAME:
-    environment:
-      POSTGRES_HOST: $MASTER_TARGET_SERVICE_NAME
-      POSTGRES_DB: $DBNAME
-      POSTGRES_PASSWORD: $PASSWORD
-      POSTGRES_USER: $USER
-"
-fi
+}
 
-info "Configured $SERVICE_NAME code for $TARGET_SERVICE_NAME access."
+. ./hooks/sql_database-relation-joined

nextcloud/hooks/sql_database-relation-joined

@@ -0,0 +1,75 @@
+#!/bin/bash
+
+. lib/common
+
+set -e
+TYPE="$(relation-get type)" || {
+    err "No ${WHITE}type${NORMAL} set in relation."
+    exit 1
+}
+PASSWORD="$(relation-get password)"
+USER="$(relation-get user)"
+DBNAME="$(relation-get dbname)"
+
+
+## This check adds purely arbitrary limits to what could be a password
+## if we need to open that more, just consider the next script where we'll
+## need to write in a PHP structure, or in YAML structure.
+
+## Note that here, "[]" chars are not accepted just because it doesn't seem evident
+## to test for those in bash.
+if ! [[ "$PASSWORD" =~ ^[a-zA-Z0-9~\`\&+=@\#^\*/\\_%\$:\;\!?.,\<\>{}()\"\'|-]*$ ]]; then
+     err "Invalid password chosen for $type database."
+     exit 1
+fi
+
+## if config is not existent
+if [ -e "$CONFIGFILE" ] && grep "^  'dbuser' => '" "$CONFIGFILE" >/dev/null; then
+
+    ## 'occ' can't be used as it will try to connect to db before running and
+    ## will fail if user/password is not correct
+
+    ## We need to get through bash, and sed interpretation, then PHP single quoted strings.
+    quoted_user="${USER//\\/\\\\\\\\\\}"
+    quoted_user="${quoted_user//\'/\\\\\'}"
+    quoted_password="${PASSWORD//\\/\\\\\\\\\\}"
+    quoted_password="${quoted_password//\'/\\\\\'}"
+    case "$TYPE" in
+        mysql)
+            nextcloud_type="mysql";;
+        postgres)
+            nextcloud_type="pgsql";;
+        *)
+            err "Unknown type '$TYPE' for database."
+            exit 1
+            ;;
+    esac
+
+    sed -ri "s/^(  'dbuser' => ')(.*)(',)$/\1${quoted_user}\3/g;\
+             s/^(  'dbpassword' => ')(.*)(',)$/\1${quoted_password}\3/g;\
+             s/^(  'dbtype' => ')(.*)(',)$/\1${nextcloud_type}\3/g;\
+             s/^(  'dbhost' => ')(.*)(',)$/\1${MASTER_TARGET_SERVICE_NAME}\3/g;\
+             " "$CONFIGFILE"
+
+else
+
+    ## These variable are not used by current docker image after first install
+
+    if [ "$TYPE" == "mysql" ]; then
+        database_env_label="DATABASE"
+    else
+        database_env_label="DB"
+    fi
+
+    config-add "\
+services:
+  $MASTER_BASE_SERVICE_NAME:
+    environment:
+      ${TYPE^^}_HOST: $MASTER_TARGET_SERVICE_NAME
+      ${TYPE^^}_${database_env_label}: $DBNAME
+      ${TYPE^^}_PASSWORD: $PASSWORD
+      ${TYPE^^}_USER: $USER
+"
+fi
+
+info "Configured $SERVICE_NAME code for $TARGET_SERVICE_NAME access."

nextcloud/hooks/web_proxy-relation-joined

@@ -1,33 +1,20 @@
 #!/bin/bash
 
+. lib/common
+
 set -e
 
 DOMAIN=$(relation-get domain) || exit 1
 URL="$(relation-get url)" || exit 1
 PROTO="${URL%%://*}"
 
-if ! trusted_domains="$(
-    compose -q --no-relations --no-init occ "$MASTER_BASE_SERVICE_NAME" \
-         config:system:get trusted_domains)"; then
-    err "Couldn't get 'trusted_domains'. Here's the ouput:"
-    echo "$trusted_domains" | prefix "  | " >&2
-
-    echo "If the code of nextcloud is already there (command occ is found), but " >&2
-    echo "the database is not yet created, this situation will arise." >&2
+nextcloud:config:simple:add overwritehost "$DOMAIN" || {
+    err "Failed to set ${WHITE}overwritehost${NORMAL} to '$DOMAIN'."
     exit 1
-fi
+}
 
-occ_opts=(
-    ## necessary as nextcloud do not detect correctly those, and behind
-    ## a proxy, it will generate a lot of URL that are not detected
-    ## by means of ``ReverseProxyPass`` on apache for instance
+nextcloud:config:simple:add overwriteprotocol "$PROTO" || {
+    err "Failed to set ${WHITE}overwriteprotocol${NORMAL} to '$PROTO'."
+    exit 1
+}
 
-    config:system:set overwritehost --value="$DOMAIN" \;
-    config:system:set overwriteprotocol --value="$PROTO"
-)
-if ! [[ $'\n'"$trusted_domains"$'\n' == *$'\n'"$MASTER_BASE_SERVICE_NAME"$'\n'* ]]; then
-    trusted_index=$(echo "$trusted_domains" | wc -l)
-    debug "Adding $MASTER_TARGET_SERVICE_NAME to ${WHITE}trusted_domains${NORMAL}."
-    occ_opts+=( \; config:system:set trusted_domains "$trusted_index" --value="$MASTER_BASE_SERVICE_NAME")
-fi
-compose --no-relations --no-init occ "$MASTER_BASE_SERVICE_NAME" "${occ_opts[@]}"

nextcloud/lib/common

@@ -124,7 +124,51 @@ occ() {
             -v "$HOST_CHARM_STORE/${CHARM_REL_PATH#${CHARM_STORE}/}/src/occ.batch:/var/www/html/occ.batch" \
             -T --rm -u www-data "$SERVICE_NAME" /var/www/html/occ.batch "$@" | cat
 
+    if [ "${PIPESTATUS[0]}" != 0 ]; then
+        err "Failure to execute these ${WHITE}occ${NORMAL} commands:"
+        printf '%s ' "$@" |
+            sed -r "s/\\;/\n/g" |
+            sed -r "s/^\s*(.*)\s*$/${WHITE}\1${NORMAL}/g" |
+            prefix "  ${DARKGRAY}>${NORMAL} " >&2
+        echo "" >&2
+        echo "" >&2
+        echo "  If the code of nextcloud is already there (command occ is found), but " >&2
+        echo "  the database is not yet created, this situation will arise." >&2
         return "${PIPESTATUS[0]}"
+    fi
+
 }
 
 
+
+nextcloud:config:simple:add() {
+    local key="$1" value="$2"
+    create_occ_if_not_exists || return 1
+
+    if ![ -e "$CONFIGFILE" ]; then
+        err "Config file '$CONFIGFILE' does not exist."
+        return 1
+    fi
+
+    if [ "$#" -lt 2 ]; then
+        err "No value specified for '$key'."
+        return 1
+    fi
+
+    ## check for \ and ' in value and key
+    if [[ "$value" =~ [\\\'] ]]; then
+        err "Unsupported value for '$key' contains a backslash or a single quote."
+        return 1
+    fi
+    if [[ "$key" =~ [\\\'] ]]; then
+        err "Key '$key' contains a backslash or a single quote."
+        return 1
+    fi
+
+    if grep "^  '$key' => '" "$CONFIGFILE" >/dev/null; then
+        sed -ri "s/^(  '$key' => ')(.*)(',)$/\1${value}\3/g" "$CONFIGFILE"
+        return 0
+    fi
+    ## Add '$key' => 'value', to the end of the file, before the closing paren.
+    sed -ri "s/^(\);)$/  '$key' => '${value}',\n\1/g" "$CONFIGFILE"
+}

nextcloud/metadata.yml

@@ -7,25 +7,18 @@ config-resources:
 provides:
   nextcloud-app:
 uses:
-  postgres-database:
+  sql-database:
     #constraint: required | recommended | optional
     #auto: pair | summon | none ## default: pair
     constraint: required
     auto: summon
     solves:
       database: "main storage"
-  mysql-database:
+  web-proxy:
     #constraint: required | recommended | optional
     #auto: pair | summon | none ## default: pair
     constraint: optional
     auto: pair
-    solves:
-      database: "main storage"
-  web-proxy:
-    #constraint: required | recommended | optional
-    #auto: pair | summon | none ## default: pair
-    constraint: required
-    auto: summon
     solves:
       proxy: "Public access"
     default-options:

odoo-tecnativa/README.org

@@ -0,0 +1,40 @@
+
+
+Odoo-tecnativa is a odoo image containing all source and add-ons because
+we want to certify the whole image.
+
+So this means there are no builds being managed by compose, and no injection
+of code.
+
+
+* Usage
+
+** dbfilter
+
+With image ~16.0~, an advanced version of ~dbfilter~ is installed. Here
+a few examples:
+
+#+begin_src yaml
+odoo:
+  # ..
+  options:
+    dbfilter:
+      ## DOMAIN_REGEX: DBFILTER
+      '^www.domain.org$': '^bar$'          ## domain `www.domain.org` can only see `bar`.
+      '^foo\.': 'foo_.*'                   ## domain starting with `foo.` can see db `foo_`
+      '^(?P<name>[^.]+)\.': '%{name}s_.*'  ## domain starting with `<PREFIX>.` can see db `PREFIX_`
+      '': 'other_.*'                       ## all domains can see db 'other_*'
+
+      ## Don't forget to configure the domains in the web-proxy part !
+  relations:
+    web-proxy:
+      apache:
+        domain: www.domain.org
+        aliases:
+        - foo.otherdomain.com
+        - bar.wiz.eu
+        - test.domain.org
+#+end_src
+
+If there's only one database seen because of the ~dbfilter~, odoo will
+use it by default.

odoo-tecnativa/README.rst

@@ -1,8 +0,0 @@
-
-
-Odoo-tecnativa is a odoo image containing all source and add-ons because
-we want to certify the whole image.
-
-So this means there are no builds being managed by compose, and no injection
-of code.
-

odoo-tecnativa/actions/load

@@ -160,26 +160,41 @@ check_input() {
     }
 }
 
+## Beware that we are not on the host, so we need to create the
+## fifo in a place we can share with the curl container.
+export TMPDIR=/var/cache/compose
+settmpdir RESTORE_TMPDIR
 
+mkfifo "$RESTORE_TMPDIR/fifo"
+
+## Using fifo because if using ``@-`` curl will load all the database
+## in memory before sending it, which is NOT desirable as the size of
+## the database can be greater than available memory. Using fifo, we
+## force the data to be streamed.
 cmd=(
     docker run -i --rm --network "$container_network"
+      -v "$RESTORE_TMPDIR/fifo:/tmp/restore/fifo"
         "$DEFAULT_CURL_IMAGE"
         -sS 
         -X POST 
         -F "master_pwd=${ADMIN_PASSWORD}" 
-        -F "backup_file=@-"
         -F "name=${dbname}"
-        "${curl_opts[@]}"
+        -F "backup_file=@/tmp/restore/fifo"
         http://${container_ip}:8069/web/database/restore
 	)
 
 ## XXXvlab: contains password, left only for advanced debug
-#debug "${cmd[@]}"
+#echo "COMMAND: ${cmd[@]}" >&2
 
-out=$(set -o pipefail; check_input | "${cmd[@]}") || {
+"${cmd[@]}" > "$RESTORE_TMPDIR/out" &
+pid=$!
+check_input > "$RESTORE_TMPDIR/fifo"
+wait "$pid" || {
     die "Posting to odoo restore API through curl was unsuccessfull."
 }
 
+out=$(cat "$RESTORE_TMPDIR/out")
+
 if [[ "$out" == *"<html>"* ]]; then
     errmsg=$(echo "$out" | grep "alert-danger")
     errmsg=${errmsg#*>}

odoo-tecnativa/hooks/init

@@ -36,6 +36,39 @@ else
     modules="base,${modules}"
 fi
 
+cmd_args=()
+
+## dbfilter management
+service_def=$(get_compose_service_def "$SERVICE_NAME") || return 1
+
+dbfilter=$(e "$service_def" | yq e -r=false '.options.dbfilter') || true
+if [ -n "$dbfilter" ]; then
+    type=$(e "$dbfilter" | yq e "type")
+    case "${type:2}" in
+        str)
+            dbfilter=$(e "$dbfilter" | yq e -o=json '.')
+            cmd_args+=("--db-filter=${dbfilter}")
+            ;;
+        map)
+            python_expr="["
+            while read-0 key val; do
+                [[ "$key" == "null" ]] && key="None"
+                python_expr+="(${key//\$/\$\$}, ${val//\$/\$\$}),"
+            done < <(e "$dbfilter" | yq -o=json -0 'to_entries | .[] | [.key, .value] | .[]')
+            python_expr+="]"
+            cmd_args+=("--db-filter=${python_expr}")
+            ;;
+        null)
+            :
+            ;;
+        *)
+            err "Invalid value for ${WHITE}dbfilter${NORMAL}: ${dbfilter}" >&2
+            echo "  It should be a string or a key/value mapping" >&2
+            exit 1
+            ;;
+    esac
+fi
+
 
 ## config file
 conf=$(options-get conf 2>/dev/null) || true
@@ -65,4 +98,10 @@ $SERVICE_NAME:
   command:
     - \"--workers=${workers}\"
     - \"-i ${modules}\"
+$(
+for arg in "${cmd_args[@]}"; do
+    echo "    - |"
+    e "$arg" | prefix "      "
+done
+)
 "

piwigo/hooks/post_deploy

@@ -77,7 +77,7 @@ if [ -f "$CONFIG" ]; then
 
     export SERVICE_NAME="$ts"
     export SERVICE_DATASTORE="$DATASTORE/$SERVICE_NAME"
-    DOCKER_BASE_IMAGE=$(service_base_docker_image "$SERVICE_NAME")
+    DOCKER_BASE_IMAGE=$(service_ensure_image_ready "$SERVICE_NAME")
     export DOCKER_BASE_IMAGE
 
     target_charm=$(get_service_charm "$ts") || exit 1

postgres/hooks/schedule_command-relation-joined

@@ -6,40 +6,67 @@
 ##  - they are launched from the host
 ##  - the target of the link is launched first, and get a chance to ``relation-set``
 ##  - both side of the scripts get to use ``relation-get``.
+##  - use actions of other side for other side's business logic
 
 . lib/common
 
-set -e
 
-## XXXvlab: should use container name here so that it could support
-## multiple postgres
-label=${SERVICE_NAME}-pg-backup
-DST=$CONFIGSTORE/$TARGET_SERVICE_NAME/etc/cron/$label
-schedule=$(relation-get schedule)
+##
+## This script has to replace `exclude-dbs` options to match definition
+## from `schedule-command` interface that is awaited by the target side
+## in the `pre_deploy` script.
+##
 
-if ! echo "$schedule" | egrep '^\s*(([0-9/,*-]+\s+){4,4}[0-9/,*-]+|@[a-z]+)\s*$' >/dev/null 2>&1; then
-    err "Unrecognized schedule '$schedule'."
+if [ -z "$RELATION_DATA_FILE" ]; then
+    err "$FUNCNAME: relation does not seems to be correctly setup."
     exit 1
 fi
 
+if ! [ -r "$RELATION_DATA_FILE" ]; then
+    err "$FUNCNAME: can't read relation's data." >&2
+    exit 1
+fi
+
+if [ "$(cat "$RELATION_DATA_FILE" | shyaml get-type)" == "str" ]; then
+    ## Cached version already there
+
+    ## Note that we rely on the fact that when the relations
+    ## options are changed in the `compose.yml` file, the relation
+    ## data file is recreated from the values of the
+    ## `compose.yml`.
+    info "Previous relation data is still valid."
+    exit 0
+fi
+
+schedule=$(relation-get schedule) || true
 exclude_dbs=$(relation-get exclude-dbs 2>/dev/null) || true
 exclude_dbs=$(echo "$exclude_dbs" | shyaml get-values 2>/dev/null |
                   nspc) || true
 
-## Warning: 'docker -v' will use HOST directory even if launched from
-##   'cron' container.
-file_put "$DST" <<EOF
-COMPOSE_LAUNCHER_OPTS=$COMPOSE_LAUNCHER_OPTS
-
-$schedule     root  lock $label -D -p 10 -c "\
-    docker run --rm \
-        -e PGHOST=${SERVICE_NAME} \
-        --network ${PROJECT_NAME}_default \
-        -e exclude_dbs=\"$exclude_dbs\" \
-        -v \"$LOCAL_DB_PASSFILE\":/root/.pgpass \
-        -v \"$HOST_CHARM_STORE/${CHARM_REL_PATH#${CHARM_STORE}/}/resources/bin/pg-backup:/usr/sbin/pg-backup\" \
-        -v \"$SERVICE_DATASTORE/var/backups/pg:/var/backups/pg\" \
-        --entrypoint pg-backup \
-        \"$DOCKER_BASE_IMAGE\"" 2>&1 | ts '\%F \%T \%Z' >> /var/log/cron/pg-backup_script.log
-EOF
-chmod +x "$DST"
+
+lock_opts=(-D -p 10)
+
+
+command=(
+    docker run --rm
+    -e PGHOST=${SERVICE_NAME}
+    --network ${PROJECT_NAME}_default
+    -e exclude_dbs="$exclude_dbs"
+    -v "$LOCAL_DB_PASSFILE":/root/.pgpass
+    -v "$HOST_CHARM_STORE/${CHARM_REL_PATH#${CHARM_STORE}/}/resources/bin/pg-backup:/usr/sbin/pg-backup"
+    -v "$SERVICE_DATASTORE/var/backups/pg:/var/backups/pg"
+    --entrypoint pg-backup
+    "$DOCKER_BASE_IMAGE"
+)
+
+quoted_command=()
+for arg in "${command[@]}"; do
+    quoted_command+=("$(printf "%q" "$arg")")
+done
+
+printf "(%s) {%s} %s\n" \
+       "$schedule" \
+       "${lock_opts[*]}" \
+       "${quoted_command[*]}" > "$RELATION_DATA_FILE"
+    echo "data: '$(cat "$RELATION_DATA_FILE")'" >&2
+

postgres/hooks/sql_database-relation-joined

@@ -0,0 +1,16 @@
+#!/bin/bash
+
+## When writing relation script, remember:
+##  - they should be idempotents
+##  - they can be launched while the dockers is already up
+##  - they are launched from the host
+##  - the target of the link is launched first, and get a chance to ``relation-set``
+##  - both side of the scripts get to use ``relation-get``.
+
+relation-set type postgres || {
+    err "Could not set relation ${WHITE}type${NORMAL} to 'postgres'."
+    exit 1
+}
+
+. hooks/postgres_database-relation-joined
+

postgres/metadata.yml

@@ -4,6 +4,7 @@ data-resources:
   - /var/lib/postgresql/data
 provides:
   postgres-database:
+  sql-database:
 
 uses:
   schedule-command:

precise/base-0k/hooks/install.d/30-customize.sh

@@ -157,3 +157,44 @@ set is hlsearch ai ic scs
 nnoremap <esc><esc> :nohls<cr>
 
 EOF
+
+protect_pre_install /usr/share/nano/yaml.nanorc
+
+cat << EOF > /usr/share/nano/yaml.nanorc
+# Supports `YAML` files
+syntax "YAML" "\.ya?ml$"
+header "^(---|===)" "%YAML"
+
+## Keys
+color brightmagenta "^\s*[\$A-Za-z0-9_-]+\:"
+color brightmagenta "^\s*@[\$A-Za-z0-9_-]+\:"
+
+## Values
+color yellow ":\s.+$"
+## Booleans
+icolor brightcyan " (y|yes|n|no|true|false|on|off)$"
+## Numbers
+color brightred " [[:digit:]]+(\.[[:digit:]]+)?"
+## Arrays
+color red "\[" "\]" ":\s+[|>]" "^\s*- "
+## Reserved
+color green "(^| )!!(binary|bool|float|int|map|null|omap|seq|set|str) "
+
+## Comments
+color cyan "#.*$"
+
+## Errors
+##color ,red ":\w.+$"
+##color ,red ":'.+$"
+##color ,red ":".+$"
+##color ,red "\s+$"
+
+## Non closed quote
+color ,red "['\"][^['\"]]*$"
+
+## Closed quotes
+color yellow "['\"].*['\"]"
+
+## Equal sign
+color white ":( |$)"
+EOF

rallly/hooks/init

@@ -2,6 +2,8 @@
 
 set -e
 
+version_gt() { test "$(echo "$@" | tr " " "\n" | sort -V | head -n 1)" != "$1"; }
+
 ## bug: https://github.com/adoptium/containers/issues/215#issuecomment-1142046045
 docker_version=$(docker info --format '{{.ServerVersion}}')
 if ! version_gt "$docker_version" 20.10.0; then

rocketchat/README.org

@@ -11,7 +11,8 @@ We are using official image. Latest tags usually.
 ** Test new version
 
 Rocket.chat has a powerfull and working database update mecanism that
-will take care of migrating database on startup.
+will take care of migrating database on startup *as long as you jump only
+from one major version to the next*.
 
 *** Get latest available versions
 

rsync-backup-target/build/Dockerfile

@@ -4,7 +4,8 @@ MAINTAINER Valentin Lab <valentin.lab@kalysto.org>
 
 ## coreutils is for ``date`` support of ``--rfc-3339=seconds`` argument.
 ## findutils is for ``find`` support of ``--newermt`` argument.
-RUN apk add rsync sudo bash openssh-server coreutils findutils
+## gawk is for ``awk`` support of unicode strings.
+RUN apk add rsync sudo bash openssh-server coreutils findutils gawk
 RUN ssh-keygen -A
 
 ## New user/group rsync/rsync with home dir in /var/lib/rsync

rsync-backup-target/build/src/usr/local/sbin/ssh-admin-cmd-validate

@@ -80,6 +80,15 @@ elif [[ "$SSH_ORIGINAL_COMMAND" =~ ^"ssh-key get-type "[a-zA-Z0-9._-]+$ ]]; then
 
     # echo "Would accept: $SSH_ORIGINAL_COMMAND" >&2
 	exec sudo /usr/local/sbin/ssh-key get-type "$label" "${ssh_args[@]:2}"
+elif [[ "$SSH_ORIGINAL_COMMAND" =~ ^"ssh-key "(enable|disable)" "[a-zA-Z0-9._-]+$ ]]; then
+	log "ACCEPTED: $SSH_ORIGINAL_COMMAND"
+
+    ## Interpret \ to allow passing spaces (want to avoid possible issue with \n)
+    #read -a ssh_args <<< "${SSH_ORIGINAL_COMMAND}"
+    ssh_args=(${SSH_ORIGINAL_COMMAND})
+
+    # echo "Would accept: $SSH_ORIGINAL_COMMAND" >&2
+	exec sudo /usr/local/sbin/ssh-key "${ssh_args[1]}" "$label" "${ssh_args[@]:2}"
 elif [[ "$SSH_ORIGINAL_COMMAND" =~ ^"request-recovery-key "[a-zA-Z0-9._-]+$ ]]; then
 	log "ACCEPTED: $SSH_ORIGINAL_COMMAND"
 

rsync-backup-target/build/src/usr/local/sbin/ssh-key

@@ -25,18 +25,115 @@ DARKPINK="${ANSI_ESC}0;35m"
 DARKCYAN="${ANSI_ESC}0;36m"
 DARKWHITE="${ANSI_ESC}0;37m"
 
+read-0() {
+    local eof= IFS=''
+    while [ "$1" ]; do
+        read -r -d '' -- "$1" || eof=1
+        shift
+    done
+    [ -z "$eof" ]
+}
+
+
+col:normalize:size() {
+  local alignment="$1" colors="$2"
+
+
+ # Encode the associative array into a string for awk
+  local col_colors_string=""
+  for key in "${!col_colors[@]}"; do
+    col_colors_string+="${key}=${col_colors[$key]} "
+  done
+
+  # Pass the string to awk
+  awk -v alignment="$alignment" -v colors="$colors" -v colorStr="$col_colors_string" -v normal="$NORMAL" '
+  BEGIN {
+    # Split the color string into key-value pairs
+    n = split(colorStr, pairs);
+    for (i = 1; i <= n; i++) {
+      split(pairs[i], kv, "=");
+      col_colors[kv[1]] = kv[2];
+    }
+
+  }
+  {
+    # Store the entire line in the lines array
+    lines[NR] = $0;
+
+    # Split the line into fields and update max lengths
+    split($0, fields);
+    for (i = 1; i <= length(fields); i++) {
+      if (length(fields[i]) > max[i]) {
+        max[i] = length(fields[i]);
+      }
+    }
+  }
+  END {
+    # Print lines with fields padded to max, apply color
+    for (i = 1; i <= NR; i++) {
+      split(lines[i], fields);
+      line = "";
+      for (j = 1; j <= length(fields); j++) {
+        # Determine alignment
+        align = substr(alignment, j, 1) == "+" ? "+" : "-";
+        color_code = substr(colors, j, 1);
+
+        color_prefix = (color_code != "-" && color_code in col_colors) ? col_colors[color_code] : "";
+        color_suffix = (color_prefix != "") ? normal : "";
+
+        # Construct line with alignment and color
+        line = line color_prefix sprintf("%" align max[j] "s ", fields[j]) color_suffix;
+      }
+      print line;
+    }
+  }'
+}
+
+declare -A col_colors=(
+    [s]=${DARKGRAY}   ## s for 'slate' (actually gray)
+    [r]=${DARKRED}
+    [g]=${DARKGREEN}
+    [y]=${DARKYELLOW}
+    [b]=${DARKBLUE}
+    [p]=${DARKPINK}
+    [c]=${DARKCYAN}
+    [w]=${DARKWHITE}
+
+    [S]=${GRAY}
+    [R]=${RED}
+    [G]=${GREEN}
+    [Y]=${YELLOW}
+    [B]=${BLUE}
+    [P]=${PINK}
+    [C]=${CYAN}
+    [W]=${WHITE}
+)
 
 ssh-key-ls() {
     local label="$1" f content
-    for f in "${RSYNC_KEY_PATH}"/backup/"$label"/*.pub; do
+    shift
+    while read-0 f; do
         [ -e "$f" ] || continue
         ident=${f##*/}
+        if [ "$f" != "${f%.disabled}" ]; then
+            enabled="${RED}x"
+            ident=${ident%.disabled}
+        else
+            enabled="${GREEN}✓"
+        fi
         ident=${ident%.pub}
-        content=$(cat "$f")
+        content=$(cat "$f") || return 1
         key=${content#* }
         key=${key% *}
-        printf "${DARKGRAY}..${NORMAL}%24s ${DARKCYAN}%s${NORMAL}\n" "${key: -24}" "$ident"
-    done
+        commentary=${content##* }
+        type=${commentary%%@*}
+        # printf "${DARKGRAY}..${NORMAL}%12s ${DARKPINK}%-7s${NORMAL} ${DARKCYAN}%s${NORMAL}\n" \
+            #        "${key: -12}" "${type}" "$ident"
+        printf "%s %s %s %s\n" "…${key: -12}" "$type" "$enabled" "$ident"
+    done < <(find "${RSYNC_KEY_PATH}"/backup/"$label" \
+                  -maxdepth 1 -mindepth 1 \
+                  \( -name \*.pub -or -name \*.pub.disabled \) -print0 | sort -z
+            ) | col:normalize:size "----" "ysGc"
 }
 
 
@@ -45,10 +142,47 @@ ssh-key-rm() {
 
     delete="${RSYNC_KEY_PATH}/backup/$label/$ident.pub"
     if ! [ -e "$delete" ]; then
+        if ! [ -e "${delete}.disabled" ]; then
             echo "Error: key '$ident' not found." >&2
             return 1
         fi
+        rm "${delete}.disabled"
+        return 0
+    fi
     rm "$delete"
+    /usr/local/sbin/ssh-update-keys
+}
+
+ssh-key-disable() {
+    local label="$1" ident="$2" disable
+
+    disable="${RSYNC_KEY_PATH}/backup/$label/$ident.pub"
+    if ! [ -e "$disable" ]; then
+        if ! [ -e "${disable}.disabled" ]; then
+            echo "Error: key '$ident' not found." >&2
+            return 1
+        fi
+        echo "Already disabled." >&2
+        return 0
+    fi
+    mv "${disable}" "${disable}.disabled"
+
+    /usr/local/sbin/ssh-update-keys
+}
+
+ssh-key-enable() {
+    local label="$1" ident="$2" enable
+
+    enable="${RSYNC_KEY_PATH}/backup/$label/$ident.pub.disabled"
+    if ! [ -e "$enable" ]; then
+        if ! [ -e "${enable%.disabled}" ]; then
+            echo "Error: key '$ident' not found." >&2
+            return 1
+        fi
+        echo "Already enabled." >&2
+        return 0
+    fi
+    mv "${enable}" "${enable%.disabled}"
 
     /usr/local/sbin/ssh-update-keys
 }
@@ -141,6 +275,11 @@ case "$1" in
         shift
         ssh-key-ls "$@"
         ;;
+    "disable"|"enable")
+        arg="$1"
+        shift
+        ssh-key-"$arg" "$@"
+        ;;
     "get-type")
         shift
         ssh-key-get-type "$@"

rsync-backup-target/hooks/init

@@ -16,12 +16,13 @@ set -e
 service_def=$(get_compose_service_def "$SERVICE_NAME")
 
 admin_keys=$(echo "$service_def" | shyaml -y get-value options.admin 2>/dev/null) || {
-    err "You must specify a ${WHITE}admin${NORMAL} struct to use this service"
+    err "You must specify a ${WHITE}admin${NORMAL} struct in ${DARKYELLOW}$SERVICE_NAME${NORMAL}'s options"
     exit 1
 }
 
 [ "$(echo "$admin_keys" | shyaml -y get-type 2>/dev/null)" == "struct" ] || {
-    err "Invalid value type for ${WHITE}admin${NORMAL}, please provide a struct"
+    err "Invalid value type for ${WHITE}admin${NORMAL} in" \
+        "${DARKYELLOW}$SERVICE_NAME${NORMAL}'s options, please provide a struct"
     exit 1
 }
 

rsync-backup/hooks/schedule_command-relation-joined

@@ -6,18 +6,33 @@
 ##  - they are launched from the host
 ##  - the target of the link is launched first, and get a chance to ``relation-set``
 ##  - both side of the scripts get to use ``relation-get``.
+##  - use actions of other side for other side's business logic
 
 . lib/common
 
-set -e
-
-schedule=$(relation-get schedule)
+if [ -z "$RELATION_DATA_FILE" ]; then
+    err "$FUNCNAME: relation does not seems to be correctly setup."
+    exit 1
+fi
 
-if ! echo "$schedule" | egrep '^\s*(([0-9/,*-]+\s+){4,4}[0-9/,*-]+|@[a-z]+)\s*$' >/dev/null 2>&1; then
-    err "Unrecognized schedule '$schedule'."
+if ! [ -r "$RELATION_DATA_FILE" ]; then
+    err "$FUNCNAME: can't read relation's data." >&2
     exit 1
 fi
 
+if [ "$(cat "$RELATION_DATA_FILE" | shyaml get-type)" == "str" ]; then
+    ## Cached version already there
+
+    ## Note that we rely on the fact that when the relations
+    ## options are changed in the `compose.yml` file, the relation
+    ## data file is recreated from the values of the
+    ## `compose.yml`.
+    info "Previous relation data is still valid."
+    exit 0
+fi
+
+set -e
+
 private_key=$(options-get private-key) || exit 1
 target=$(options-get target) || exit 1
 ident=$(options-get ident) || exit 1
@@ -29,21 +44,29 @@ host_path_key="$SERVICE_CONFIGSTORE${local_path_key}"
 echo "$private_key" | file_put "$host_path_key/id_rsa"
 chmod 600 "$host_path_key/id_rsa"
 
-label="${SERVICE_NAME}"
-DST=$CONFIGSTORE/$TARGET_CHARM_NAME/etc/cron/$label
-
-## Warning: using '\' in heredoc will be removed in the final cron file, which
-##   is totally wanted: cron does not support multilines.
-file_put "$DST" <<EOF
-$schedule    root   lock $label -v -D -p 10 -k -c "\
-    docker run --rm \
-        -e LABEL_HOSTNAME=\"$ident\" \
-        -v \"$RSYNC_CONFIG_DIR:/etc/rsync\" \
-        -v \"$host_path_key:$local_path_key\" \
-        -v \"$HOST_DATASTORE:/mnt/source\" \
-        -v \"$HOST_COMPOSE_YML_FILE:/mnt/source/compose.yml\" \
-        --network ${PROJECT_NAME}_default \
-        \"$DOCKER_BASE_IMAGE\" \
-        /mnt/source \"$target\"" 2>&1 | ts '\%F \%T' >> /var/log/cron/${label}_script.log
-EOF
-chmod +x "$DST"
+schedule=$(relation-get schedule) || true
+
+lock_opts=(-D -p 10 -k)
+
+command=(
+    docker run --rm
+    -e LABEL_HOSTNAME="$ident"
+    -v "$RSYNC_CONFIG_DIR:/etc/rsync"
+    -v "$host_path_key:$local_path_key"
+    -v "$HOST_DATASTORE:/mnt/source"
+    -v "$HOST_COMPOSE_YML_FILE:/mnt/source/compose.yml"
+    --network ${PROJECT_NAME}_default
+    "$DOCKER_BASE_IMAGE"
+    /mnt/source "$target"
+)
+
+quoted_command=()
+for arg in "${command[@]}"; do
+    quoted_command+=("$(printf "%q" "$arg")")
+done
+
+printf "(%s) {%s} %s\n" \
+       "$schedule" \
+       "${lock_opts[*]}" \
+       "${quoted_command[*]}" > "$RELATION_DATA_FILE"
+

sftp/lib/common

@@ -127,7 +127,7 @@ make_build_script() {
 
         while read-0 key; do
             keys+="$key"$'\n'
-        done < <(echo "$user_def" | shyaml get-values-0 -q keys)
+        done < <(echo "$user_def" | shyaml -q get-values-0 keys)
         if [ "$keys" ]; then
             code+="mkdir -p \"/home/$user/.ssh\""$'\n'
             code+="cat <<EOF > /home/$user/.ssh/authorized_keys"$'\n'