You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

2288 lines
75 KiB

9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
9 years ago
  1. #!/bin/bash
  2. ##
  3. ## TODO:
  4. ## - subordinate container should really be able to modify base image of their master
  5. ## - this could be done through docker-update
  6. ## - I'm not happy with the current build using 'build/' directory, this should be
  7. ## changed to:
  8. ## - always have a base image (specified in metadata), and always have hooks/install
  9. ## executed and merge in image (like docker-build-charm).
  10. ## - container base image is ALWAYS the image of the master container... this brings
  11. ## questions about a double way to express inheritage (through relations as it is
  12. ## implemented now, or through this base-image ?)
  13. ## - the name of the scripts for relation (aka relation_name-relation-joined) is bad as
  14. ## reading the name in a hooks/ dir, there are no way to know if we are the target or
  15. ## the base of the relation.
  16. ## - we could leverage a 'relations/' dir on the root of the charm, with both:
  17. ## 'relations/provide/relation_name' and 'relations/receive/relation_name'
  18. ## - a very bad point with the actual naming is that we can't have a providing AND
  19. ## receiving a relation with same name.
  20. ## - The cache system should keep md5 of docker-compose and other things between runs
  21. ## - The cache system should use underlying function that have only arguments inputs.
  22. ## This will allow to cache completely without issues function in time.
  23. ## - would probably need instrospection in charm custom action to know if these need
  24. ## init or relations to be set up.
  25. #:-
  26. [ -e /etc/shlib ] && . /etc/shlib || {
  27. echo "Unsatisfied dependency. Please install 'kal-shlib-core'."
  28. exit 1
  29. }
  30. #:-
  31. include pretty
  32. include parse
  33. depends shyaml docker
  34. version=0.1
  35. usage="$exname [COMPOSE_OPTS] [ACTION [ACTION_OPTS]]"
  36. help="\
  37. $WHITE$exname$NORMAL jobs is to run various shell scripts to build
  38. a running orchestrated and configured docker containers. These shell
  39. scripts will have the opportunity to build a 'docker-compose.yml'.
  40. Once init script and relations scripts are executed, $WHITE$exname$NORMAL
  41. delegate the launching to ${WHITE}docker-compose${NORMAL} by providing it
  42. the final 'docker-compose.yml'.
  43. $WHITE$exname$NORMAL also leverage charms to offer some additional custom
  44. actions per charm, which are simply other scripts that can be
  45. run without launching ${WHITE}docker-compose${NORMAL}.
  46. In compose message, color coding is enforced as such:
  47. - ${DARKCYAN}action$NORMAL,
  48. - ${DARKBLUE}relation$NORMAL,
  49. - ${DARKYELLOW}charm${NORMAL}/${DARKYELLOW}service${NORMAL},
  50. - ${WHITE}option-name${NORMAL}/${WHITE}command-name${NORMAL}/${WHITE}Section-Title${NORMAL}
  51. $WHITE$exname$NORMAL reads '/etc/compose.conf' for global variables, and
  52. '/etc/compose.local.conf' for local host adjustements.
  53. ${WHITE}$exname Options${NORMAL}:
  54. -h, --help Print this message and quit
  55. (ignoring any other options)
  56. -V, --version Print current version and quit
  57. (ignoring any other options)
  58. -v, --verbose Be more verbose
  59. -d, --debug Print full debugging information (sets also verbose)
  60. "
  61. [[ "${BASH_SOURCE[0]}" != "${0}" ]] && SOURCED=true
  62. export CACHEDIR=/var/cache/compose
  63. export VARDIR=/var/lib/compose
  64. md5_compat() { md5sum | cut -c -32; }
  65. quick_cat_file() { quick_cat_stdin < "$1"; }
  66. quick_cat_stdin() { local IFS=''; while read -r line; do echo "$line"; done ; }
  67. export -f quick_cat_file quick_cat_stdin md5_compat
  68. clean_cache() {
  69. local i=0
  70. for f in $(ls -t "$CACHEDIR/"*.cache.* 2>/dev/null | tail -n +500); do
  71. ((i++))
  72. rm -f "$f"
  73. done
  74. if (( i > 0 )); then
  75. debug "${WHITE}Cleaned cache:${NORMAL} Removed $((i)) elements (current cache size is $(du -sh "$CACHEDIR" | cut -f 1))"
  76. fi
  77. }
  78. trap_add "EXIT" clean_cache
  79. usage="$exname CHARM"'
  80. Deploy and manage a swarm of containers to provide services based on
  81. a ``compose.yml`` definition and charms from a ``charm-store``.
  82. '
  83. export DEFAULT_COMPOSE_FILE
  84. ##
  85. ## Merge YAML files
  86. ##
  87. export _merge_yaml_common_code="
  88. import sys
  89. import yaml
  90. try:
  91. # included in standard lib from Python 2.7
  92. from collections import OrderedDict
  93. except ImportError:
  94. # try importing the backported drop-in replacement
  95. # it's available on PyPI
  96. from ordereddict import OrderedDict
  97. ## Ensure that there are no collision with legacy OrderedDict
  98. ## that could be used for omap for instance.
  99. class MyOrderedDict(OrderedDict):
  100. pass
  101. yaml.add_representer(
  102. MyOrderedDict,
  103. lambda cls, data: cls.represent_dict(data.items()))
  104. yaml.add_constructor(
  105. yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG,
  106. lambda cls, node: MyOrderedDict(cls.construct_pairs(node)))
  107. def fc(filename):
  108. with open(filename) as f:
  109. return f.read()
  110. def merge(*args):
  111. # sys.stderr.write('%r\n' % (args, ))
  112. args = [arg for arg in args if arg is not None]
  113. if len(args) == 0:
  114. return None
  115. if len(args) == 1:
  116. return args[0]
  117. if all(isinstance(arg, (int, basestring, bool)) for arg in args):
  118. return args[-1]
  119. elif all(isinstance(arg, list) for arg in args):
  120. res = []
  121. for arg in args:
  122. for elt in arg:
  123. if elt in res:
  124. res.remove(elt)
  125. res.append(elt)
  126. return res
  127. elif all(isinstance(arg, dict) for arg in args):
  128. keys = set()
  129. for arg in args:
  130. keys |= set(arg.keys())
  131. dct = {}
  132. for key in keys:
  133. sub_args = []
  134. for arg in args:
  135. if key in arg:
  136. sub_args.append(arg)
  137. try:
  138. dct[key] = merge(*(a[key] for a in sub_args))
  139. except NotImplementedError as e:
  140. raise NotImplementedError(
  141. e.args[0],
  142. '%s.%s' % (key, e.args[1]) if e.args[1] else key,
  143. e.args[2])
  144. if dct[key] is None:
  145. del dct[key]
  146. return dct
  147. else:
  148. raise NotImplementedError(
  149. 'Unsupported types: %s'
  150. % (', '.join(list(set(arg.__class__.__name__ for arg in args)))), '', args)
  151. return None
  152. def merge_cli(*args):
  153. try:
  154. c = merge(*args)
  155. except NotImplementedError as e:
  156. sys.stderr.write('%s. Conflicting key is %r. Values are:\n%s\n' % (e.args[0], e.args[1], e.args[2]))
  157. exit(1)
  158. if c is not None:
  159. print '%s' % yaml.dump(c, default_flow_style=False)
  160. "
  161. merge_yaml() {
  162. if ! [ -r "$state_tmpdir/merge_yaml.py" ]; then
  163. cat <<EOF > "$state_tmpdir/merge_yaml.py"
  164. $_merge_yaml_common_code
  165. merge_cli(*(yaml.load(fc(f)) for f in sys.argv[1:]))
  166. EOF
  167. fi
  168. python "$state_tmpdir/merge_yaml.py" "$@"
  169. }
  170. export -f merge_yaml
  171. merge_yaml_str() {
  172. local entries="$@"
  173. if ! [ -r "$state_tmpdir/merge_yaml_str.py" ]; then
  174. cat <<EOF > "$state_tmpdir/merge_yaml_str.py"
  175. $_merge_yaml_common_code
  176. merge_cli(*(yaml.load(f) for f in sys.argv[1:]))
  177. EOF
  178. fi
  179. python "$state_tmpdir/merge_yaml_str.py" "$@"
  180. }
  181. export -f merge_yaml_str
  182. yaml_key_val_str() {
  183. local entries="$@"
  184. if ! [ -r "$state_tmpdir/yaml_key_val_str.py" ]; then
  185. cat <<EOF > "$state_tmpdir/yaml_key_val_str.py"
  186. $_merge_yaml_common_code
  187. print '%s' % yaml.dump({
  188. yaml.load(sys.argv[1]):
  189. yaml.load(sys.argv[2])}, default_flow_style=False)
  190. EOF
  191. fi
  192. python "$state_tmpdir/yaml_key_val_str.py" "$@"
  193. }
  194. export -f yaml_key_val_str
  195. ##
  196. ## Docker
  197. ##
  198. docker_has_image() {
  199. local image="$1"
  200. images=$(docker images -q "$image" 2>/dev/null) || {
  201. err "docker images call has failled unexpectedly."
  202. return 1
  203. }
  204. [ "$images" ]
  205. }
  206. export -f docker_has_image
  207. cmd_on_base_image() {
  208. local charm="$1"
  209. shift
  210. base_image=$(service_base_docker_image "$charm") || return 1
  211. docker run -i --entrypoint /bin/bash "$base_image" -c "$*"
  212. }
  213. export -f cmd_on_base_image
  214. cached_cmd_on_base_image() {
  215. local charm="$1" cache_file="$state_tmpdir/$FUNCNAME.cache.$(echo "$*" | md5_compat)"
  216. shift
  217. if [ -e "$cache_file" ]; then
  218. # debug "$FUNCNAME: cache hit ($*)"
  219. quick_cat_stdin < "$cache_file"
  220. return 0
  221. fi
  222. base_image=$(service_base_docker_image "$charm") || return 1
  223. result=$(cmd_on_base_image "$charm" "$@") || return 1
  224. echo "$result" | tee "$cache_file"
  225. }
  226. export -f cached_cmd_on_base_image
  227. ##
  228. ## Generic
  229. ##
  230. str_matches() {
  231. local str="$1"
  232. shift
  233. for pattern in "$@"; do
  234. eval "[[ \"$str\" == $pattern ]]" && return 0
  235. done
  236. return 1
  237. }
  238. gen_password() {
  239. local l=( {a..z} {A..Z} {0..9} ) nl="${#l[@]}" size=${1:-16}
  240. while ((size--)); do
  241. echo -n "${l[$((RANDOM * nl / 32768))]}"
  242. done
  243. echo
  244. }
  245. export -f gen_password
  246. file_put() {
  247. local TARGET="$1"
  248. mkdir -p "$(dirname "$TARGET")" &&
  249. cat - > "$TARGET"
  250. }
  251. export -f file_put
  252. file_put_0() {
  253. local TARGET="$1"
  254. mkdir -p "$(dirname "$TARGET")" &&
  255. cat > "$TARGET"
  256. }
  257. export -f file_put_0
  258. fetch_file() {
  259. local src="$1"
  260. case "$src" in
  261. *"://"*)
  262. err "Unsupported target scheme."
  263. return 1
  264. ;;
  265. *)
  266. ## Try direct
  267. if ! [ -r "$src" ]; then
  268. err "File '$src' not found/readable."
  269. return 1
  270. fi
  271. cat "$src" || return 1
  272. ;;
  273. esac
  274. }
  275. export -f fetch_file
  276. ## receives stdin content to decompress on stdout
  277. ## stdout content should be tar format.
  278. uncompress_file() {
  279. local filename="$1"
  280. ## Warning, the content of the file is already as stdin, the filename
  281. ## is there to hint for correct decompression.
  282. case "$filename" in
  283. *".gz")
  284. gunzip
  285. ;;
  286. *".bz2")
  287. bunzip2
  288. ;;
  289. *)
  290. cat
  291. ;;
  292. esac
  293. }
  294. export -f uncompress_file
  295. get_file() {
  296. local src="$1"
  297. fetch_file "$src" | uncompress_file "$src"
  298. }
  299. export -f get_file
  300. ##
  301. ## Common database lib
  302. ##
  303. _clean_docker() {
  304. local _DB_NAME="$1" container_id="$2"
  305. (
  306. set +e
  307. debug "Removing container $_DB_NAME"
  308. docker stop "$container_id"
  309. docker rm "$_DB_NAME"
  310. rm -vf "/tmp/${_DB_NAME}.state"
  311. )
  312. }
  313. export -f _clean_docker
  314. are_files_locked_in_dir() {
  315. local dir="$1" device hdev ldev
  316. device=$(stat -c %d "$dir") || {
  317. err "Can't stat %d."
  318. return 1
  319. }
  320. device=$(printf "%04x" $device)
  321. hdev=${device:0:2}
  322. ldev=${device:2:2}
  323. inodes=$(find "$dir" -printf ':%i:\n')
  324. found=
  325. while read -r inode; do
  326. debug "try inode:$inode"
  327. if [[ "$inodes" == *":$inode:"* ]]; then
  328. found=1
  329. break
  330. fi
  331. done < <(cat /proc/locks | grep " $hdev:$ldev:" | sed -r "s/^.*$hdev:$ldev:([0-9]+).*$/\1/g")
  332. [ "$found" ]
  333. }
  334. export -f are_files_locked_in_dir
  335. export _PID="$$"
  336. ensure_db_docker_running () {
  337. _DB_NAME="db_${DB_NAME}_${_PID}"
  338. if [ -e "/tmp/${_DB_NAME}.state" ]; then
  339. export DOCKER_IP="$(cat "/tmp/${_DB_NAME}.state")"
  340. debug "Re-using previous docker/connection '$DOCKER_IP'."
  341. _set_db_params "$DOCKER_IP"
  342. return 0
  343. fi
  344. if [ -e "/tmp/${_DB_NAME}.working" ]; then
  345. ## avoid recursive calls.
  346. if [ -z "$DOCKER_IP" ]; then
  347. err "Currently figuring up DOCKER_IP, please set it yourself before this call if needed."
  348. return 1
  349. else
  350. debug "Ignoring recursive call."
  351. fi
  352. return 0
  353. fi
  354. touch "/tmp/${_DB_NAME}.working"
  355. docker rm "$_DB_NAME" 2>/dev/null || true
  356. host_db_working_dir="$DATASTORE/${SERVICE_NAME}$DB_DATADIR"
  357. if is_db_locked; then
  358. info "Some process is using '$host_db_working_dir'. Trying to find a docker that would do this..."
  359. found=
  360. for docker_id in $(docker ps -q); do
  361. has_volume_mounted=$(
  362. docker inspect \
  363. --format "{{range .Mounts}}{{if eq .Destination \"$DB_DATADIR\"}}{{.Source}}{{end}}{{end}}" \
  364. "$docker_id")
  365. if [ "$has_volume_mounted" == "$host_db_working_dir" ]; then
  366. found="$docker_id"
  367. break
  368. fi
  369. done
  370. if [ -z "$found" ]; then
  371. err "Please shutdown any other docker using this directory."
  372. return 1
  373. fi
  374. export container_id="$found"
  375. info "Found docker $docker_id is already running."
  376. else
  377. verb "Database is not locked."
  378. if ! docker_has_image "$DOCKER_BASE_IMAGE"; then
  379. docker pull "$DOCKER_BASE_IMAGE"
  380. fi
  381. docker_opts=
  382. if [ -f "$DB_PASSFILE" ]; then
  383. verb "Found and using '$DB_PASSFILE'."
  384. docker_opts="$db_docker_opts -v $SERVER_ROOT_PREFIX$DB_PASSFILE:$DB_PASSFILE"
  385. fi
  386. debug docker run -d \
  387. --name "$_DB_NAME" \
  388. $docker_opts \
  389. -v "$host_db_working_dir:$DB_DATADIR" \
  390. "$DOCKER_BASE_IMAGE"
  391. if ! container_id=$(
  392. docker run -d \
  393. --name "$_DB_NAME" \
  394. $docker_opts \
  395. -v "$host_db_working_dir:$DB_DATADIR" \
  396. "$DOCKER_BASE_IMAGE"
  397. ); then
  398. err "'docker run' failed !"
  399. _clean_docker "$_DB_NAME" "$container_id"
  400. rm "/tmp/${_DB_NAME}.working"
  401. return 1
  402. fi
  403. trap_add EXIT,ERR "_clean_docker '$_DB_NAME' '$container_id'"
  404. fi
  405. if DOCKER_IP=$(wait_for_docker_ip "$container_id"); then
  406. echo "$DOCKER_IP" > "/tmp/${_DB_NAME}.state"
  407. debug "written /tmp/${_DB_NAME}.state"
  408. rm "/tmp/${_DB_NAME}.working"
  409. _set_db_params "$DOCKER_IP"
  410. return 0
  411. else
  412. err "Db not found. Docker logs follows:"
  413. docker logs --tail=5 "$container_id" 2>&1 | prefix " | " >&2
  414. rm "/tmp/${_DB_NAME}.working"
  415. return 1
  416. fi
  417. }
  418. export -f ensure_db_docker_running
  419. ## Require to set $db_docker_opts if needed, and $DB_PASSFILE
  420. ##
  421. _dcmd() {
  422. local docker_opts command="$1"
  423. shift
  424. debug "Db> $command $@"
  425. if [ -f "$DB_PASSFILE" ]; then
  426. verb "Found and using '$DB_PASSFILE'."
  427. db_docker_opts="$db_docker_opts -v $SERVER_ROOT_PREFIX$DB_PASSFILE:$DB_PASSFILE"
  428. fi
  429. debug docker run -i --rm \
  430. $db_docker_opts \
  431. --entrypoint "$command" "$DOCKER_BASE_IMAGE" $db_cmd_opts "$@"
  432. docker run -i --rm \
  433. $db_docker_opts \
  434. --entrypoint "$command" "$DOCKER_BASE_IMAGE" $db_cmd_opts "$@"
  435. }
  436. export -f _dcmd
  437. ## Executes code through db
  438. dcmd () {
  439. [ "$DB_NAME" ] || print_syntax_error "$FUNCNAME: You must provide \$DB_NAME."
  440. [ "$DB_DATADIR" ] || print_syntax_error "$FUNCNAME: You must provide \$DB_DATADIR."
  441. [ "$DB_PASSFILE" ] || print_syntax_error "$FUNCNAME: You must provide \$DB_PASSFILE."
  442. [ "$_PID" ] || print_syntax_error "$FUNCNAME: You must provide \$_PID."
  443. [ "$(type -t is_db_locked)" == "function" ] || print_syntax_error "$FUNCNAME: You must provide function 'is_db_locked'."
  444. [ "$(type -t _set_db_params)" == "function" ] || print_syntax_error "$FUNCNAME: You must provide function '_set_db_params'."
  445. [ "$(type -t ddb)" == "function" ] || print_syntax_error "$FUNCNAME: You must provide function 'ddb'."
  446. ensure_db_docker_running </dev/null || return 1
  447. _dcmd "$@"
  448. }
  449. export -f dcmd
  450. ## Warning: requires a ``ddb`` matching current database to be checked
  451. wait_for_docker_ip() {
  452. local name=$1 timeout=5 timeout_count=0 DOCKER_IP= DB_OK=
  453. while [ -z "$DOCKER_IP" ]; do
  454. sleep 1
  455. DOCKER_IP=$(docker inspect --format='{{.NetworkSettings.IPAddress}}' "$name" 2>/dev/null)
  456. verb "[1/2] Waiting for docker $name... ($[timeout_count + 1]/$timeout)"
  457. ((timeout_count++)) || true
  458. if [ "$timeout_count" == "$timeout" ]; then
  459. err "${RED}timeout error${NORMAL}(${timeout}s):"\
  460. "Could not find '$name' docker" \
  461. "container's IP."
  462. return 1
  463. fi
  464. done
  465. verb "[1/2] Found docker $name IP: $DOCKER_IP"
  466. timeout_count=0
  467. DB_OK=
  468. _set_db_params "$DOCKER_IP"
  469. while [ -z "$DB_OK" ]; do
  470. sleep 1
  471. echo "SELECT 1;" | ddb >/dev/null && DB_OK=1
  472. verb "[2/2] Waiting for db service from docker $name... ($[timeout_count + 1]/$timeout)"
  473. ((timeout_count++)) || true
  474. if [ "$timeout_count" == "$timeout" ]; then
  475. err "${RED}timeout error${NORMAL}(${timeout}s):"\
  476. "Could not connect to db on $DOCKER_IP." \
  477. "container's IP."
  478. return 1
  479. fi
  480. done
  481. verb "[2/2] Db is ready !"
  482. echo "$DOCKER_IP"
  483. return 0
  484. }
  485. export -f wait_for_docker_ip
  486. ##
  487. ## Arrays
  488. ##
  489. array_values_to_stdin() {
  490. local e
  491. if [ "$#" -ne "1" ]; then
  492. print_syntax_warning "$FUNCNAME: need one argument."
  493. return 1
  494. fi
  495. var="$1"
  496. eval "for e in \"\${$var[@]}\"; do echo -en \"\$e\\0\"; done"
  497. }
  498. array_keys_to_stdin() {
  499. local e
  500. if [ "$#" -ne "1" ]; then
  501. print_syntax_warning "$FUNCNAME: need one argument."
  502. return 1
  503. fi
  504. var="$1"
  505. eval "for e in \"\${!$var[@]}\"; do echo -en \"\$e\\0\"; done"
  506. }
  507. array_kv_to_stdin() {
  508. local e
  509. if [ "$#" -ne "1" ]; then
  510. print_syntax_warning "$FUNCNAME: need one argument."
  511. return 1
  512. fi
  513. var="$1"
  514. eval "for e in \"\${!$var[@]}\"; do echo -n \"\$e\"; echo -en '\0'; echo -n \"\${$var[\$e]}\"; echo -en '\0'; done"
  515. }
  516. array_pop() {
  517. local narr="$1" nres="$2"
  518. for key in $(eval "echo \${!$narr[@]}"); do
  519. eval "$nres=\${$narr[\"\$key\"]}"
  520. eval "unset $narr[\"\$key\"]"
  521. return 0
  522. done
  523. }
  524. export -f array_pop
  525. array_member() {
  526. local src elt
  527. src="$1"
  528. elt="$2"
  529. while read-0 key; do
  530. if [ "$(eval "echo -n \"\${$src[\$key]}\"")" == "$elt" ]; then
  531. return 0
  532. fi
  533. done < <(array_keys_to_stdin "$src")
  534. return 1
  535. }
  536. export -f array_member
  537. ##
  538. ## Internal Process
  539. ##
  540. get_docker_compose_links() {
  541. local service="$1" cache_file="$state_tmpdir/$FUNCNAME.cache.$1" \
  542. links charm charm_part master_charm
  543. if [ -z "$service" ]; then
  544. print_syntax_error "$FUNCNAME: Please specify a service as first argument."
  545. return 1
  546. fi
  547. if [ -e "$cache_file" ]; then
  548. # debug "$FUNCNAME: cache hit ($*)"
  549. cat "$cache_file"
  550. return 0
  551. fi
  552. master_charm=$(get_top_master_charm_for_service "$service") || return 1
  553. deps=()
  554. while read-0 relation_name target_service relation_config tech_dep; do
  555. master_target_charm="$(get_top_master_charm_for_service "$target_service")"
  556. [ "$master_charm" == "$master_target_charm" ] && continue
  557. if [ "$tech_dep" == "reversed" ]; then
  558. deps+=("$(echo -en "$master_target_charm:\n links:\n - $master_charm")")
  559. elif [ "$tech_dep" == "True" ]; then
  560. deps+=("$(echo -en "$master_charm:\n links:\n - $master_target_charm")")
  561. fi
  562. done < <(get_compose_relations "$service") || return 1
  563. merge_yaml_str "${deps[@]}" | tee "$cache_file"
  564. }
  565. _get_docker_compose_opts() {
  566. local service="$1" cache_file="$state_tmpdir/$FUNCNAME.cache.$1" \
  567. links charm charm_part master_charm
  568. if [ -z "$service" ]; then
  569. print_syntax_error "$FUNCNAME: Please specify a service as first argument."
  570. return 1
  571. fi
  572. if [ -e "$cache_file" ]; then
  573. # debug "$FUNCNAME: cache hit ($*)"
  574. cat "$cache_file"
  575. return 0
  576. fi
  577. compose_def="$(get_compose_service_def "$service")" || return 1
  578. master_charm="$(get_top_master_charm_for_service "$service")"
  579. docker_compose_opts=$(echo "$compose_def" | shyaml get-value "docker-compose" 2>/dev/null)
  580. if [ "$docker_compose_opts" ]; then
  581. yaml_key_val_str "$master_charm" "$docker_compose_opts"
  582. fi | tee "$cache_file"
  583. }
  584. ##
  585. ## By Reading the metadata.yml, we create a docker-compose.yml mixin.
  586. ## Some metadata.yml (of subordinates) will indeed modify other
  587. ## services than themselves.
  588. _get_docker_compose_service_mixin() {
  589. local service="$1" cache_file="$state_tmpdir/$FUNCNAME.cache.$1" links charm charm_part
  590. if [ -z "$service" ]; then
  591. print_syntax_error "$FUNCNAME: Please specify a service as first argument."
  592. return 1
  593. fi
  594. if [ -e "$cache_file" ]; then
  595. # debug "$FUNCNAME: cache hit ($*)"
  596. cat "$cache_file"
  597. return 0
  598. fi
  599. master_charm=$(get_top_master_charm_for_service "$service") || return 1
  600. ## The compose part
  601. links_yaml=$(get_docker_compose_links "$service") || return 1
  602. docker_compose_options=$(_get_docker_compose_opts "$service") || return 1
  603. ## the charm part
  604. #debug "Get charm name from service name $DARKYELLOW$service$NORMAL."
  605. charm=$(get_service_charm "$service") || return 1
  606. charm_part=$(get_docker_compose_mixin_from_metadata "$charm") || return 1
  607. ## Merge results
  608. if [ "$charm_part" ]; then
  609. charm_yaml="$(yaml_key_val_str "$master_charm" "$charm_part")" || return 1
  610. merge_yaml_str "$links_yaml" "$charm_yaml" "$docker_compose_options" || return 1
  611. else
  612. echo "$links_yaml"
  613. fi | tee "$cache_file"
  614. }
  615. export -f _get_docker_compose_service_mixin
  616. ##
  617. ## Get full `docker-compose.yml` format for all listed services (and
  618. ## their deps)
  619. ##
  620. ## @export
  621. ## @cache: !system !nofail +stdout
  622. get_docker_compose () {
  623. local cache_file="$state_tmpdir/$FUNCNAME.cache.$(echo "$*" | md5_compat)" entries services service start
  624. if [ -e "$cache_file" ]; then
  625. # debug "$FUNCNAME: cache hit ($*)"
  626. cat "$cache_file"
  627. return 0
  628. fi
  629. ##
  630. ## Adding sub services configurations
  631. ##
  632. declare -A entries
  633. start_compilation=$SECONDS
  634. debug "Compiling 'docker-compose.conf' base for $DARKYELLOW$@$NORMAL..."
  635. for target_service in "$@"; do
  636. start=$SECONDS
  637. services=$(get_ordered_service_dependencies "$target_service") || return 1
  638. debug " $DARKYELLOW$target_service$NORMAL deps:$DARKYELLOW" $services "$NORMAL$GRAY(in $((SECONDS - start))s)$NORMAL"
  639. for service in $services; do
  640. if [ "${entries[$service]}" ]; then
  641. ## Prevent double inclusion of same service if this
  642. ## service is deps of two or more of your
  643. ## requirements.
  644. continue
  645. fi
  646. ## mark the service as "loaded" as well as it's containers
  647. ## if this is a subordinate service
  648. start_service=$SECONDS
  649. entries[$service]=$(_get_docker_compose_service_mixin "$service") || return 1
  650. debug " Applied $DARKYELLOW$service$NORMAL charm metadata mixins $GRAY(in $((SECONDS - start_service))s)$NORMAL"
  651. done
  652. debug " ..finished all mixins for $DARKYELLOW$target_service$NORMAL $GRAY(in $((SECONDS - start))s)$NORMAL"
  653. done
  654. merge_yaml_str "${entries[@]}" > "$cache_file"
  655. export _current_docker_compose="$(cat "$cache_file")"
  656. echo "$_current_docker_compose"
  657. debug " ..compilation of base 'docker-compose.conf' done $GRAY(in $((SECONDS - start_compilation))s)$NORMAL" || true
  658. # debug " ** ${WHITE}docker-compose.conf${NORMAL}:"
  659. # debug "$_current_docker_compose"
  660. }
  661. export -f get_docker_compose
  662. _get_compose_service_def_cached () {
  663. local service="$1" docker_compose="$2" cache_file="$CACHEDIR/$FUNCNAME.cache.$(echo "$*" | md5_compat)"
  664. if [ -e "$cache_file" ]; then
  665. #debug "$FUNCNAME: STATIC cache hit"
  666. cat "$cache_file"
  667. touch "$cache_file"
  668. return 0
  669. fi
  670. service_def_base="charm: $service"
  671. value=$(echo "$docker_compose" | shyaml get-value "$service" 2>/dev/null)
  672. merge_yaml <(echo "$service_def_base") <(echo "$value") | tee "$cache_file"
  673. }
  674. export -f _get_compose_service_def_cached
  675. ## XXXvlab: a lot to be done to cache the results
  676. get_compose_service_def () {
  677. local service="$1" docker_compose cache_file="$state_tmpdir/$FUNCNAME.cache.$1" \
  678. result
  679. if [ -e "$cache_file" ]; then
  680. #debug "$FUNCNAME: SESSION cache hit"
  681. cat "$cache_file"
  682. return 0
  683. fi
  684. [ -z "$service" ] && print_syntax_error "Missing service as first argument."
  685. docker_compose=$(cat "$COMPOSE_YML_FILE") || return 1
  686. result=$(_get_compose_service_def_cached "$service" "$docker_compose") || return 1
  687. echo "$result" | tee "$cache_file"
  688. }
  689. export -f get_compose_service_def
  690. _get_service_charm_cached () {
  691. local service="$1" service_def="$2" cache_file="$CACHEDIR/$FUNCNAME.cache.$(echo "$*" | md5_compat)"
  692. if [ -e "$cache_file" ]; then
  693. # debug "$FUNCNAME: cache hit $1"
  694. cat "$cache_file"
  695. touch "$cache_file"
  696. return 0
  697. fi
  698. charm=$(echo "$service_def" | shyaml get-value charm 2>/dev/null)
  699. if [ -z "$charm" ]; then
  700. err "Missing charm in service $DARKYELLOW$service$NORMAL definition."
  701. return 1
  702. fi
  703. echo "$charm" | tee "$cache_file"
  704. }
  705. export -f _get_service_charm_cached
  706. get_service_charm () {
  707. local service="$1"
  708. if [ -z "$service" ]; then
  709. print_syntax_error "$FUNCNAME: Please specify a service as first argument."
  710. return 1
  711. fi
  712. service_def=$(get_compose_service_def "$service") || return 1
  713. _get_service_charm_cached "$service" "$service_def"
  714. }
  715. export -f get_service_charm
  716. ## built above the docker-compose abstraction, so it relies on the
  717. ## full docker-compose.yml to be already built.
  718. get_service_def () {
  719. local service="$1" def
  720. if [ -z "$_current_docker_compose" ]; then
  721. print_syntax_error "$FUNCNAME is meant to be called after"\
  722. "\$_current_docker_compose has been calculated."
  723. fi
  724. def=$(echo "$_current_docker_compose" | shyaml get-value "$service" 2>/dev/null)
  725. if [ -z "$def" ]; then
  726. err "No definition for service $DARKYELLOW$service$NORMAL in compiled 'docker-compose.conf'."
  727. return 1
  728. fi
  729. echo "$def"
  730. }
  731. export -f get_service_def
  732. ## Return the base docker image name of a service
  733. service_base_docker_image() {
  734. local service="$1" cache_file="$state_tmpdir/$FUNCNAME.cache.$1" \
  735. master_charm charm service_image service_build service_dockerfile
  736. if [ -e "$cache_file" ]; then
  737. # debug "$FUNCNAME: cache hit ($*)"
  738. cat "$cache_file"
  739. return 0
  740. fi
  741. master_charm="$(get_top_master_charm_for_service "$service")" || {
  742. err "Could not compute base charm for service $DARKYELLOW$service$NORMAL."
  743. return 1
  744. }
  745. service_def="$(get_service_def "$master_charm")" || {
  746. err "Could not get docker-compose service definition for $DARKYELLOW$master_charm$NORMAL."
  747. return 1
  748. }
  749. service_image=$(echo "$service_def" | shyaml get-value image 2>/dev/null)
  750. if [ "$?" != 0 ]; then
  751. service_build=$(echo "$service_def" | shyaml get-value build 2>/dev/null)
  752. if [ "$?" != 0 ]; then
  753. err "Service $DARKYELLOW$service$NORMAL has no ${WHITE}image${NORMAL} nor ${WHITE}build${NORMAL} parameter."
  754. echo "$service_def" >&2
  755. return 1
  756. fi
  757. service_dockerfile="$CHARM_STORE/${service_build}/Dockerfile"
  758. if ! [ -e "$service_dockerfile" ]; then
  759. err "No Dockerfile found in '$service_dockerfile' location."
  760. return 1
  761. fi
  762. grep '^FROM' "$service_dockerfile" | xargs echo | cut -f 2 -d " "
  763. else
  764. echo "$service_image"
  765. fi | tee "$cache_file"
  766. }
  767. export -f service_base_docker_image
  768. get_charm_relation_def () {
  769. local charm="$1" relation_name="$2" cache_file="$state_tmpdir/$FUNCNAME.cache.$1.$2" \
  770. relation_def metadata
  771. if [ -e "$cache_file" ]; then
  772. # debug "$FUNCNAME: cache hit ($*)"
  773. cat "$cache_file"
  774. return 0
  775. fi
  776. metadata="$(get_charm_metadata "$charm")" || return 1
  777. relation_def="$(echo "$metadata" | shyaml get-value "provides.${relation_name}" 2>/dev/null)"
  778. echo "$relation_def" | tee "$cache_file"
  779. }
  780. export -f get_charm_relation_def
  781. get_charm_tech_dep_orientation_for_relation() {
  782. local charm="$1" relation_name="$2" cache_file="$state_tmpdir/$FUNCNAME.cache.$1.$2" \
  783. relation_def metadata value
  784. if [ -e "$cache_file" ]; then
  785. # debug "$FUNCNAME: cache hit ($*)"
  786. cat "$cache_file"
  787. return 0
  788. fi
  789. relation_def=$(get_charm_relation_def "$charm" "$relation_name" 2>/dev/null)
  790. value=$(echo "$relation_def" | shyaml get-value 'tech-dep' 2>/dev/null)
  791. value=${value:-True}
  792. echo "$value" | tee "$cache_file"
  793. }
  794. export -f get_charm_tech_dep_orientation_for_relation
  795. ##
  796. ## Use compose file to get deps, and relation definition in metadata.yml
  797. ## for tech-dep attribute.
  798. get_service_deps() {
  799. local service="$1" cache_file="$state_tmpdir/$FUNCNAME.cache.$1"
  800. if [ -e "$cache_file" ]; then
  801. # debug "$FUNCNAME: cache hit ($*)"
  802. cat "$cache_file"
  803. return 0
  804. fi
  805. (
  806. set -o pipefail
  807. get_compose_relations "$service" | \
  808. while read-0 relation_name target_service _relation_config tech_dep; do
  809. echo "$target_service"
  810. done | tee "$cache_file"
  811. ) || return 1
  812. }
  813. export -f get_service_deps
  814. _rec_get_depth() {
  815. local elt=$1 dep deps max cache_file="$state_tmpdir/$FUNCNAME.cache.$1"
  816. [ "${depths[$elt]}" ] && return 0
  817. if [ -e "$cache_file" ]; then
  818. # debug "$FUNCNAME: cache hit ($*)"
  819. depths[$elt]=$(cat "$cache_file")
  820. return 0
  821. fi
  822. visited[$elt]=1
  823. #debug "Setting visited[$elt]"
  824. #debug "Asking for $DARKYELLOW$elt$NORMAL dependencies"
  825. deps=$(get_service_deps "$elt") || {
  826. debug "Failed get_service_deps $elt"
  827. return 1
  828. }
  829. # debug "$elt deps are:" $deps
  830. max=0
  831. for dep in $deps; do
  832. [ "${visited[$dep]}" ] && {
  833. #debug "Already computing $dep"
  834. continue
  835. }
  836. _rec_get_depth "$dep" || return 1
  837. #debug "Requesting depth[$dep]"
  838. if (( ${depths[$dep]} > max )); then
  839. max="${depths[$dep]}"
  840. fi
  841. done
  842. # debug "Setting depth[$elt] to $((max + 1))"
  843. depths[$elt]=$((max + 1))
  844. echo "${depths[$elt]}" > $cache_file
  845. }
  846. export -f _rec_get_depth
  847. get_ordered_service_dependencies() {
  848. local services=("$@") cache_file="$state_tmpdir/$FUNCNAME.cache.$(echo "$*" | md5_compat)"
  849. if [ -e "$cache_file" ]; then
  850. # debug "$FUNCNAME: cache hit ($*)"
  851. cat "$cache_file"
  852. return 0
  853. fi
  854. #debug "Figuring ordered deps of $DARKYELLOW$services$NORMAL"
  855. if [ -z "${services[*]}" ]; then
  856. print_syntax_error "$FUNCNAME: no arguments"
  857. return 1
  858. fi
  859. declare -A depths
  860. declare -A visited
  861. heads=("${services[@]}")
  862. while [ "${#heads[@]}" != 0 ]; do
  863. array_pop heads head
  864. _rec_get_depth "$head" || return 1
  865. done
  866. i=0
  867. while [ "${#depths[@]}" != 0 ]; do
  868. for key in "${!depths[@]}"; do
  869. value="${depths[$key]}"
  870. if [ "$value" == "$i" ]; then
  871. echo "$key"
  872. unset depths[$key]
  873. fi
  874. done
  875. i=$((i + 1))
  876. done | tee "$cache_file"
  877. }
  878. export -f get_ordered_service_dependencies
  879. run_service_hook () {
  880. local services="$1" action="$2" loaded
  881. declare -A loaded
  882. for service in $services; do
  883. for subservice in $(get_ordered_service_dependencies "$service"); do
  884. if [ "${loaded[$subservice]}" ]; then
  885. ## Prevent double inclusion of same service if this
  886. ## service is deps of two or more of your
  887. ## requirements.
  888. continue
  889. fi
  890. charm=$(get_service_charm "$subservice") || return 1
  891. TARGET_SCRIPT="$CHARM_STORE/$charm/hooks/$action"
  892. [ -e "$TARGET_SCRIPT" ] || continue
  893. PROJECT_NAME=$(get_default_project_name) || return 1
  894. Wrap -d "$YELLOW$action$NORMAL hook of charm $DARKYELLOW$charm$NORMAL" <<EOF || return 1
  895. cd "$CHARM_STORE/$charm"
  896. export SERVICE_NAME=$subservice
  897. export IMAGE_NAME=$(echo "${PROJECT_NAME}" | tr -d "_-")_\${SERVICE_NAME}
  898. export CONTAINER_NAME=\${IMAGE_NAME}_1
  899. export CHARM_NAME="$charm"
  900. export PROJECT_NAME=$PROJECT_NAME
  901. export DOCKER_BASE_IMAGE=$(service_base_docker_image "$charm")
  902. export SERVICE_DATASTORE="$DATASTORE/$charm"
  903. export SERVICE_CONFIGSTORE="$CONFIGSTORE/$charm"
  904. "$TARGET_SCRIPT"
  905. EOF
  906. loaded[$subservice]=1
  907. done
  908. done
  909. return 0
  910. }
  911. relation-get () {
  912. local key="$1"
  913. cat "$RELATION_DATA_FILE" | shyaml get-value "$key" 2>/dev/null
  914. if [ "$?" != 0 ]; then
  915. err "The key $WHITE$key$NORMAL was not found in relation's data."
  916. return 1
  917. fi
  918. }
  919. export -f relation-get
  920. relation-base-compose-get () {
  921. local key="$1"
  922. echo "$RELATION_BASE_COMPOSE_DEF" | shyaml get-value "options.$key" 2>/dev/null
  923. if [ "$?" != 0 ]; then
  924. err "The key $WHITE$key$NORMAL was not found in base service compose definition.."
  925. return 1
  926. fi
  927. }
  928. export -f relation-base-compose-get
  929. relation-target-compose-get () {
  930. local key="$1"
  931. echo "$RELATION_BASE_COMPOSE_DEF" | shyaml get-value "options.$key" 2>/dev/null
  932. if [ "$?" != 0 ]; then
  933. err "The key $WHITE$key$NORMAL was not found in base service compose definition.."
  934. return 1
  935. fi
  936. }
  937. export -f relation-base-compose-get
  938. relation-set () {
  939. local key="$1" value="$2"
  940. if [ -z "$RELATION_DATA_FILE" ]; then
  941. err "$FUNCNAME: relation does not seems to be correctly setup."
  942. return 1
  943. fi
  944. if ! [ -r "$RELATION_DATA_FILE" ]; then
  945. err "$FUNCNAME: can't read relation's data." >&2
  946. return 1
  947. fi
  948. _config_merge "$RELATION_DATA_FILE" <(echo "$key: $value")
  949. }
  950. export -f relation-set
  951. _config_merge() {
  952. local config_filename="$1" merge_to_file="$2"
  953. touch "$config_filename" &&
  954. merge_yaml "$config_filename" "$merge_to_file" > "$config_filename.tmp" || return 1
  955. mv "$config_filename.tmp" "$config_filename"
  956. }
  957. export -f _config_merge
  958. ## XXXvlab; this can be used only in relation, I'd like to use it in init.
  959. config-add() {
  960. local metadata="$1"
  961. _config_merge "$RELATION_CONFIG" <(echo "$metadata")
  962. }
  963. export -f config-add
  964. ## XXXvlab; this can be used only in relation, I'd like to use it in init.
  965. init-config-add() {
  966. local metadata="$1"
  967. _config_merge "$state_tmpdir/to-merge-in-docker-compose.yml" <(echo "$metadata")
  968. }
  969. export -f init-config-add
  970. logstdout() {
  971. local name="$1"
  972. sed -r 's%^%'"${name}"'> %g'
  973. }
  974. export -f logstdout
  975. logstderr() {
  976. local name="$1"
  977. sed -r 's%^(.*)$%'"${RED}${name}>${NORMAL} \1"'%g'
  978. }
  979. export -f logstderr
  980. _run_service_relation () {
  981. local relation_name="$1" service="$2" target_service="$3" relation_config="$4" relation_dir services
  982. charm=$(get_service_charm "$service") || return 1
  983. target_charm=$(get_service_charm "$target_service") || return 1
  984. base_script_name=$(echo "$relation_name" | tr "-" "_" )-relation-joined
  985. script_name="hooks/${base_script_name}"
  986. [ ! -e "$CHARM_STORE/$target_charm/$script_name" -a ! -e "$CHARM_STORE/$charm/$script_name" ] &&
  987. return 0
  988. relation_dir=$(get_relation_data_dir "$charm" "$target_charm" "$relation_name") || return 1
  989. RELATION_DATA_FILE=$(get_relation_data_file "$charm" "$target_charm" "$relation_name" "$relation_config") || return 1
  990. RELATION_BASE_COMPOSE_DEF=$(get_compose_service_def "$service") || return 1
  991. RELATION_TARGET_COMPOSE_DEF=$(get_compose_service_def "$target_service") || return 1
  992. export BASE_SERVICE_NAME=$service
  993. export TARGET_SERVICE_NAME=$target_service
  994. export BASE_CHARM_NAME=$charm
  995. export TARGET_CHARM_NAME=$target_charm
  996. PROJECT_NAME=$(get_default_project_name) || return 1
  997. MASTER_BASE_CHARM_NAME=$(get_top_master_charm_for_service "$service") || return 1
  998. MASTER_TARGET_CHARM_NAME=$(get_top_master_charm_for_service "$target_service") || return 1
  999. export RELATION_DATA_FILE RELATION_BASE_COMPOSE_DEF RELATION_TARGET_COMPOSE_DEF
  1000. export MASTER_BASE_CHARM_NAME MASTER_TARGET_CHARM_NAME PROJECT_NAME
  1001. target_errlvl=0
  1002. if ! [ -e "$CHARM_STORE/$target_charm/$script_name" ]; then
  1003. verb "No relation script '$script_name' in $DARKYELLOW$target_charm$NORMAL."
  1004. else
  1005. verb "Running ${DARKBLUE}$relation_name${NORMAL} relation-joined script" \
  1006. "for target charm $DARKYELLOW$target_charm$NORMAL"
  1007. RELATION_CONFIG="$relation_dir/config_provider"
  1008. DOCKER_BASE_IMAGE=$(service_base_docker_image "$target_service") || return 1
  1009. export DOCKER_BASE_IMAGE RELATION_CONFIG RELATION_DATA
  1010. {
  1011. (
  1012. cd "$CHARM_STORE/$target_charm"
  1013. SERVICE_NAME=$target_service
  1014. SERVICE_DATASTORE="$DATASTORE/$target_charm"
  1015. SERVICE_CONFIGSTORE="$CONFIGSTORE/$target_charm"
  1016. export SERVICE_NAME DOCKER_BASE_IMAGE SERVICE_DATASTORE SERVICE_CONFIGSTORE
  1017. "$script_name"
  1018. echo "$?" > "$relation_dir/target_errlvl"
  1019. ) | logstdout "$DARKYELLOW$target_charm$NORMAL/$DARKBLUE$relation_name$NORMAL (joined) ${GREEN}@${NORMAL}"
  1020. } 3>&1 1>&2 2>&3 | logstderr "$DARKYELLOW$target_charm$NORMAL/$DARKBLUE$relation_name$NORMAL (joined) ${RED}@${NORMAL}" 3>&1 1>&2 2>&3
  1021. target_errlvl="$(cat "$relation_dir/target_errlvl")" || {
  1022. err "Relation script '$script_name' in $DARKYELLOW$target_charm$NORMAL" \
  1023. "failed before outputing an errorlevel."
  1024. ((target_errlvl |= "1" ))
  1025. }
  1026. if [ -e "$RELATION_CONFIG" ]; then
  1027. debug "Merging some new config info in $DARKYELLOW$target_service$NORMAL"
  1028. _config_merge "$state_tmpdir/to-merge-in-docker-compose.yml" "$RELATION_CONFIG" &&
  1029. rm "$RELATION_CONFIG"
  1030. ((target_errlvl |= "$?"))
  1031. fi
  1032. fi
  1033. if [ "$target_errlvl" == 0 ]; then
  1034. errlvl=0
  1035. if [ -e "$CHARM_STORE/$charm/$script_name" ]; then
  1036. verb "Running ${DARKBLUE}$relation_name${NORMAL} relation-joined script" \
  1037. "for charm $DARKYELLOW$charm$NORMAL"
  1038. RELATION_CONFIG="$relation_dir/config_providee"
  1039. RELATION_DATA="$(cat "$RELATION_DATA_FILE")"
  1040. DOCKER_BASE_IMAGE=$(service_base_docker_image "$service") || return 1
  1041. export DOCKER_BASE_IMAGE RELATION_CONFIG RELATION_DATA
  1042. {
  1043. (
  1044. cd "$CHARM_STORE/$charm"
  1045. SERVICE_NAME=$service
  1046. SERVICE_DATASTORE="$DATASTORE/$charm"
  1047. SERVICE_CONFIGSTORE="$CONFIGSTORE/$charm"
  1048. export SERVICE_NAME DOCKER_BASE_IMAGE SERVICE_DATASTORE SERVICE_CONFIGSTORE
  1049. "$script_name"
  1050. echo "$?" > "$relation_dir/errlvl"
  1051. ) | logstdout "$DARKYELLOW$charm$NORMAL/$DARKBLUE$relation_name$NORMAL (joined) ${GREEN}@${NORMAL}"
  1052. } 3>&1 1>&2 2>&3 | logstderr "$DARKYELLOW$charm$NORMAL/$DARKBLUE$relation_name$NORMAL (joined) ${RED}@$NORMAL" 3>&1 1>&2 2>&3
  1053. errlvl="$(cat "$relation_dir/errlvl")" || {
  1054. err "Relation script '$script_name' in $DARKYELLOW$charm$NORMAL" \
  1055. "failed before outputing an errorlevel."
  1056. ((errlvl |= "1" ))
  1057. }
  1058. if [ -e "$RELATION_CONFIG" ]; then
  1059. _config_merge "$state_tmpdir/to-merge-in-docker-compose.yml" "$RELATION_CONFIG" &&
  1060. rm "$RELATION_CONFIG"
  1061. ((errlvl |= "$?" ))
  1062. fi
  1063. if [ "$errlvl" != 0 ]; then
  1064. err "Relation $DARKBLUE$relation_name$NORMAL on $DARKYELLOW$charm$NORMAL failed to run properly."
  1065. fi
  1066. else
  1067. verb "No relation script '$script_name' in charm $DARKYELLOW$charm$NORMAL. Ignoring."
  1068. fi
  1069. else
  1070. err "Relation $DARKBLUE$relation_name$NORMAL on $DARKYELLOW$target_charm$NORMAL failed to run properly."
  1071. fi
  1072. if [ "$target_errlvl" == 0 -a "$errlvl" == 0 ]; then
  1073. debug "Relation $DARKBLUE$relation_name$NORMAL is established" \
  1074. "between $DARKYELLOW$service$NORMAL and $DARKYELLOW$target_service$NORMAL."
  1075. return 0
  1076. else
  1077. return 1
  1078. fi
  1079. }
  1080. export -f _run_service_relation
  1081. _get_compose_relations_cached () {
  1082. local compose_service_def="$1" cache_file="$CACHEDIR/$FUNCNAME.cache.$(echo "$*" | md5_compat)" \
  1083. relation_name relation_def target_service
  1084. if [ -e "$cache_file" ]; then
  1085. #debug "$FUNCNAME: STATIC cache hit $1"
  1086. cat "$cache_file"
  1087. touch "$cache_file"
  1088. return 0
  1089. fi
  1090. (
  1091. set -o pipefail
  1092. if [ "$compose_service_def" ]; then
  1093. while read-0 relation_name relation_def; do
  1094. (
  1095. case "$(echo "$relation_def" | shyaml get-type 2>/dev/null)" in
  1096. "str")
  1097. target_service="$(echo "$relation_def" | shyaml get-value 2>/dev/null)"
  1098. tech_dep="$(get_charm_tech_dep_orientation_for_relation "$target_service" "$relation_name")"
  1099. echo -en "$relation_name\0$target_service\0\0$tech_dep\0"
  1100. ;;
  1101. "sequence")
  1102. while read-0 target_service; do
  1103. tech_dep="$(get_charm_tech_dep_orientation_for_relation "$target_service" "$relation_name")"
  1104. echo -en "$relation_name\0$target_service\0\0$tech_dep\0"
  1105. done < <(echo "$relation_def" | shyaml get-values-0 2>/dev/null)
  1106. ;;
  1107. "struct")
  1108. while read-0 target_service relation_config; do
  1109. tech_dep="$(get_charm_tech_dep_orientation_for_relation "$target_service" "$relation_name")"
  1110. echo -en "$relation_name\0$target_service\0$relation_config\0$tech_dep\0"
  1111. done < <(echo "$relation_def" | shyaml key-values-0 2>/dev/null)
  1112. ;;
  1113. esac
  1114. ) </dev/null >> "$cache_file"
  1115. done < <(echo "$compose_service_def" | shyaml key-values-0 relations 2>/dev/null)
  1116. fi
  1117. )
  1118. if [ "$?" != 0 ]; then
  1119. err "Error while looking for compose relations."
  1120. rm -f "$cache_file" ## no cache
  1121. return 1
  1122. fi
  1123. [ -e "$cache_file" ] && cat "$cache_file"
  1124. return 0
  1125. }
  1126. export -f _get_compose_relations_cached
  1127. get_compose_relations () {
  1128. local service="$1" cache_file="$state_tmpdir/$FUNCNAME.cache.$1" \
  1129. compose_def
  1130. if [ -e "$cache_file" ]; then
  1131. #debug "$FUNCNAME: SESSION cache hit $1"
  1132. cat "$cache_file"
  1133. return 0
  1134. fi
  1135. compose_def="$(get_compose_service_def "$service")" || return 1
  1136. _get_compose_relations_cached "$compose_def" > "$cache_file"
  1137. if [ "$?" != 0 ]; then
  1138. err "Error while looking for compose relations."
  1139. rm -f "$cache_file" ## no cache
  1140. return 1
  1141. fi
  1142. cat "$cache_file"
  1143. }
  1144. export -f get_compose_relations
  1145. run_service_relations () {
  1146. local services="$1" loaded
  1147. declare -A loaded
  1148. for service in $(get_ordered_service_dependencies $services); do
  1149. # debug "Upping dep's relations of ${DARKYELLOW}$service${NORMAL}:"
  1150. for subservice in $(get_service_deps "$service") "$service"; do
  1151. [ "${loaded[$subservice]}" ] && continue
  1152. # debug " Relations of ${DARKYELLOW}$subservice${NORMAL}:"
  1153. while read-0 relation_name target_service relation_config tech_dep; do
  1154. export relation_config
  1155. Wrap -d "Building $DARKYELLOW$subservice$NORMAL --$DARKBLUE$relation_name$NORMAL--> $DARKYELLOW$target_service$NORMAL" <<EOF || return 1
  1156. _run_service_relation "$relation_name" "$subservice" "$target_service" "\$relation_config"
  1157. EOF
  1158. done < <(get_compose_relations "$subservice") || return 1
  1159. loaded[$subservice]=1
  1160. done
  1161. done
  1162. }
  1163. export -f run_service_relations
  1164. _run_service_action_direct() {
  1165. local service="$1" action="$2" charm script _dummy
  1166. shift; shift
  1167. read-0 charm script || true ## against 'set -e' that could be setup in parent scripts
  1168. if read-0 _dummy || [ "$_dummy" ]; then
  1169. print_syntax_error "$FUNCNAME: too many arguments in action descriptor"
  1170. return 1
  1171. fi
  1172. compose_file=$(get_compose_yml_location) || return 1
  1173. export action_errlvl_file="$state_tmpdir/action-$service-$charm-$action-errlvl"
  1174. export state_tmpdir
  1175. {
  1176. (
  1177. set +e ## Prevents unwanted leaks from parent shell
  1178. cd "$CHARM_STORE/$charm"
  1179. export COMPOSE_CONFIG=$(cat "$compose_file")
  1180. export METADATA_CONFIG=$(cat "$CHARM_STORE/$charm/metadata.yml")
  1181. export SERVICE_NAME=$service
  1182. export ACTION_NAME=$action
  1183. export CONTAINER_NAME=$(get_top_master_charm_for_service "$service")
  1184. export DOCKER_BASE_IMAGE=$(service_base_docker_image "$CONTAINER_NAME")
  1185. export SERVICE_DATASTORE="$DATASTORE/$service"
  1186. export SERVICE_CONFIGSTORE="$CONFIGSTORE/$service"
  1187. exname="$exname $ACTION_NAME $SERVICE_NAME" \
  1188. stdbuf -oL -eL "$script" "$@"
  1189. echo "$?" > "$action_errlvl_file"
  1190. ) | logstdout "$DARKYELLOW$charm$NORMAL/${DARKCYAN}$action${NORMAL} ${GREEN}@${NORMAL}"
  1191. } 3>&1 1>&2 2>&3 | logstderr "$DARKYELLOW$charm$NORMAL/${DARKCYAN}$action${NORMAL} ${RED}@$NORMAL" 3>&1 1>&2 2>&3
  1192. if ! [ -e "$action_errlvl_file" ]; then
  1193. err "Action $DARKYELLOW$service$NORMAL:$DARKCYAN$action$NORMAL has failed without having time" \
  1194. "to output an errlvl"
  1195. return 1
  1196. fi
  1197. return "$(cat "$action_errlvl_file")"
  1198. }
  1199. export -f _run_service_action_direct
  1200. _run_service_action_relation() {
  1201. local service="$1" action="$2" charm target_charm relation_name target_script relation_config _dummy
  1202. shift; shift
  1203. read-0 charm target_charm relation_name target_script relation_config || true
  1204. if read-0 _dummy || [ "$_dummy" ]; then
  1205. print_syntax_error "$FUNCNAME: too many arguments in action descriptor"
  1206. return 1
  1207. fi
  1208. export RELATION_DATA_FILE=$(get_relation_data_file "$charm" "$target_charm" "$relation_name" "$relation_config")
  1209. compose_file=$(get_compose_yml_location) || return 1
  1210. export action_errlvl_file="$state_tmpdir/action-$service-$charm-$action-errlvl"
  1211. export state_tmpdir
  1212. {
  1213. (
  1214. set +e ## Prevents unwanted leaks from parent shell
  1215. cd "$CHARM_STORE/$charm"
  1216. export METADATA_CONFIG=$(cat "$CHARM_STORE/$charm/metadata.yml")
  1217. export SERVICE_NAME=$service
  1218. export RELATION_TARGET_CHARM="$target_charm"
  1219. export RELATION_BASE_CHARM="$charm"
  1220. export ACTION_NAME=$action
  1221. export CONTAINER_NAME=$(get_top_master_charm_for_service "$service")
  1222. export DOCKER_BASE_IMAGE=$(service_base_docker_image "$CONTAINER_NAME")
  1223. export SERVICE_DATASTORE="$DATASTORE/$service"
  1224. export SERVICE_CONFIGSTORE="$CONFIGSTORE/$service"
  1225. exname="$exname $ACTION_NAME $SERVICE_NAME" \
  1226. stdbuf -oL -eL "$target_script" "$@"
  1227. echo "$?" > "$action_errlvl_file"
  1228. ) | logstdout "$DARKYELLOW$charm$NORMAL/${DARKCYAN}$action${NORMAL} ${GREEN}@${NORMAL}"
  1229. } 3>&1 1>&2 2>&3 | logstderr "$DARKYELLOW$charm$NORMAL/${DARKCYAN}$action${NORMAL} ${RED}@$NORMAL" 3>&1 1>&2 2>&3
  1230. if ! [ -e "$action_errlvl_file" ]; then
  1231. err "Action $DARKYELLOW$service$NORMAL:$DARKCYAN$action$NORMAL has failed without having time" \
  1232. "to output an errlvl"
  1233. return 1
  1234. fi
  1235. return "$(cat "$action_errlvl_file")"
  1236. }
  1237. export -f _run_service_action_relation
  1238. get_relation_data_dir() {
  1239. local charm="$1" target_charm="$2" relation_name="$3" cache_file="$state_tmpdir/$FUNCNAME.cache.$1"
  1240. if [ -e "$cache_file" ]; then
  1241. # debug "$FUNCNAME: cache hit ($*)"
  1242. cat "$cache_file"
  1243. return 0
  1244. fi
  1245. project=$(get_default_project_name) || return 1
  1246. relation_dir="$VARDIR/relations/$project/$charm-$target_charm/$relation_name"
  1247. if ! [ -d "$relation_dir" ]; then
  1248. mkdir -p "$relation_dir" || return 1
  1249. chmod go-rwx "$relation_dir" || return 1 ## protecting this directory
  1250. fi
  1251. echo "$relation_dir" || tee "$cache_file"
  1252. }
  1253. export -f get_relation_data_dir
  1254. get_relation_data_file() {
  1255. local charm="$1" target_charm="$2" relation_name="$3" relation_config="$4"
  1256. relation_dir=$(get_relation_data_dir "$charm" "$target_charm" "$relation_name") || return 1
  1257. relation_data_file="$relation_dir/data"
  1258. new=
  1259. if [ -e "$relation_data_file" ]; then
  1260. ## Has reference changed ?
  1261. new_md5=$(echo "$relation_config" | md5_compat)
  1262. if [ "$new_md5" != "$(cat "$relation_data_file.md5_ref" 2>/dev/null)" ]; then
  1263. new=true
  1264. fi
  1265. else
  1266. new=true
  1267. fi
  1268. if [ "$new" ]; then
  1269. echo "$relation_config" > "$relation_data_file"
  1270. chmod go-rwx "$relation_data_file" ## protecting this file
  1271. echo "$relation_config" | md5_compat > "$relation_data_file.md5_ref"
  1272. fi
  1273. echo "$relation_data_file"
  1274. }
  1275. export -f get_relation_data_file
  1276. has_service_action () {
  1277. local service="$1" action="$2" cache_file="$state_tmpdir/$FUNCNAME.cache.$1" \
  1278. target_script charm target_charm
  1279. if [ -e "$cache_file" ]; then
  1280. # debug "$FUNCNAME: cache hit ($*)"
  1281. cat "$cache_file"
  1282. return 0
  1283. fi
  1284. charm=$(get_service_charm "$service") || return 1
  1285. ## Action directly provided ?
  1286. target_script="$CHARM_STORE/$charm/actions/$action"
  1287. if [ -x "$target_script" ]; then
  1288. echo -en "direct\0$charm\0$target_script" | tee "$cache_file"
  1289. return 0
  1290. fi
  1291. ## Action provided by relation ?
  1292. while read-0 relation_name target_service relation_config tech_dep; do
  1293. target_charm=$(get_service_charm "$target_service") || return 1
  1294. target_script="$CHARM_STORE/$target_charm/actions/relations/$relation_name/$action"
  1295. if [ -x "$target_script" ]; then
  1296. echo -en "relation\0$charm\0$target_charm\0$relation_name\0$target_script\0$relation_config" | tee "$cache_file"
  1297. return 0
  1298. fi
  1299. done < <(get_compose_relations "$service")
  1300. return 1
  1301. # master=$(get_top_master_charm_for_service "$charm")
  1302. # [ "$master" == "$charm" ] && return 1
  1303. # has_service_action "$master" "$action"
  1304. }
  1305. export -f has_service_action
  1306. run_service_action () {
  1307. local service="$1" action="$2"
  1308. shift ; shift
  1309. {
  1310. if ! read-0 action_type; then
  1311. info "Service $DARKYELLOW$service$NORMAL does not have any action $DARKCYAN$action$NORMAL defined."
  1312. info " Add an executable script to '$CHARM_STORE/$charm/actions/$action' to implement action."
  1313. return 1
  1314. fi
  1315. Section "running $DARKYELLOW$service$NORMAL/$DARKCYAN$action$NORMAL ($action_type)"; Feed
  1316. "_run_service_action_${action_type}" "$service" "$action" "$@"
  1317. } < <(has_service_action "$service" "$action")
  1318. }
  1319. export -f run_service_action
  1320. get_compose_relation_config() {
  1321. local service=$1 relation_config cache_file="$state_tmpdir/$FUNCNAME.cache.$1"
  1322. if [ -e "$cache_file" ]; then
  1323. # debug "$FUNCNAME: cache hit ($*)"
  1324. cat "$cache_file"
  1325. return 0
  1326. fi
  1327. compose_service_def=$(get_compose_service_def "$service") || return 1
  1328. echo "$compose_service_def" | shyaml get-value "relations" 2>/dev/null | tee "$cache_file"
  1329. }
  1330. export -f get_compose_relation_config
  1331. # ## Return key-values-0
  1332. # get_compose_relation_config_for_service() {
  1333. # local service=$1 relation_name=$2 relation_config
  1334. # compose_service_relations=$(get_compose_relation_config "$service") || return 1
  1335. # if ! relation_config=$(
  1336. # echo "$compose_service_relations" |
  1337. # shyaml get-value "${relation_name}" 2>/dev/null); then
  1338. # err "Couldn't find $DARKYELLOW${service}$NORMAL/${WHITE}${relation_name}$NORMAL" \
  1339. # "relation config in compose configuration."
  1340. # return 1
  1341. # fi
  1342. # if [ -z "$relation_config" ]; then
  1343. # err "Relation ${WHITE}mysql-database$NORMAL is empty in compose configuration."
  1344. # return 1
  1345. # fi
  1346. # if ! echo "$relation_config" | shyaml key-values-0 2>/dev/null; then
  1347. # err "No key/values in ${DARKBLUE}mysql-database$NORMAL of compose config."
  1348. # return 1
  1349. # fi
  1350. # }
  1351. # export -f get_compose_relation_config_for_service
  1352. _get_master_charm_for_service_cached () {
  1353. local service="$1" charm="$2" metadata="$3" cache_file="$CACHEDIR/$FUNCNAME.cache.$(echo "$*" | md5_compat)" \
  1354. charm requires master_charm target_charm target_service service_def
  1355. if [ -e "$cache_file" ]; then
  1356. # debug "$FUNCNAME: STATIC cache hit ($1)"
  1357. cat "$cache_file"
  1358. touch "$cache_file"
  1359. return 0
  1360. fi
  1361. if [ "$(echo "$metadata" | shyaml get-value "subordinate" 2>/dev/null)" != "True" ]; then
  1362. ## just return charm name
  1363. echo "$charm" | tee "$cache_file"
  1364. return 0
  1365. fi
  1366. ## fetch the container relation
  1367. requires="$(echo "$metadata" | shyaml get-value "requires" 2>/dev/null)"
  1368. if [ -z "$requires" ]; then
  1369. die "Charm $DARKYELLOW$charm$NORMAL is a subordinate but does not have any 'requires' " \
  1370. "section."
  1371. fi
  1372. master_charm=
  1373. while read-0 relation_name relation; do
  1374. if [ "$(echo "$relation" | shyaml get-value "scope" 2>/dev/null)" == "container" ]; then
  1375. # debug "$DARKYELLOW$service$NORMAL's relation" \
  1376. # "$DARKBLUE${relation_name}$NORMAL is a container."
  1377. interface="$(echo "$relation" | shyaml get-value "interface" 2>/dev/null)"
  1378. if [ -z "$interface" ]; then
  1379. err "No ${WHITE}$interface${NORMAL} set for relation $relation_name."
  1380. return 1
  1381. fi
  1382. ## Action provided by relation ?
  1383. target_service=
  1384. while read-0 relation_name candidate_target_service _relation_config _tech_dep; do
  1385. [ "$interface" == "$relation_name" ] && {
  1386. target_service="$candidate_target_service"
  1387. break
  1388. }
  1389. done < <(get_compose_relations "$service")
  1390. if [ -z "$target_service" ]; then
  1391. err "Couldn't find ${WHITE}relations.$interface${NORMAL} in" \
  1392. "${DARKYELLOW}$service$NORMAL compose definition."
  1393. return 1
  1394. fi
  1395. master_charm=$(get_service_charm "$target_service") || return 1
  1396. break
  1397. fi
  1398. done < <(echo "$requires" | shyaml key-values-0 2>/dev/null)
  1399. if [ -z "$master_charm" ]; then
  1400. die "Charm $DARKYELLOW$charm$NORMAL is a subordinate but does not have any relation with" \
  1401. " ${WHITE}scope${NORMAL} set to 'container'."
  1402. fi
  1403. echo "$master_charm" | tee "$cache_file"
  1404. }
  1405. export -f _get_master_charm_for_service_cached
  1406. get_master_charm_for_service() {
  1407. local service="$1" cache_file="$state_tmpdir/$FUNCNAME.cache.$1" \
  1408. charm metadata result
  1409. if [ -e "$cache_file" ]; then
  1410. # debug "$FUNCNAME: SESSION cache hit ($*)"
  1411. cat "$cache_file"
  1412. return 0
  1413. fi
  1414. charm=$(get_service_charm "$service") || return 1
  1415. metadata=$(get_charm_metadata "$charm") || return 1
  1416. result=$(_get_master_charm_for_service_cached "$service" "$charm" "$metadata") || return 1
  1417. echo "$result" | tee "$cache_file"
  1418. }
  1419. export -f get_master_charm_for_service
  1420. get_top_master_charm_for_service() {
  1421. local service="$1" cache_file="$state_tmpdir/$FUNCNAME.cache.$1" \
  1422. current_service
  1423. if [ -e "$cache_file" ]; then
  1424. # debug "$FUNCNAME: cache hit ($*)"
  1425. cat "$cache_file"
  1426. return 0
  1427. fi
  1428. current_service="$service"
  1429. while true; do
  1430. master_service=$(get_master_charm_for_service "$current_service") || return 1
  1431. [ "$master_service" == "$current_service" ] && break
  1432. current_service="$master_service"
  1433. done
  1434. echo "$current_service" | tee "$cache_file"
  1435. return 0
  1436. }
  1437. export -f get_top_master_charm_for_service
  1438. get_charm_metadata() {
  1439. local charm="$1" metadata_file
  1440. metadata_file="$CHARM_STORE/$charm/metadata.yml"
  1441. if ! [ -e "$metadata_file" ]; then
  1442. return 0 ## No metadata file is as if metadata was empty
  1443. fi
  1444. if ! [ -d "$CHARM_STORE/$charm" ]; then
  1445. die "No charm $charm was found in charm store."
  1446. fi
  1447. cat "$metadata_file"
  1448. }
  1449. export -f get_charm_metadata
  1450. ##
  1451. ## The result is a mixin that is not always a complete valid
  1452. ## docker-compose entry (thinking of subordinates). The result
  1453. ## will be merge with master charms.
  1454. _get_docker_compose_mixin_from_metadata_cached() {
  1455. local charm="$1" metadata="$2" cache_file="$CACHEDIR/$FUNCNAME.cache.$(echo "$*" | md5_compat)" \
  1456. metadata_file metadata volumes docker_compose subordinate image
  1457. if [ -e "$cache_file" ]; then
  1458. #debug "$FUNCNAME: STATIC cache hit $1"
  1459. cat "$cache_file"
  1460. touch "$cache_file"
  1461. return 0
  1462. fi
  1463. mixin=
  1464. if [ "$metadata" ]; then
  1465. ## resources to volumes
  1466. volumes=$(
  1467. for resource_type in data config; do
  1468. while read-0 resource; do
  1469. eval "echo \" - \$${resource_type^^}STORE/\$charm\$resource:\$resource:rw\""
  1470. done < <(echo "$metadata" | shyaml get-values-0 "${resource_type}-resources" 2>/dev/null)
  1471. done
  1472. while read-0 resource; do
  1473. if [[ "$resource" == *:* ]]; then
  1474. echo " - $resource:rw"
  1475. else
  1476. echo " - $resource:$resource:rw"
  1477. fi
  1478. done < <(echo "$metadata" | shyaml get-values-0 "host-resources" 2>/dev/null)
  1479. while read-0 resource; do
  1480. echo " - $CHARM_STORE/$charm/resources$resource:$resource:rw"
  1481. done < <(echo "$metadata" | shyaml get-values-0 "charm-resources" 2>/dev/null)
  1482. ) || return 1
  1483. if [ "$volumes" ]; then
  1484. mixin=$(echo -en "volumes:\n$volumes")
  1485. fi
  1486. docker_compose=$(echo "$metadata" | shyaml get-value "docker-compose" 2>/dev/null)
  1487. if [ "$docker_compose" ]; then
  1488. mixin=$(merge_yaml_str "$mixin" "$docker_compose")
  1489. fi
  1490. if [ "$(echo "$metadata" | shyaml get-value "subordinate" 2>/dev/null)" == "True" ]; then
  1491. subordinate=true
  1492. fi
  1493. fi
  1494. image=$(echo "$metadata" | shyaml get-value "docker-image" 2>/dev/null)
  1495. image_or_build_statement=
  1496. if [ "$image" ]; then
  1497. if [ "$subordinate" ]; then
  1498. err "Subordinate charm can not have a ${WHITE}docker-image${NORMAL} value."
  1499. return 1
  1500. fi
  1501. image_or_build_statement="image: $image"
  1502. elif [ -d "$CHARM_STORE/$charm/build" ]; then
  1503. if [ "$subordinate" ]; then
  1504. err "Subordinate charm can not have a 'build' sub directory."
  1505. return 1
  1506. fi
  1507. image_or_build_statement="build: $charm/build"
  1508. fi
  1509. if [ "$image_or_build_statement" ]; then
  1510. mixin=$(merge_yaml_str "$mixin" "$image_or_build_statement")
  1511. fi
  1512. echo "$mixin" | tee "$cache_file"
  1513. }
  1514. export -f _get_docker_compose_mixin_from_metadata_cached
  1515. get_docker_compose_mixin_from_metadata() {
  1516. local charm="$1" cache_file="$state_tmpdir/$FUNCNAME.cache.$1"
  1517. if [ -e "$cache_file" ]; then
  1518. #debug "$FUNCNAME: SESSION cache hit ($*)"
  1519. cat "$cache_file"
  1520. return 0
  1521. fi
  1522. metadata="$(get_charm_metadata "$charm")" || return 1
  1523. mixin=$(_get_docker_compose_mixin_from_metadata_cached "$charm" "$metadata") || return 1
  1524. echo "$mixin" | tee "$cache_file"
  1525. }
  1526. export -f get_docker_compose_mixin_from_metadata
  1527. _save() {
  1528. local name="$1"
  1529. cat - | tee -a "$docker_compose_dir/.data/$name"
  1530. }
  1531. export -f _save
  1532. get_default_project_name() {
  1533. if [ "$DEFAULT_PROJECT_NAME" ]; then
  1534. echo "$DEFAULT_PROJECT_NAME"
  1535. return 0
  1536. fi
  1537. compose_yml_location="$(get_compose_yml_location)" || return 1
  1538. if [ "$compose_yml_location" ]; then
  1539. if normalized_path=$(readlink -e "$compose_yml_location"); then
  1540. echo "$(basename "$(dirname "$normalized_path")")"
  1541. return 0
  1542. fi
  1543. fi
  1544. echo "project"
  1545. return 0
  1546. }
  1547. export -f get_default_project_name
  1548. launch_docker_compose() {
  1549. docker_compose_tmpdir=$(mktemp -d -t tmp.XXXXXXXXXX)
  1550. #debug "Creating temporary docker-compose directory in '$docker_compose_tmpdir'."
  1551. # trap_add EXIT "debug \"Removing temporary docker-compose directory in $docker_compose_tmpdir.\";\
  1552. # rm -rf \"$docker_compose_tmpdir\""
  1553. trap_add EXIT "rm -rf \"$docker_compose_tmpdir\""
  1554. project=$(get_default_project_name)
  1555. mkdir -p "$docker_compose_tmpdir/$project"
  1556. docker_compose_dir="$docker_compose_tmpdir/$project"
  1557. if [ -z "$SERVICE_PACK" ]; then
  1558. export SERVICE_PACK=$(get_default_target_services $SERVICE_PACK)
  1559. fi
  1560. get_docker_compose $SERVICE_PACK > "$docker_compose_dir/docker-compose.yml" || return 1
  1561. if [ -e "$state_tmpdir/to-merge-in-docker-compose.yml" ]; then
  1562. # debug "Merging some config data in docker-compose.yml:"
  1563. # debug "$(cat $state_tmpdir/to-merge-in-docker-compose.yml)"
  1564. _config_merge "$docker_compose_dir/docker-compose.yml" "$state_tmpdir/to-merge-in-docker-compose.yml" || return 1
  1565. fi
  1566. if [ -z "$(echo $(cat "$docker_compose_dir/docker-compose.yml"))" ]; then
  1567. die "Unexpected empty docker-compose."
  1568. fi
  1569. ## XXXvlab: could be more specific and only link the needed charms
  1570. ln -sf "$CHARM_STORE/"* "$docker_compose_dir/"
  1571. mkdir "$docker_compose_dir/.data"
  1572. {
  1573. {
  1574. cd "$docker_compose_dir"
  1575. debug "${WHITE}docker-compose.yml$NORMAL for $DARKYELLOW$SERVICE_PACK$NORMAL:"
  1576. debug "$(cat "$docker_compose_dir/docker-compose.yml" | prefix " $GRAY|$NORMAL ")"
  1577. debug "${WHITE}Launching$NORMAL: docker-compose $@"
  1578. docker-compose "$@"
  1579. echo "$?" > "$docker_compose_dir/.data/errlvl"
  1580. } | _save stdout
  1581. } 3>&1 1>&2 2>&3 | _save stderr 3>&1 1>&2 2>&3
  1582. if tail -n 1 "$docker_compose_dir/.data/stderr" | egrep "Service .+ failed to build: Error getting container [0-9a-f]+ from driver devicemapper: (open|Error mounting) /dev/mapper/docker-.*: no such file or directory$" >/dev/null 2>&1; then
  1583. err "Detected bug https://github.com/docker/docker/issues/4036 ... "
  1584. err "Please re-launch your command, or switch from 'devicemapper' driver to 'overlayfs' or 'aufs'."
  1585. fi
  1586. docker_compose_errlvl="$(cat "$docker_compose_dir/.data/errlvl" 2>/dev/null)"
  1587. if [ -z "$docker_compose_errlvl" ]; then
  1588. err "Something went wrong before you could gather docker-compose errorlevel."
  1589. return 1
  1590. fi
  1591. return "$docker_compose_errlvl"
  1592. }
  1593. export -f launch_docker_compose
  1594. get_compose_yml_location() {
  1595. parent=$(while ! [ -e "./compose.yml" ]; do
  1596. [ "$PWD" == "/" ] && exit 0
  1597. cd ..
  1598. done; echo "$PWD"
  1599. )
  1600. if [ "$parent" ]; then
  1601. echo "$parent/compose.yml"
  1602. return 0
  1603. fi
  1604. if [ "$DEFAULT_COMPOSE_FILE" ]; then
  1605. if ! [ -e "$DEFAULT_COMPOSE_FILE" ]; then
  1606. err "No 'compose.yml' was found in current or parent dirs," \
  1607. "and \$DEFAULT_COMPOSE_FILE points to an unexistent file."
  1608. die "Please provide a 'compose.yml' file."
  1609. fi
  1610. echo "$DEFAULT_COMPOSE_FILE"
  1611. return 0
  1612. fi
  1613. err "No 'compose.yml' was found in current or parent dirs, and no \$DEFAULT_COMPOSE_FILE was set."
  1614. die "Please provide a 'compose.yml' file."
  1615. return 1
  1616. }
  1617. export -f get_compose_yml_location
  1618. get_default_target_services() {
  1619. local services=("$@")
  1620. if [ -z "${services[*]}" ]; then
  1621. if [ "$DEFAULT_SERVICES" ]; then
  1622. debug "No service provided, using $WHITE\$DEFAULT_SERVICES$NORMAL variable." \
  1623. "Target services: $DARKYELLOW$DEFAULT_SERVICES$NORMAL"
  1624. services="$DEFAULT_SERVICES"
  1625. else
  1626. err "No service provided."
  1627. return 1
  1628. fi
  1629. fi
  1630. echo "${services[*]}"
  1631. }
  1632. export -f get_default_target_services
  1633. get_master_services() {
  1634. local loaded master_service
  1635. declare -A loaded
  1636. for service in "$@"; do
  1637. master_service=$(get_top_master_charm_for_service "$service") || return 1
  1638. if [ "${loaded[$master_service]}" ]; then
  1639. continue
  1640. fi
  1641. echo "$master_service"
  1642. loaded["$master_service"]=1
  1643. done | xargs echo
  1644. }
  1645. export -f get_master_services
  1646. _setup_state_dir() {
  1647. export state_tmpdir=$(mktemp -d -t tmp.XXXXXXXXXX)
  1648. #debug "Creating temporary state directory in '$state_tmpdir'."
  1649. # trap_add EXIT "debug \"Removing temporary state directory in $state_tmpdir.\";\
  1650. # rm -rf \"$state_tmpdir\""
  1651. trap_add EXIT "rm -rf \"$state_tmpdir\""
  1652. }
  1653. get_docker_compose_opts_list() {
  1654. local action="$1" cache_file="$CACHEDIR/$FUNCNAME.cache.$(echo "$action"; cat "$(which docker-compose)" | md5_compat)"
  1655. if [ -e "$cache_file" ]; then
  1656. cat "$cache_file"
  1657. return 0
  1658. fi
  1659. docker_compose_help_msg=$(docker-compose "$action" --help 2>/dev/null) || return 1
  1660. echo "$docker_compose_help_msg" | grep '^Options:' -A 20000 |
  1661. tail -n +2 |
  1662. egrep "^\s+-" |
  1663. sed -r 's/\s+((((-[a-zA-Z]|--[a-zA-Z0-9-]+)( [A-Z=]+|=[^ ]+)?)(, )?)+)\s+.*$/\1/g' |
  1664. tee "$cache_file"
  1665. }
  1666. _MULTIOPTION_REGEX='^((-[a-zA-Z]|--[a-zA-Z0-9-]+)(, )?)+'
  1667. _MULTIOPTION_REGEX_LINE_FILTER=$_MULTIOPTION_REGEX'(\s|=)'
  1668. get_docker_compose_multi_opts_list() {
  1669. local action="$1" opts_list
  1670. opts_list=$(get_docker_compose_opts_list "$action") || return 1
  1671. echo "$opts_list" | egrep "$_MULTIOPTION_REGEX_LINE_FILTER" |
  1672. sed -r "s/^($_MULTIOPTION_REGEX)(\s|=).*$/\1/g" |
  1673. tr ',' "\n" | xargs echo
  1674. }
  1675. get_docker_compose_single_opts_list() {
  1676. local action="$1" opts_list
  1677. opts_list=$(get_docker_compose_opts_list "$action") || return 1
  1678. echo "$opts_list" | egrep -v "$_MULTIOPTION_REGEX_LINE_FILTER" |
  1679. tr ',' "\n" | xargs echo
  1680. }
  1681. clean_cache() {
  1682. local i=0
  1683. for f in $(ls -t "$CACHEDIR/"*.cache.* 2>/dev/null | tail -n +500); do
  1684. ((i++))
  1685. rm -f "$f"
  1686. done
  1687. if (( i > 0 )); then
  1688. debug "${WHITE}Cleaned cache:${NORMAL} Removed $((i)) elements (current cache size is $(du -sh "$CACHEDIR" | cut -f 1))"
  1689. fi
  1690. }
  1691. [ "$SOURCED" ] && return 0
  1692. if [ -z "$DISABLE_SYSTEM_CONFIG_FILE" ]; then
  1693. if [ -r /etc/default/charm ]; then
  1694. . /etc/default/charm
  1695. fi
  1696. if [ -r "/etc/default/$exname" ]; then
  1697. . "/etc/default/$exname"
  1698. fi
  1699. ## XXXvlab: should provide YML config opportunities in possible parent dirs ?
  1700. ## userdir ? and global /etc/compose.yml ?
  1701. . /etc/compose.conf || exit 1
  1702. . /etc/compose.local.conf || exit 1
  1703. fi
  1704. _setup_state_dir
  1705. mkdir -p "$CACHEDIR" || exit 1
  1706. DOCKER_HOST_NET=$(ip addr show docker0 | grep 'inet ' | xargs echo | cut -f 2 -d " ") || {
  1707. die "Couldn't determine docker host net"
  1708. }
  1709. DOCKER_HOST_IP=$(echo "$DOCKER_HOST_NET" | cut -f 1 -d "/") || {
  1710. die "Couldn't determine docker host ip"
  1711. }
  1712. export DOCKER_HOST_NET DOCKER_HOST_IP
  1713. ##
  1714. ## Argument parsing
  1715. ##
  1716. remainder_args=()
  1717. compose_opts=()
  1718. action_opts=()
  1719. no_hooks=
  1720. no_init=
  1721. action=
  1722. stage="main" ## switches from 'main', to 'action', 'remainder'
  1723. is_docker_compose_action=
  1724. DC_MATCH_MULTI=
  1725. DC_MATCH_SINGLE=
  1726. while [ "$#" != 0 ]; do
  1727. case "$stage" in
  1728. "main")
  1729. case "$1" in
  1730. --help|-h)
  1731. no_init=true ; no_hooks=true ; no_relations=true
  1732. print_help
  1733. exit 0
  1734. ;;
  1735. --verbose|-v)
  1736. export VERBOSE=true
  1737. ;;
  1738. --version|-V)
  1739. print_version
  1740. exit 0
  1741. ;;
  1742. -f)
  1743. [ -e "$2" ] || die "File $2 doesn't exists"
  1744. export DEFAULT_COMPOSE_FILE="$2"
  1745. shift
  1746. ;;
  1747. -p)
  1748. export DEFAULT_PROJECT_NAME="$2"
  1749. shift
  1750. ;;
  1751. --no-relations)
  1752. export no_relations=true
  1753. ;;
  1754. --no-hooks)
  1755. export no_hooks=true
  1756. ;;
  1757. --no-init)
  1758. export no_init=true
  1759. ;;
  1760. --debug)
  1761. export DEBUG=true
  1762. export VERBOSE=true
  1763. ;;
  1764. --*|-*)
  1765. compose_opts+=("$1")
  1766. ;;
  1767. *)
  1768. action="$1"
  1769. stage="action"
  1770. DC_MATCH_MULTI=$(get_docker_compose_multi_opts_list "$action") &&
  1771. DC_MATCH_SINGLE="$(get_docker_compose_single_opts_list "$action")"
  1772. if [ "$DC_MATCH_MULTI" ]; then
  1773. DC_MATCH_SINGLE="$DC_MATCH_SINGLE $(echo "$DC_MATCH_MULTI" | sed -r 's/( |$)/=\* /g')"
  1774. fi
  1775. is_docker_compose_action=true
  1776. ;;
  1777. esac
  1778. ;;
  1779. "action")
  1780. case "$1" in
  1781. --help|-h)
  1782. no_init=true ; no_hooks=true ; no_relations=true
  1783. action_opts+=("$1")
  1784. ;;
  1785. --*|-*)
  1786. if [ "$is_docker_compose_action" ]; then
  1787. if str_matches "$1" $DC_MATCH_MULTI; then
  1788. action_opts=("$1" "$2")
  1789. shift;
  1790. elif str_matches "$1" $DC_MATCH_SINGLE; then
  1791. action_opts=("$1")
  1792. else
  1793. err "Unknown option '$1'. Please check help."
  1794. docker-compose "$action" --help >&2
  1795. exit 1
  1796. fi
  1797. fi
  1798. ;;
  1799. *)
  1800. action_posargs+=("$1")
  1801. stage="remainder"
  1802. ;;
  1803. esac
  1804. ;;
  1805. "remainder")
  1806. remainder_args+=("$@")
  1807. break 3;;
  1808. esac
  1809. shift
  1810. done
  1811. [ "${compose_opts[*]}" ] && debug "Main docker-compose opts: ${compose_opts[*]}"
  1812. [ "${action_opts[*]}" ] && debug "Action $DARKCYAN$action$NORMAL with opts: ${action_opts[*]}"
  1813. [ "${remainder_args[*]}" ] && debug "Remainder args: ${remainder_args[*]}"
  1814. ##
  1815. ## Actual code
  1816. ##
  1817. export CHARM_STORE=${CHARM_STORE:-/srv/charm-store}
  1818. export DOCKER_DATASTORE=${DOCKER_DATASTORE:-/srv/docker-datastore}
  1819. COMPOSE_YML_FILE=$(get_compose_yml_location) || exit 1
  1820. export COMPOSE_YML_FILE
  1821. debug "Found $WHITE$exname$NORMAL YAML file in '$COMPOSE_YML_FILE'."
  1822. output=$(shyaml get-value < "$COMPOSE_YML_FILE" 2>&1)
  1823. if [ "$?" != 0 ]; then
  1824. err "Invalid YAML in '$COMPOSE_YML_FILE':"
  1825. while IFS='' read -r line1 && IFS='' read -r line2; do
  1826. echo "$line1 $GRAY($line2)$NORMAL"
  1827. done < <(echo "$output" | grep ^yaml.scanner -A 100 |
  1828. sed -r 's/^ in "<stdin>", //g' | sed -r 's/^yaml.scanner.[a-zA-Z]+: //g') |
  1829. prefix " $GRAY|$NORMAL "
  1830. exit 1
  1831. fi
  1832. if ! [ -d "$CHARM_STORE" ]; then
  1833. err "Charm store path $YELLOW$CHARM_STORE$NORMAL does not exists. "
  1834. err "Please check your $YELLOW\$CHARM_STORE$NORMAL variable value."
  1835. exit 1
  1836. fi
  1837. if [ -z "$(cd "$CHARM_STORE"; ls)" ]; then
  1838. err "no available charms in charm store $YELLOW$CHARM_STORE$NORMAL. Either:"
  1839. err " - check $YELLOW\$CHARM_STORE$NORMAL variable value"
  1840. err " - download charms in $CHARM_STORE"
  1841. print_error "Charm store is empty. Cannot continue."
  1842. fi
  1843. ##
  1844. ## Get services in command line.
  1845. ##
  1846. is_service_action=
  1847. case "$action" in
  1848. up|build|start|stop|config)
  1849. services="$(get_default_target_services "${action_posargs[@]}")" || exit 1
  1850. orig_services="${action_posargs[@]:1}"
  1851. ;;
  1852. run)
  1853. services="${action_posargs[0]}"
  1854. ;;
  1855. "")
  1856. services=
  1857. ;;
  1858. *)
  1859. if is_service_action=$(has_service_action "${action_posargs[0]}" "$action"); then
  1860. {
  1861. read-0 action_type
  1862. case "$action_type" in
  1863. "relation")
  1864. read-0 _ target_service relation_name
  1865. debug "Found action $DARKYELLOW${action_posargs[0]}$NORMAL/$DARKBLUE$relation_name$NORMAL/$DARKCYAN$action$NORMAL (in $DARKYELLOW$target_service$NORMAL)"
  1866. ;;
  1867. "direct")
  1868. debug "Found action $DARKYELLOW${action_posargs[0]}$NORMAL.$DARKCYAN$action$NORMAL"
  1869. ;;
  1870. esac
  1871. } < <(has_service_action "${action_posargs[0]}" "$action")
  1872. services="${action_posargs[0]}"
  1873. else
  1874. services="$(get_default_target_services "${action_posargs[@]}")"
  1875. fi
  1876. ;;
  1877. esac
  1878. get_docker_compose $services >/dev/null || { ## precalculate variable \$_current_docker_compose
  1879. err "Fails to compile base 'docker-compose.conf'"
  1880. exit 1
  1881. }
  1882. ##
  1883. ## Pre-action
  1884. ##
  1885. full_init=
  1886. case "$action" in
  1887. up|run)
  1888. full_init=true
  1889. ;;
  1890. "")
  1891. full_init=
  1892. ;;
  1893. *)
  1894. if [ "$is_service_action" ]; then
  1895. full_init=true
  1896. fi
  1897. ;;
  1898. esac
  1899. if [ "$full_init" ]; then
  1900. ## init in order
  1901. if [ -z "$no_init" ]; then
  1902. Section initialisation
  1903. run_service_hook "$services" init || exit 1
  1904. fi
  1905. ## Get relations
  1906. if [ -z "$no_relations" ]; then
  1907. run_service_relations "$services" || exit 1
  1908. fi
  1909. ## XXXvlab: to be removed when all relation and service stuff is resolved
  1910. if [ -z "$no_hooks" ]; then
  1911. ordered_services=$(get_ordered_service_dependencies $services) || exit 1
  1912. for service in $ordered_services; do
  1913. charm=$(get_service_charm "$service") || exit 1
  1914. for script in "$CHARM_STORE/$charm/hooks.d/"*.sh; do
  1915. [ -e "$script" ] || continue
  1916. [ -x "$script" ] || { echo "compose: script $script is not executable." >&2; exit 1; }
  1917. (
  1918. debug "Launching '$script'."
  1919. cd "$(dirname "$script)")";
  1920. "$script" "$@"
  1921. ) || { echo "compose: hook $script failed. Stopping." >&2; exit 1; }
  1922. done
  1923. done
  1924. fi
  1925. fi
  1926. export SERVICE_PACK="$services"
  1927. ##
  1928. ## Docker-compose
  1929. ##
  1930. case "$action" in
  1931. up|start|stop|build)
  1932. master_services=$(get_master_services $SERVICE_PACK) || exit 1
  1933. launch_docker_compose "${compose_opts[@]}" "$action" "${action_opts[@]}" $master_services
  1934. ;;
  1935. run)
  1936. master_service=$(get_master_services $SERVICE_PACK) || exit 1
  1937. launch_docker_compose "${compose_opts[@]}" "$action" "${action_opts[@]}" "$master_service" "${remainder_args[@]}"
  1938. ;;
  1939. # enter)
  1940. # master_service=$(get_master_services $SERVICE_PACK) || exit 1
  1941. # [ "${remainder_args[*]}" ] || remainder_args=("/bin/bash" "-c" "export TERM=xterm; exec bash")
  1942. # docker exec -ti "${action_opts[@]}" "$master_service" "${remainder_args[@]}"
  1943. # ;;
  1944. config)
  1945. ## removing the services
  1946. launch_docker_compose "${compose_opts[@]}" "$action" "${action_opts[@]}" "${remainder_args[@]}"
  1947. warn "Runtime configuration modification (from relations) are not included here."
  1948. ;;
  1949. *)
  1950. if [ "$is_service_action" ]; then
  1951. run_service_action "$SERVICE_PACK" "$action" "${remainder_args[@]}"
  1952. else
  1953. launch_docker_compose "${compose_opts[@]}" "$action" "${action_opts[@]}" "${remainder_args[@]}"
  1954. fi
  1955. ;;
  1956. esac