Add docker_build for building Docker image using Bazel
docker_build is a Skylark rule that describe a docker image. You can
replace Dockerfile by a docker_build to use Bazel's incrementality model.
--
MOS_MIGRATED_REVID=99160762
diff --git a/tools/build_defs/docker/BUILD b/tools/build_defs/docker/BUILD
new file mode 100644
index 0000000..3b6ebd9
--- /dev/null
+++ b/tools/build_defs/docker/BUILD
@@ -0,0 +1,79 @@
+licenses(["notice"]) # Apache 2.0
+
+TEST_DATA = [
+ "//tools/build_defs/docker/testdata:base_with_entrypoint",
+ "//tools/build_defs/docker/testdata:base_with_volume",
+ "//tools/build_defs/docker/testdata:derivative_with_cmd",
+ "//tools/build_defs/docker/testdata:derivative_with_shadowed_cmd",
+ "//tools/build_defs/docker/testdata:derivative_with_volume",
+ "//tools/build_defs/docker/testdata:files_base",
+ "//tools/build_defs/docker/testdata:files_with_files_base",
+ "//tools/build_defs/docker/testdata:files_with_tar_base",
+ "//tools/build_defs/docker/testdata:tar_base",
+ "//tools/build_defs/docker/testdata:tar_with_files_base",
+ "//tools/build_defs/docker/testdata:tar_with_tar_base",
+ "//tools/build_defs/docker/testdata:generated_tarball",
+ "//tools/build_defs/docker/testdata:with_env",
+ "//tools/build_defs/docker/testdata:with_double_env",
+ "//tools/build_defs/docker/testdata:link_with_files_base",
+]
+
+sh_test(
+ name = "build_test",
+ size = "medium",
+ srcs = [
+ "build_test.sh",
+ ],
+ data = TEST_DATA + ["testenv.sh"],
+ deps = [
+ "//src/test/shell:bashunit",
+ ],
+)
+
+# Used by docker_build and friends
+py_library(
+ name = "archive",
+ srcs = ["archive.py"],
+ visibility = ["//tools/build_defs/docker:__subpackages__"],
+)
+
+py_binary(
+ name = "rewrite_json",
+ srcs = ["rewrite_json.py"],
+ visibility = ["//visibility:public"],
+ deps = ["//third_party/py/gflags"],
+)
+
+py_test(
+ name = "rewrite_json_test",
+ srcs = ["rewrite_json_test.py"],
+ deps = [
+ ":rewrite_json",
+ ],
+)
+
+py_binary(
+ name = "build_layer",
+ srcs = ["build_layer.py"],
+ visibility = ["//visibility:public"],
+ deps = [
+ ":archive",
+ "//third_party/py/gflags",
+ ],
+)
+
+py_binary(
+ name = "sha256",
+ srcs = ["sha256.py"],
+ visibility = ["//visibility:public"],
+)
+
+py_binary(
+ name = "create_image",
+ srcs = ["create_image.py"],
+ visibility = ["//visibility:public"],
+ deps = [
+ ":archive",
+ "//third_party/py/gflags",
+ ],
+)
diff --git a/tools/build_defs/docker/README.md b/tools/build_defs/docker/README.md
new file mode 100644
index 0000000..6c698a8
--- /dev/null
+++ b/tools/build_defs/docker/README.md
@@ -0,0 +1,321 @@
+# Docker support for Bazel
+
+## Overview
+
+These build rules are used for building [Docker](https://www.docker.com)
+images. Such images are easy to modify and deploy system image for
+deploying application easily on cloud providers.
+
+As traditional Dockerfile-based `docker build`s effectively execute a series
+of commands inside of Docker containers, saving the intermediate results as
+layers; this approach is unsuitable for use in Bazel for a variety of reasons.
+
+The docker_build rule constructs a tarball that is compatible with
+`docker save/load`, and creates a single layer out of each BUILD rule in the chain.
+
+* [Basic Example](#basic-example)
+* [Build Rule Reference](#reference)
+ * [`docker_build`](#docker_build)
+* [Future work](#future)
+
+<a name="basic-example"></a>
+## Basic Example
+
+Consider the following BUILD file in `//third_party/debian`:
+
+```python
+filegroup(
+ name = "ca_certificates",
+ srcs = ["ca_certificates.deb"],
+)
+
+# Example when you have all your dependencies in your repository.
+# We have an example on how to fetch them from the web later in this
+# document.
+filegroup(
+ name = "openjdk-7-jre-headless",
+ srcs = ["openjdk-7-jre-headless.deb"],
+)
+
+docker_build(
+ name = "wheezy",
+ tars = ["wheezy.tar"],
+)
+```
+
+The `wheezy` target in that BUILD file roughly corresponds to the Dockerfile:
+
+```docker
+FROM scratch
+ADD wheezy.tar /
+```
+
+You can then build up subsequent layers via:
+
+```python
+docker_build(
+ name = "base",
+ base = "//third_party/debian:wheezy",
+ debs = ["//third_party/debian:ca_certificates"],
+)
+
+docker_build(
+ name = "java",
+ base = ":base",
+ debs = ["//third_party/debian:openjdk-7-jre-headless"],
+)
+```
+
+## Metadata
+
+You can set layer metadata on these same rules by simply adding (supported) arguments to the rule, for instance:
+
+```python
+docker_build(
+ name = "my-layer",
+ entrypoint = ["foo", "bar", "baz"],
+ ...
+)
+```
+
+Will have a similar effect as the Dockerfile construct:
+
+```docker
+ENTRYPOINT ["foo", "bar", "baz"]
+```
+
+For the set of supported metadata, and ways to construct layers, see here.
+
+
+### Using
+
+Suppose you have a `docker_build` target `//my/image:helloworld`:
+
+```python
+docker_build(
+ name = "helloworld",
+ ...
+)
+```
+
+You can build this with `bazel build my/image:helloworld`.
+This will produce the file `bazel-genfiles/my/image/helloworld.tar`.
+You can load this into my local Docker client by running
+`docker load -i bazel-genfiles/my/image/helloworld.tar`, or simply
+`bazel run my/image:helloworld`.
+
+
+Upon success you should be able to run `docker images` and see:
+
+```
+REPOSITORY TAG IMAGE ID ...
+blaze/my_image helloworld d3440d7f2bde ...
+```
+
+You can now use this docker image with the name `bazel/my_image:helloworld` or
+tag it with another name, for example:
+`docker tag bazel/my_image:helloworld gcr.io/my-project/my-awesome-image:v0.9`
+
+__Nota Bene:__ the `docker images` command will show a really old timestamp
+because `docker_build` remove all timestamps from the build to make it
+reproducible.
+
+## Pulling images and deb files from the internet
+
+If you do not want to check in base image in your repository, you can use
+[external repositories](http://bazel.io/docs/external.html). For instance,
+you could create various layer with `external` labels:
+
+```python
+load("/tools/build_defs/docker/docker", "docker_build")
+
+docker_build(
+ name = "java",
+ base = "@docker-debian//:wheezy",
+ debs = ["@openjdk-7-jre-headless//file"],
+)
+```
+
+Using the WORKSPACE file to add the actual files:
+
+```python
+new_http_archive(
+ name = "docker-debian",
+ url = "https://codeload.github.com/tianon/docker-brew-debian/zip/e9bafb113f432c48c7e86c616424cb4b2f2c7a51",
+ build_file = "debian.BUILD",
+ type = "zip",
+ sha256 = "515d385777643ef184729375bc5cb996134b3c1dc15c53acf104749b37334f68",
+)
+
+http_file(
+ name = "openjdk-7-jre-headless",
+ url = "http://security.debian.org/debian-security/pool/updates/main/o/openjdk-7/openjdk-7-jre-headless_7u79-2.5.5-1~deb7u1_amd64.deb",
+ sha256 = "b632f0864450161d475c012dcfcc37a1243d9ebf7ff9d6292150955616d71c23",
+)
+```
+
+With the following `debian.BUILD` file:
+
+```python
+load("/tools/build_defs/docker/docker", "docker_build")
+
+# Extract .xz files
+genrule(
+ name = "wheezy_tar",
+ srcs = ["docker-brew-debian-e9bafb113f432c48c7e86c616424cb4b2f2c7a51/wheezy/rootfs.tar.xz"],
+ outs = ["wheezy_tar.tar"],
+ cmd = "cat $< | xzcat >$@",
+)
+
+docker_build(
+ name = "wheezy",
+ tars = [":wheezy_tar"],
+ visibility = ["//visibility:public"],
+)
+```
+
+<a name="reference"></a>
+## Build Rule Reference [reference]
+
+<a name="docker_build"></a>
+### `docker_build`
+
+`docker_build(name, base, data_path, directory, files, tars, debs,
+symlinks, entrypoint, cmd, env, ports, volumes)`
+
+<table>
+ <thead>
+ <tr>
+ <th>Attribute</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><code>name</code></td>
+ <td>
+ <code>Name, required</code>
+ <p>A unique name for this rule.</p>
+ </td>
+ </tr>
+ <tr>
+ <td><code>base</code></td>
+ <td>
+ <code>File, optional</code>
+ <p>
+ The base layers on top of which to overlay this layer, equivalent to
+ FROM.
+ </p>
+ </td>
+ </tr>
+ <tr>
+ <td><code>data_path</code></td>
+ <td>
+ <code>String, optional</code>
+ <p>Root path of the files.</p>
+ <p>
+ The directory structure from the files is preserved inside the
+ docker image but a prefix path determined by `data_path`
+ is removed from the the directory structure. This path can
+ be absolute from the workspace root if starting with a `/` or
+ relative to the rule's directory. It is set to `.` by default.
+ </p>
+ </td>
+ </tr>
+ <tr>
+ <td><code>directory</code></td>
+ <td>
+ <code>String, optional</code>
+ <p>Target directory.</p>
+ <p>
+ The directory in which to expand the specified files, defaulting to '/'.
+ Only makes sense accompanying one of files/tars/debs.
+ </p>
+ </td>
+ </tr>
+ <tr>
+ <td><code>files</code></td>
+ <td>
+ <code>List of files, optional</code>
+ <p>File to add to the layer.</p>
+ <p>
+ A list of files that should be included in the docker image.
+ </p>
+ </td>
+ </tr>
+ <tr>
+ <td><code>tars</code></td>
+ <td>
+ <code>List of files, optional</code>
+ <p>Tar file to extract in the layer.</p>
+ <p>
+ A list of tar files whose content should be in the docker image.
+ </p>
+ </td>
+ </tr>
+ <tr>
+ <td><code>debs</code></td>
+ <td>
+ <code>List of files, optional</code>
+ <p>Debian package to install.</p>
+ <p>
+ A list of debian package that will be installed in the docker image.
+ </p>
+ </td>
+ </tr>
+ <tr>
+ <td><code>symlinks</code></td>
+ <td>
+ <code>Dictionary, optional</code>
+ <p>Symlinks to create in the docker image.</p>
+ <p>
+ <code>
+ symlinks = {
+ "/path/to/link": "/path/to/target",
+ ...
+ },
+ </code>
+ </p>
+ </td>
+ </tr>
+ <tr>
+ <td><code>entrypoint</code></td>
+ <td>
+ <code>String or string list, optional</code>
+ <p><a href="https://docs.docker.com/reference/builder/#entrypoint">List
+ of entrypoints to add in the layer.</a></p>
+ </td>
+ </tr>
+ <tr>
+ <td><code>cmd</code></td>
+ <td>
+ <code>String or string list, optional</code>
+ <p><a href="https://docs.docker.com/reference/builder/#cmd">List
+ of commands to execute in the layer.</a></p>
+ </td>
+ </tr>
+ <tr>
+ <td><code>ports</code></td>
+ <td>
+ <code>String list, optional</code>
+ <p><a href="https://docs.docker.com/reference/builder/#expose">List
+ of ports to expose.</a></p>
+ </td>
+ </tr>
+ <tr>
+ <td><code>volumes</code></td>
+ <td>
+ <code>String list, optional</code>
+ <p><a href="https://docs.docker.com/reference/builder/#volumes">List
+ of volumes to mount.</a></p>
+ </td>
+ </tr>
+ </tbody>
+ </tbody>
+</table>
+
+<a name="future"></a>
+# Future work
+
+In the future, we would like to provide better integration with docker
+repositories: pull and push docker image.
diff --git a/tools/build_defs/docker/archive.py b/tools/build_defs/docker/archive.py
new file mode 100644
index 0000000..0f9c428
--- /dev/null
+++ b/tools/build_defs/docker/archive.py
@@ -0,0 +1,194 @@
+# Copyright 2015 Google Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Archive manipulation library for the Docker rules."""
+
+import os
+from StringIO import StringIO
+import tarfile
+
+
+class SimpleArFile(object):
+ """A simple AR file reader.
+
+ This enable to read AR file (System V variant) as described
+ in https://en.wikipedia.org/wiki/Ar_(Unix).
+
+ The standard usage of this class is:
+
+ with SimpleArFile(filename) as ar:
+ nextFile = ar.next()
+ while nextFile:
+ print nextFile.filename
+ nextFile = ar.next()
+
+ Upon error, this class will raise a ArError exception.
+ """
+ # TODO(dmarting): We should use a standard library instead but python 2.7
+ # does not have AR reading library.
+
+ class ArError(Exception):
+ pass
+
+ class SimpleArFileEntry(object):
+ """Represent one entry in a AR archive.
+
+ Attributes:
+ filename: the filename of the entry, as described in the archive.
+ timestamp: the timestamp of the file entry.
+ owner_id, group_id: numeric id of the user and group owning the file.
+ mode: unix permission mode of the file
+ size: size of the file
+ data: the content of the file.
+ """
+
+ def __init__(self, f):
+ self.filename = f.read(16).strip()
+ if self.filename.endswith('/'): # SysV variant
+ self.filename = self.filename[:-1]
+ self.timestamp = int(f.read(12).strip())
+ self.owner_id = int(f.read(6).strip())
+ self.group_id = int(f.read(6).strip())
+ self.mode = int(f.read(8).strip(), 8)
+ self.size = int(f.read(10).strip())
+ if f.read(2) != '\x60\x0a':
+ raise self.ArError('Invalid AR file header')
+ self.data = f.read(self.size)
+
+ MAGIC_STRING = '!<arch>\n'
+
+ def __init__(self, filename):
+ self.filename = filename
+
+ def __enter__(self):
+ self.f = open(self.filename, 'rb')
+ if self.f.read(len(self.MAGIC_STRING)) != self.MAGIC_STRING:
+ raise self.ArError('Not a ar file: ' + self.filename)
+ return self
+
+ def __exit__(self, t, v, traceback):
+ self.f.close()
+
+ def next(self):
+ """Read the next file. Returns None when reaching the end of file."""
+ if self.f.tell() == os.fstat(self.f.fileno()).st_size:
+ return None
+ return self.SimpleArFileEntry(self.f)
+
+
+class TarFileWriter(object):
+ """A wrapper to write tar files."""
+
+ def __init__(self, name):
+ self.tar = tarfile.open(name=name, mode='w')
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, t, v, traceback):
+ self.close()
+
+ def add_file(self, name, kind=tarfile.REGTYPE, content=None, link=None,
+ file_content=None, uid=0, gid=0, uname='', gname='', mtime=0,
+ mode=None):
+ """Add a file to the current tar.
+
+ Args:
+ name: the name of the file to add.
+ kind: the type of the file to add, see tarfile.*TYPE.
+ content: a textual content to put in the file.
+ link: if the file is a link, the destination of the link.
+ file_content: file to read the content from. Provide either this
+ one or `content` to specifies a content for the file.
+ uid: owner user identifier.
+ gid: owner group identifier.
+ uname: owner user names.
+ gname: owner group names.
+ mtime: modification time to put in the archive.
+ mode: unix permission mode of the file, default 0644 (0755).
+ """
+ if not name.startswith('.') and not name.startswith('/'):
+ name = './' + name
+ tarinfo = tarfile.TarInfo(name)
+ tarinfo.mtime = mtime
+ tarinfo.uid = uid
+ tarinfo.gid = gid
+ tarinfo.uname = uname
+ tarinfo.gname = gname
+ tarinfo.type = kind
+ if mode is None:
+ tarinfo.mode = 0644 if kind == tarfile.REGTYPE else 0755
+ else:
+ tarinfo.mode = mode
+ if link:
+ tarinfo.linkname = link
+ if content:
+ tarinfo.size = len(content)
+ self.tar.addfile(tarinfo, StringIO(content))
+ elif file_content:
+ with open(file_content, 'rb') as f:
+ tarinfo.size = os.fstat(f.fileno()).st_size
+ self.tar.addfile(tarinfo, f)
+ else:
+ self.tar.addfile(tarinfo)
+
+ def add_tar(self, tar, rootuid=None, rootgid=None,
+ numeric=False, name_filter=None):
+ """Merge a tar content into the current tar, stripping timestamp.
+
+ Args:
+ tar: the name of tar to extract and put content into the current tar.
+ rootuid: user id that we will pretend is root (replaced by uid 0).
+ rootgid: group id that we will pretend is root (replaced by gid 0).
+ numeric: set to true to strip out name of owners (and just use the
+ numeric values).
+ name_filter: filter out file by names. If not none, this method will be
+ called for each file to add, given the name and should return true if
+ the file is to be added to the final tar and false otherwise.
+ """
+ compression = os.path.splitext(tar)[-1][1:]
+ if compression == 'tgz':
+ compression = 'gz'
+ elif compression == 'bzip2':
+ compression = 'bz2'
+ elif compression not in ['gz', 'bz2']:
+ # Unfortunately xz format isn't supported in py 2.7 :(
+ compression = ''
+ with tarfile.open(name=tar, mode='r:' + compression) as intar:
+ for tarinfo in intar:
+ if name_filter is None or name_filter(tarinfo.name):
+ tarinfo.mtime = 0
+ if rootuid is not None and tarinfo.uid == rootuid:
+ tarinfo.uid = 0
+ tarinfo.uname = 'root'
+ if rootgid is not None and tarinfo.gid == rootgid:
+ tarinfo.gid = 0
+ tarinfo.gname = 'root'
+ if numeric:
+ tarinfo.uname = ''
+ tarinfo.gname = ''
+ name = tarinfo.name
+ if not name.startswith('/') and not name.startswith('.'):
+ tarinfo.name = './' + name
+
+ if tarinfo.isfile():
+ self.tar.addfile(tarinfo, intar.extractfile(tarinfo.name))
+ else:
+ self.tar.addfile(tarinfo)
+
+ def close(self):
+ """Close the output tar file.
+
+ This class should not be used anymore after calling that method.
+ """
+ self.tar.close()
diff --git a/tools/build_defs/docker/build_layer.py b/tools/build_defs/docker/build_layer.py
new file mode 100644
index 0000000..b9aa431
--- /dev/null
+++ b/tools/build_defs/docker/build_layer.py
@@ -0,0 +1,167 @@
+# Copyright 2015 Google Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""This tool build docker layer tar file from a list of inputs."""
+
+import os
+import os.path
+import sys
+import tarfile
+import tempfile
+
+from tools.build_defs.docker import archive
+from third_party.py import gflags
+
+gflags.DEFINE_string(
+ 'output', None,
+ 'The output file, mandatory')
+gflags.MarkFlagAsRequired('output')
+
+gflags.DEFINE_multistring(
+ 'file', [],
+ 'A file to add to the layer')
+
+gflags.DEFINE_multistring(
+ 'tar', [],
+ 'A tar file to add to the layer')
+
+gflags.DEFINE_multistring(
+ 'deb', [],
+ 'A debian package to add to the layer')
+
+gflags.DEFINE_multistring(
+ 'link', [],
+ 'Add a symlink a inside the layer ponting to b if a:b is specified')
+gflags.RegisterValidator(
+ 'link',
+ lambda l: all(value.find(':') > 0 for value in l),
+ message='--link value should contains a : separator')
+
+gflags.DEFINE_string(
+ 'file_path', '',
+ 'The path to strip from the files passed using the --file option')
+
+gflags.DEFINE_string(
+ 'directory', None,
+ 'Directory in which to store the file inside the layer')
+
+FLAGS = gflags.FLAGS
+
+
+class DockerLayer(object):
+ """A class to generates a Docker layer."""
+
+ class DebError(Exception):
+ pass
+
+ def __init__(self, output, file_path, directory):
+ self.file_path = file_path
+ self.directory = directory
+ self.output = output
+
+ def __enter__(self):
+ self.tarfile = archive.TarFileWriter(self.output)
+ return self
+
+ def __exit__(self, t, v, traceback):
+ self.tarfile.close()
+
+ def add_file(self, f):
+ """Add a file to the layer.
+
+ Args:
+ f: the file to add to the layer
+
+ `f` will be copied into the `self.directory` sub-directory of the layer.
+ Directory structure beyond `self.file_path` will be preserved (so, if
+ `f`'s path is `a/b/c/file`, `directory` is `d/e` and `file_path` is `a/b`,
+ the path of the file in the layer will be `d/e/c/file`).
+ """
+ dest = f
+ # TODO(mattmoor): Consider applying the working directory to all four
+ # options, not just files...
+ if dest.startswith(self.file_path):
+ dest = dest[len(self.file_path):]
+ if self.directory:
+ dest = self.directory + '/' + dest
+ dest = dest.lstrip('/') # Remove leading slashes
+ self.tarfile.add_file(dest, file_content=f)
+
+ def add_tar(self, tar):
+ """Add a tar file to the layer.
+
+ All files presents in that tar will be added to the layer under
+ the same paths. The current user uid and gid will be replaced
+ by 0 (to make like we are running as root) and no user name nor
+ group names will be added to the layer.
+
+ Args:
+ tar: the tar file to add to the layer
+ """
+ self.tarfile.add_tar(tar,
+ rootuid=os.getuid(),
+ rootgid=os.getgid(),
+ numeric=True)
+
+ def add_link(self, symlink, destination):
+ """Add a symbolic link pointing to `destination` in the layer.
+
+ Args:
+ symlink: the name of the symbolic link to add.
+ destination: where the symbolic link point to.
+ """
+ self.tarfile.add_file(symlink, tarfile.SYMTYPE, link=destination)
+
+ def add_deb(self, deb):
+ """Extract a debian package in the layer.
+
+ All files presents in that debian package will be added to the
+ layer under the same paths. No user name nor group names will
+ be added to the layer.
+
+ Args:
+ deb: the tar file to add to the layer
+
+ Raises:
+ DebError: if the format of the deb archive is incorrect.
+
+ This method does not support LZMA (data.tar.xz or data.tar.lzma)
+ for the data in the deb package. Using Python 3 would fix it.
+ """
+ with archive.SimpleArFile(deb) as arfile:
+ current = arfile.next()
+ while current and not current.filename.startswith('data.'):
+ current = arfile.next()
+ if not current:
+ raise self.DebError(deb + ' does not contains a data file!')
+ tmpfile = tempfile.mkstemp(suffix=os.path.splitext(current.filename)[-1])
+ with open(tmpfile[1], 'wb') as f:
+ f.write(current.data)
+ self.add_tar(tmpfile[1])
+ os.remove(tmpfile[1])
+
+
+def main(unused_argv):
+ with DockerLayer(FLAGS.output, FLAGS.file_path, FLAGS.directory) as layer:
+ for f in FLAGS.file:
+ layer.add_file(f)
+ for tar in FLAGS.tar:
+ layer.add_tar(tar)
+ for deb in FLAGS.deb:
+ layer.add_deb(deb)
+ for link in FLAGS.link:
+ l = link.split(':', 1)
+ layer.add_link(l[0], l[1])
+
+if __name__ == '__main__':
+ main(FLAGS(sys.argv))
diff --git a/tools/build_defs/docker/build_test.sh b/tools/build_defs/docker/build_test.sh
new file mode 100755
index 0000000..3cd3cfe
--- /dev/null
+++ b/tools/build_defs/docker/build_test.sh
@@ -0,0 +1,279 @@
+#!/bin/bash
+
+# Copyright 2015 Google Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Unit tests for docker_build
+
+DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+source ${DIR}/testenv.sh || { echo "testenv.sh not found!" >&2; exit 1; }
+
+readonly PLATFORM="$(uname -s | tr 'A-Z' 'a-z')"
+if [ "${PLATFORM}" = "darwin" ]; then
+ readonly MAGIC_TIMESTAMP="$(date -r 0 "+%b %e %Y")"
+else
+ readonly MAGIC_TIMESTAMP="$(date --date=@0 "+%F %R")"
+fi
+
+function EXPECT_CONTAINS() {
+ local complete="${1}"
+ local substring="${2}"
+ local message="${3:-Expected '${substring}' not found in '${complete}'}"
+
+ echo "${complete}" | grep -Fsq -- "${substring}" \
+ || fail "$message"
+}
+
+function check_property() {
+ local property="${1}"
+ local tarball="${2}"
+ local layer="${3}"
+ local expected="${4}"
+ local test_data="${TEST_DATA_DIR}/${tarball}.tar"
+
+ local metadata="$(tar xOf "${test_data}" "./${layer}/json")"
+
+ # This would be much more accurate if we had 'jq' everywhere.
+ EXPECT_CONTAINS "${metadata}" "\"${property}\": ${expected}"
+}
+
+function check_no_property() {
+ local property="${1}"
+ local tarball="${2}"
+ local layer="${3}"
+ local test_data="${TEST_DATA_DIR}/${tarball}.tar"
+
+ tar xOf "${test_data}" "./${layer}/json" >$TEST_log
+ expect_not_log "\"${property}\":"
+}
+
+function check_size() {
+ check_property Size "${@}"
+}
+
+function check_id() {
+ check_property id "${@}"
+}
+
+function check_parent() {
+ check_property parent "${@}"
+}
+
+function check_entrypoint() {
+ check_property Entrypoint "${@}"
+}
+
+function check_cmd() {
+ check_property Cmd "${@}"
+}
+
+function check_ports() {
+ check_property ExposedPorts "${@}"
+}
+
+function check_volumes() {
+ check_property Volumes "${@}"
+}
+
+function check_env() {
+ check_property Env "${@}"
+}
+
+function check_layers() {
+ local input=${1}
+ shift 1
+ local expected_layers=(${*})
+
+ local expected_layers_sorted=(
+ $(for i in ${expected_layers[*]}; do echo $i; done | sort)
+ )
+ local test_data="${TEST_DATA_DIR}/${input}.tar"
+
+ # Verbose output for testing.
+ tar tvf "${test_data}"
+
+ local actual_layers=(
+ $(tar tvf ${test_data} | tr -s ' ' | cut -d' ' -f 4- | sort \
+ | cut -d'/' -f 2 | grep -E '^[0-9a-f]+$' | sort | uniq))
+
+ # Verbose output for testing.
+ echo Expected: ${expected_layers_sorted[@]}
+ echo Actual: ${actual_layers[@]}
+
+ check_eq "${#expected_layers[@]}" "${#actual_layers[@]}"
+
+ local index=0
+ local parent=
+ while [ "${index}" -lt "${#expected_layers[@]}" ]
+ do
+ # Check that the nth sorted layer matches
+ check_eq "${expected_layers_sorted[$index]}" "${actual_layers[$index]}"
+
+ # Grab the ordered layer and check it.
+ local layer="${expected_layers[$index]}"
+
+ # Verbose output for testing.
+ echo Checking layer: "${layer}"
+
+ local listing="$(tar xOf "${test_data}" "./${layer}/layer.tar" | tar tv)"
+
+ # Check that all files in the layer, if any, have the magic timestamp
+ check_eq "$(echo "${listing}" | grep -Fv "${MAGIC_TIMESTAMP}")" ""
+
+ check_id "${input}" "${layer}" "\"${layer}\""
+
+ # Check that the layer contains its predecessor as its parent in the JSON.
+ if [[ -n "${parent}" ]]; then
+ check_parent "${input}" "${layer}" "\"${parent}\""
+ fi
+
+ # Check that the layer's size metadata matches the layer's tarball's size.
+ local layer_size=$(tar xOf "${test_data}" "./${layer}/layer.tar" | wc -c | xargs)
+ check_size "${input}" "${layer}" "${layer_size}"
+
+ index=$((index + 1))
+ parent=$layer
+ done
+}
+
+function test_files_base() {
+ check_layers "files_base" \
+ "240dd12c02aee796394ce18eee3108475f7d544294b17fc90ec54e983601fe1b"
+}
+
+function test_files_with_files_base() {
+ check_layers "files_with_files_base" \
+ "240dd12c02aee796394ce18eee3108475f7d544294b17fc90ec54e983601fe1b" \
+ "a9fd8cab2b9831ca2a13f371c04667a7698ef3baa90f3e820c4568d774cc69ab"
+}
+
+function test_tar_base() {
+ check_layers "tar_base" \
+ "83e8285de55c00f74f45628f75aec4366b361913be486e2e96af1a7b05211094"
+
+ # Check that this layer doesn't have any entrypoint data by looking
+ # for *any* entrypoint.
+ check_no_property "Entrypoint" "tar_base" \
+ "83e8285de55c00f74f45628f75aec4366b361913be486e2e96af1a7b05211094"
+}
+
+function test_tar_with_tar_base() {
+ check_layers "tar_with_tar_base" \
+ "83e8285de55c00f74f45628f75aec4366b361913be486e2e96af1a7b05211094" \
+ "f2878819ee41f261d2ed346e92c1fc2096e9eaa51e3e1fb32c7da1a21be77029"
+}
+
+function test_files_with_tar_base() {
+ check_layers "files_with_tar_base" \
+ "83e8285de55c00f74f45628f75aec4366b361913be486e2e96af1a7b05211094" \
+ "c96f2793f6ade79f8f4a4cfe46f31752de14f3b1eae7f27aa0c7440f78f612f3"
+}
+
+function test_tar_with_files_base() {
+ check_layers "tar_with_files_base" \
+ "240dd12c02aee796394ce18eee3108475f7d544294b17fc90ec54e983601fe1b" \
+ "2f1d1cc52ab8e72bf5affcac1a68a86c7f75679bf58a2b2a6fefdbfa0d239651"
+}
+
+function test_base_with_entrypoint() {
+ check_layers "base_with_entrypoint" \
+ "3cf09865c613d49e5fa6a1f7027744e51da662139ea833f8e757f70c8f75a554"
+
+ check_entrypoint "base_with_entrypoint" \
+ "3cf09865c613d49e5fa6a1f7027744e51da662139ea833f8e757f70c8f75a554" \
+ '["/bar"]'
+
+ # Check that the base layer has a port exposed.
+ check_ports "base_with_entrypoint" \
+ "3cf09865c613d49e5fa6a1f7027744e51da662139ea833f8e757f70c8f75a554" \
+ '{"8080/tcp": {}}'
+}
+
+function test_derivative_with_shadowed_cmd() {
+ check_layers "derivative_with_shadowed_cmd" \
+ "3cf09865c613d49e5fa6a1f7027744e51da662139ea833f8e757f70c8f75a554" \
+ "46e302dc2cb5c19baaeb479e8142ab1bb12ca77b3d7a0ecd379304413e6c5b28"
+}
+
+function test_derivative_with_cmd() {
+ check_layers "derivative_with_cmd" \
+ "3cf09865c613d49e5fa6a1f7027744e51da662139ea833f8e757f70c8f75a554" \
+ "46e302dc2cb5c19baaeb479e8142ab1bb12ca77b3d7a0ecd379304413e6c5b28" \
+ "968891207e14ab79a7ab3c71c796b88a4321ec30b9a74feb1d7c92d5a47c8bc2"
+
+ check_entrypoint "derivative_with_cmd" \
+ "968891207e14ab79a7ab3c71c796b88a4321ec30b9a74feb1d7c92d5a47c8bc2" \
+ '["/bar"]'
+
+ # Check that the middle layer has our shadowed arg.
+ check_cmd "derivative_with_cmd" \
+ "46e302dc2cb5c19baaeb479e8142ab1bb12ca77b3d7a0ecd379304413e6c5b28" \
+ '["shadowed-arg"]'
+
+ # Check that our topmost layer excludes the shadowed arg.
+ check_cmd "derivative_with_cmd" \
+ "968891207e14ab79a7ab3c71c796b88a4321ec30b9a74feb1d7c92d5a47c8bc2" \
+ '["arg1", "arg2"]'
+
+ # Check that the topmost layer has the ports exposed by the bottom
+ # layer, and itself.
+ check_ports "derivative_with_cmd" \
+ "968891207e14ab79a7ab3c71c796b88a4321ec30b9a74feb1d7c92d5a47c8bc2" \
+ '{"80/tcp": {}, "8080/tcp": {}}'
+}
+
+function test_derivative_with_volume() {
+ check_layers "derivative_with_volume" \
+ "f86da639a9346bec6d3a821ad1f716a177a8ff8f71d66f8b70238ce7e7ba51b8" \
+ "839bbd055b732c784847b3ec112d88c94f3bb752147987daef916bc956f9adf0"
+
+ # Check that the topmost layer has the ports exposed by the bottom
+ # layer, and itself.
+ check_volumes "derivative_with_volume" \
+ "f86da639a9346bec6d3a821ad1f716a177a8ff8f71d66f8b70238ce7e7ba51b8" \
+ '{"/logs": {}}'
+
+ check_volumes "derivative_with_volume" \
+ "839bbd055b732c784847b3ec112d88c94f3bb752147987daef916bc956f9adf0" \
+ '{"/asdf": {}, "/blah": {}, "/logs": {}}'
+}
+
+function test_generated_tarball() {
+ check_layers "generated_tarball" \
+ "54b8328604115255cc76c12a2a51939be65c40bf182ff5a898a5fb57c38f7772"
+}
+
+function test_with_env() {
+ check_layers "with_env" \
+ "f86da639a9346bec6d3a821ad1f716a177a8ff8f71d66f8b70238ce7e7ba51b8" \
+ "80b94376a90de45256c3e94c82bc3812bc5cbd05b7d01947f29e6805e8cd7018"
+
+ check_env "with_env" \
+ "80b94376a90de45256c3e94c82bc3812bc5cbd05b7d01947f29e6805e8cd7018" \
+ '["bar=blah blah blah", "foo=/asdf"]'
+}
+
+function test_with_double_env() {
+ check_layers "with_double_env" \
+ "f86da639a9346bec6d3a821ad1f716a177a8ff8f71d66f8b70238ce7e7ba51b8" \
+ "80b94376a90de45256c3e94c82bc3812bc5cbd05b7d01947f29e6805e8cd7018" \
+ "548e1d847a1d051e3cb3af383b0ebe40d341c01c97e735ae5a78ee3e10353b93"
+
+ # Check both the aggregation and the expansion of embedded variables.
+ check_env "with_double_env" \
+ "548e1d847a1d051e3cb3af383b0ebe40d341c01c97e735ae5a78ee3e10353b93" \
+ '["bar=blah blah blah", "baz=/asdf blah blah blah", "foo=/asdf"]'
+}
+
+run_suite "build_test"
diff --git a/tools/build_defs/docker/create_image.py b/tools/build_defs/docker/create_image.py
new file mode 100644
index 0000000..b86bc11
--- /dev/null
+++ b/tools/build_defs/docker/create_image.py
@@ -0,0 +1,137 @@
+# Copyright 2015 Google Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""This tool creates a docker image from a layer and the various metadata."""
+
+import sys
+import tarfile
+
+from tools.build_defs.docker import archive
+from third_party.py import gflags
+
+# Hardcoded docker versions that we are claiming to be.
+DATA_FORMAT_VERSION = '1.0'
+
+gflags.DEFINE_string(
+ 'output', None,
+ 'The output file, mandatory')
+gflags.MarkFlagAsRequired('output')
+
+gflags.DEFINE_string(
+ 'metadata', None,
+ 'The JSON metadata file for this image, mandatory.')
+gflags.MarkFlagAsRequired('metadata')
+
+gflags.DEFINE_string(
+ 'layer', None,
+ 'The tar file for the top layer of this image, mandatory.')
+gflags.MarkFlagAsRequired('metadata')
+
+gflags.DEFINE_string(
+ 'id', None,
+ 'The hex identifier of this image (hexstring or @filename), mandatory.')
+gflags.MarkFlagAsRequired('id')
+
+gflags.DEFINE_string(
+ 'base', None,
+ 'The base image file for this image.')
+
+gflags.DEFINE_string(
+ 'repository', None,
+ 'The name of the repository to add this image.')
+
+gflags.DEFINE_string(
+ 'name', None,
+ 'The symbolic name of this image.')
+
+FLAGS = gflags.FLAGS
+
+
+def _base_name_filter(name):
+ """Do not add multiple times 'top' and 'repositories' when merging images."""
+ filter_names = ['top', 'repositories']
+ return all([not name.endswith(s) for s in filter_names])
+
+
+def create_image(output, identifier,
+ base=None, layer=None, metadata=None,
+ name=None, repository=None):
+ """Creates a Docker image.
+
+ Args:
+ output: the name of the docker image file to create.
+ identifier: the identifier of the top layer for this image.
+ base: a base layer (optional) to merge to current layer.
+ layer: the layer content (a tar file).
+ metadata: the json metadata file for the top layer.
+ name: symbolic name for this docker image.
+ repository: repository name for this docker image.
+ """
+ tar = archive.TarFileWriter(output)
+ # Write our id to 'top' as we are now the topmost layer.
+ tar.add_file('top', content=identifier)
+ # Each layer is encoded as a directory in the larger tarball of the form:
+ # {id}\
+ # layer.tar
+ # VERSION
+ # json
+ # Create the directory for us to now fill in.
+ tar.add_file(identifier + '/', tarfile.DIRTYPE)
+ # VERSION generally seems to contain 1.0, not entirely sure
+ # what the point of this is.
+ tar.add_file(identifier + '/VERSION', content=DATA_FORMAT_VERSION)
+ # Add the layer file
+ tar.add_file(identifier + '/layer.tar', file_content=layer)
+ # Now the json metadata
+ tar.add_file(identifier + '/json', file_content=metadata)
+ # Merge the base if any
+ if base:
+ tar.add_tar(base, name_filter=_base_name_filter)
+ # In addition to N layers of the form described above, there is
+ # a single file at the top of the image called repositories.
+ # This file contains a JSON blob of the form:
+ # {
+ # 'repo':{
+ # 'tag-name': 'top-most layer hex',
+ # ...
+ # },
+ # ...
+ # }
+ if repository:
+ tar.add_file('repositories', content='\n'.join([
+ '{',
+ ' "%s": {' % repository,
+ ' "%s": "%s"' % (name, identifier),
+ ' }',
+ '}']))
+
+
+# Main program to create a docker image. It expect to be run with:
+# create_image --output=output_file \
+# --id=@identifier \
+# [--base=base] \
+# --layer=layer.tar \
+# --metadata=metadata.json \
+# --name=myname --repository=repositoryName
+# See the gflags declaration about the flags argument details.
+def main(unused_argv):
+ identifier = FLAGS.id
+ if identifier.startswith('@'):
+ with open(identifier[1:], 'r') as f:
+ identifier = f.read()
+ create_image(FLAGS.output, identifier, FLAGS.base,
+ FLAGS.layer, FLAGS.metadata,
+ FLAGS.name, FLAGS.repository)
+
+if __name__ == '__main__':
+ main(FLAGS(sys.argv))
diff --git a/tools/build_defs/docker/docker.bzl b/tools/build_defs/docker/docker.bzl
new file mode 100644
index 0000000..6ad6d4f
--- /dev/null
+++ b/tools/build_defs/docker/docker.bzl
@@ -0,0 +1,360 @@
+# Copyright 2015 Google Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Rules for manipulation Docker images."""
+
+# Filetype to restrict inputs
+tar_filetype = FileType([".tar", ".tar.gz", ".tgz", ".tar.xz"])
+deb_filetype = FileType([".deb"])
+
+# Docker files are tarballs, should we allow other extensions than tar?
+docker_filetype = tar_filetype
+
+# This validates the two forms of value accepted by
+# ENTRYPOINT and CMD, turning them into a canonical
+# python list form.
+#
+# The Dockerfile construct:
+# ENTRYPOINT "/foo"
+# Results in:
+# "Entrypoint": [
+# "/bin/sh",
+# "-c",
+# "\"/foo\""
+# ],
+# Whereas:
+# ENTRYPOINT ["/foo", "a"]
+# Results in:
+# "Entrypoint": [
+# "/foo",
+# "a"
+# ],
+# NOTE: prefacing a command with 'exec' just ends up with the former
+def _validate_command(name, argument):
+ if type(argument) == "string":
+ return ["/bin/sh", "-c", argument]
+ elif type(argument) == "list":
+ return argument
+ elif argument:
+ fail("The %s attribute must be a string or list, if specified." % name)
+ else:
+ return None
+
+def _short_path_dirname(path):
+ """Returns the directory's name of the short path of an artifact."""
+ sp = path.short_path
+ return sp[:sp.rfind("/")]
+
+def _build_layer(ctx):
+ """Build the current layer for appending it the base layer."""
+ # Compute the relative path
+ data_path = ctx.attr.data_path
+ if not data_path:
+ data_path = _short_path_dirname(ctx.outputs.out)
+ elif data_path[0] == "/":
+ data_path = data_path[1:]
+ else: # relative path
+ data_path = _short_path_dirname(ctx.outputs.out) + "/" + data_path
+
+ layer = ctx.new_file(ctx.label.name + ".layer")
+ build_layer = ctx.executable._build_layer
+ args = [
+ "--file_path=" + data_path,
+ "--output=" + layer.path,
+ "--directory=" + ctx.attr.directory
+ ]
+ args += ["--file=" + f.path for f in ctx.files.files]
+ args += ["--tar=" + f.path for f in ctx.files.tars]
+ args += ["--deb=" + f.path for f in ctx.files.debs]
+ args += ["--link=%s:%s" % (k, ctx.attr.symlinks[k])
+ for k in ctx.attr.symlinks]
+
+ ctx.action(
+ executable = build_layer,
+ arguments = args,
+ inputs = ctx.files.files + ctx.files.tars + ctx.files.debs,
+ outputs = [layer],
+ mnemonic="DockerLayer"
+ )
+ return layer
+
+def _sha256(ctx, artifact):
+ """Create an action to compute the SHA-256 of an artifact."""
+ out = ctx.new_file(artifact.basename + ".sha256")
+ ctx.action(
+ executable = ctx.executable._sha256,
+ arguments = [artifact.path, out.path],
+ inputs = [artifact],
+ outputs = [out],
+ mnemonic = "SHA256")
+ return out
+
+def _get_base_artifact(ctx):
+ if ctx.files.base:
+ if hasattr(ctx.attr.base, "docker_image"):
+ return ctx.attr.base.docker_image
+ if len(ctx.files.base) != 1:
+ fail("base attribute should be a single tar file.")
+ return ctx.files.base[0]
+
+def _metadata_action(ctx, layer, name, output):
+ """Generate the action to create the JSON metadata for the layer."""
+ rewrite_tool = ctx.executable._rewrite_tool
+ env = ctx.attr.env
+ args = [
+ "--output=%s" % output.path,
+ "--layer=%s" % layer.path,
+ "--name=@%s" % name.path,
+ "--entrypoint=%s" % ",".join(ctx.attr.entrypoint),
+ "--command=%s" % ",".join(ctx.attr.cmd),
+ "--env=%s" % ",".join(["%s=%s" % (k, env[k]) for k in env]),
+ "--ports=%s" % ",".join(ctx.attr.ports),
+ "--volumes=%s" % ",".join(ctx.attr.volumes)
+ ]
+ inputs = [layer, rewrite_tool, name]
+ base = _get_base_artifact(ctx)
+ if base:
+ args += ["--base=%s" % base.path]
+ inputs += [base]
+
+ ctx.action(
+ executable = rewrite_tool,
+ arguments = args,
+ inputs = inputs,
+ outputs = [output],
+ mnemonic = "RewriteJSON")
+
+def _compute_layer_name(ctx, layer):
+ """Compute the layer's name.
+
+ This function synthesize a version of its metadata where in place
+ of its final name, we use the SHA256 of the layer blob.
+
+ This makes the name of the layer a function of:
+ - Its layer's SHA256
+ - Its metadata
+ - Its parent's name.
+ Assuming the parent's name is derived by this same rigor, then
+ a simple induction proves the content addressability.
+
+ Args:
+ ctx: Rule context.
+ layer: The layer's artifact for which to compute the name.
+ Returns:
+ The artifact that will contains the name for the layer.
+ """
+ metadata = ctx.new_file(ctx.label.name + ".metadata-name")
+ layer_sha = _sha256(ctx, layer)
+ _metadata_action(ctx, layer, layer_sha, metadata)
+ return _sha256(ctx, metadata)
+
+def _metadata(ctx, layer, name):
+ """Create the metadata for the new docker image."""
+ metadata = ctx.new_file(ctx.label.name + ".metadata")
+ _metadata_action(ctx, layer, name, metadata)
+ return metadata
+
+def _create_image(ctx, layer, name, metadata):
+ """Create the new image."""
+ create_image = ctx.executable._create_image
+ args = [
+ "--output=" + ctx.outputs.out.path,
+ "--metadata=" + metadata.path,
+ "--layer=" + layer.path,
+ "--id=@" + name.path,
+ # We label at push time, so we only put a single name in this file:
+ # bazel/package:target => {the layer being appended}
+ # TODO(dmarting): Does the name makes sense? We could use the
+ # repositoryName/package instead. (why do we need to replace
+ # slashes?)
+ "--repository=bazel/" + ctx.label.package.replace("/", "_"),
+ "--name=" + ctx.label.name
+ ]
+ inputs = [layer, metadata, name]
+ # If we have been provided a base image, add it.
+ base = _get_base_artifact(ctx)
+ if base:
+ args += ["--base=%s" % base.path]
+ inputs += [base]
+ ctx.action(
+ executable = create_image,
+ arguments = args,
+ inputs = inputs,
+ use_default_shell_env = True,
+ outputs = [ctx.outputs.out]
+ )
+
+def _docker_build_impl(ctx):
+ """Implementation for the docker_build rule."""
+ layer = _build_layer(ctx)
+ name = _compute_layer_name(ctx, layer)
+ metadata = _metadata(ctx, layer, name)
+ _create_image(ctx, layer, name, metadata)
+ ctx.file_action(
+ content = "\n".join([
+ "#!/bin/bash -eu",
+ "docker load -i " + ctx.outputs.out.short_path
+ ]),
+ output = ctx.outputs.executable,
+ executable = True)
+ return struct(runfiles = ctx.runfiles(files = [ctx.outputs.out]),
+ docker_image = ctx.outputs.out)
+
+docker_build_ = rule(
+ implementation = _docker_build_impl,
+ attrs = {
+ "base": attr.label(allow_files=docker_filetype),
+ "data_path": attr.string(),
+ "directory": attr.string(default="/"),
+ "tars": attr.label_list(allow_files=tar_filetype),
+ "debs": attr.label_list(allow_files=deb_filetype),
+ "files": attr.label_list(allow_files=True),
+ "symlinks": attr.string_dict(),
+ "entrypoint": attr.string_list(),
+ "cmd": attr.string_list(),
+ "env": attr.string_dict(),
+ "ports": attr.string_list(), # Skylark doesn't support int_list...
+ "volumes": attr.string_list(),
+ # Implicit dependencies.
+ "_build_layer": attr.label(
+ default=Label("//tools/build_defs/docker:build_layer"),
+ cfg=HOST_CFG,
+ executable=True,
+ allow_files=True),
+ "_create_image": attr.label(
+ default=Label("//tools/build_defs/docker:create_image"),
+ cfg=HOST_CFG,
+ executable=True,
+ allow_files=True),
+ "_rewrite_tool": attr.label(
+ default=Label("//tools/build_defs/docker:rewrite_json"),
+ cfg=HOST_CFG,
+ executable=True,
+ allow_files=True),
+ "_sha256": attr.label(
+ default=Label("//tools/build_defs/docker:sha256"),
+ cfg=HOST_CFG,
+ executable=True,
+ allow_files=True)
+ },
+ outputs = {
+ "out": "%{name}.tar",
+ },
+ executable = True)
+
+# Produces a new docker image tarball compatible with 'docker load', which
+# is a single additional layer atop 'base'. The goal is to have relatively
+# complete support for building docker image, from the Dockerfile spec.
+#
+# Only 'name' is required. All other fields have sane defaults.
+#
+# docker_build(
+# name="...",
+# visibility="...",
+#
+# # The base layers on top of which to overlay this layer,
+# # equivalent to FROM.
+# base="//another/build:rule",]
+#
+# # The base directory of the files, defaulted to this package.
+# # All files structure relatively to that path will be preserved.
+# # A leading '/' mean the workspace root and this path is relative
+# # to the current package by default.
+# data_path="...",
+#
+# # The directory in which to expand the specified files,
+# # defaulting to '/'.
+# # Only makes sense accompanying one of files/tars/debs.
+# directory="...",
+#
+# # The set of archives to expand, or packages to install
+# # within the chroot of this layer
+# files=[...],
+# tars=[...],
+# debs=[...],
+#
+# # The set of symlinks to create within a given layer.
+# symlinks = {
+# "/path/to/link": "/path/to/target",
+# ...
+# },
+#
+# # https://docs.docker.com/reference/builder/#entrypoint
+# entrypoint="...", or
+# entrypoint=[...], -- exec form
+#
+# # https://docs.docker.com/reference/builder/#cmd
+# cmd="...", or
+# cmd=[...], -- exec form
+#
+# # https://docs.docker.com/reference/builder/#expose
+# ports=[...],
+#
+# # TODO(mattmoor): NYI
+# # https://docs.docker.com/reference/builder/#maintainer
+# maintainer="...",
+#
+# # TODO(mattmoor): NYI
+# # https://docs.docker.com/reference/builder/#user
+# # NOTE: the normal directive affects subsequent RUN, CMD,
+# # and ENTRYPOINT
+# user="...",
+#
+# # https://docs.docker.com/reference/builder/#volume
+# volumes=[...],
+#
+# # TODO(mattmoor): NYI
+# # https://docs.docker.com/reference/builder/#workdir
+# # NOTE: the normal directive affects subsequent RUN, CMD,
+# # ENTRYPOINT, ADD, and COPY
+# workdir="...",
+#
+# # https://docs.docker.com/reference/builder/#env
+# env = {
+# "var1": "val1",
+# "var2": "val2",
+# ...
+# "varN": "valN",
+# },
+#
+# # NOTE: Without a motivating use case, there is little reason to support:
+# # https://docs.docker.com/reference/builder/#onbuild
+# )
+def docker_build(**kwargs):
+ """Package a docker image.
+
+ This rule generates a sequence of genrules the last of which is named 'name',
+ so the dependency graph works out properly. The output of this rule is a
+ tarball compatible with 'docker save/load' with the structure:
+ {layer-name}:
+ layer.tar
+ VERSION
+ json
+ ...
+ repositories
+ top # an implementation detail of our rules, not consumed by Docker.
+ This rule appends a single new layer to the tarball of this form provided
+ via the 'base' parameter.
+
+ The images produced by this rule are always named 'blaze/tmp:latest' when
+ loaded (an internal detail). The expectation is that the images produced
+ by these rules will be uploaded using the 'docker_push' rule below.
+
+ Args:
+ **kwargs: See above.
+ """
+ if "cmd" in kwargs:
+ kwargs["cmd"] = _validate_command("cmd", kwargs["cmd"])
+ if "entrypoint" in kwargs:
+ kwargs["entrypoint"] = _validate_command("entrypoint", kwargs["entrypoint"])
+ docker_build_(**kwargs)
diff --git a/tools/build_defs/docker/rewrite_json.py b/tools/build_defs/docker/rewrite_json.py
new file mode 100644
index 0000000..a189971
--- /dev/null
+++ b/tools/build_defs/docker/rewrite_json.py
@@ -0,0 +1,225 @@
+# Copyright 2015 Google Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""This package manipulates Docker image layer metadata."""
+from collections import namedtuple
+import copy
+import json
+import os
+import os.path
+import sys
+import tarfile
+
+from third_party.py import gflags
+
+gflags.DEFINE_string(
+ 'name', None, 'The name of the current layer')
+
+gflags.DEFINE_string(
+ 'base', None, 'The parent image')
+
+gflags.DEFINE_string(
+ 'output', None, 'The output file to generate')
+
+gflags.DEFINE_string(
+ 'layer', None, 'The current layer tar')
+
+gflags.DEFINE_list(
+ 'entrypoint', None,
+ 'Override the "Entrypoint" of the previous layer')
+
+gflags.DEFINE_list(
+ 'command', None,
+ 'Override the "Cmd" of the previous layer')
+
+gflags.DEFINE_list(
+ 'ports', None,
+ 'Augment the "ExposedPorts" of the previous layer')
+
+gflags.DEFINE_list(
+ 'volumes', None,
+ 'Augment the "Volumes" of the previous layer')
+
+gflags.DEFINE_list(
+ 'env', None,
+ 'Augment the "Env" of the previous layer')
+
+FLAGS = gflags.FLAGS
+
+_MetadataOptionsT = namedtuple(
+ 'MetadataOptionsT',
+ ['name', 'parent', 'size', 'entrypoint', 'cmd', 'env', 'ports', 'volumes'])
+
+
+class MetadataOptions(_MetadataOptionsT):
+ """Docker image layer metadata options."""
+
+ def __new__(cls, name=None, parent=None, size=None,
+ entrypoint=None, cmd=None, env=None,
+ ports=None, volumes=None):
+ """Constructor."""
+ return super(MetadataOptions, cls).__new__(
+ cls, name=name, parent=parent, size=size,
+ entrypoint=entrypoint, cmd=cmd, env=env,
+ ports=ports, volumes=volumes)
+
+
+_DOCKER_VERSION = '1.5.0'
+
+_PROCESSOR_ARCHITECTURE = 'amd64'
+
+_OPERATING_SYSTEM = 'linux'
+
+
+def Resolve(value, environment):
+ """Resolves environment variables embedded in the given value."""
+ outer_env = os.environ
+ try:
+ os.environ = environment
+ return os.path.expandvars(value)
+ finally:
+ os.environ = outer_env
+
+
+def RewriteMetadata(data, options):
+ """Rewrite and return a copy of the input data according to options.
+
+ Args:
+ data: The dict of Docker image layer metadata we're copying and rewriting.
+ options: The changes this layer makes to the overall image's metadata, which
+ first appears in this layer's version of the metadata
+
+ Returns:
+ A deep copy of data, which has been updated to reflect the metadata
+ additions of this layer.
+
+ Raises:
+ Exception: a required option was missing.
+ """
+ output = copy.deepcopy(data)
+
+ if not options.name:
+ raise Exception('Missing required option: name')
+ output['id'] = options.name
+
+ if options.parent:
+ output['parent'] = options.parent
+ elif data:
+ raise Exception('Expected empty input object when parent is omitted')
+
+ if options.size:
+ output['Size'] = options.size
+ elif 'Size' in output:
+ del output['Size']
+
+ if 'config' not in output:
+ output['config'] = {}
+
+ if options.entrypoint:
+ output['config']['Entrypoint'] = options.entrypoint
+ if options.cmd:
+ output['config']['Cmd'] = options.cmd
+
+ output['docker_version'] = _DOCKER_VERSION
+ output['architecture'] = _PROCESSOR_ARCHITECTURE
+ output['os'] = _OPERATING_SYSTEM
+
+ if options.env:
+ environ_dict = {}
+ # Build a dictionary of existing environment variables (used by Resolve).
+ for kv in output['config'].get('Env', []):
+ (k, v) = kv.split('=', 1)
+ environ_dict[k] = v
+ # Merge in new environment variables, resolving references.
+ for kv in options.env:
+ (k, v) = kv.split('=', 1)
+ # Resolve handles scenarios like "PATH=$PATH:...".
+ v = Resolve(v, environ_dict)
+ environ_dict[k] = v
+ output['config']['Env'] = [
+ '%s=%s' % (k, environ_dict[k]) for k in sorted(environ_dict.keys())]
+
+ if options.ports:
+ if 'ExposedPorts' not in output['config']:
+ output['config']['ExposedPorts'] = {}
+ for p in options.ports:
+ if '/' in p:
+ # The port spec has the form 80/tcp, 1234/udp
+ # so we simply use it as the key.
+ output['config']['ExposedPorts'][p] = {}
+ else:
+ # Assume tcp
+ output['config']['ExposedPorts'][p + '/tcp'] = {}
+
+ if options.volumes:
+ if 'Volumes' not in output['config']:
+ output['config']['Volumes'] = {}
+ for p in options.volumes:
+ output['config']['Volumes'][p] = {}
+
+ # TODO(mattmoor): comment, created, container_config
+
+ # container_config contains information about the container
+ # that was used to create this layer, so it shouldn't
+ # propagate from the parent to child. This is where we would
+ # annotate information that can be extract by tools like Blubber
+ # or Quay.io's UI to gain insight into the source that generated
+ # the layer. A Dockerfile might produce something like:
+ # # (nop) /bin/sh -c "apt-get update"
+ # We might consider encoding the fully-qualified bazel build target:
+ # //tools/build_defs/docker:image
+ # However, we should be sensitive to leaking data through this field.
+ if 'container_config' in output:
+ del output['container_config']
+
+ return output
+
+
+def GetTarFile(f, name):
+ """Return the content of a file inside a tar file."""
+ with tarfile.open(f, 'r') as tar:
+ tarinfo = tar.getmember(name)
+ if not tarinfo:
+ return ''
+ return tar.extractfile(tarinfo).read()
+
+
+def main(unused_argv):
+ parent = ''
+ base_json = '{}'
+ if FLAGS.base:
+ parent = GetTarFile(FLAGS.base, './top')
+ base_json = GetTarFile(FLAGS.base, './%s/json' % parent)
+ data = json.loads(base_json)
+
+ name = FLAGS.name
+ if name.startswith('@'):
+ with open(name[1:], 'r') as f:
+ name = f.read()
+
+ output = RewriteMetadata(data, MetadataOptions(
+ name=name,
+ parent=parent,
+ size=os.path.getsize(FLAGS.layer),
+ entrypoint=FLAGS.entrypoint,
+ cmd=FLAGS.command,
+ env=FLAGS.env,
+ ports=FLAGS.ports,
+ volumes=FLAGS.volumes))
+
+ with open(FLAGS.output, 'w') as fp:
+ json.dump(output, fp, sort_keys=True)
+ fp.write('\n')
+
+if __name__ == '__main__':
+ main(FLAGS(sys.argv))
diff --git a/tools/build_defs/docker/rewrite_json_test.py b/tools/build_defs/docker/rewrite_json_test.py
new file mode 100644
index 0000000..8bb711d
--- /dev/null
+++ b/tools/build_defs/docker/rewrite_json_test.py
@@ -0,0 +1,621 @@
+# Copyright 2015 Google Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Testing for rewrite_json."""
+
+import unittest
+
+from tools.build_defs.docker.rewrite_json import _DOCKER_VERSION
+from tools.build_defs.docker.rewrite_json import _OPERATING_SYSTEM
+from tools.build_defs.docker.rewrite_json import _PROCESSOR_ARCHITECTURE
+from tools.build_defs.docker.rewrite_json import MetadataOptions
+from tools.build_defs.docker.rewrite_json import RewriteMetadata
+
+
+class RewriteJsonTest(unittest.TestCase):
+ """Testing for rewrite_json."""
+
+ def testNewEntrypoint(self):
+ in_data = {
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor'
+ }
+ }
+ name = 'deadbeef'
+ parent = 'blah'
+ entrypoint = ['/bin/bash']
+ expected = {
+ 'id': name,
+ 'parent': parent,
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'Entrypoint': entrypoint
+ },
+ 'docker_version': _DOCKER_VERSION,
+ 'architecture': _PROCESSOR_ARCHITECTURE,
+ 'os': _OPERATING_SYSTEM,
+ }
+
+ actual = RewriteMetadata(in_data, MetadataOptions(
+ name=name, entrypoint=entrypoint, parent=parent))
+ self.assertEquals(expected, actual)
+
+ def testOverrideEntrypoint(self):
+ in_data = {
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'Entrypoint': ['/bin/sh', 'does', 'not', 'matter'],
+ }
+ }
+ name = 'deadbeef'
+ parent = 'blah'
+ entrypoint = ['/bin/bash']
+ expected = {
+ 'id': name,
+ 'parent': parent,
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'Entrypoint': entrypoint
+ },
+ 'docker_version': _DOCKER_VERSION,
+ 'architecture': _PROCESSOR_ARCHITECTURE,
+ 'os': _OPERATING_SYSTEM,
+ }
+
+ actual = RewriteMetadata(in_data, MetadataOptions(
+ name=name, entrypoint=entrypoint, parent=parent))
+ self.assertEquals(expected, actual)
+
+ def testNewCmd(self):
+ in_data = {
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'Entrypoint': ['/bin/bash'],
+ }
+ }
+ name = 'deadbeef'
+ parent = 'blah'
+ cmd = ['/bin/bash']
+ expected = {
+ 'id': name,
+ 'parent': parent,
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'Entrypoint': ['/bin/bash'],
+ 'Cmd': cmd
+ },
+ 'docker_version': _DOCKER_VERSION,
+ 'architecture': _PROCESSOR_ARCHITECTURE,
+ 'os': _OPERATING_SYSTEM,
+ }
+
+ actual = RewriteMetadata(in_data, MetadataOptions(
+ name=name, cmd=cmd, parent=parent))
+ self.assertEquals(expected, actual)
+
+ def testOverrideCmd(self):
+ in_data = {
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'Entrypoint': ['/bin/bash'],
+ 'Cmd': ['does', 'not', 'matter'],
+ }
+ }
+ name = 'deadbeef'
+ parent = 'blah'
+ cmd = ['does', 'matter']
+ expected = {
+ 'id': name,
+ 'parent': parent,
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'Entrypoint': ['/bin/bash'],
+ 'Cmd': cmd
+ },
+ 'docker_version': _DOCKER_VERSION,
+ 'architecture': _PROCESSOR_ARCHITECTURE,
+ 'os': _OPERATING_SYSTEM,
+ }
+
+ actual = RewriteMetadata(in_data, MetadataOptions(
+ name=name, cmd=cmd, parent=parent))
+ self.assertEquals(expected, actual)
+
+ def testOverrideBoth(self):
+ in_data = {
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'Entrypoint': ['/bin/sh'],
+ 'Cmd': ['does', 'not', 'matter'],
+ }
+ }
+ name = 'deadbeef'
+ parent = 'blah'
+ entrypoint = ['/bin/bash', '-c']
+ cmd = ['my-command', 'my-arg1', 'my-arg2']
+ expected = {
+ 'id': name,
+ 'parent': parent,
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'Entrypoint': entrypoint,
+ 'Cmd': cmd
+ },
+ 'docker_version': _DOCKER_VERSION,
+ 'architecture': _PROCESSOR_ARCHITECTURE,
+ 'os': _OPERATING_SYSTEM,
+ }
+
+ actual = RewriteMetadata(in_data, MetadataOptions(
+ name=name, entrypoint=entrypoint, cmd=cmd, parent=parent))
+ self.assertEquals(expected, actual)
+
+ def testOverrideParent(self):
+ name = 'me!'
+ parent = 'parent'
+ # In the typical case, we expect the parent to
+ # come in as the 'id', and our grandparent to
+ # be its 'parent'.
+ in_data = {
+ 'id': parent,
+ 'parent': 'grandparent',
+ }
+ expected = {
+ 'id': name,
+ 'parent': parent,
+ 'config': {},
+ 'docker_version': _DOCKER_VERSION,
+ 'architecture': _PROCESSOR_ARCHITECTURE,
+ 'os': _OPERATING_SYSTEM,
+ }
+
+ actual = RewriteMetadata(in_data, MetadataOptions(
+ name=name, parent=parent))
+ self.assertEquals(expected, actual)
+
+ def testNewSize(self):
+ # Size is one of the few fields that, when omitted,
+ # should be removed.
+ in_data = {
+ 'id': 'you',
+ 'Size': '124',
+ }
+ name = 'me'
+ parent = 'blah'
+ size = '4321'
+ expected = {
+ 'id': name,
+ 'parent': parent,
+ 'Size': size,
+ 'config': {},
+ 'docker_version': _DOCKER_VERSION,
+ 'architecture': _PROCESSOR_ARCHITECTURE,
+ 'os': _OPERATING_SYSTEM,
+ }
+
+ actual = RewriteMetadata(in_data, MetadataOptions(
+ name=name, size=size, parent=parent))
+ self.assertEquals(expected, actual)
+
+ def testOmitSize(self):
+ # Size is one of the few fields that, when omitted,
+ # should be removed.
+ in_data = {
+ 'id': 'you',
+ 'Size': '124',
+ }
+ name = 'me'
+ parent = 'blah'
+ expected = {
+ 'id': name,
+ 'parent': parent,
+ 'config': {},
+ 'docker_version': _DOCKER_VERSION,
+ 'architecture': _PROCESSOR_ARCHITECTURE,
+ 'os': _OPERATING_SYSTEM,
+ }
+
+ actual = RewriteMetadata(in_data, MetadataOptions(
+ name=name, parent=parent))
+ self.assertEquals(expected, actual)
+
+ def testOmitName(self):
+ # Name is required.
+ with self.assertRaises(Exception):
+ RewriteMetadata({}, MetadataOptions(name=None))
+
+ def testStripContainerConfig(self):
+ # Size is one of the few fields that, when omitted,
+ # should be removed.
+ in_data = {
+ 'id': 'you',
+ 'container_config': {},
+ }
+ name = 'me'
+ parent = 'blah'
+ expected = {
+ 'id': name,
+ 'parent': parent,
+ 'config': {},
+ 'docker_version': _DOCKER_VERSION,
+ 'architecture': _PROCESSOR_ARCHITECTURE,
+ 'os': _OPERATING_SYSTEM,
+ }
+
+ actual = RewriteMetadata(in_data, MetadataOptions(
+ name=name, parent=parent))
+ self.assertEquals(expected, actual)
+
+ def testEmptyBase(self):
+ in_data = {}
+ name = 'deadbeef'
+ entrypoint = ['/bin/bash', '-c']
+ cmd = ['my-command', 'my-arg1', 'my-arg2']
+ size = '999'
+ expected = {
+ 'id': name,
+ 'config': {
+ 'Entrypoint': entrypoint,
+ 'Cmd': cmd,
+ 'ExposedPorts': {
+ '80/tcp': {}
+ }
+ },
+ 'docker_version': _DOCKER_VERSION,
+ 'architecture': _PROCESSOR_ARCHITECTURE,
+ 'os': _OPERATING_SYSTEM,
+ 'Size': size,
+ }
+
+ actual = RewriteMetadata(in_data, MetadataOptions(
+ name=name, entrypoint=entrypoint, cmd=cmd, size=size,
+ ports=['80']))
+ self.assertEquals(expected, actual)
+
+ def testOmitParentWithBase(self):
+ # Our input data should be empty when parent is omitted
+ in_data = {
+ 'id': 'you',
+ }
+ with self.assertRaises(Exception):
+ RewriteMetadata(in_data, MetadataOptions(name='me'))
+
+ def testNewPort(self):
+ in_data = {
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor'
+ }
+ }
+ name = 'deadbeef'
+ parent = 'blah'
+ port = '80'
+ expected = {
+ 'id': name,
+ 'parent': parent,
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'ExposedPorts': {
+ port + '/tcp': {}
+ }
+ },
+ 'docker_version': _DOCKER_VERSION,
+ 'architecture': _PROCESSOR_ARCHITECTURE,
+ 'os': _OPERATING_SYSTEM,
+ }
+
+ actual = RewriteMetadata(in_data, MetadataOptions(
+ name=name, parent=parent, ports=[port]))
+ self.assertEquals(expected, actual)
+
+ def testAugmentPort(self):
+ in_data = {
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'ExposedPorts': {
+ '443/tcp': {}
+ }
+ }
+ }
+ name = 'deadbeef'
+ parent = 'blah'
+ port = '80'
+ expected = {
+ 'id': name,
+ 'parent': parent,
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'ExposedPorts': {
+ '443/tcp': {},
+ port + '/tcp': {}
+ }
+ },
+ 'docker_version': _DOCKER_VERSION,
+ 'architecture': _PROCESSOR_ARCHITECTURE,
+ 'os': _OPERATING_SYSTEM,
+ }
+
+ actual = RewriteMetadata(in_data, MetadataOptions(
+ name=name, parent=parent, ports=[port]))
+ self.assertEquals(expected, actual)
+
+ def testMultiplePorts(self):
+ in_data = {
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor'
+ }
+ }
+ name = 'deadbeef'
+ parent = 'blah'
+ port1 = '80'
+ port2 = '8080'
+ expected = {
+ 'id': name,
+ 'parent': parent,
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'ExposedPorts': {
+ port1 + '/tcp': {},
+ port2 + '/tcp': {}
+ }
+ },
+ 'docker_version': _DOCKER_VERSION,
+ 'architecture': _PROCESSOR_ARCHITECTURE,
+ 'os': _OPERATING_SYSTEM,
+ }
+
+ actual = RewriteMetadata(in_data, MetadataOptions(
+ name=name, parent=parent, ports=[port1, port2]))
+ self.assertEquals(expected, actual)
+
+ def testPortCollision(self):
+ port = '80'
+ in_data = {
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'ExposedPorts': {
+ port + '/tcp': {}
+ }
+ }
+ }
+ name = 'deadbeef'
+ parent = 'blah'
+ expected = {
+ 'id': name,
+ 'parent': parent,
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'ExposedPorts': {
+ port + '/tcp': {}
+ }
+ },
+ 'docker_version': _DOCKER_VERSION,
+ 'architecture': _PROCESSOR_ARCHITECTURE,
+ 'os': _OPERATING_SYSTEM,
+ }
+
+ actual = RewriteMetadata(in_data, MetadataOptions(
+ name=name, parent=parent, ports=[port]))
+ self.assertEquals(expected, actual)
+
+ def testPortWithProtocol(self):
+ in_data = {
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor'
+ }
+ }
+ name = 'deadbeef'
+ parent = 'blah'
+ port = '80/tcp'
+ expected = {
+ 'id': name,
+ 'parent': parent,
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'ExposedPorts': {
+ port: {}
+ }
+ },
+ 'docker_version': _DOCKER_VERSION,
+ 'architecture': _PROCESSOR_ARCHITECTURE,
+ 'os': _OPERATING_SYSTEM,
+ }
+
+ actual = RewriteMetadata(in_data, MetadataOptions(
+ name=name, parent=parent, ports=[port]))
+ self.assertEquals(expected, actual)
+
+ def testNewVolume(self):
+ in_data = {
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor'
+ }
+ }
+ name = 'deadbeef'
+ parent = 'blah'
+ volume = '/logs'
+ expected = {
+ 'id': name,
+ 'parent': parent,
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'Volumes': {
+ volume: {}
+ }
+ },
+ 'docker_version': _DOCKER_VERSION,
+ 'architecture': _PROCESSOR_ARCHITECTURE,
+ 'os': _OPERATING_SYSTEM,
+ }
+
+ actual = RewriteMetadata(in_data, MetadataOptions(
+ name=name, parent=parent, volumes=[volume]))
+ self.assertEquals(expected, actual)
+
+ def testAugmentVolume(self):
+ in_data = {
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'Volumes': {
+ '/original': {}
+ }
+ }
+ }
+ name = 'deadbeef'
+ parent = 'blah'
+ volume = '/data'
+ expected = {
+ 'id': name,
+ 'parent': parent,
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'Volumes': {
+ '/original': {},
+ volume: {}
+ }
+ },
+ 'docker_version': _DOCKER_VERSION,
+ 'architecture': _PROCESSOR_ARCHITECTURE,
+ 'os': _OPERATING_SYSTEM,
+ }
+
+ actual = RewriteMetadata(in_data, MetadataOptions(
+ name=name, parent=parent, volumes=[volume]))
+ self.assertEquals(expected, actual)
+
+ def testMultipleVolumes(self):
+ in_data = {
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor'
+ }
+ }
+ name = 'deadbeef'
+ parent = 'blah'
+ volume1 = '/input'
+ volume2 = '/output'
+ expected = {
+ 'id': name,
+ 'parent': parent,
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'Volumes': {
+ volume1: {},
+ volume2: {}
+ }
+ },
+ 'docker_version': _DOCKER_VERSION,
+ 'architecture': _PROCESSOR_ARCHITECTURE,
+ 'os': _OPERATING_SYSTEM,
+ }
+
+ actual = RewriteMetadata(in_data, MetadataOptions(
+ name=name, parent=parent, volumes=[volume1, volume2]))
+ self.assertEquals(expected, actual)
+
+ def testEnv(self):
+ in_data = {
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor'
+ }
+ }
+ name = 'deadbeef'
+ parent = 'blah'
+ env = [
+ 'baz=blah',
+ 'foo=bar',
+ ]
+ expected = {
+ 'id': name,
+ 'parent': parent,
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'Env': env,
+ },
+ 'docker_version': _DOCKER_VERSION,
+ 'architecture': _PROCESSOR_ARCHITECTURE,
+ 'os': _OPERATING_SYSTEM,
+ }
+
+ actual = RewriteMetadata(in_data, MetadataOptions(
+ name=name, env=env, parent=parent))
+ self.assertEquals(expected, actual)
+
+ def testEnvResolveReplace(self):
+ in_data = {
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'Env': [
+ 'foo=bar',
+ 'baz=blah',
+ 'blah=still around',
+ ],
+ }
+ }
+ name = 'deadbeef'
+ parent = 'blah'
+ env = [
+ 'baz=replacement',
+ 'foo=$foo:asdf',
+ ]
+ expected = {
+ 'id': name,
+ 'parent': parent,
+ 'config': {
+ 'User': 'mattmoor',
+ 'WorkingDir': '/usr/home/mattmoor',
+ 'Env': [
+ 'baz=replacement',
+ 'blah=still around',
+ 'foo=bar:asdf',
+ ],
+ },
+ 'docker_version': _DOCKER_VERSION,
+ 'architecture': _PROCESSOR_ARCHITECTURE,
+ 'os': _OPERATING_SYSTEM,
+ }
+
+ actual = RewriteMetadata(in_data, MetadataOptions(
+ name=name, env=env, parent=parent))
+ self.assertEquals(expected, actual)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/tools/build_defs/docker/sha256.py b/tools/build_defs/docker/sha256.py
new file mode 100644
index 0000000..992062f
--- /dev/null
+++ b/tools/build_defs/docker/sha256.py
@@ -0,0 +1,27 @@
+# Copyright 2015 Google Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""A wrapper to have a portable SHA-256 tool."""
+
+# TODO(dmarting): instead of this tool we should make SHA-256 of artifacts
+# available in Skylark.
+import hashlib
+import sys
+
+if __name__ == "__main__":
+ if len(sys.argv) != 3:
+ print "Usage: %s input output" % sys.argv[0]
+ sys.exit(-1)
+ with open(sys.argv[2], "w") as outputfile:
+ with open(sys.argv[1], "rb") as inputfile:
+ outputfile.write(hashlib.sha256(inputfile.read()).hexdigest())
diff --git a/tools/build_defs/docker/testdata/BUILD b/tools/build_defs/docker/testdata/BUILD
new file mode 100644
index 0000000..4957110
--- /dev/null
+++ b/tools/build_defs/docker/testdata/BUILD
@@ -0,0 +1,132 @@
+package(
+ default_visibility = [
+ "//tools/build_rules/docker:__subpackages__",
+ ],
+)
+
+load("/tools/build_rules/docker/docker", "docker_build")
+
+docker_build(
+ name = "files_base",
+ files = ["foo"],
+)
+
+docker_build(
+ name = "files_with_files_base",
+ base = ":files_base",
+ files = ["bar"],
+)
+
+docker_build(
+ name = "tar_base",
+ tars = ["one.tar"],
+)
+
+docker_build(
+ name = "tar_with_tar_base",
+ base = ":tar_base",
+ tars = ["two.tar"],
+)
+
+docker_build(
+ name = "files_with_tar_base",
+ base = ":tar_base",
+ files = ["bar"],
+)
+
+docker_build(
+ name = "tar_with_files_base",
+ base = ":files_base",
+ tars = ["two.tar"],
+)
+
+# TODO(mattmoor): Test scalar entrypoint
+docker_build(
+ name = "base_with_entrypoint",
+ entrypoint = ["/bar"],
+ files = ["bar"],
+ ports = ["8080"],
+ tars = ["two.tar"],
+)
+
+# TODO(mattmoor): Test scalar cmd
+docker_build(
+ name = "derivative_with_shadowed_cmd",
+ base = ":base_with_entrypoint",
+ cmd = ["shadowed-arg"],
+ files = ["foo"],
+)
+
+docker_build(
+ name = "derivative_with_cmd",
+ base = ":derivative_with_shadowed_cmd",
+ cmd = [
+ "arg1",
+ "arg2",
+ ],
+ ports = ["80/tcp"],
+ tars = ["one.tar"],
+)
+
+docker_build(
+ name = "base_with_volume",
+ files = [
+ "bar",
+ "foo",
+ ],
+ volumes = ["/logs"],
+)
+
+docker_build(
+ name = "derivative_with_volume",
+ base = ":base_with_volume",
+ volumes = [
+ "/asdf",
+ "/blah",
+ ],
+)
+
+py_binary(
+ name = "extras_gen",
+ srcs = ["extras_gen.py"],
+ deps = ["//tools/build_rules/docker:archive"],
+)
+
+genrule(
+ name = "extras",
+ outs = ["extras.tar"],
+ cmd = "$(location :extras_gen) $@",
+ tools = [":extras_gen"],
+)
+
+docker_build(
+ name = "generated_tarball",
+ tars = [
+ ":extras",
+ ],
+)
+
+docker_build(
+ name = "with_env",
+ base = ":base_with_volume",
+ env = {
+ "foo": "/asdf",
+ "bar": "blah blah blah",
+ },
+)
+
+docker_build(
+ name = "with_double_env",
+ base = ":with_env",
+ env = {
+ "baz": "${foo} $bar",
+ },
+)
+
+docker_build(
+ name = "link_with_files_base",
+ base = ":files_base",
+ symlinks = {
+ "/usr/bin/java": "/bar",
+ },
+)
diff --git a/tools/build_defs/docker/testdata/bar b/tools/build_defs/docker/testdata/bar
new file mode 100644
index 0000000..d6d9d34
--- /dev/null
+++ b/tools/build_defs/docker/testdata/bar
@@ -0,0 +1 @@
+blah
\ No newline at end of file
diff --git a/tools/build_defs/docker/testdata/extras_gen.py b/tools/build_defs/docker/testdata/extras_gen.py
new file mode 100644
index 0000000..fc9719f
--- /dev/null
+++ b/tools/build_defs/docker/testdata/extras_gen.py
@@ -0,0 +1,39 @@
+# Copyright 2015 Google Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""A simple cross-platform helper to create a timestamped tar file."""
+import datetime
+import sys
+import tarfile
+
+from tools.build_defs.docker import archive
+
+if __name__ == '__main__':
+ mtime = int(datetime.datetime.now().strftime('%s'))
+ with archive.TarFileWriter(sys.argv[1]) as f:
+ f.add_file('./', tarfile.DIRTYPE,
+ uname='root', gname='root', mtime=mtime)
+ f.add_file('./usr/', tarfile.DIRTYPE,
+ uname='root', gname='root', mtime=mtime)
+ f.add_file('./usr/bin/', tarfile.DIRTYPE,
+ uname='root', gname='root', mtime=mtime)
+ f.add_file('./usr/bin/java', tarfile.SYMTYPE,
+ link='/path/to/bin/java',
+ uname='root', gname='root', mtime=mtime)
+ f.add_file('./etc/', tarfile.DIRTYPE,
+ uname='root', gname='root', mtime=mtime)
+ f.add_file('./etc/nsswitch.conf',
+ content='hosts: files dns\n',
+ uname='root', gname='root', mtime=mtime)
+ f.add_file('./tmp/', tarfile.DIRTYPE,
+ uname='root', gname='root', mtime=mtime)
diff --git a/tools/build_defs/docker/testdata/foo b/tools/build_defs/docker/testdata/foo
new file mode 100644
index 0000000..5e40c08
--- /dev/null
+++ b/tools/build_defs/docker/testdata/foo
@@ -0,0 +1 @@
+asdf
\ No newline at end of file
diff --git a/tools/build_defs/docker/testdata/one.tar b/tools/build_defs/docker/testdata/one.tar
new file mode 100644
index 0000000..ba13e09
--- /dev/null
+++ b/tools/build_defs/docker/testdata/one.tar
Binary files differ
diff --git a/tools/build_defs/docker/testdata/two.tar b/tools/build_defs/docker/testdata/two.tar
new file mode 100644
index 0000000..1842824
--- /dev/null
+++ b/tools/build_defs/docker/testdata/two.tar
Binary files differ
diff --git a/tools/build_defs/docker/testenv.sh b/tools/build_defs/docker/testenv.sh
new file mode 100755
index 0000000..6274d4d
--- /dev/null
+++ b/tools/build_defs/docker/testenv.sh
@@ -0,0 +1,25 @@
+#!/bin/bash
+
+# Copyright 2015 Google Inc. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Integration test for docker, test environment.
+
+[ -z "$TEST_SRCDIR" ] && { echo "TEST_SRCDIR not set!" >&2; exit 1; }
+
+# Load the unit-testing framework
+source "${TEST_SRCDIR}/src/test/shell/unittest.bash" || \
+ { echo "Failed to source unittest.bash" >&2; exit 1; }
+
+readonly TEST_DATA_DIR="${TEST_SRCDIR}/tools/build_defs/docker/testdata"
