From e782a6f5ea06b534437c5777774088fe3fff2794 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rn-Michael=20Miehe?= Date: Sat, 21 Sep 2019 00:54:08 +0200 Subject: [PATCH 1/4] Major README refactoring --- README.md | 88 +++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 60 insertions(+), 28 deletions(-) diff --git a/README.md b/README.md index ed9e763..71ae483 100644 --- a/README.md +++ b/README.md @@ -1,51 +1,87 @@ # autodoc -[`autodoc`](https://github.com/ldericher/autodoc) is a simple [CI](https://en.wikipedia.org/wiki/Continuous_integration) script, primarily aimed at document creation. +[`autodoc`](https://github.com/ldericher/autodoc) is a simple [CI](https://en.wikipedia.org/wiki/Continuous_integration) system optimized for document creation. -## Basics +## Quick Start Guide -`autodoc` relies upon [inotify-tools](https://github.com/rvoicilas/inotify-tools) to recursively watch a Linux file system directory. +1. Install [Docker CE](https://docs.docker.com/install/) -For each file change, `autodoc` searches corresponding build instruction files (Makefiles etc.) and kicks off build processes accordingly. +1. Clone or download the `autodoc` repository, open a terminal inside the [example_docs](https://github.com/ldericher/autodoc/tree/master/example_docs) directory -## Usage +1. Deploy an `autodoc` container: + ```bash + docker run --rm -it \ + --name autodoc \ + --volume "${PWD}":/docs \ + --user "$(id -u):$(id -g)" \ + ldericher/autodoc + ``` -`autodoc` is designed to run in a server-side, containerized context. + The contents of the directory are now being watched by `autodoc`! -### Deploy a container + When deploying an `autodoc` container, mount your document root to `/docs`. You *should* also set the container's UID and GID. -`autodoc` can be pulled from the docker hub using `docker pull ldericher/autodoc`. +1. Edit some stuff, save, then watch the magic (and the terminal output) -When deploying an `autodoc` container, mount your document root to `/docs`. You *should* also set the container's UID and GID. +## Where not to use `autodoc` -#### Included software +`autodoc` is **not** a solution for Continuous Integration of large scale systems software! `autodoc` excels at building a large number of independent, small files. -TODO `ldericher/autodoc` contains `pandoc`. +## Prime use case: Nextcloud -#### tl;dr +Nextcloud is a "safe home for all your data" that can [easily be deployed using docker-compose](https://hub.docker.com/_/nextcloud). +Add an `autodoc` container to create directories where PDFs are automatically held up to date for all your documents. This extends upon the "[Base version - apache](https://hub.docker.com/_/nextcloud#base-version---apache)" of the Nextcloud compose deployment. -Deploy an `autodoc` instance in your current working dir: +```yaml +version: '2' - docker run --name autodoc -d -v "${PWD}":/docs --user "$(id -u):$(id -g)" ldericher/autodoc +volumes: + documents: -### Automating builds +services: + app: + volumes: + - documents:/opt/autodoc -Example automated builds can be found [here](https://github.com/ldericher/autodoc/tree/master/example_docs). + autodoc: + restart: always + image: ldericher/autodoc + user: "UID:GID" + volumes: + - documents:/docs +``` -In general, just put a build instruction file into any (sub-)directory watched by `autodoc` and add your source files. +The "user" key should be set to the same numeric IDs used for the nextcloud worker processes! To find the right IDs, issue `docker-compose exec app sh -c 'id -u www-data; id -g www-data'`. +For the apache containers, this should evaluate to "33:33". + +To begin, add the mounted `/opt/autodoc` as a local external storage to your Nextcloud instance. +You might need to setup the permissions on your new volume using `docker-compose exec app chown -R www-data:www-data /opt/autodoc`. + +## Basic functionality + +`autodoc` uses `inotifywait` from [inotify-tools](https://github.com/rvoicilas/inotify-tools) to recursively watch a Linux file system directory. + +For each file change, `autodoc` searches relevant build instruction files (Makefiles etc.) and kicks off build processes accordingly. + +## Concept: Source patterns + +To avoid unnecessary rebuilds and self-triggering, `autodoc` uses "source patterns" to filter for the relevant build instructions. +A source pattern is a `bash` regular expression matching any filename that should be regarded as a "source file" to the build instruction file. + +For instance, if a Makefile instructs how to build from Markdown source files, that Makefile's source pattern should likely be `\.md$`. + +## Creating an automated build + +In general, just put your source files into any (sub-)directory watched by `autodoc`. Add a build instruction file. On each file change, its containing directory is searched for a build instruction file. Watched parent directories are also probed for further build instructions. Every relevant instruction file will be executed as found. You may combine build instruction systems to your liking. -#### SRCPAT concept, "relevant" build instructions +## Build instruction systems -To avoid unnecessary rebuilds and self-triggering, `autodoc` uses "source patterns" to decide which build instructions are relevant. - -For instance, if a build instruction file describes building anything from Markdown files, its source pattern should be something like `\.md$` to match files with ".md" as last extension. Source patterns are `bash` regular expressions. - -#### GNU Make (Makefiles) +### GNU Make (Makefiles) `autodoc` supports standard Makefiles. `Makefile`s must contain a SRCPAT annotation comment as follows, where `` is the source pattern as above. @@ -56,7 +92,7 @@ For instance, if a build instruction file describes building anything from Markd If there are multiple SRCPAT annotations, the lowermost one will be used. -##### Advanced options +##### Advanced Make options You may add a PHONY target "autodoc" which will be built *instead* of the default target. @@ -65,7 +101,3 @@ You may add a PHONY target "autodoc" which will be built *instead* of the defaul autodoc: @echo "Hello World!" ``` - -## What not to use `autodoc` for - -`autodoc` excels at building a large number of independent small files. It is **not** a solution for Continuous Integration of large scale software systems! From ac0d9d08b530b863cda21fb447cf3b761cbd779e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rn-Michael=20Miehe?= Date: Sat, 21 Sep 2019 00:55:03 +0200 Subject: [PATCH 2/4] minor bugfixes, Makefile name parameter for do_make --- build/autodoc.sh | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/build/autodoc.sh b/build/autodoc.sh index a2323fc..8673f21 100755 --- a/build/autodoc.sh +++ b/build/autodoc.sh @@ -4,27 +4,28 @@ g_watchroot="$(readlink -f "${1:-.}")" # compile using bare make command -do_make() { # $1:DIR $2:OBJECT +do_make() { # $1:DIR $2:MAKEFILE $3:OBJECT # extract params local dir="$1" - local object="$2" + local makefile="$2" + local object="$3" # extract Makefile 'source pattern' - local srcpat="$(grep -E "^#@SRCPAT" "${dir}/Makefile" | tail -n 1 | sed -r "s/^#@SRCPAT\s+//")" + local srcpat="$(grep -E "^#@SRCPAT" "${dir}/${makefile}" | tail -n 1 | sed -r "s/^#@SRCPAT\s+//")" if [ -z "${srcpat}" ]; then - echo "Empty source pattern! Makefile needs '#@SRCPAT' annotation!" + echo "Empty source pattern, check '#@SRCPAT' annotation!" elif [[ "${object}" =~ ${srcpat} ]]; then # check for autodoc target - local target="$(grep -E "^autodoc:" "${dir}/Makefile" | sed -r "s/:.*$//")" - local target="${target:-all}" + local target="$(grep -E "^autodoc:" "${dir}/${makefile}" | sed -r "s/:.*$//")" + # local target="${target:-all}" # fallback target: all - echo "SRCPAT OK, building '${target}'." - make --no-print-directory -C "${dir}" -j "${target}" + echo "Making '${target}'." + make --no-print-directory -C "${dir}" -j ${target} else - echo "SRCPAT mismatch '${srcpat}'." + echo "SRCPAT '${srcpat}' mismatch." fi } @@ -39,16 +40,15 @@ do_compile() { # $1:DIR $2:OBJECT $3:DONE if [ -r "${dir}/Makefile" ]; then # Makefile found - echo -n "Using '${dir}/Makefile'. " - do_make "${dir}" "${object}" + echo -n "Found '${dir}/Makefile'. " + do_make "${dir}" "Makefile" "${object}" local done="1" fi - # search parent dir for more build instructions + # never leave $g_watchroot if [ "${dir}" != "${g_watchroot}" ]; then - # never leave $g_watchroot - local dir="$(dirname "${dir}")" - do_compile "${dir}" "${object}" "${done}" + # search parent dir for more build instructions + do_compile "$(dirname "${dir}")" "${object}" "${done}" elif [ "${done}" == "0" ]; then # hit $g_watchroot From 2dbb26a31861eeab6bd946d4f59cd40ee38ea557 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rn-Michael=20Miehe?= Date: Tue, 24 Sep 2019 11:13:54 +0200 Subject: [PATCH 3/4] Another README refactor --- README.md | 52 ++++++++++++++++++++++++++++------------------------ 1 file changed, 28 insertions(+), 24 deletions(-) diff --git a/README.md b/README.md index 71ae483..888aa97 100644 --- a/README.md +++ b/README.md @@ -2,31 +2,43 @@ [`autodoc`](https://github.com/ldericher/autodoc) is a simple [CI](https://en.wikipedia.org/wiki/Continuous_integration) system optimized for document creation. -## Quick Start Guide +In general, any file-sharing solution -- preferably on top of `docker-compose` -- can be made into an automatic document distribution system by adding an `autodoc` instance. -1. Install [Docker CE](https://docs.docker.com/install/) +## Quick Start Guide using Docker -1. Clone or download the `autodoc` repository, open a terminal inside the [example_docs](https://github.com/ldericher/autodoc/tree/master/example_docs) directory +The `autodoc` image [available on Docker Hub](https://hub.docker.com/r/ldericher/autodoc) is based on [pandocker](https://hub.docker.com/r/ldericher/pandocker) providing Ubuntu's TeXlive `LaTeX` and `pandoc` in a simple box. -1. Deploy an `autodoc` container: - ```bash - docker run --rm -it \ - --name autodoc \ +01. Install [Docker CE](https://docs.docker.com/install/) + +01. Clone or download the `autodoc` repository, open a terminal inside the [example_docs](https://github.com/ldericher/autodoc/tree/master/example_docs) directory + +01. Deploy an `autodoc` container: + + ```bash + docker run --rm -it \ --volume "${PWD}":/docs \ --user "$(id -u):$(id -g)" \ ldericher/autodoc - ``` + ``` - The contents of the directory are now being watched by `autodoc`! + The contents of the directory are now being watched by `autodoc`! - When deploying an `autodoc` container, mount your document root to `/docs`. You *should* also set the container's UID and GID. + When deploying an `autodoc` container, just mount your document root to `/docs`. You *should* also set the container's UID and GID. These are seen above. -1. Edit some stuff, save, then watch the magic (and the terminal output) +01. Edit some stuff, save -- and watch the magic happen (and the terminal output). -## Where not to use `autodoc` + On each file change, `autodoc` searches relevant build instruction files (Makefiles etc.) and kicks off build processes accordingly. + +### How *not* to use `autodoc` `autodoc` is **not** a solution for Continuous Integration of large scale systems software! `autodoc` excels at building a large number of independent, small files. +### Deploying without Docker + +`autodoc` only hard-depends on `inotifywait` from [inotify-tools](https://github.com/rvoicilas/inotify-tools) to recursively watch Linux file system directories. + +You will usually want to install a `LaTeX` distribution and setup `pandoc`. + ## Prime use case: Nextcloud Nextcloud is a "safe home for all your data" that can [easily be deployed using docker-compose](https://hub.docker.com/_/nextcloud). @@ -54,15 +66,9 @@ services: The "user" key should be set to the same numeric IDs used for the nextcloud worker processes! To find the right IDs, issue `docker-compose exec app sh -c 'id -u www-data; id -g www-data'`. For the apache containers, this should evaluate to "33:33". -To begin, add the mounted `/opt/autodoc` as a local external storage to your Nextcloud instance. +To begin, add the mounted `/opt/autodoc` as a 'local type' external storage to your Nextcloud instance. You might need to setup the permissions on your new volume using `docker-compose exec app chown -R www-data:www-data /opt/autodoc`. -## Basic functionality - -`autodoc` uses `inotifywait` from [inotify-tools](https://github.com/rvoicilas/inotify-tools) to recursively watch a Linux file system directory. - -For each file change, `autodoc` searches relevant build instruction files (Makefiles etc.) and kicks off build processes accordingly. - ## Concept: Source patterns To avoid unnecessary rebuilds and self-triggering, `autodoc` uses "source patterns" to filter for the relevant build instructions. @@ -83,8 +89,8 @@ You may combine build instruction systems to your liking. ### GNU Make (Makefiles) -`autodoc` supports standard Makefiles. -`Makefile`s must contain a SRCPAT annotation comment as follows, where `` is the source pattern as above. +`autodoc` supports GNU Makefiles. +However, Makefiles must contain a SRCPAT annotation comment as follows, where `` is a source pattern as above. ```Makefile #@SRCPAT @@ -92,9 +98,7 @@ You may combine build instruction systems to your liking. If there are multiple SRCPAT annotations, the lowermost one will be used. -##### Advanced Make options - -You may add a PHONY target "autodoc" which will be built *instead* of the default target. +You *may* add a PHONY target "autodoc" which will be built *instead* of the default target. ```Makefile .PHONY: autodoc From bd09f25d0c7bff3a11d3c3088cb270bcd8bc3846 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=C3=B6rn-Michael=20Miehe?= Date: Tue, 24 Sep 2019 11:37:39 +0200 Subject: [PATCH 4/4] various QoL changes --- build/autodoc.sh | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/build/autodoc.sh b/build/autodoc.sh index 8673f21..312e036 100755 --- a/build/autodoc.sh +++ b/build/autodoc.sh @@ -10,23 +10,31 @@ do_make() { # $1:DIR $2:MAKEFILE $3:OBJECT local makefile="$2" local object="$3" - # extract Makefile 'source pattern' + # check Makefile 'source pattern' local srcpat="$(grep -E "^#@SRCPAT" "${dir}/${makefile}" | tail -n 1 | sed -r "s/^#@SRCPAT\s+//")" if [ -z "${srcpat}" ]; then - echo "Empty source pattern, check '#@SRCPAT' annotation!" + echo -n "Empty source pattern, check '#@SRCPAT' annotation! " + return 1 elif [[ "${object}" =~ ${srcpat} ]]; then # check for autodoc target local target="$(grep -E "^autodoc:" "${dir}/${makefile}" | sed -r "s/:.*$//")" - # local target="${target:-all}" # fallback target: all - echo "Making '${target}'." + if [ -z "${target}" ]; then + echo "Running 'make'. " + else + echo "Making '${target}'. " + fi + make --no-print-directory -C "${dir}" -j ${target} else - echo "SRCPAT '${srcpat}' mismatch." + echo -n "SRCPAT '${srcpat}' mismatch. " + return 1 fi + + return 0 } # compile a directory @@ -40,9 +48,9 @@ do_compile() { # $1:DIR $2:OBJECT $3:DONE if [ -r "${dir}/Makefile" ]; then # Makefile found - echo -n "Found '${dir}/Makefile'. " - do_make "${dir}" "Makefile" "${object}" - local done="1" + echo -n "Found '${dir}/Makefile': " + do_make "${dir}" "Makefile" "${object}" \ + && local done="1" fi # never leave $g_watchroot