Merge pull request #28 from ninech/enhanced_configuration

Make the default configuration cluster ready

.gitignore

@@ -0,0 +1 @@
+*.sql.gz

README.md

@@ -1,13 +1,16 @@
 # netbox-docker
 
-[![Build Status](https://travis-ci.org/ninech/netbox-docker.svg?branch=master)](https://travis-ci.org/ninech/netbox-docker)
+[![Build Status](https://travis-ci.org/ninech/netbox-docker.svg?branch=master)][travis]
 
-This repository houses the components needed to build NetBox as a Docker container.
+This repository houses the components needed to build Netbox as a Docker container.
-Images built using this code are released to [Docker Hub](https://hub.docker.com/r/ninech/netbox) every night.
+Images built using this code are released to [Docker Hub][netbox-dockerhub] every night.
+
+[travis]: https://travis-ci.org/ninech/netbox-docker
+[netbox-dockerhub]: https://hub.docker.com/r/ninech/netbox/tags/
 
 ## Quickstart
 
-To get NetBox up and running:
+To get Netbox up and running:
 
 ```
 $ git clone -b master https://github.com/ninech/netbox-docker.git
@@ -30,7 +33,7 @@ $ open "http://$(docker-compose port nginx 80)/"
 $ xdg-open "http://$(docker-compose port nginx 80)/" &>/dev/null &
 ```
 
-Alternatively, use something like [Reception](https://github.com/ninech/reception) to
+Alternatively, use something like [Reception][docker-reception] to
 connect to _docker-compose_ projects.
 
 Default credentials:
@@ -39,6 +42,8 @@ Default credentials:
 * Password: **admin**
 * API Token: **0123456789abcdef0123456789abcdef01234567**
 
+[docker-reception]: https://github.com/ninech/reception
+
 ## Dependencies
 
 This project relies only on *Docker* and *docker-compose* meeting this requirements:
@@ -51,13 +56,74 @@ To ensure this, compare the output of `docker --version` and `docker-compose --v
 ## Configuration
 
 You can configure the app using environment variables. These are defined in `netbox.env`.
+Read [Environment Variables in Compose][compose-env] to understand about the various possibilities to overwrite these variables.
+(The easiest solution being simply adjusting that file.)
+
+To find all possible variables, have a look at the [configuration.docker.py][docker-config] and [docker-entrypoint.sh][entrypoint] files.
+Generally, the environment variables are called the same as their respective Netbox configuration variables.
+Variables which are arrays are usually composed by putting all the values into the same environment variables with the values separated by a whitespace ("` `").
+For example defining `ALLOWED_HOSTS=localhost ::1 127.0.0.1` would allows access to Netbox through `http://localhost`, `http://[::1]` and `http://127.0.0.1`.
+
+[compose-env]: https://docs.docker.com/compose/environment-variables/
+
+### Production
+
+The default settings are optimized for (local) development environments.
+You should therefore adjust the configuration for production setups, at least the following variables:
+
+* `ALLOWED_HOSTS`: Add all URLs that lead to your netbox instance.
+* `DB_*`: Use a persistent database.
+* `EMAIL_*`: Use your own mailserver.
+* `MAX_PAGE_SIZE`: Use the recommended default of 1000.
+* `SUPERUSER_*`: Only define those variables during the initial setup, and drop them once the DB is set up.
+
+### Running on Docker Swarm / Kubernetes / OpenShift
+
+You may run this image in a cluster such as Docker Swarm, Kubernetes or OpenShift, but this is advanced level.
+
+In this case, we encourage you to statically configure Netbox by starting from [Netbox's example config file][default-config], and mounting it into your container using the mechanism provided by your container platform (i.e. [Docker Swarm configs][swarm-config], [Kubernetes secrets][k8s-secrets], [OpenShift configmaps][openshift-config]).
+
+But if you rather continue to configure your application through environment variables, you may continue to use [the built-in configuration file][docker-config].
+We discourage storing secrets in environment variables, as environment variable are passed on to all sub-processes and may leak easily into other systems, e.g. error collecting tools that often collect all environment variables whenever an error occurs.
+
+Therefore we *strongly advise* to make use of the secrets mechanism provided by your container platform (i.e. [Docker Swarm secrets][swarm-secrets], [Kubernetes secrets][k8s-secrets], [OpenShift secrets][openshift-secrets]).
+[The configuration file][docker-config] and [the entrypoint script][entrypoint] try to load the following secrets from the respective files.
+If a secret is defined by an environment variable and in the respective file at the same time, then the value from the environment variable is used.
+
+* `SUPERUSER_PASSWORD`: `/run/secrets/superuser_password`
+* `SUPERUSER_API_TOKEN`: `/run/secrets/superuser_api_token`
+* `DB_PASSWORD`: `/run/secrets/db_password`
+* `SECRET_KEY`: `/run/secrets/secret_key`
+* `EMAIL_PASSWORD`: `/run/secrets/email_password`
+* `NAPALM_PASSWORD`: `/run/secrets/napalm_password`
+
+Please also consider [the advice about running Netbox in production](#production) above!
+
+[docker-config]: https://github.com/ninech/netbox-docker/blob/master/docker/configuration.docker.py
+[default-config]: https://github.com/digitalocean/netbox/blob/develop/netbox/netbox/configuration.example.py
+[entrypoint]: https://github.com/ninech/netbox-docker/blob/master/docker/docker-entrypoint.sh
+[swarm-config]: https://docs.docker.com/engine/swarm/configs/
+[swarm-secrets]: https://docs.docker.com/engine/swarm/secrets/
+[openshift-config]: https://docs.openshift.org/latest/dev_guide/configmaps.html
+[openshift-secrets]: https://docs.openshift.org/latest/dev_guide/secrets.html
+[k8s-secrets]: https://kubernetes.io/docs/concepts/configuration/secret/
+
+#### A Note On OpenShift
+
+OpenShift usually is configured with specific restriction regarding root users.
+[Special care][openshift-root] has to be taken when building images for OpenShift.
+The Docker Image that may be built using this project (and which is available on Docker Hub) might not yet run without further customization on OpenShift.
+If you have this running on OpenShift, it would be nice if you could open a PR with the changes you needed to make.
+Or if you didn't do any changes and it just worked, that you could confirm this so that we can remove this notice.
+
+[openshift-root]: https://docs.openshift.org/latest/creating_images/guidelines.html#openshift-specific-guidelines
 
 ## Version
 
 The `docker-compose.yml` file is prepared to run a specific version of Netbox.
 To use this feature, set the environment-variable `VERSION` before launching `docker-compose`, as shown below.
 `VERSION` may be set to the name of
-[any tag of the `ninech/netbox` Docker image](https://hub.docker.com/r/ninech/netbox/tags/).
+[any tag of the `ninech/netbox` Docker image on Docker Hub][netbox-dockerhub].
 
 ```
 $ export VERSION=v2.2.6
@@ -66,9 +132,8 @@ $ docker-compose up -d
 ```
 
 You can also build a specific version of the Netbox image. This time, `VERSION` indicates any valid
-[Git Reference](https://git-scm.com/book/en/v2/Git-Internals-Git-References) declared on
+[Git Reference][git-ref] declared on [the 'digitalocean/netbox' Github repository][netbox-github].
-[the Netbox Github repository](https://github.com/digitalocean/netbox/releases).
+Most commonly you will specify a tag or branch name.
-Most commonly you will specify a tag name or a branch name.
 
 ```
 $ export VERSION=develop
@@ -79,6 +144,47 @@ $ docker-compose up -d
 Hint: If you're building a specific version by tag name, the `--no-cache` argument is not strictly necessary.
 This can increase the build speed if you're just adjusting the config, for example.
 
+[git-ref]: https://git-scm.com/book/en/v2/Git-Internals-Git-References
+[netbox-github]: https://github.com/digitalocean/netbox/releases
+
+## Troubleshooting
+
+This section is a collection of some common issues and how to resolve them.
+If your issue is not here, look through [the existing issues][issues] and eventually create a new issue.
+
+[issues]: (https://github.com/ninech/netbox-docker/issues)
+
+### Docker Compose basics
+
+* You can see all running containers belonging to this project using `docker-compose ps`.
+* You can see the logs by running `docker-compose logs -f`.
+  Running `docker-compose logs -f netbox` will just show the logs for netbox.
+* You can stop everything using `docker-compose stop`.
+* You can clean up everything using `docker-compose down -v --remove-orphans`. **This will also remove any related data.**
+* You can enter the shell of the running Netbox container using `docker-compose exec netbox /bin/bash`. Now you have access to `./manage.py`, e.g. to reset a password.
+* To access the database run `docker-compose exec postgres sh -c 'psql -U $POSTGRES_USER $POSTGRES_DB'`
+* To create a database backup run `docker-compose exec postgres sh -c 'pg_dump -U $POSTGRES_USER $POSTGRES_DB' | gzip > db_dump.sql.gz`
+* To restore that database backup run `gunzip -c db_dump.sql.gz | docker exec -i $(docker-compose ps -q postgres) sh -c 'psql -U $POSTGRES_USER $POSTGRES_DB'`.
+
+### Getting a "Bad Request (400)"
+
+> When connecting to the Netbox instance, I get a "Bad Request (400)" error.
+
+This usually happens when the `ALLOWED_HOSTS` variable is not set correctly.
+
+### How to upgrade
+
+> How do I update to a newer version?
+
+It should be sufficient to pull the latest image from Docker Hub, stopping the container and starting it up again:
+
+```bash
+docker-compose pull netbox
+docker-compose stop netbox
+docker-compose rm -f netbox
+docker-compose up -d netbox
+```
+
 ## Rebuilding & Publishing images
 
 `./build.sh` is used to rebuild the Docker image:
@@ -104,7 +210,7 @@ You can use the following ENV variables to customize the build:
 
 ## Tests
 
-To run the bundled test, use the `docker-compose.test.yml` file.
+To run the test coming with Netbox, use the `docker-compose.test.yml` file as such:
 
 ```
 $ docker-compose -f docker-compose.test.yml run --rm app

docker-compose.test.yml

@@ -15,7 +15,6 @@ services:
     postgres:
       image: postgres:9.6-alpine
       env_file: postgres.env
-
 volumes:
   netbox-static-files:
     driver: local

docker-compose.yml

@@ -12,13 +12,15 @@ services:
         volumes:
         - netbox-nginx-config:/etc/netbox-nginx/
         - netbox-static-files:/opt/netbox/netbox/static
+        - netbox-media-files:/opt/netbox/netbox/media
+        - netbox-report-files:/opt/netbox/netbox/reports
     nginx:
         image: nginx:1.11-alpine
         command: nginx -g 'daemon off;' -c /etc/netbox-nginx/nginx.conf
         depends_on:
         - netbox
         ports:
-        - 80
+        - 8080
         volumes:
         - netbox-static-files:/opt/netbox/netbox/static
         - netbox-nginx-config:/etc/netbox-nginx/
@@ -31,3 +33,7 @@ volumes:
         driver: local
     netbox-nginx-config:
         driver: local
+    netbox-media-files:
+        driver: local
+    netbox-report-files:
+        driver: local

docker/configuration.docker.py

@@ -1,4 +1,21 @@
 import os
+import socket
+
+# For reference see http://netbox.readthedocs.io/en/latest/configuration/mandatory-settings/
+# Based on https://github.com/digitalocean/netbox/blob/develop/netbox/netbox/configuration.example.py
+
+# Read secret from file
+def read_secret(secret_name):
+    try:
+        f = open('/run/secrets/' + secret_name, 'r', encoding='utf-8')
+    except EnvironmentError:
+        return ''
+    else:
+        with f:
+            return f.readline().strip()
+
+BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
 #########################
 #                       #
 #   Required settings   #
@@ -9,13 +26,14 @@ import os
 # access to the server via any other hostnames. The first FQDN in the list will be treated as the preferred name.
 #
 # Example: ALLOWED_HOSTS = ['netbox.example.com', 'netbox.internal.local']
-ALLOWED_HOSTS = os.environ.get('ALLOWED_HOSTS', '').split(' ')
+ALLOWED_HOSTS = os.environ.get('ALLOWED_HOSTS', socket.gethostname()).split(' ')
 
 # PostgreSQL database configuration.
 DATABASE = {
     'NAME': os.environ.get('DB_NAME', 'netbox'),         # Database name
     'USER': os.environ.get('DB_USER', ''),               # PostgreSQL username
-    'PASSWORD': os.environ.get('DB_PASSWORD', ''),       # PostgreSQL password
+    'PASSWORD': os.environ.get('DB_PASSWORD', read_secret('db_password')),
+                                                         # PostgreSQL password
     'HOST': os.environ.get('DB_HOST', 'localhost'),      # Database server
     'PORT': os.environ.get('DB_PORT', ''),               # Database port (leave blank for default)
 }
@@ -24,7 +42,7 @@ DATABASE = {
 # For optimal security, SECRET_KEY should be at least 50 characters in length and contain a mix of letters, numbers, and
 # symbols. NetBox will not run without this defined. For more information, see
 # https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-SECRET_KEY
-SECRET_KEY = os.environ.get('SECRET_KEY', '')
+SECRET_KEY = os.environ.get('SECRET_KEY', read_secret('secret_key'))
 
 #########################
 #                       #
@@ -38,16 +56,51 @@ ADMINS = [
     # ['John Doe', 'jdoe@example.com'],
 ]
 
+# Optionally display a persistent banner at the top and/or bottom of every page. HTML is allowed. To display the same
+# content in both banners, define BANNER_TOP and set BANNER_BOTTOM = BANNER_TOP.
+BANNER_TOP = os.environ.get('BANNER_TOP', '')
+BANNER_BOTTOM = os.environ.get('BANNER_BOTTOM', '')
+
+# Text to include on the login page above the login form. HTML is allowed.
+BANNER_LOGIN = os.environ.get('BANNER_LOGIN', '')
+
+# Base URL path if accessing NetBox within a directory. For example, if installed at http://example.com/netbox/, set:
+# BASE_PATH = 'netbox/'
+BASE_PATH = os.environ.get('BASE_PATH', '')
+
+# API Cross-Origin Resource Sharing (CORS) settings. If CORS_ORIGIN_ALLOW_ALL is set to True, all origins will be
+# allowed. Otherwise, define a list of allowed origins using either CORS_ORIGIN_WHITELIST or
+# CORS_ORIGIN_REGEX_WHITELIST. For more information, see https://github.com/ottoyiu/django-cors-headers
+CORS_ORIGIN_ALLOW_ALL = os.environ.get('CORS_ORIGIN_ALLOW_ALL', False)
+CORS_ORIGIN_WHITELIST = os.environ.get('CORS_ORIGIN_WHITELIST', '').split(' ')
+CORS_ORIGIN_REGEX_WHITELIST = [
+    # r'^(https?://)?(\w+\.)?example\.com$',
+]
+
+# Set to True to enable server debugging. WARNING: Debugging introduces a substantial performance penalty and may reveal
+# sensitive information about your installation. Only enable debugging while performing testing. Never enable debugging
+# on a production system.
+DEBUG = os.environ.get('DEBUG', False)
+
 # Email settings
 EMAIL = {
     'SERVER': os.environ.get('EMAIL_SERVER', 'localhost'),
     'PORT': os.environ.get('EMAIL_PORT', 25),
     'USERNAME': os.environ.get('EMAIL_USERNAME', ''),
-    'PASSWORD': os.environ.get('EMAIL_PASSWORD', ''),
+    'PASSWORD': os.environ.get('EMAIL_PASSWORD', read_secret('email_password')),
     'TIMEOUT': os.environ.get('EMAIL_TIMEOUT', 10),  # seconds
     'FROM_EMAIL': os.environ.get('EMAIL_FROM', ''),
 }
 
+# Enforcement of unique IP space can be toggled on a per-VRF basis.
+# To enforce unique IP space within the global table (all prefixes and IP addresses not assigned to a VRF),
+# set ENFORCE_GLOBAL_UNIQUE to True.
+ENFORCE_GLOBAL_UNIQUE = os.environ.get('ENFORCE_GLOBAL_UNIQUE', False)
+
+# Enable custom logging. Please see the Django documentation for detailed guidance on configuring custom logs:
+#   https://docs.djangoproject.com/en/1.11/topics/logging/
+LOGGING = {}
+
 # Setting this to True will permit only authenticated users to access any part of NetBox. By default, anonymous users
 # are permitted to access most data in NetBox (excluding secrets) but not make any changes.
 LOGIN_REQUIRED = os.environ.get('LOGIN_REQUIRED', False)
@@ -59,13 +112,37 @@ BASE_PATH = os.environ.get('BASE_PATH', '')
 # Setting this to True will display a "maintenance mode" banner at the top of every page.
 MAINTENANCE_MODE = os.environ.get('MAINTENANCE_MODE', False)
 
+# An API consumer can request an arbitrary number of objects =by appending the "limit" parameter to the URL (e.g.
+# "?limit=1000"). This setting defines the maximum limit. Setting it to 0 or None will allow an API consumer to request
+# all objects by specifying "?limit=0".
+MAX_PAGE_SIZE = int(os.environ.get('MAX_PAGE_SIZE', 1000))
+
+# The file path where uploaded media such as image attachments are stored. A trailing slash is not needed. Note that
+# the default value of this setting is derived from the installed location.
+MEDIA_ROOT = os.environ.get('MEDIA_ROOT', os.path.join(BASE_DIR, 'media'))
+
 # Credentials that NetBox will use to access live devices.
 NAPALM_USERNAME = os.environ.get('NAPALM_USERNAME', '')
-NAPALM_PASSWORD = os.environ.get('NAPALM_PASSWORD', '')
+NAPALM_PASSWORD = os.environ.get('NAPALM_PASSWORD', read_secret('napalm_password'))
+
+# NAPALM timeout (in seconds). (Default: 30)
+NAPALM_TIMEOUT = os.environ.get('NAPALM_TIMEOUT', 30)
+
+# NAPALM optional arguments (see http://napalm.readthedocs.io/en/latest/support/#optional-arguments). Arguments must
+# be provided as a dictionary.
+NAPALM_ARGS = {}
 
 # Determine how many objects to display per page within a list. (Default: 50)
 PAGINATE_COUNT = os.environ.get('PAGINATE_COUNT', 50)
 
+# When determining the primary IP address for a device, IPv6 is preferred over IPv4 by default. Set this to True to
+# prefer IPv4 instead.
+PREFER_IPV4 = os.environ.get('PREFER_IPV4', False)
+
+# The file path where custom reports will be stored. A trailing slash is not needed. Note that the default value of
+# this setting is derived from the installed location.
+REPORTS_ROOT = os.environ.get('REPORTS_ROOT', os.path.join(BASE_DIR, 'reports'))
+
 # Time zone (default: UTC)
 TIME_ZONE = os.environ.get('TIME_ZONE', 'UTC')
 
@@ -77,9 +154,3 @@ TIME_FORMAT = os.environ.get('TIME_FORMAT', 'g:i a')
 SHORT_TIME_FORMAT = os.environ.get('SHORT_TIME_FORMAT', 'H:i:s')
 DATETIME_FORMAT = os.environ.get('DATETIME_FORMAT', 'N j, Y g:i a')
 SHORT_DATETIME_FORMAT = os.environ.get('SHORT_DATETIME_FORMAT', 'Y-m-d H:i')
-
-# Enforcement of unique IP space can be toggled on a per-VRF basis.
-# To enforce unique IP space within the global table (all prefixes and IP addresses not assigned to a VRF),
-# set ENFORCE_GLOBAL_UNIQUE to True.
-ENFORCE_GLOBAL_UNIQUE = os.environ.get('ENFORCE_GLOBAL_UNIQUE')
-

docker/docker-entrypoint.sh

@@ -8,18 +8,26 @@ while ! ./manage.py migrate 2>&1; do
 done
 
 # create superuser silently
-if [[ -z ${SUPERUSER_NAME} ]]; then
+if [ -z ${SUPERUSER_NAME+x} ]; then
   SUPERUSER_NAME='admin'
 fi
-if [[ -z ${SUPERUSER_EMAIL} ]]; then
+if [ -z ${SUPERUSER_EMAIL+x} ]; then
   SUPERUSER_EMAIL='admin@example.com'
 fi
-if [[ -z ${SUPERUSER_PASSWORD} ]]; then
+if [ -z ${SUPERUSER_PASSWORD+x} ]; then
+  if [ -f "/run/secrets/superuser_password" ]; then
+    SUPERUSER_PASSWORD="$(< /run/secrets/superuser_password)"
+  else
     SUPERUSER_PASSWORD='admin'
   fi
-if [[ -z ${SUPERUSER_API_TOKEN} ]]; then
+fi
+if [ -z ${SUPERUSER_API_TOKEN+x} ]; then
+  if [ -f "/run/secrets/superuser_api_token" ]; then
+    SUPERUSER_API_TOKEN="$(< /run/secrets/superuser_api_token)"
+  else
     SUPERUSER_API_TOKEN='0123456789abcdef0123456789abcdef01234567'
   fi
+fi
 
 echo "💡 Username: ${SUPERUSER_NAME}, E-Mail: ${SUPERUSER_EMAIL}, Password: ${SUPERUSER_PASSWORD}, Token: ${SUPERUSER_API_TOKEN}"
 

docker/nginx.conf

@@ -15,10 +15,8 @@ http {
     client_max_body_size 10M;
 
     server {
-        listen 80;
+        listen      8080;
-
         server_name localhost;
-
         access_log  off;
 
         location /static/ {

netbox.env

@@ -1,18 +1,18 @@
-SUPERUSER_NAME=admin
-SUPERUSER_EMAIL=admin@example.com
-SUPERUSER_PASSWORD=admin
-SUPERUSER_API_TOKEN=0123456789abcdef0123456789abcdef01234567
 ALLOWED_HOSTS=localhost 0.0.0.0 127.0.0.1 [::1] netbox nginx netboxdocker.docker nginx.netboxdocker.docker
 DB_NAME=netbox
 DB_USER=netbox
 DB_PASSWORD=J5brHrAXFLQSif0K
 DB_HOST=postgres
-SECRET_KEY=r8OwDznj!!dci#P9ghmRfdu1Ysxm0AiPeDCQhKE+N_rClfWNj
 EMAIL_SERVER=localhost
 EMAIL_PORT=25
-EMAIL_USERNAME=foo
+EMAIL_USERNAME=netbox
-EMAIL_PASSWORD=bar
+EMAIL_PASSWORD=
-EMAIL_TIMEOUT=10
+EMAIL_TIMEOUT=5
 EMAIL_FROM=netbox@bar.com
-NETBOX_USERNAME=guest
+NAPALM_TIMEOUT=5
-NETBOX_PASSWORD=guest
+MAX_PAGE_SIZE=0
+SECRET_KEY=r8OwDznj!!dci#P9ghmRfdu1Ysxm0AiPeDCQhKE+N_rClfWNj
+SUPERUSER_NAME=admin
+SUPERUSER_EMAIL=admin@example.com
+SUPERUSER_PASSWORD=admin
+SUPERUSER_API_TOKEN=0123456789abcdef0123456789abcdef01234567