Infrastructure as Code
Table of Contents
- 1. Infrastructure as Code
- 1.1. NixOS over ansible
- 1.2. Why you should never ever use NixOS
- 1.3. Why you should never ever use NixOS: a rebuttal
- 1.4. https://web.archive.org/web/20210707233008/sandervanderburg.blogspot.com/2011/11/on-nix-nixos-and-filesystem-hierarchy.html
- 1.5. Programabilidad en distintos entornos (config as code)
- 1.6. Pulumi
- 1.7. Domen Kožar Why Puppet/Chef/Ansible aren’t good enough and we can do better
- 1.8. The Dhall configuration language
- 1.9. Infrastructure as SQL on AWS: IaSQL is Now Open Source and SaaS
1. Infrastructure as Code
1.1. NixOS over ansible
1.2. Why you should never ever use NixOS
1.2.1. Table Of Contents
I’ll start my article from the biggest myth about NixOS. And soon you’ll know why.
DevOps-friendly. Declarative specs and safe upgrades make NixOS a great system for DevOps use. NixOps, the NixOS cloud deployment tool, allows you to provision and manage networks of NixOS machines in environments like Amazon EC2 and VirtualBox. (c) NixOS.org
I will continuously update this article each time I’ll face a new weird and challenging task that makes your life much harder if you’re using NixOS instead of traditional Linux distributions.
Well, I’m a lucky guy… I have no choice right now and have to spend most of my working time on NixOS automation. To this moment of time I have more than 10 years experience of using different Linux distributions and more than 6 months of NixOS experience. And most of my experience with NixOS is bad experience.
1.2.1.1. Why you may want to use NixOS.
I’ll try to write something here. Hope, I’ll be able to…
- Determinism.
It is declared and provided to you by the system design, that you’ll get the same version of any software package from the derivation. Lie!
All it’s determinism is about that it deterministically gives you a lot of issues, which you need to overcome manually without any support.
You may install Nix tools on Linux and OS X, for example. But it is not always building the same derivation results on Linux and NixOS itself. So, it is not OS agnostic. There’re a lot of examples of such behavior, for example,fetchgit
andfetchFromGitHub
functions:
- fetchgit/fetchFromGitHub fail to fetch submodules from “git@server:repo” URLs.
- Use
fetchFromGitHub
instead offetchgit
when the source is a GitHub clone URL. - fetchgit -> fetchFromGitHub where possible.
- How to manually replicate/reproduce/obtain the sha256 hash specified in Nix with fetchgit or fetchFromGitHub?.
Meanwhile in the official manual to the current and latest version-18-09-1373-50fb6820759 they are still using
fetchgit
.
- fetchgit/fetchFromGitHub fail to fetch submodules from “git@server:repo” URLs.
1.2.1.2. Why you should never ever use NixOS.
Here I’m gonna collect all my daily pain in a single place, which I hope will help people not to make somebody’s else mistakes. This is my personal view on the problem space and it can not be affiliated with the company or project where I’m working right now.
- Small community.
Sure, there are a lot of enthusiasts who are playing with NixOS by installing it on their workstations of VMs, but in comparison with Ubuntu, CentOS, Debian or RedHat:
That means you will not be able:
- To get fast or enterprise support.
- To to hire people to manage it.
- To get fast or enterprise support.
- Documentation.
Official documentation is awful. Or it is just absent. NixOS is older than Docker. They are solving the same task, but NixOS is way less popular. Try to think why.
NIX CONFIGURATION LANGUAGE It is functional language which name is also Nix and it is a core of this system configuration. If you’re coming from OOP world it will be very difficult to understand that “language”. You must go to swallow Nix pills to even try to understand something about it.
SOFTWARE MANAGEMENT “Derivation” is a way of describing software distribution in NixOS. Each derivation represents software, plugin or a library. Common syntax of describing it looks like this (nginx derivation example):
{ stdenv , fetchurl , openssl , zlib , pcre , libxml2 , libxslt , gd , geoip , withDebug ? false , withStream ? true , withMail ? false , modules ? [] , version , sha256 , ... }: with stdenv .lib; stdenv .mkDerivation { name = "nginx- ${version } "; src = fetchurl { url = "https://nginx.org/download/nginx- ${version } .tar.gz"; inherit sha256; }; buildInputs = [ openssl zlib pcre libxml2 libxslt gd geoip ] ++ concatMap (mod: mod .inputs or []) modules; configureFlags = [ "--with-http_ssl_module" "--with-http_v2_module" "--with-http_realip_module" "--with-http_addition_module" "--with-http_xslt_module" "--with-http_geoip_module" "--with-http_sub_module" "--with-http_dav_module" "--with-http_flv_module" "--with-http_mp4_module" "--with-http_gunzip_module" "--with-http_gzip_static_module" "--with-http_auth_request_module" "--with-http_random_index_module" "--with-http_secure_link_module" "--with-http_degradation_module" "--with-http_stub_status_module" "--with-threads" "--with-pcre-jit" # Install destination problems # "--with-http_perl_module" ] ++ optional withDebug [ "--with-debug" ] ++ optional withStream [ "--with-stream" "--with-stream_geoip_module" "--with-stream_realip_module" "--with-stream_ssl_module" "--with-stream_ssl_preread_module" ] ++ optional withMail [ "--with-mail" "--with-mail_ssl_module" ] ++ optional (gd != null) "--with-http_image_filter_module" ++ optional ( with stdenv .hostPlatform; isLinux || isFreeBSD) "--with-file-aio" ++ map (mod: "--add-module= ${mod .src } ") modules; NIX_CFLAGS_COMPILE = [ "-I ${libxml2 .dev } /include/libxml2" ] ++ optional stdenv .isDarwin "-Wno-error=deprecated-declarations"; preConfigure = (concatMapStringsSep " \n " (mod: mod .preConfigure or "") modules); hardeningEnable = optional ( !stdenv .isDarwin) "pie"; enableParallelBuilding = true; postInstall = '' mv $out/sbin $out/bin ''; meta = { description = "A reverse proxy and lightweight webserver"; homepage = http://nginx.org; license = licenses .bsd2; platforms = platforms .all; maintainers = with maintainers; [ thoughtpolice raskin fpletz ]; }; }
Interesting, isn’t it?
If yes, get ready, you’ll not be able to live in the system without writing something like that. Nix Packages GitHub repository will be your second home, where you’ll be looking for examples of Nix expressions, configuration approaches and other examples.
- Configuration management.
System state and configuration described in a special file
configuration.nix
. It’s like an entry point to endless amount of functions and dependencies for your OS.
Want to customize something in the OS, be ready, you’ll need to write your customsystemd
service to do that. Aaand, yes, service configuration file also need to be declared using Nix expression language.
All standard system configuration files like/etc/fstab
, for example are generated by a weird code and read only.
Forget about Chef, Puppet or Ansible! Only Nix expressions! Only hardcore!
- Kernel upgrade.
It is not possible just to upgrade the kernel from “ver1” to “ver2”. New kernel will bring whole set of system packages and their dependencies with it. Do not know if it is safe. Will test it soon.
- Cloud support.
Nix is not ready for a cloud. At all. I’m always facing the following situations on NixOS 18.03:
- OS is not booting from the AMI.
- OS is not supporting newest AWS instance types (like m5.*, for example).
- NixOS 18.03 is not always able to resize your hard drive when it’s boot.
- Do not even try to use it in Auto Scaling group and passing configuration.nix content through user-data. You may easily get to the loop, when instance start building something new from the derivations during it’s boot process, 100% CPU utilizations will require AutoScaling Group to launch additional instances that also will build something, all the instances are unhealthy and Auto Scaling group will start terminating the first instance in a loop. Your service will never start.
- OS is not booting from the AMI.
- Cache.
To speedup software installation in NixOS the community provides you with Nix Binary Cache, where they put everything they’ve built successfully. Every version of every derivation.
Problems start happening when you spinning up your personal cache, which is used, for example to store your proprietary build artifacts. You always need to keep eye on the community channel version you’re using as a base for your builds. And yes, your own cache size will also grow very fast!
Some very talented developers start using Nix binary cache to cache everything’s possible. For example NPM or Yarn packages. Good luck!
- Security.
People are thinking, that NixOS deterministic approach of dealing with software is very secure. Maybe yes, it is really not possible to hijack the build results once the derivation bin build. But… You always need to pay for it…
During the upgrade from 18.03 to 18.09 Nix community decided to change Docker tools they’ve been use to calculate derivations checksums where you were dealing with Docker images.
As a result we rewrote all Docker based derivation declarations and more over to rebuild them all. It was really painful and unmanaged migration.
- Windows support.
Just forget about it.
- System requirements.
It requires a lot of CPU power to build it’s stuff and really a lot of space to store its multiple versions. Much more, than you need in traditional Linux distros.
- Once invested.
- Try to change a technology stack as soon as possible.
- Do not listen anybody.
- Do not reinvent the wheel.
- Just waste it.
And, God, do not try to run Nix in Docker! It’s really, not, REALLY bad idea!
- Try to change a technology stack as soon as possible.
- What to do next.
Run away! Not, seriously! Run. From. It. Away.
When you try to summarize all your thoughts around the topic you want to cover in a post, you’re starting from simple Google search. One of results to my “nixos pros and cons” query: NixOS alternatives. Kinda joke, but really, think twice when you’re making decision to invest your time or money in it. And I would also think twice about WHY colleagues are advising it to me.
- Docker is the best alternative.
The link is in previous paragraph. Your choice is traditional Linux and Docker! Yes, you’ll have the same deterministic and secure result and even more more with less pain and and much pleasure.
- Additional reading.
1.3. Why you should never ever use NixOS: a rebuttal
1.4. https://web.archive.org/web/20210707233008/sandervanderburg.blogspot.com/2011/11/on-nix-nixos-and-filesystem-hierarchy.html
The main reason to refrain from using these hierarchies is that they don’t provide isolation. For example the /usr/lib directory contains a wide collection of shared libraries belonging to many packages. To determine to which package a particular library belongs, you need to have a look in the database of the package manager. Because most libraries can be found in the same location, e.g. /usr/lib, it is very tempting to forget specifying a dependency, because they can still be implicitly found.
The purpose of the Nix package manager is to achieve purity, which means that the build result of a package should exclusively depends on the source code and input parameters. Purity ensures that the build of a package can be reproduced anywhere and that a package can be transferred to any machine we want, with the guarantee that the dependencies are present and correct.
Nix achieves purity by using the filesystem as a database and to store packages in isolation in a special directory called the Nix store. For example:
/nix/store/r8vvq9kq18pz08v249h8my6r9vs7s0n3-firefox-8.0.1
The path above refers to the Mozilla Firefox web browser. The first part of the directory name: r8vvq9kq18pz08v249h8my6r9vs7s0n3 is a hash-code derived from all build time dependencies. By using this naming convention, components can be stored safely in isolation from each other, because no component shares the same name. If Firefox is compiled with a different version of GCC or linked to a different version of the GTK+ library, the hash code will differ and thus not interfere with another variant.
1.5. Programabilidad en distintos entornos (config as code)
- En los entornos menos críticos y más vivos, quieres un build y test todos los días, que tire toda la infraestructura y la vuelva a levantar, y usas metaprogramación
- En los entornos de Producción todo es muy conservador y no quieres usar metaprogramación sino ver el código generado, copiarlo, examinarlo, entenderlo, ejecutarlo
- Tradeoff flexibilidad vs portabilidad, smart vs dumb
1.6. Pulumi
https://www.pulumi.com/docs/intro/vs/terraform/
Feature | Pulumi | Terraform |
---|---|---|
Language Support | Python, TypeScript, JavaScript, Go, C#, F# | HashiCorp Configuration Language (HCL) |
IDE Support | Code completion, strong typing, error squiggles, rich resource documentation, etc. | Limited |
State Management | Managed through Pulumi Service by default, self-managed options available. | Self-managed by default, managed SaaS offering available. |
Provider Support | Native cloud providers with 100% same-day resource coverage plus Terraform-based providers for additional coverage. | Support across multiple IaaS, SaaS, and PaaS providers. |
Cloud Native Support | Richly typed. Includes CRDs & in-cluster operator support for GitOps delivery. | Core API typed. Generic support for CRD. |
Dynamic Provider Support | Yes | No |
OSS License | Apache License 2.0 | Mozilla Public License 2.0 |
Infrastructure Reuse and Modularity | Flexible. Reuse functions, classes, packages, and Pulumi components. | Constrained. Can only reuse Terraform modules. |
Testing and Validation | Unit, property, and integration testing. Supports popular test frameworks. | Integration testing only |
Modes of Execution | Run CLI commands or initiate commands programmatically with Automation API. | Run CLI commands or perform remote runs with SaaS offering. |
Embed within Application Code | Yes, via Automation API | No |
Third-party CI/CD Tools Support | Yes | Yes |
Policy as Code | Yes | Yes |
Secrets Management | Yes. Secrets are encrypted in transit and in the state file. | No. Secrets are stored in a separate product (Vault). There is no way to encrypt them in the state file. |
Audit Capabilities | Yes | Limited |
Adopt Existing Resources | Yes. Generates code as part of the import process. | Yes. No code generation capabilities. |
Aliases | Yes | Limited |
Transformations | Yes | No |
Import Code from other IaC Tools | Yes | No |
1.7. Domen Kožar Why Puppet/Chef/Ansible aren’t good enough and we can do better
https://www.domenkozar.com/2014/03/11/why-puppet-chef-ansible-arent-good-enough-and-we-can-do-better/
Al final Nix es un gestor de paquetes que se crea un “entorno virtual” para gestionar las dependencias al estilo virtualenv/snap/appimage
1.8. The Dhall configuration language
https://dhall-lang.org/#
Lenguaje de programación para generar JSON y YAML como si fuesen código (Infrastructure as Code)