Infrastructure as Code

Table of Contents

1. Infrastructure as Code

1.2. Why you should never ever use NixOS

1.2.1. Table Of Contents

I’ll start my article from the biggest myth about NixOS. And soon you’ll know why.

DevOps-friendly. Declarative specs and safe upgrades make NixOS a great system for DevOps use. NixOps, the NixOS cloud deployment tool, allows you to provision and manage networks of NixOS machines in environments like Amazon EC2 and VirtualBox. (c) NixOS.org

I will continuously update this article each time I’ll face a new weird and challenging task that makes your life much harder if you’re using NixOS instead of traditional Linux distributions.
Well, I’m a lucky guy… I have no choice right now and have to spend most of my working time on NixOS automation. To this moment of time I have more than 10 years experience of using different Linux distributions and more than 6 months of NixOS experience. And most of my experience with NixOS is bad experience.

1.2.1.1. Why you may want to use NixOS.

I’ll try to write something here. Hope, I’ll be able to…

  1. Determinism.

    It is declared and provided to you by the system design, that you’ll get the same version of any software package from the derivation. Lie!
    All it’s determinism is about that it deterministically gives you a lot of issues, which you need to overcome manually without any support.
    You may install Nix tools on Linux and OS X, for example. But it is not always building the same derivation results on Linux and NixOS itself. So, it is not OS agnostic. There’re a lot of examples of such behavior, for example, fetchgit and fetchFromGitHub functions:

    Meanwhile in the official manual to the current and latest version-18-09-1373-50fb6820759 they are still using fetchgit.

1.2.1.2. Why you should never ever use NixOS.

Here I’m gonna collect all my daily pain in a single place, which I hope will help people not to make somebody’s else mistakes. This is my personal view on the problem space and it can not be affiliated with the company or project where I’m working right now.

  1. Small community.

    Sure, there are a lot of enthusiasts who are playing with NixOS by installing it on their workstations of VMs, but in comparison with Ubuntu, CentOS, Debian or RedHat:
    That means you will not be able:

    • To get fast or enterprise support.
    • To to hire people to manage it.
  2. Documentation.

    Official documentation is awful. Or it is just absent. NixOS is older than Docker. They are solving the same task, but NixOS is way less popular. Try to think why.
    NIX CONFIGURATION LANGUAGE It is functional language which name is also Nix and it is a core of this system configuration. If you’re coming from OOP world it will be very difficult to understand that “language”. You must go to swallow Nix pills to even try to understand something about it.
    SOFTWARE MANAGEMENT “Derivation” is a way of describing software distribution in NixOS. Each derivation represents software, plugin or a library. Common syntax of describing it looks like this (nginx derivation example):

       { stdenv , fetchurl , openssl , zlib , pcre , libxml2 , libxslt
       , gd , geoip
       , withDebug  ?  false
       , withStream  ?  true
       , withMail  ?  false
       , modules  ? []
       , version , sha256 ,  ...
      }:
       with stdenv .lib;
      stdenv .mkDerivation {
        name  =  "nginx- ${version } ";
        src  = fetchurl {
          url  =  "https://nginx.org/download/nginx- ${version } .tar.gz";
           inherit sha256;
        };
        buildInputs  = [ openssl zlib pcre libxml2 libxslt gd geoip ]
           ++ concatMap (mod: mod .inputs or []) modules;
        configureFlags  = [
           "--with-http_ssl_module"
           "--with-http_v2_module"
           "--with-http_realip_module"
           "--with-http_addition_module"
           "--with-http_xslt_module"
           "--with-http_geoip_module"
           "--with-http_sub_module"
           "--with-http_dav_module"
           "--with-http_flv_module"
           "--with-http_mp4_module"
           "--with-http_gunzip_module"
           "--with-http_gzip_static_module"
           "--with-http_auth_request_module"
           "--with-http_random_index_module"
           "--with-http_secure_link_module"
           "--with-http_degradation_module"
           "--with-http_stub_status_module"
           "--with-threads"
           "--with-pcre-jit"
           # Install destination problems
           # "--with-http_perl_module"
        ]  ++ optional withDebug [
           "--with-debug"
        ]  ++ optional withStream [
           "--with-stream"
           "--with-stream_geoip_module"
           "--with-stream_realip_module"
           "--with-stream_ssl_module"
           "--with-stream_ssl_preread_module"
        ]  ++ optional withMail [
           "--with-mail"
           "--with-mail_ssl_module"
        ]
           ++ optional (gd  !=  null)  "--with-http_image_filter_module"
           ++ optional ( with stdenv .hostPlatform; isLinux  || isFreeBSD)  "--with-file-aio"
           ++ map (mod:  "--add-module= ${mod .src } ") modules;
        NIX_CFLAGS_COMPILE  = [  "-I ${libxml2 .dev } /include/libxml2" ]  ++ optional stdenv .isDarwin  "-Wno-error=deprecated-declarations";
        preConfigure  = (concatMapStringsSep  " \n " (mod: mod .preConfigure or  "") modules);
        hardeningEnable  = optional ( !stdenv .isDarwin)  "pie";
        enableParallelBuilding  =  true;
        postInstall  =  ''
           mv $out/sbin $out/bin
         '';
        meta  = {
          description  =  "A reverse proxy and lightweight webserver";
          homepage     =  http://nginx.org;
          license      = licenses .bsd2;
          platforms    = platforms .all;
          maintainers  =  with maintainers; [ thoughtpolice raskin fpletz ];
        };
      }
    

    Interesting, isn’t it?
    If yes, get ready, you’ll not be able to live in the system without writing something like that. Nix Packages GitHub repository will be your second home, where you’ll be looking for examples of Nix expressions, configuration approaches and other examples.

  3. Configuration management.

    System state and configuration described in a special file configuration.nix. It’s like an entry point to endless amount of functions and dependencies for your OS.
    Want to customize something in the OS, be ready, you’ll need to write your custom systemd service to do that. Aaand, yes, service configuration file also need to be declared using Nix expression language.
    All standard system configuration files like /etc/fstab, for example are generated by a weird code and read only.
    Forget about Chef, Puppet or Ansible! Only Nix expressions! Only hardcore!

  4. Kernel upgrade.

    It is not possible just to upgrade the kernel from “ver1” to “ver2”. New kernel will bring whole set of system packages and their dependencies with it. Do not know if it is safe. Will test it soon.

  5. Cloud support.

    Nix is not ready for a cloud. At all. I’m always facing the following situations on NixOS 18.03:

    • OS is not booting from the AMI.
    • OS is not supporting newest AWS instance types (like m5.*, for example).
    • NixOS 18.03 is not always able to resize your hard drive when it’s boot.
    • Do not even try to use it in Auto Scaling group and passing configuration.nix content through user-data. You may easily get to the loop, when instance start building something new from the derivations during it’s boot process, 100% CPU utilizations will require AutoScaling Group to launch additional instances that also will build something, all the instances are unhealthy and Auto Scaling group will start terminating the first instance in a loop. Your service will never start.
  6. Cache.

    To speedup software installation in NixOS the community provides you with Nix Binary Cache, where they put everything they’ve built successfully. Every version of every derivation.
    Problems start happening when you spinning up your personal cache, which is used, for example to store your proprietary build artifacts. You always need to keep eye on the community channel version you’re using as a base for your builds. And yes, your own cache size will also grow very fast!
    Some very talented developers start using Nix binary cache to cache everything’s possible. For example NPM or Yarn packages. Good luck!

  7. Security.

    People are thinking, that NixOS deterministic approach of dealing with software is very secure. Maybe yes, it is really not possible to hijack the build results once the derivation bin build. But… You always need to pay for it…
    During the upgrade from 18.03 to 18.09 Nix community decided to change Docker tools they’ve been use to calculate derivations checksums where you were dealing with Docker images.
    As a result we rewrote all Docker based derivation declarations and more over to rebuild them all. It was really painful and unmanaged migration.

  8. Windows support.

    Just forget about it.

  9. System requirements.

    It requires a lot of CPU power to build it’s stuff and really a lot of space to store its multiple versions. Much more, than you need in traditional Linux distros.

  10. Once invested.
    • Try to change a technology stack as soon as possible.
    • Do not listen anybody.
    • Do not reinvent the wheel.
    • Just waste it.

    And, God, do not try to run Nix in Docker! It’s really, not, REALLY bad idea!

  11. What to do next.

    Run away! Not, seriously! Run. From. It. Away.
    When you try to summarize all your thoughts around the topic you want to cover in a post, you’re starting from simple Google search. One of results to my “nixos pros and cons” query: NixOS alternatives. Kinda joke, but really, think twice when you’re making decision to invest your time or money in it. And I would also think twice about WHY colleagues are advising it to me.

  12. Docker is the best alternative.

    The link is in previous paragraph. Your choice is traditional Linux and Docker! Yes, you’ll have the same deterministic and secure result and even more more with less pain and and much pleasure.

  13. Additional reading.

1.3. Why you should never ever use NixOS: a rebuttal

1.4. https://web.archive.org/web/20210707233008/sandervanderburg.blogspot.com/2011/11/on-nix-nixos-and-filesystem-hierarchy.html

The main reason to refrain from using these hierarchies is that they don’t provide isolation. For example the /usr/lib directory contains a wide collection of shared libraries belonging to many packages. To determine to which package a particular library belongs, you need to have a look in the database of the package manager. Because most libraries can be found in the same location, e.g. /usr/lib, it is very tempting to forget specifying a dependency, because they can still be implicitly found.
The purpose of the Nix package manager is to achieve purity, which means that the build result of a package should exclusively depends on the source code and input parameters. Purity ensures that the build of a package can be reproduced anywhere and that a package can be transferred to any machine we want, with the guarantee that the dependencies are present and correct.
Nix achieves purity by using the filesystem as a database and to store packages in isolation in a special directory called the Nix store. For example:
/nix/store/r8vvq9kq18pz08v249h8my6r9vs7s0n3-firefox-8.0.1
The path above refers to the Mozilla Firefox web browser. The first part of the directory name: r8vvq9kq18pz08v249h8my6r9vs7s0n3 is a hash-code derived from all build time dependencies. By using this naming convention, components can be stored safely in isolation from each other, because no component shares the same name. If Firefox is compiled with a different version of GCC or linked to a different version of the GTK+ library, the hash code will differ and thus not interfere with another variant.

1.5. Programabilidad en distintos entornos (config as code)

Configs suck, use code

  • En los entornos menos críticos y más vivos, quieres un build y test todos los días, que tire toda la infraestructura y la vuelva a levantar, y usas metaprogramación
  • En los entornos de Producción todo es muy conservador y no quieres usar metaprogramación sino ver el código generado, copiarlo, examinarlo, entenderlo, ejecutarlo
  • Tradeoff flexibilidad vs portabilidad, smart vs dumb

1.6. Pulumi

https://www.pulumi.com/docs/intro/vs/terraform/

Feature Pulumi Terraform
Language Support Python, TypeScript, JavaScript, Go, C#, F# HashiCorp Configuration Language (HCL)
IDE Support Code completion, strong typing, error squiggles, rich resource documentation, etc. Limited
State Management Managed through Pulumi Service by default, self-managed options available. Self-managed by default, managed SaaS offering available.
Provider Support Native cloud providers with 100% same-day resource coverage plus Terraform-based providers for additional coverage. Support across multiple IaaS, SaaS, and PaaS providers.
Cloud Native Support Richly typed. Includes CRDs & in-cluster operator support for GitOps delivery. Core API typed. Generic support for CRD.
Dynamic Provider Support Yes No
OSS License Apache License 2.0 Mozilla Public License 2.0
Infrastructure Reuse and Modularity Flexible. Reuse functions, classes, packages, and Pulumi components. Constrained. Can only reuse Terraform modules.
Testing and Validation Unit, property, and integration testing. Supports popular test frameworks. Integration testing only
Modes of Execution Run CLI commands or initiate commands programmatically with Automation API. Run CLI commands or perform remote runs with SaaS offering.
Embed within Application Code Yes, via Automation API No
Third-party CI/CD Tools Support Yes Yes
Policy as Code Yes Yes
Secrets Management Yes. Secrets are encrypted in transit and in the state file. No. Secrets are stored in a separate product (Vault). There is no way to encrypt them in the state file.
Audit Capabilities Yes Limited
Adopt Existing Resources Yes. Generates code as part of the import process. Yes. No code generation capabilities.
Aliases Yes Limited
Transformations Yes No
Import Code from other IaC Tools Yes No

1.7. Domen Kožar Why Puppet/Chef/Ansible aren’t good enough and we can do better

https://www.domenkozar.com/2014/03/11/why-puppet-chef-ansible-arent-good-enough-and-we-can-do-better/
Al final Nix es un gestor de paquetes que se crea un “entorno virtual” para gestionar las dependencias al estilo virtualenv/snap/appimage

1.8. The Dhall configuration language

https://dhall-lang.org/#
Lenguaje de programación para generar JSON y YAML como si fuesen código (Infrastructure as Code)

1.9. Infrastructure as SQL on AWS: IaSQL is Now Open Source and SaaS

Author: Julian Lopez Carballal

Created: 2024-10-21 Mon 10:24