bash

Table of Contents

1. bash

1.1. Introductory

1.1.1. Games

1.1.1.2. TICKLER Codédex | Start your coding adventure!   track

Command line version cooming soon…

1.1.2. Resumen

Path relativo absoluto
Flags ls -l, –help -h
Poner o no / al final del directorio, poner o no poner el nombre del archivo cuando copias, directorio actual, cd
cp, mv, cd, ls
chmod, chown
nano
fg, bg cuando haces C-z con nano

  • Atajos de teclado
    C-c → cortar proceso y limpiar comando actual
    Tab → autocompletar todo
    C-r → buscar historial
    C-d → salir de la terminal

1.1.3.

1.1.8. Tools to simplify the Command Line

https://n-o-d-e.net/cli.html
https://n-o-d-e.net/cli2.html

1.3. Trucos

  • stty sane → cuando te peta la terminal y no puedes escribir
  • setxkbmap es, setxkbmap us → Para poner el teclado en español / en inglés americano
  • xkill → kill seleccionando gráficamente la ventana
  • sudo timedatectl set-timezone "Europe/Madrid" → Zona horaria de Madrid
  • Sacar tu IP pública dig +short myip.opendns.com @resolver1.opendns.com, curl ifconfig.co
  • Cambiar fecha de modificación de un archivo
    touch -d "30 days ago"
1.3.0.1. adduser sencillo, no interactivo
adduser --disabled-password --gecos "" <username>
1.3.0.2. Crear un directorio si no existe
[ -d foo ] || mkdir foo
1.3.0.3. Argumentos de línea de comandos con getopts
curl cheat.sh/bash+getopts2
# question_id: 402377
# `getopt` and `getopts` are different beasts, and people seem to have a
# bit of misunderstanding of what they do.  `getopts` is a built-in
# command to `bash` to process command-line options in a loop and assign
# each found option and value in turn to built-in variables, so you can
# further process them.  `getopt`, however, is an external utility
# program, and it _doesn't actually process your options for you_ the
# way that e.g. bash `getopts`, the Perl `Getopt` module or the Python
# `optparse`/`argparse` modules do.  All that `getopt` does is
# canonicalize the options that are passed in — i.e. convert them to a
# more standard form, so that it's easier for a shell script to process
# them.  For example, an application of `getopt` might convert the
# following:

 myscript -ab infile.txt -ooutfile.txt

# into this:

 myscript -a -b -o outfile.txt infile.txt

# You have to do the actual processing yourself.  You don't have to use
# `getopt` at all if you make various restrictions on the way you can
# specify options:
#
  1. only put one option per argument;
  2. all options go before any positional parameters (i.e. non-option arguments);
  3. for options with values (e.g. `-o` above), the value has to go as a separate argument (after a space).

https://unix.stackexchange.com/questions/182285/calling-bash-getopts-from-a-function-fails-the-2nd-time
The answer is that getopts keeps the pointer in $OPTIND and therefore the trick is to add local OPTIND or OPTIND=1 to the beginning of the getopt_test function.

OPTIND=1
1.3.0.3.1. http://abhipandey.com/2016/03/getopt-vs-getopts/

Generally, try to stay away from getopt for the following reasons:

  1. External utility
  2. Traditional versions can’t handle empty argument strings, or arguments with embedded whitespace
  3. For the loops to work perfectly, you must provide the values in the same sequence as the for loop itself; which is very hard to control
  4. Mostly a way to standardize the parameters
1.3.0.4. Sacar información de un ordenador
separator="============================================="
archivo=dump.txt
echo $separator >> $archivo 2>&1
cat /proc/cpuinfo >> $archivo 2>&1
echo $separator >> $archivo 2>&1
cat /proc/meminfo >> $archivo 2>&1
echo $separator >> $archivo 2>&1
lspci >> $archivo 2>&1
echo $separator >> $archivo 2>&1
cat /proc/bus/pci/devices >> $archivo 2>&1
echo $separator >> $archivo 2>&1
lsblk >> $archivo 2>&1
echo $separator >> $archivo 2>&1
cat /proc/partitions >> $archivo 2>&1
echo $separator >> $archivo 2>&1
( sudo dmidecode || dmidecode ) >> $archivo 2>&1
echo $separator >> $archivo 2>&1
( sudo cat /sys/devices/virtual/dmi/id/* || cat /sys/devices/virtual/dmi/id/* ) >> $archivo 2>&1
echo $separator >> $archivo 2>&1
1.3.0.5. Resolver una lista de hostnames
cat hostnames.txt | sed -E "s%https?://%%;s%/.+%%" | xargs -I _ host _ | grep -E 'has (IPv6 )?address'
1.3.0.6. Procesar el resultado de un comando y sacarlo en tiempo real (streaming)
result=$(command | tee /dev/tty)

1.3.2. Sin clasificar

  • nl para añadir números de línea echo -e "one\ntwo\nthree" | nl
  • flock para crear locks en scripts de bash
  • https://www.shellcheck.net/ → finds bugs in your shell scripts (linting)

  • Ctrl+R para buscar hacia delante comandos

      stty stop ^P # Ponerlo en ~/.bashrc para que Ctrl+S sirva para buscar hacia delante (Ctrl+R hacia atrás)

1.3.3. Progreso al copiar

1.3.3.1. Sacar el progreso

Para zsh

src="/media/extract2"; dst="/media/julian/DVD/extract2"; while true; do  echo $(( ($(du -d 1 $dst | tail -n 1 | awk '{print $1}') * 100.0) / $(du -d 1 $src | tail -n 1 | awk '{print $1}') )); sleep 5; done

Para bash

src="~/Documents"; dst="/media/pi/Linux Backup/linux/Documents"; while true; do  echo $(( ( $(du -d 1 "$dst" |
tail -n 1 | awk '{print $1}') * 100) / $(du -d 1 "$src" | tail -n 1 | awk '{print $1}') )); sleep 5; done
1.3.3.2. progreso al copiar un disco entero
  while (true); do echo "scale=4; $(df /dev/dm-3 | tail -n 1 | awk '{print $3}')*100.0/469969564.0" | bc; sleep 5; clear; done
  # Donde 469969564.0 es el tamaño en bloques usados del dispositivo de la copia

Disclaimer: como es un método un poco chapucero, es posible que se quede
por debajo o por encima del 100% en algunos casos

1.3.3.3. progreso al copiar de una carpeta a otra

Funciona mejor en zsh, con más decimales

src=""; dst=""; while true; do  echo $(( ($(du -d 1 $dst | tail -n 1 | awk '{print $1}') * 100.0) / $(du -d 1 $src | tail -n 1 | awk '{print $1}') )); sleep 5; done

1.3.4. Lista de un directorio a org-mode

Tienes que recorrer el árbol en preorder
https://stackoverflow.com/questions/76169133/how-do-i-do-a-pre-order-file-traversal-in-bash

findfile_preorder()(
    shopt -s dotglob nullglob
    for p; do
        printf '%s\n' "$p"
        [[ ! -L $p && -d $p ]] && findfile_preorder "$p"/*
    done
)

findfile_preorder Documents | awk -F/ '{for (i=1; i<NF; i++) printf("*"); print " " $0}'

findfile_preorder Documents | awk -F/ '{for (i=1; i<NF; i++) printf("*"); print " " $0}' | sed -E 's/^\*+ (.+)/echo -e "&\\n:PROPERTIES:\\n:ID: $(uuidgen --sha1 --namespace @url --name \\\"\1\\\")\\n:END:\\n\1"/' | bash

1.3.5. Errores de certificados

https://askubuntu.com/questions/1095266/apt-get-update-failed-because-certificate-verification-failed-because-handshake

sudo chmod 755 /etc /etc/ssl /etc/ssl/certs
sudo chmod 644 /etc/ssl/certs/ca-certificates.crt

1.3.6. Sacar el número de columnas de un csv

  • Si el número de columnas varía dentro de un mismo csv
grep "GPS" trackLog-2021-nov.-01_17-48-32.csv | grep -n -o "," | sort -n | uniq -c | cut -d : -f 1
# GPS es el nombre de una columna que sabes que va a existir siempre

1.3.7. Sacar el nombre del ordenador

https://unix.stackexchange.com/questions/574808/get-the-hardware-model-name-in-linux

if [[ -d /system/app/ && -d /system/priv-app ]]; then
    model="$(getprop ro.product.brand) $(getprop ro.product.model)"

elif [[ -f /sys/devices/virtual/dmi/id/product_name ||
    -f /sys/devices/virtual/dmi/id/product_version ]]; then
    model=$(< /sys/devices/virtual/dmi/id/product_name)
    model+=" $(< /sys/devices/virtual/dmi/id/product_version)"

elif [[ -f /sys/firmware/devicetree/base/model ]]; then
    model=$(< /sys/firmware/devicetree/base/model)

elif [[ -f /tmp/sysinfo/model ]]; then
    model=$(< /tmp/sysinfo/model)
fi
echo $model

HP ZBook Firefly 15 G7 Mobile Workstation

1.3.8. Y/n Prompt bash

read -p "Continue? [Y/n] " yn

case $yn in
    [yY]) true ;;
    [nN]) exit ;;
    * ) true ;;
esac

1.3.9. Sacar número de líneas máximo para configurar el formateo

find . -iname "*.py" | xargs -I % wc -L "%" | sort -nr | head -n 20

1.3.10. Renombrar con un nombre aleatorio

for f in *.jpg; do
  random_string=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 30 | head -n 1);
  nuevo_nombre="${random_string}.jpg"
  mv "./$f" "$nuevo_nombre"
done

1.3.11. Generar 20 strings aleatorias de longitud 30

cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 30 | head -n 20

1.3.12. Sacar tamaño de imágenes

find . -iname '*.jpg' | xargs -I _ bash -c "echo -en \"_\t\t\t\"; exiftool _ | grep 'Image Size' | awk '{print \$4}'" | sort

1.4. herramientas

1.4.1. misc

  • agrep Fuzzy grep
  • sort, unique, head, tail
  • Regex con sed y con grep siempre usar -E (extended) para que haga cosas normales y esperables y no tener que poner 4 backslashes
  • seq start step end
  • wc wordcount para contar líneas, caracteres, …
  • tee se usa mucho pero ni idea cómo va
    Por ejemplo si haces result=$(command | tee /dev/tty) puedes tener a la vez el resultado del comando y la salida en tiemo real
  • tac es como cat sèver la orep

1.4.2. grep

1.4.2.1. Colores con (rip)grep y less

https://linuxcommando.blogspot.com/2007/10/grep-with-color-output.html

grep --color=always  abc  a_file.txt | less -R
rg -p | less -R

1.4.3. sed

1.4.4. find

find ~+
para que te saque el path completo

1.4.4.1. Eliminar archivos por extensión
  list=`find . -name *.pyc`
  readarray -t goodlist <<<"$list"
  for ((i=0; i<= ${#goodlist}; i++)); do rm "${goodlist[i]}"; done
1.4.4.2. Ejecutar cosas contra los matches find
  find -iname "*sync-conflict*" -exec sh -c 'rm {}' \;
1.4.4.3. hash de una carpeta (determinista)

Si no le pones el LC_ALL=C te depende del locale en el que lo ejecutes

find /path/to/folder/ -type f -print0 | LC_ALL=C sort -z | xargs -0 sha1sum | sha1sum
# Para comprobar que es igual a un hash
[[ $(find . -type f -not -name '*csv' -print0 | LC_ALL=C sort -z | xargs -0 sha1sum | sha1sum) = "009d6a0a9544379f70e365ea904c9835905cee89  -" ]]
# Si sólamente te interesa comparar si dos carpetas son iguales, pongamos tfrecord1 y tfrecord2:
find tfrecord1 -type f -print0 | LC_ALL=C sort -z | xargs -0 sha1sum | awk '{print $1}' | sha1sum
find tfrecord2 -type f -print0 | LC_ALL=C sort -z | xargs -0 sha1sum | awk '{print $1}' | sha1sum

Faltaría hacerlo genérico

1.4.4.4. Operadores booleanos (or de varias condiciones)

Por defecto te los une con un and

find Documents \( -iname "*pdf" -o -iname "*.epub" \)

https://man7.org/linux/man-pages/man1/find.1.html → OPERATORS

1.4.4.5. Permisos recursivos de una carpeta
  find folder/ -type f -exec bash -c "chmod 644 \"{}\"" \;
  chmod 664 -R folder/ # Suele ser más rápido que lo anterior
  find orgzly/ -type d -exec bash -c "chmod 755 \"{}\"" \; # Para poder acceder a los directorios

1.4.5. awk

Use cut when possible instead of awk

cat /etc/passwd | cut -d ':' --output-delimiter=" " -f 1,5-7

https://github.com/FreedomBen/awk-hack-the-planet → excellent awk resource, github repo + videos

  • -F → set Field Separator
$NF # Último campo
$(NF-n) # Campo n empezando por la izquierda
FS # (Input) Field Separator
OFS # Output Field Separator
awk 'BEGIN{FS=":"; OFS=","} {print $0}' /etc/passwd # Change field separator from : to ;
awk "/pattern/ action"
Se pueden sumar columnas numéricas
length($0) → Longitud de la línea, se puede utilizar como una condición
length($0) < 8
{ if ($NF == "/bin/bash") print $0 }
Multiple conditions with &&
df -a 2>/dev/null | awk '{ if ($0 ~ /\/dev\/loop/ && length($1) < 11) print $1"\t"$2+$3}'
match($0, /o/) → Matches entries with "o"
awk 'NR==7 NR==11, {print NR, $0}' # Print from 7 through 11, printing also line number
NF → Number of Fields, NR → Number of Records
Pasar variables de entorno a awk con -v avar="$var", luego la puedes utilizar como avar (sin $) en awk
1.4.5.1. Print last N columns

1.4.6. less

less -N # para que tenga numero de linea
less -R # para que interprete las secuencias de escape de colores

1.4.7. tail

tail -F <archivo> # Sigue al archivo según se añaden cosas, con --retry si se corta
tail  -s <n> -f <log> # follows the log refreshing each n seconds

1.4.8. Pequeñas y efectivas

  • inotify para disparar acciones en función de cambios en el sistema
  • notify-send para lanzar notificaciones en escritorio
    • dunst es un pelín más sofisticado y te permite poner botones que lanzan acciones
  • tmux para lanzar procesos y comprobar cómo van, hacer splits en la propia terminal
  • termux puede hacer de todo en android (menús, notificaciones, escanear nfcs…)
  • pv para que pinte el progreso (también vale tqdm)

1.4.9. fzf

1.4.10. pipes en paralelo (DAGs)

Las partes secuenciales se expresan bien con operadores tipo pipe (|)
https://github.com/flonatel/pipexec
https://news.ycombinator.com/item?id=39656056
https://www2.dmst.aueb.gr/dds/sw/dgsh/

Tenía un ejemplo de parallel pero no sé si es este

now="$(date +'%Y-%m-%d %H:%M:%S')"
echo "file|link|status" > "link_rot $now.csv"
rg -H "https?://" \
    | sed -E 's%(.+\.org)\:.+?(https?://[^] ]+).+?%\1|\2%g' \
    | parallel --colsep '\|' curl -o /dev/null -s -w '{1}\|{2}\|%{http_code}\\n' '{2}' \
    | tee -a  "link_rot $now.csv"

1.5. Resumen

1.5.1. Atajos de teclado

stty --all

  Ctrl+C → Stop Current Comand (SIGINT)
  Ctrl+D → End of input (exit shell)
  Ctrl+\ → SIGQUIT, mas estricto que SIGINT
  Ctrl+Z → SIGTSTP, detiene el proceso
  Ctrl+S → Halt output to screen
  Ctrl+Q → Restart output to screen
  Ctrl+J → New line
  Ctrl+L → Clear
  Esc+Enter → New line without executing command

https://ss64.com/bash/syntax-keyboard.html
Autocompletado

  TAB     → General
  Esc+?   → Todas las posibilidades
  Esc+/   → Archivo
  Ctrl+X/ → Todos los archivos
  Esc+~   → Usuario
  Ctrl+X~ → Todos los usuarios
  Esc+$   → Variable
  Ctrl+X$ → Todas las variables
  Esc+@   → Host
  Ctrl_X@ → Todos los hosts

1.5.2. Procesos

  • $ <comando> → Ctrl+Z lo detiene → $ bg lo ejecuta en segundo
    plano, $ fg lo ejecuta en primer plano
  • comando & → correr en segundo plano
1.5.2.1. double forking

(setsid command &)

  • This uses ( &) to fork to background
  • setsid to detach from the controlling tty.

https://superuser.com/a/172476

1.5.2.2. Ejecutar un proceso redirigiendo logs
  export MY_VAR="foobar"; run_my_command > "log_`date -Iseconds`.txt" 2>&1

1.5.3. [[]]

1.5.4. (( ))

  • Aritmética de enteros
  • En zsh también tiene aritmética de coma flotante

1.5.6. Aritmética de coma flotante

  bc <<< 'scale=2; 100/3'
  echo "scale=2; 100/3" | bc

1.5.7. Shell Parameter Expansion / Substitution

https://www.gnu.org/software/bash/manual/bash.html#Shell-Parameter-Expansion
https://tldp.org/LDP/abs/html/parameter-substitution.html

https://stackoverflow.com/questions/3601515/how-to-check-if-a-variable-is-set-in-bash

if [ -z ${var+x} ]; then echo "var is unset"; else echo "var is set to '$var'"; fi

if [ -z "$var" ]; then echo "var is blank"; else echo "var is set to '$var'"; fi

1.5.8. arrays, variables ${var#pattern}

1.5.8.1. Extensiones de archivos
full_path="/var/log/Xorg.0.log.old"
filename=${full_path##*/}
full_extension=${filename#*.}
extension=${filename##*.}
1.5.8.2. Todas las separaciones posibles
var=first:middle:last
${var##*:} → last
${var#*:} → middle:last
${var%%:*} → first
${var%:*} → first:middle

var=one:two
${var##*:}, ${var#*:} → two
${var%%:*}, ${var%:*} → one

var=only
todas dan simplemente only
1.5.8.3. Arrays asociativos

Depende de zsh o bash

declare -A array
for dir in ~/.local/share/venvs/*; do if echo $dir | grep -v -e shims -e versions 1>/dev/null; then; array["$dir"]=$(readlink $dir/project); fi; done
echo ${array["/home/julian/.local/share/venvs/133063051db23bd4a339770311f9c270"]}

/home/julian/code/conv

https://stackoverflow.com/questions/48650075/how-to-assign-a-value-to-dynamic-associative-array-in-bash

arr=my_array
declare -A "$arr"
declare "$arr[foo]=bar"
echo "${my_array[foo]}" bar

This provides the indirection you need. The argument $arr[foo]=bar is expanded to my_array[foo]=bar, which is a valid assignment expression to be processed by declare.

1.5.9. variables especiales $@, $?, IFS, PS1

1.5.10. Redirecciones, stdout, stderr

1.5.10.1. Illustrated Redirection Tutorial [Bash Hackers Wiki]

https://wiki.bash-hackers.org/howto/redirection_tutorial

1.5.11. UI

1.6. Variables de entorno específicas a servicios de systemd

https://youtu.be/9kFE5WVL5O0
Variables de entorno especificas a servicios de systemd
Utilizar cuando pase a un servicio las cosas de orgzly-integrations

1.7. Alternative shell pipeline tools

Tools that work with an alternative to the UNIX text stream (JSON, CSV, HTML)

1.7.1. jq

1.7.1.2. jq Manual (development version)

https://stedolan.github.io/jq/manual/

1.7.1.3. jq Cheat Sheet

https://gist.github.com/olih/f7437fb6962fb3ee9fe95bda8d2c8fa4

JSON        -> [{"text": ...}, {"text": ...}, {"text": ...}]
jq_schema   -> ".[].text"

JSON        -> {"key": [{"text": ...}, {"text": ...}, {"text": ...}]}
jq_schema   -> ".key[].text"

JSON        -> ["...", "...", "..."]
jq_schema   -> ".[]"
1.7.1.5. jj: faster jq
1.7.1.6. trucos jq

Streaming agrupando entradas

(echo 1 2 3; sleep 2; echo 4 5 6) | jq --stream '[., input, input] | flatten'

1.7.2. rq

https://github.com/dflemstr/rq

The goal is to make ad-hoc exploration of data sets easy without having to use more heavy-weight tools like SQL/MapReduce/custom programs. rq fills a similar niche as tools like awk or sed, but works with structured (record) data instead of text. It is similar to jq, but supports more record formats and has a far more advanced query engine.

1.7.3. hq - like jq, but for HTML

https://github.com/ludovicianul/hq

1.7.4. jc: converts the output of popular tools to JSON

https://github.com/kellyjonbrazil/jc
Idea behind the tool:
Bringing the Unix Philosophy to the 21st Century - Brazil’s Blog

1.7.6. jqnatividad/qsv: CSVs sliced, diced & analyzed. (Based on Rust+Polars)

1.7.8. fx: Terminal JSON viewer & processor

https://fx.wtf/

1.7.9. Miller

https://github.com/johnkerl/miller
Miller is like awk, sed, cut, join, and sort for data formats such as CSV, TSV, JSON, JSON Lines, and positionally-indexed.

1.7.10. Tools for editing/visualizing binary data?

binary formats could also have viewers and editors. Those destined for human creation and consumption generally are, but data exchange formats rarely have the same love, and as such stay unreadable. They don’t have to be. They can be drawn on the screen or even directly edited. We just need to write the tools.

1.8. Es bash un “lenguaje de programación serio”?

1.8.1. Programación más seria en bash (unofficial bash strict mode)

http://redsymbol.net/articles/unofficial-bash-strict-mode/
https://stackoverflow.com/questions/685435/trace-of-executed-programs-called-by-a-bash-script
https://linoxide.com/change-bash-prompt-variable-ps1/

# Para que al dar error salga del script y no siga ejecutando cosas a lo loco, igual que haría cualquier lenguaje de programación decente:
set -e
# Variables no definidas lanzan un error:
set -u
# Hacer que las pipes devuelvan el estado del comando
set -o pipefail
true | false | true  # Esto devuelve 0 sin pipefail (coge el true de la derecha) y 1 con pipefail (coge el false de en medio)

IFS=$'\n\t' # No separa por espacios

Todo junto:
#+begin_src bash
#!/bin/bash
set -euo pipefail
IFS=$'\n\t'
1.8.1.1. Debugging en bash
# DEBUGGING
# Para debugear, imprime lo que se está ejecutando en cada momento y su número de línea
# !Importante! Poner ', nunca " (con " se evalúa en el momento, y $LINENO devuelve la línea en la que defines PS4)
# !Importante! No poner comandos en PS4 porque si no se ejecutan infinitamente (Un print de PS4 necesita evaluar un comando, que genera otro print de PS4, que evalua el comando otra vez, ...)

# Versión con menos cosas, para scripts individuales que no llaman a funciones ni a otros archivos
# EPOCHREALTIME funciona desde bash 5, si no no se imprime nada
PS4='+[\D{%d-%m-%y %H:%M:%S}${EPOCHREALTIME##*.}] ${LINENO}:'
 # Versión completa
PS4='+[\D{%d-%m-%y %H:%M:%S}${EPOCHREALTIME##*.}] ${FUNCNAME[0]}() $BASH_SOURCE:${BASH_LINENO[0]}: '
set -x # Imprime lo de arriba cada vez que se ejecuta algo

# Tracebacks en bash
function traceback() {
    i=0; while caller $i ;do printf "%$i""s"; ((i++)) ;done
}
1.8.1.2. Ejecutar línea por línea un script

https://stackoverflow.com/questions/9080431/how-execute-bash-script-line-by-line

set -x
trap read debug
1.8.1.3. Poner excepciones de verdad en bash?

Lo mismo se puede combinar de alguna manera un PS4 chulo con set -x y lanzarlo sólamente en un trap, aunque creo que en ese caso sólamente tienes la última excepción
Funciona mejor con callee

1.8.3. In defense of bash scripts - Bash is SUPER

  • Simple
    When describing shell scripts as simple (not “simple to use”nk ), I mean that there are virtually no dependencies needed in order to run a simple bash script
  • Ubiquitous
  • Portable
    You really can’t get more portable than shell scripts. There’s no need to install text editors or update packages or install dependencies
  • Efficient
    e.g.: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
  • Robust
    Piping functions is robusts. Since monads are like pipes, and a monad is just a monoid in the category of endofunctors, it follows that bash scripting, when done well, is like functional programming
1.8.3.1. Friends don’t let friends program in shell script

Why?

  • Shell script is ugly.
  • Shell script is unreliable.
  • Shell script doesn’t have exceptions.
  • Shell script is hard to debug.
  • Shell script is hard to test.
  • Shell script isn’t object oriented.
  • Shell script is full of idiosyncratic cruft.
  • Shell script doesn’t have any decent data structures.
  • Shell script doesn’t scale.
  • Shell script encourages code duplication.

1.8.4. https://cuddly-octo-palm-tree.com/posts/2021-10-31-better-bash-functions/

  • Use () instead of {} to use lexical scope instead of dynamic scope (lexical scope means callee functions cannot access variables created in the scope of the caller function, unlike dynamic scope)
  • Use traps for safe cleanup

1.8.6. Next Generation Shells (concept)

https://news.ycombinator.com/item?id=28487009

1.8.6.1. NGS (Language)

https://ngs-lang.org/
NGS: next generation shell para sustituir bash, habla de carencias de bash

  • shell pipes do feel ergonomic but conceptually unidirectional untyped byte streams are not that impressive these days.
  • The “problem” with general-purpose languages is that they don’t have ergonomic facilities for DevOps.
    run an external process, get the exit code of the external process, work with files
  • The “problem” with bash:
    extremely poor error handling, missing data structures, syntax.
    A related observation is that other languages also had afterthoughts such as “use strict” which very roughly corresponds to set -eu in bash.
  • NGS design
    https://console.substack.com/p/console-92?s=r
    Small scripts don’t pay / Small scripts stay small
    You don’t need to pay for something that is possible in the language when you don’t use it.
    Example: main() function. It provides a very ergonomic way to parse the command line arguments. When you don’t care about arguments - just don’t define main(), the script runs top to bottom.
    Syntax compression
    The more frequently used a feature is, the shorter the syntax will be.
    Small number of concepts
    For example, NGS does not have classes. There are types and methods.
  • Features
    exceptions, types with inheritance, multiple dispatch, guards (additional conditions to invoke a method), basic functional programming support, patterns, (new, in progress) facility for deeply nested data manipulation, automatic parsing of command line arguments when invoking the optional main(), threads, domain-specific syntax and facilities for running external programs and file manipulation.
  • Further NGS design
    https://ngs-lang.org/doc/latest/man/ngslang.1.html
1.8.6.1.1. https://ngs-lang.org/doc/latest/man/ngslang.1.html → Language Principles
  • Systems engineers’ language
    NGS is a domain-specific language. It is aimed to solve common system tasks in a convenient manner «more structured text than bash, namely json»
  • Uniformity
    NGS tries to be uniform wherever possible to minimize surprises. The idea here is to match between expected and actual behaviour of given method plus parameters.
  • Power
    As a rule, trade-offs between power and not allowing to shoot yourself in the foot are resolved in favor of the power solution. The language is aimed at experienced engineers which use their own judgement. The language should be powerful enough to shoot all feet in the building at once.
  • Simple methods naming for less guess work
    For example, the multimethod +:
    • 1 + 2 adds the numbers
    • arr1 + arr2 adds (concatenates) arrays
    • hash1 + hash2 produces merged hash.
  • Extensibility
    • fetch('your_file.super-format') can be extended to decode your format.
    • read, which reads from a file, can be extended to support HTTP or S3.
    • Most of the syntax (for example my_var.my_field or my_var[my_index]) is just sugar for calling methods. This behaviour lets you, the user, define any operator for existing or your custom types.
  • Simplicity
    Very small number of core concepts in the language:
    • Types with a simple type system, geared only toward multiple dispatch. No classes.
    • Multimethods, which allow using same method name for operations on different types, as in the + example above.
    • Closures
    • Exceptions
  • Familiarity
    Many concepts and syntax constructs come from other languages.
1.8.6.2. Oil Shell - Why Create a New Unix Shell? (2021)
  1. Shouldn’t we discourage people from writing shell scripts?
    1. It won’t work. Like PHP, the shell language is useful, ubiquitous, flawed, and being improved.
    2. Even if a new line of shell never gets written, there will still be a huge installed base of shell scripts that you may need to understand (e.g. when they don’t work).
    3. Shell is still the best tool for many jobs. (cloud/devops)
1.8.6.2.2. The Internet Was Designed With a Narrow Waist

Narrow waist facilitates interoperability

  1. Narrow Waist

    The narrow waist (of an hourglass) is a software concept that solves an interoperability problem, avoiding an O(M × N) explosion. All of these are narrow waists:

    • Interchange formats like JSON
    • Networking protocols like HTTP
    • Operating system interfaces like Win32 and POSIX
    • Instruction set architectures like x86, and arguably WebAssembly.
    • «Language Server Protocol, Debugger Adapter Protocol and Treesitter are also examples»

    O(M × N) code explosion — A system may need bespoke code to fill in every cell of a grid, like M algorithms and N data structures, or M languages and N operating systems. This problem can often be mitigated by better software architecture, e.g. with protocols, interchange formats, or intermediate representations.

1.8.6.7. Comparison of NGS, osh (oilshell), elvish shell, nushell
  • NGS
    • small scripts don’t pay the price that bigger scripts may
    • The more frequently used a feature is, the shorter the syntax will be.
    • The more frequently used a feature is, the shorter the syntax will be.
  • oilshell
    • backwards-compatible with bash
  • elvish shell
  • nushell
1.8.6.8. Charm: We build tools to make the command line glamorous

More like extending bash, specially with Bubble Tea

1.8.9. Bringing the Unix Philosophy to the 21st Century - Brazil’s Blog

https://blog.kellybrazil.com/2019/11/26/bringing-the-unix-philosophy-to-the-21st-century/
Estandarizar la salida de los programas en bash para que se puedan encadenar más fácilmente

1.8.10. Please stop writing shell scripts

https://pythonspeed.com/articles/shell-scripts/

1.8.10.1. The problem with shell scripts
  1. Errors don’t stop execution
  2. Unknown variables cause no errors
  3. Pipes don’t catch errors
  4. Subshells are weird
1.8.10.2. Some bad reasons to use shell scripts
  1. It’s always there!
    If you’re packaging an app writen in X, use X with is shell convenience packages!
  2. Just write correct code!
    In practice:
    1. You’re probably not working alone; it’s unlikely everyone on your team has the relevant expertise.
    2. Everyone gets tired, gets distracted, and otherwise ends up making mistakes.
    3. Almost every complex shell script I’ve seen was lacking the set -euo pipefail invocation, and adding it after the fact is quite difficult (usually impossible).
    4. I’m not sure I’ve ever seen an automated test for a shell script. I’m sure they exist, but they’re quite rare.
  3. Shellcheck will catch all these bugs!
    Will catch some of the problems, but not all
1.8.10.3. Shell scripts are fine in some situations
  • For one-off scripts that you are manually supervising, you can get away with laxer practices.
  • Sometimes you really have no guarantees that another programming language is available, and you need to use the shell to get things going.
  • For sufficiently simple cases, just running a few commands sequentially, with no subshells, conditional logic, or loops, set -euo pipefail is sufficient (and make sure you use shellcheck -o all)

1.8.11. Shell utils

  • plumbum Make bash scripts using python (example)
    • SHTK alternative to plumbum
  • sh sh is a full-fledged subprocess replacement for Python 2.6 - 3.8, PyPy and PyPy3 that allows you to call any program as if it were a function

1.9. (Rust) replacements for gnu utils

1.9.1. Ver “Rust Programs Every Linux User Should Know About” en YouTube

https://youtu.be/dQa9mveTSV4
replacements for gnu utils

  • ls with syntax highlight rwx
  • cat with syntax highlight
  • find but simpler
  • ripgrep
  • ps with syntax highlight
  • faster cp

Author: Julian Lopez Carballal

Created: 2024-09-16 Mon 05:47